Rewriting the rulebook
Eclipse Intent: “Being Agile does not mean being short-sighted”
Eclipse Intent is a subproject of Mylyn that promises to make it easier to keep documentation up-to-date, with the bother of development artifacts.
JAXenter caught up with Project Lead Alex Lagarde to discuss the importance of documentation and how Intent can help.
Documentation is surely not the most popular domain in programming. From your point of view, what are the most common reasons – or excuses – for not writing doc?
We could compare the current reluctance to document software with the massive rejection of unit testing in the 90’s: people think it’s useless because it takes so much time and does not bring any direct value to the end-user.
I think that many people consider that documentation is useless because of the Manifesto for Agile Software Development, which promotes the value of “Working software over comprehensive documentation”. But in my opinion this sentence is often misunderstood: it just means that the best documentation in the world is no excuse if your project fails to deliver a working software.
Moreover, the lack of tooling to browse through/update documentation makes documentation-related tasks long and painful. Any developer who had to maintain documentation up-to-date with the changes made on their software will agree: documentation update is quite a burden. When you make modifications on the code, how many of you take the time to browse through the hundreds of pages of documentation just to find where to document these changes, and check the whole documentation consistency afterwards? No one, and thus documentation becomes useless, as it does not reflect the recent changes you made on your software.
Sooner or later, no one will trust and hence read it any more. Consequently, the design choices your team made will vanish, although you spent a lot of time trying to document these choices.This is the main reason for which documentation is not popular: although it is time-consuming, you are never sure that it is accurate and useful.
How would you try to convince a developer to write doc, especially if there are severe time constraints on a project?
Let’s continue with the testing example: what convinced people that writing test was as important as writing code?
First of all, the context matters: people should be aware that documentation is important and understand the issues caused by missing or unsynchronized doc. Fortunately, we are not the only ones claiming that documentation is important, even in Agile processes: Kurt Solarte (IBM) talks about the importance of Agile Documentation and the “maintenance documentation gap”, Scott Ambler talks about “Agile/Lean Documentation “ and defines a set of best practices to update the doc, Andreas Rüping wrote a book about “Agile documentation”...
So people are starting to get interested in the improvements we can make (in process, in tooling) to make documentation update more efficient. I will just add that being Agile does not mean being short-sighted: remember that if your team's primary goal is indeed to develop software, its secondary goal is to enable your next effort, “preparing for the next game” as Cockburn would say. Yes, documentation can preserve knowledge within a team, and prevent developers from re-inventing things when team members leave and new people join.
If you think about your activity as a developer, you will realize that you spend most of your time trying to understand other people's code. Don't you think you would gain time by having a clear and up-to-date documentation about the code you are studying? Yes? Why don't you write it for your own software then?
Thinking that doc is a waste of time is exactly like thinking tests are a waste of time: it takes effort to write, but it allows to save time when your software evolves.
But if we refer to the testing example, what really convinced people was not the theoretical talk: it was the emergence of tooling allowing to quickly write tests (such as Junit). That's where Intent comes into play.
How does Intent lower the barrier for writing good and useful documentation?
Intent is a documentation environment integrated to the Eclipse IDE. So first of all you will be able to edit your documentation directly inside your IDE, without having to switch to another tool. Moreover, you will be able to open an Intent editor on any sub-part of the documentation, no matter how small. The idea is to follow the “loading off your brain” mantra shared by all the Mylyn tools: we only show to the user what he currently needs to see, and not a hundred of pages document.
As explained previously, there is an inconvenient truth about documentation: although we're in 2012, performing a task as simple as updating a documentation according to changes made on the code is totally manual, long and painful. That's exactly why we decided to create the Mylyn Intent project: to provide tooling for keeping your documentation synchronized with any technical artifact (Java code, models, C++ code, plugin dependencies, a code sample on the internet...).
How does this tooling work?
To do so, we simply maintain formal links between each documentation part and the software artifacts it is describing. Whenever you modify your code, Intent will use these formal links to determine which parts of the documentation should be updated, avoiding you a painful manual research. Intent will place error markers on all the outdated documentation parts, that you will be able to see and open in one click in the Eclipse Problem View. Then you can graphically visualize the differences between what's specified in the Intent document and the corresponding technical artifact (e.g. “The class has been renamed”), and decide to update the documentation or the code.
So when do I update the outdated parts?
You are free to update the outdated documentation parts whenever you want. It's up to you: just choose the most efficient way of updating documentation according to your process. The Disciplined Agile Delivery describes several processes for updating your documentation: you may want to update it right away (“document continuously”), wait for the code to stabilise (e.g. the issue is closed), update the documentation just before a release (“document late”) or when people complain (“document only when it hurts”)...
So Intent is basically a synchronization tool?
Intent is much more as it is a bridge between technical artifacts and high-level concerns; it is a perfect medium to define high-level constraints on your software. As a matter of fact, I'm sure that you already defined such constraints: your team shares the same dev guidelines, processes and good practices.
But most of the time these constraints are not expressed formally: best case scenario, they are expressed in natural language inside a documentation file. Consequently, reviewing the code to check those is a very hard and painful job, and can lead you to endorse the bad role when having to explain to your developers that they do not code like you want them to. Moreover, as these constraints are not expressed formally, you are never sure that you fulfilled all them.
What does Intent do more specifically?
Intent provides tooling for automatically checking these constraints and guidelines. For example, you can plug inside Intent a rule that says that “Any bug should not be closed if the corresponding test has not been written”, "Any feature displayed to the end-user should be presented inside the User Guide with a screenshot"... As soon as such a rule is broken, Intent displays validation issues in the Eclipse Problem View, pointing out the elements that broke it. Consequently, as a developer you can know before committing that you broke some of the dev guidelines, and fix the issue right away.
To sum-up, Intent will help you write an up-to-date documentation (by maintaining formal links between documentation & code) and automatically checks high-level constraints. In result, you will be able to rely on the documentation to store your team knowledge. You will gain time when searching for information (no need to ask other team members), and in code review (as Intent checks a lot of dev guidelines automatically).
Intent is a subproject under Mylyn Docs. How does it tie into the general purpose of Mylyn/Tasks?
The Mylyn top-level project describes itself as an Application Lifecycle Management (ALM) framework for Eclipse. Each Mylyn project brings tooling for answering ALM concerns.
As a Mylyn citizen, Intent brings additional value to the ALM prime objectives:
- “Increases productivity, as the team shares best practices for development”: Intent helps the team sharing the same best practices and automatically checks them.
- “Developers need focus only on current business requirements”: Intent allows to consider only the parts of the documentation that are relevant to your current task (“loading off your brain”). Only the relevant sub-parts of the documentation will be embedded inside the Context of a Mylyn Task. Moreover, it is way easier to retrieve all the requirements related to the code you are currently interested in.
- “Breaks boundaries through collaboration and smooth information flow“: as explained later in this article, Intent allows to share documentation and ideas in real-time.
- “Cuts maintenance time by synchronizing application and design”: that's exactly what Intent aims to do.
Moreover, documentation is an important aspect of ALM. That's why the “Mylyn docs” component was created, holding sub-projects such as Wikitext or Intent.
A central premise or motto of the Intent project is what Donald Knuth stated in his book “Literate Programming”: Instead of imagining that our main task is to instruct a computer what to do, let us concentrate rather on explaining to human beings what we want a computer to do.” The book was written back in 1984 – why do you think this statement is still valid?
Donald E. Knuth has brought most of the concepts used in Intent. Literate Programming is a methodology that combines a programming language with a documentation language inside a same document, the Literate Program. Indeed, the main idea is to treat a program as a piece of literature, addressed to human beings rather than to a computer.
Of course, this still makes sense, even more than in 1984: if code was supposed to be read only by computers, we would write it using assembler languages. But we learned that coding is a social task: your code is meant to be read by several other people (including the future you), and you should focus on being as readable as you can when writing code. According to Knuth, keeping the intents behind the software, the reasons that lead you to make a design choice or write a piece of code, is as important as the code itself. Such intents cannot be properly captured inside the code (e.g. through javadoc), because the code structure is too narrowed to present high-level specifications.
But that idea isn’t implemented directly in Intent, or is it?
We have three main differences with the Literate programming approach:
Literate Programming was about mixing documentation and C code inside a same document. With Intent, you can mix documentation with any kind of technical artifact (models, Java code, C++ code, workspace configuration...). For the Juno release, we focused on what could be called the Literate Modeling approach, mixing documentation and any kind of EMF models. We already have business opportunities in this field, where Intent is used to document Enterprise Architectures, System Engineering models (using SysML)...
With Literate Programming, you could not directly modify the code: you had to update your Literate Program, and then re-generate the code. With Intent, we consider that code should still be edited through with its dedicated tooling, models should still be edited using modelers... Just use the right tool for the right task.
With Literate Programming, you had to make your code and your documentation evolve in the same time. In Intent, we consider that documentation and code have different lifecycles. It is acceptable to have a documentation that does not reflect the changes made a few days ago. So when the documentation or the code evolves, we just put synchronization issues on the outdated parts, letting you choose when it is appropriate to change it.
How does Intent facilitate document sharing?
We believe that documenting is a social task. Sharing documentation in real-time is not just a nice-to-have feature, but a real need, that allows real-time reviews and feedbacks on your specifications and ideas. Sharing documents through e-mails or VCS is not appropriated for this real-time collaboration need, so we designed Intent to allow live collaboration around docs.
Intent relies on a client/server architecture: we have a central Repository, in which we store the Intent document as a model, and many other useful information (like all the validation and synchronization issues, traceability information...). Each feature of Intent is managed by a client of this Repository: the Eclipse editor is just one among others. Each client communicates with others through a notification mechanism ensured by the Repository.
This Repository concept is totally abstract, and can be implemented with many concrete back-ends. For the Eclipse Kepler release, we are working on a CDO-based Intent Repository. If you are not familiar with CDO, you should definitely give it a try, it is a great piece of technology. In a nutshell, CDO allows live sharing of models, with real-time updates and fine-grained lock management to avoid conflicts.
With a CDO-based Repository, many users can read and edit the same Intent document. As soon as a user saves his document, the changes are propagated through the Repository, and other users see those changes in real-time.
We are currently deploying this technology on the Obeo SmartEA tool, by connecting Intent to a CDO server storing Enterprise Architecture models. It is a very good use case for Intent, as EA models contain complex and business-critical information that needs to be rigorously documented. Intent represents a nice solution allowing to easily keep the EA models and the documentation up-to-date, and to ease the impact analysis when simulating migration of the Enterprise Architecture. The next step will be to work on performances, to be able to handle huge models and Intent documents.
Can you explain what “Ariadne” is about?
Ariadne is an open-source tool allowing to represent many technical artifacts (Java code, GIT repositories and commits, bugzilla issues, plugin dependencies, checkstyle configuration...) as Models. It also features visualization mechanisms providing you a bird-eye view of your projects.
By integrating Ariadne with Intent, we will be able to link pieces of an Intent document with these technical artifacts. For example, if you change the location of the git Repository in which your example projects are stored, Intent will raise synchronization issues in the part of the documentation that explains where to check-out the example projects.
The most powerful feature of Ariadne is that, by linking all these technical artifacts together, you can write complex constraints on your project: for example, you can make sure that any developer works with the expected target platform, checkstyle rules, mylyn connectors...You can also check that any git commit references a bugzilla issue, automatically generate API changes...
Your session at EclipseCon Europe 2012 was packed, and surely not only because your session had a catchy title… Do you think there is a growing awareness of the importance of documentation?
Definitely. As I explained earlier, more and more influential people are stressing out how important documentation is and try to make developers aware of it.
Especially in open-source projects, in which transparency and external contributions are major concerns, people should take care of their documentation. And I think that is why more and more people are getting interested in Intent.
The Eclipse Foundation has recently launched an ongoing effort to improve its processes and technologies, in order to lower the barrier for external contributions. In that context, we should define tools and processes to help people write meaningful contribution guides. I really think that Intent & Ariadne can help lowering the barrier of contribution.
Can you give us a short preview of Intent in 2013?
Our prime objective is to make developers consider documentation as something useful and easy to maintain, that they can use to formalize their best practices and share their knowledge. Hence the roadmap for the Eclipse Kepler release is straightforward (see “Mylyn Intent: the Shape of things to come” for more details):
- Improve the tooling, for example dropping a Java Class inside the documentation will automatically create a link between the doc and the Class. We also plan to provide new quick-fixes for updating the documentation, create Eclipse Views allowing to see what parts of your project are documented, provide new export formats (PDF, HTML Bootstrap)
- Work on the CDO support (see question 6) to provide the best collaborative experience possible.
- Work on the integration with Ariadne, as we know that code synchronization capabilities is the most wanted feature by the eclipse community.
Although we have a strong vision of what we want, Intent is an open-source project, and we are willing to postpone any feature to answer to the community needs. Do not hesitate to spam us on the Intent forum with questions, ideas, feedbacks…
There is still a lot of work to do on this awesome topic, and we will gladly welcome external contributions or committers. I’m convinced that Intent is the future tooling for managing our documentation.