How much testing is enough? Initial steps in testing analytics
It’s never easy to know quite when you’ve finished testing. That’s why a group of researchers have developed an Eclipse plug-in that evaluates your testing habits.
As a Software Engineering researcher at the Delft University of Technology, the Netherlands, I often get asked “How much development time needs to be spent on unit testing to ensure quality?”
Surprisingly, this question isn’t only coming from students, but mostly from seasoned software engineers in the industry. While everybody is aware of the need to unit-test, it seems most of us do not have a clear idea of when to stop testing.
Science has no answer
Unfortunately, scientific literature doesn’t make us much wiser here. In his seminal paper “The Mythical Man Month”, Fred Brooks estimated almost 40 years ago that around 50% of development effort is testing.
However, it is questionable whether this rough estimation still holds, as a lot has changed in Software Engineering over the last 40 years: Just think of Junit or Eclipse. Nowadays, in an agile mindset developers want their tests to “fail fast” so that they have immediate feedback on the code they write.
Finding individual answers instead
With software development having changed dramatically over the last few years, now is the time to undertake a large-scale study on how software engineers do unit testing. And you as a Java Eclipse developer can help us! A team of researchers at the TU Delft has created WatchDog, an Eclipse plug-in that transparently records how you interact with Eclipse: how much time you spend on developing test code, how long it takes to execute your tests and how frequently you execute those tests.
WatchDog adds its WatchDog Statistics view to your Eclipse, in which you can check your personal answers to the above questions. Figure 1 is an example of the interactive statistics WatchDog generates on the fly for you.
In that short coding session, I had my Eclipse open for 20 minutes, and about 12 minutes I was actually actively working in it. About 3 minutes went to reading code, and about 2 minutes to writing code. The rest of the time I was busy setting up the project and configuring Eclipse. I spent 4 minutes on production code, and 1.5 minutes on test code (both reading and writing it). Which tells me I should probably put more effort into my tests.
No way! What about privacy?
Watchdog only collects statistical information on your testing activities and won’t go publishing the data with a reference to you. You can take my word for it: WatchDog is open source. On the WatchDog homepage we show exactly what data we collect. And you’re invited to use a tool like WireShark to verify the information on the website is really what we collect.
Participating is also totally worth it because in February 2015 every participant will get a personal report on his or her own testing behavior. What can you learn from this report? Well, see for yourself.
Interested in learning about your own testing behaviour and advancing science? Point your Eclipse towards the update site and let the testing fun begin.