Time estimation for software testing
Sand running through the bulbs of an hourglass image via Shutterstock
Testing is a very important part of a development process that allows to achieve the level of quality which enables the product to be released commercially. However, thoroughly going through every potential risk and covering it with test cases can take a long time. Often developers are forced to find the right balance between the time spent on testing and a release date.
If the testing process is correctly performed by an experienced professional, it is entirely possible to achieve the level of quality that satisfies all requirements without going beyond the time and budget constraints of the project. However, if time estimation was wrong or too imprecise, it is very easy to miss the designated release date or deliver a product that is not in accordance with the standards set by the market. Therefore, the importance of software testing estimation should not be underestimated.
There are various approaches and estimation techniques in software testing. We created this article from our own practical experience at Apriorit to give you a better idea of how to do software testing estimation when starting a new project.
Statistically speaking, testing occupies 20 percent of the overall development time for a single-component application, 20 to 30 percent for a two-component application and 30 to 35 percent for an application with GUI. For a distributed application with GUI the number can be as high as 35 to 50 percent. However, these numbers are very general and in practice, each project and each agile development model has its own unique set of risks and challenges that need to be accounted for.
To produce a more accurate estimate, we need to use a decomposition method. In other words, you should split your testing process into smaller chunks and estimate the time required to complete each of them separately. This is one of the most effective methods for software testing.
Generally, we can distinguish five main stages of the testing process for a new product:
General process planning
Test plan and test case development
Test environment configuration
Test case execution and debugging after initial run or after product changes
General process planning
Process planning consists of two parts: project research and choosing a testing strategy.
The analysis of project documentation, initial meeting and discussion with management are all part of the research process. Each company has its own approach regarding how to conduct research and how much time one needs to spend on it, but we recommend you take 1-2 days to complete this task.
While it may not take the whole day to complete the research, this way you will give yourself some space to get ready for potential risks and unexpected questions and new ideas.
Additional time for this stage may be required if:
Project is very big and has a high volume of documentation
Team members have no previous experience working on similar projects, requiring additional time to familiarize themselves with the topic
Team members are geographically dispersed and have different operating schedules, requiring additional time to set up meetings.
Choosing a testing strategy
Your testing strategy includes a number of factors that need to be clearly defined at the very beginning of the testing process, including:
The type of testing performed
Qualification, required for developing test plans and test cases
How many QA specialists will be assigned to the project and what skills they need to possess
Experience has taught us that the best method is to create a predefined sample pattern that includes all types of testing you can perform and then make your selection.
The pattern is adjusted depending on the type of project and current quality requirements. Such adjustments can take up to an hour.
It also worth noting that designating QA specialists at this stage allows you to take into account their personal skills and experience, making QA estimation more precise. It is also a good opportunity to include days off and vacations into a testing schedule.
Test plan and test case development
Test plan and test case development are rather extensive activities that will require a lot of time and effort from your QA specialist. Test plan should always be derived from a strategy, defined in a previous stage. Your testing strategy indicates the kinds of test cases that should be developed and what priority they should have.
A good rule of thumb is that each single product requirement should be covered by at least five test cases. This allows you to get an approximate number of test cases and also to estimate the time required to create them.
On average, a single test case requires 10 minutes of development, although this heavily depends on the complexity of your test plan. Generally, test plan without any test cases requires a couple of days to develop. Additional time for creating test cases should be taken into account if your project requires them.
At this stage, the following factors should be taken into consideration:
Experience of your QA specialist. If this is the first time they produce test plan or test cases, more time should be included in the estimation.
Familiarity with used technology. Your specialists may require additional research, if they are not familiar with the technology used in this project. Such research could take up to several weeks depending on the qualification of QA specialists and complexity of material they need to cover.
Test data creation. A project may require test data. Time spent on its creation should be included in the estimation.
Order of development. Test cases should be developed in the same order in which they are going to be run. You can run either tests covering the most important features and modules of the project first, or tests covering parts of the project that are currently ready for testing.
Test plans should be created based on the order in which you decide to run tests. There are several factors that should be taken into account during the estimation process:
Previous experience with similar projects. In this case, you can take old test plans and test cases and use them as a basis for the new ones, which should help you gain some time.
Review time. Time for reviewing both new and any altered old test plans and test cases should be specifically designated. Experience, schedule and workload of a specialist designated to do reviews will impact the amount of time need to handle this. Therefore, these factors should be taken into account when coming up with an estimation.
Time environment configuration
Estimated time for test environment configuration depends on the following:
Equipment availability. If the company does not possess the necessary equipment to set up testing environment, time required to get it should be considered.
Installation and configuration time. How long does installing and configuring all the necessary equipment take depending on the experience of your specialists? A qualified system administrator will be able to perform the required work much faster than a QA specialist can.
For a middle-sized project, setting up and configuring a test environment which consists of relatively common software and hardware usually takes anywhere between an hour and a day depending on the complexity of the environment.
Test case execution and debugging
Although time varies depending on the complexity of the test case, as a rule of thumb, it takes a QA specialist approximately five minutes on average to execute one. However, if testing is done by an inexperienced QA specialist, it is safer to estimate 10 minutes for a single test case.
The biggest challenge at this stage is the fact that the number of found bugs and how easily they can be reproduced can hardly be accurately predicted. It may take several hours to find a cause of a complex bug. A high amount of bugs also implies more time for creating bug reports. All this time should be included in an estimation along with various other risks. In practice, it is usually best to add 20-25 percent on top of a final estimation to account for all possible risks and challenges.
Test case debugging after first run or after changes were introduced to the product usually takes about 10-15 percent of an overall time for test case and test plan creation. Thorough test case review can shave off this time dramatically, however, the time for such a review should also be taken into account.
For regression testing, the following factors should be taken into account:
The type of testing required. Sometimes, just the smoke testing will be enough, while in other cases, repeated full testing is needed.
Changes to test cases and test plan.
Estimation time can be cut off by assigning regression testing to the specialist who previously tested the product.
Software should be tested on all supported operating systems. For client-server applications, it is also important to test various combinations of supported systems. However, testing every single possible combination is often impractical because of the large time and resource investment it requires.
In this case, it is best to choose system combinations that will allow you to cover every supported operating system without repetition. For an intermediate product version, full testing should be performed on the systems important to the user, while the rest of supported systems should be covered by smoke tests.
Creating a system combination matrix allows to easily decide what system combinations need to be tested and what type of tests will be performed; it also allows you to produce an accurate time estimation.
Time estimation for software testing is a very tough topic. While a lot of advanced testing estimation techniques are available, and there are a lot of variables to consider and risks to account for, it still can be quite hard to produce exact results. However, it doesn’t mean that producing accurate estimates is impossible.
By following the estimation template that takes into account the necessary time to complete all the steps mentioned in this article, and being mindful of every factor that could have an impact on each step, you will be able to produce fairly accurate estimates that will give you an accurate estimation most of the time. By using the software testing estimation techniques mentioned above, the accuracy of your estimation will increase in no time.