Hussan Munir; Misagh Moayyed MSE-2012:94, pp. 118. COM/School of Computing, 2012.
Context: Test-Driven development (TDD) is a software development approach where test cases are
written before actual development of the code in iterative cycles. TDD has gained attention of many
software practitioners during the last decade since it has suggested several benefits in the software
development process. However, empirical evidence of its dominance in terms of internal code quality,
external code quality and productivity is fairly limited.
Objectives: The aim behind conducting this study is to explore what has been achieved so far in the
field of Test-driven development. The study reports the benefits and limitation of TDD compared to
TLD and the outcome variables in all the reported studies along with their measurement criteria.
Additionally, an experiment is conducted to see the impact of Test-driven development (TDD) on
internal code quality, external code quality and productivity compared to Test-Last development
Methods: In this study two research methodologies are used specifically systematic literature review
according to Kitchenham guidelines and controlled pilot experiment. In systematic literature review
number of article sources are considered and used, including Inspec, Compendex, ACM, IEEE
Xplore, Science direct (Elsevier) and ISI web of science. A review protocol is created first to ensure
the objectivity and repeatability of the whole process. Second, a controlled experiment is conducted
with professional software developers to explore the assumed benefits of Test-Driven development
(TDD) compared to Test-Last development (TLD).
Results: 9 distinct categories related to Test-driven development (TDD) are found that are
investigated and reported in the literature. All the reported experiments revealing very little or no
difference in internal code quality, external code quality and productivity in Test-Driven development
(TDD) over Test-Last development (TLD). However, results were found contradictory when research
methods are taken into account because case studies tend to find more positive results in the favor
Test-Driven development (TDD) compared to experiments possibly due to the fact that experiment are
mostly conducted in artificially created software development environment and mostly with students
as a test subjects. On the other hand, experimental results and statistical analysis show no statistically
significant result in the favor TDD compared to TLD. All the values found related to number of
acceptance test cases passed (Mann-Whitney U test Exact Sig. 0.185), McCabe’s Cyclomatic
complexity (Mann-Whitney U test Exact Sig. 0.063), Branch coverage (Mann-Whitney U test Exact
Sig. 0.212), Productivity in terms of number of lines of code per person hours (Independent sample Ttest
Sig. 0.686), productivity in terms number of user stories implemented per person hours
(Independent sample T-test Sig. 0.835) in experiment are statistically insignificant. However, static
code analysis (Independent sample T-test Sig. 0.03) result was found statistically significant but due to
the low statistical power of test it was not possible to reject the null hypothesis. The results of the
survey revealed that the majority of developers in the experiment prefer TLD over TDD, given the
lesser required level of learning curve as well as the minimum effort needed to understand and employ
TLD compared to TDD
Conclusion: Systematic literature review confirms that the reported benefits of TDD development
compared to Test-Last development are very small. However, case studies tend to find more positive
results in the favor of Test-Driven development (TDD) compared to Test-Last development (TLD).
Similarly, experimental findings are also confirming the fact that TDD has small benefits over TLD.
However, given the small effect size there is an indication that (Test-Driven development) TDD
endorses less complex code compared to Test-Last development (TLD).