Skip to main content
SpringerLink
Log in

The effectiveness of test-driven development: an industrial case study

  • Published:
Software Quality Journal Aims and scope Submit manuscript

Abstract

Test-driven development (TDD) is a software development practice, where test cases are incrementally written before implementing the production code. This paper presents the results of a multi-case study investigating the effectiveness of TDD within an industrial environment. Three comparable medium-sized projects were observed during their development cycle. Two projects were driven without TDD practice, while the third one introduced TDD into the development process. The effectiveness of TDD was expressed in terms of external code quality, productivity, and maintainability. Our results indicate that the TDD developers produced higher quality code that is easier to maintain, although we did observe a reduction in productivity.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1

Similar content being viewed by others

Notes

  1. http://www.extremeprogramming.org/rules/unittests.html

  2. In incremental development, the releases are defined by beginning with one small, functional subsystem and then adding functionality with each new release (Pflegeer and Atlee 2006).

  3. JUnit—A regression testing framework written by Erich Gamma and Kent Beck, http://www.junit.org.

  4. CCCC—C and C ++ Code Counter http://cccc.sourceforge.net/, a free software tool for the measurement of source code related metrics by Tim Littlefair.

  5. Session Initiation Protocol, Internet Engineering Task Force (IETF), RFC 3261, http://datatracker.ietf.org/doc/rfc3261/).

  6. Primavera Progress Reporter is a web-based program that connects project team members across projects, throughout an enterprise.

  7. ISO/IEC 9,126. (2001). Software engineering—product quality—part 1: Quality model. ISO 2001.

  8. Cyclomatic complexity is a measure of the complexity of code related to the number of ways there are to traverse a piece of code (McCabe 1976).

  9. An unacceptable behavior of the software is defined as a failure. A fault is a defect in the program that, when executed with certain input data, causes a failure.

  10. See Albrecht and Gaffney (1983) for details.

  11. Unit test code was excluded.

  12. Sanchez et. al. 2007 reported the average ratio across all ten releases as 0.61.

References

  • Albrecht, A. J., & Gaffney, J. E., Jr. (1983). Software function, source lines of code, and development effort prediction: A software science validation. IEEE Transactions on Software Engineering, SE-9(6), 639–648.

    Google Scholar 

  • Beck, K. (2001). Aim, fire. IEEE Software, 18(5), 87–89.

    Article  Google Scholar 

  • Beck, K. (2003). Test driven development: by example. USA: Addison Wesley Professional.

    Google Scholar 

  • Bhat, T., Nagappan, N. (2006). Evaluating the efficacy of test-driven development: industrial case studies. Proceedings of International Symposium, Empirical Software Engineering (ISESE 06), ACM Press, 356–363.

  • Crispin, Lisa. (2006). Driving software quality: how test-driven development impacts software quality. IEEE Software, 23(6), 70–71. doi:10.1109/MS.2006.157.

    Article  Google Scholar 

  • Cusumano, M. A., & Selby, R. W. (1997). How microsoft builds software. Communications of the ACM, 40(6), 53–61.

    Article  Google Scholar 

  • Damm, L.-O., & Lundberg, L. (2006). Results from introducing component-level test automation and test-driven development. Journal of Systems and Software, 79, 1001–1014. Elsevier Science Inc.

    Article  Google Scholar 

  • Edwards, S. H. (2003). Using test-driven development in the classroom: providing students with concrete feedback on performance. Proceedings of International Conference on Education and Information Systems: Technologies and Applications (EISTA ‘03).

  • Erdogmus, H., Morisio, M., & Torchiano, M. (2005). On the effectiveness of the test-first approach to programming. IEEE Transactions on Software Engineering, 31(3), 226–237.

    Article  Google Scholar 

  • Fenton, N. E., & Ohlsson, N. (2000). Quantitative analysis of faults and failures in a complex software system. IEEE Transactions on Software Engineering, 26(8), 797–814.

    Article  Google Scholar 

  • Fenton, N. E., & Pfleeger, S. L. (1998). Software metrics: A rigorous & practical approach (2nd ed.). Boston, USA: PWS Publishing Company.

    Google Scholar 

  • Fowler, M., Beck, K., Brant, J., Opdyke, W., & Roberts, D. (1999). Refactoring: Improving the design of existing code. USA: Addison Wesley.

    Google Scholar 

  • George, B., Williams, L. (2003). An initial investigation of test driven development in industry. ACM Symposium on Applied Computing (SAC), March.

  • Janzen, D. S., & Saiedian, H. (2008). Does test-driven development really improve software design quality? IEEE Software, 25(2), 77–84.

    Article  Google Scholar 

  • Jeffries, R. E. (1999). Extreme Testing. Software Testing and Quality Engineering, 1(2), 23–26, March/April.

  • Jeffries, R., Melnik, G. (2007). TDD: The art of fearless programming. IEEE Software, May/June, 24–30.

  • Jovanović, D., Dogša, T. (2003). Comparison of software development models and their usage in computer-telephony systems. In ConTEL 2003: Proceedings of the 7th international conference on telecommunications, University of Zagreb, Faculty of Electrical Engineering and Computing, Zagreb, Croatia (vol. 2, pp. 587–592), June 11–13.

  • Kitchenham, B., Pickard, L., & Pfleeger, S. L. (1995). Case studies for method and tool evaluation. IEEE Software, 12(4), 52–62.

    Article  Google Scholar 

  • Madeyski, L. (2005). Preliminary analysis of the effects of pair programming and test-driven development on the external code quality. Frontiers in Artificial Intelligence and Applications.Proceeding of the 2005 conference on Software Engineering, vol. 130, pp. 113–123.

  • Madeyski, L., Szała, Ł. (2007). The impact of test-driven development on software development productivity—An empirical study. Software Process Improvement 14th European Conference, EuroSPI 2007, Potsdam, Germany, September 26–28, 2007 in Proceedings lecture notes in computer science, Springer Berlin/Heidelberg, vol. 4764.

  • Malik, S., Palencia, J. R. (1999). Synchronize and Stabilize vs. Open-source–An analysis of 2 team-organization models when developing consumer level shrink-wrapped software. Computer Science 95.314A Research Report, Carleton University, Ottawa, Ontario, Canada.

  • Martin, R. C., Newkirk, J. W., Kess, R. S. (2002). Agile software development, principles, patterns, and practices (1st ed.). Prentice Hall, Englewood Cliffs.

  • Maurer, F., Martel, S. (2002). On the productivity of agile software practices: An industrial case study. Technical report, University of Calgary, Department of Computer Science, Calgary, Alberta, Canada.

  • Maximilien, M., Williams, L. (2003). Assessing test-driven development at IBM. Proceedings of 25th International Conference on Software Engineering (pp. 564–569).

  • McCabe, T. J. (1976). A complexity measure. IEEE Transactions on Software Engineering, 2(4), 308–320.

    Article  MathSciNet  Google Scholar 

  • Mueller, M. M., & Hagner, O. (2002). Experiment about Test-First Programming. IEE Proceedings–Software, 149(5), 131–136.

    Google Scholar 

  • Newkirk, J., Vorontsov, A. (2004). Test-driven development in microsoft NET. Microsoft Press.

  • Pfleeger, S. L., & Atlee, J. M. (2006). Software engineering: Theory and practice. New Jersey, USA: Prentice Hall.

    Google Scholar 

  • Sanchez, J., Williams, L., Maximilien, E. M. (2007). A Longitudinal Study of the Use of a Test-Driven development Practice in Industry. AGILE 2007 Conference (AGILE 2007), 13–17 August, Washington, DC, USA. IEEE Computer Society 2007, ISBN 0-7695-2872-4.

  • Shadish, W. R., Cook, T. D., & Campbell, D. T. (2002). Experimental and Quasi-experimental designs for generalized causal inference. Boston, MA: Houghton Mifflin Company.

    Google Scholar 

  • Siniaalto, M., Abrahamsson, P. (2007). A comparative case study on the impact of test-driven development on program design and test coverage. Proceedings of the First International Symposium on Empirical Software Engineering and Measurement, IEEE Computer Society, Washington, pp. 275–284, ISBN ~ ISSN: 1938–6451.

  • Stephens, M., Rosenberg, D. (2004). The Irony of Extreme Programming. Dr. Dobb’s Journal, May 2004.

Download references

Author information

Authors and Affiliations

  1. Faculty of Electrical Engineering and Computer Science, Centre for Verification and Validation of Systems, University of Maribor, Smetanova ul. 17, SI-2000, Maribor, Slovenia

    Tomaž Dogša

  2. Agileon d.o.o., Cesta XIV. Divizije 51, SI-2000, Maribor, Slovenia

    David Batič

Authors
  1. Tomaž Dogša
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. David Batič
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to David Batič.

Appendices

Appendix 1

See Table 3.

Table 3 Projects details
Full size table

Appendix 2

See Table 4.

Table 4 Survey after TDD training
Full size table

Appendix 3

See Table 5.

Table 5 Final survey
Full size table

Rights and permissions

Reprints and permissions

About this article

Cite this article

Dogša, T., Batič, D. The effectiveness of test-driven development: an industrial case study. Software Qual J 19, 643–661 (2011). https://doi.org/10.1007/s11219-011-9130-2

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11219-011-9130-2

Keywords

Search

Navigation