Skip to main content
SpringerLink
Log in

Operational definition and automated inference of test-driven development with Zorro

  • Published:
Automated Software Engineering Aims and scope Submit manuscript

Abstract

Test-driven development (TDD) is a style of development named for its most visible characteristic: the design and implementation of test cases prior to the implementation of the code required to make them pass. Many claims have been made for TDD: that it can improve implementation as well as design quality, that it can improve productivity, that it results in 100% coverage, and so forth. However, research to validate these claims has yielded mixed and sometimes contradictory results. We believe that at least part of the reason for these results stems from differing interpretations of the TDD development style, along with an inability to determine whether programmers actually follow whatever definition of TDD is in use.

Zorro is a system designed to automatically determine whether a developer is complying with an operational definition of Test-Driven Development (TDD) practices. Automated recognition of TDD can benefit the software development community in a variety of ways, from inquiry into the “true nature” of TDD, to pedagogical aids to support the practice of test-driven development, to support for more rigorous empirical studies on the effectiveness of TDD in both laboratory and real world settings.

This paper describes the Zorro system, its operational definition of TDD, the analyses made possible by Zorro, two empirical evaluations of the system, and an attempted case study. Our research shows that it is possible to define an operational definition of TDD that is amenable to automated recognition, and illustrates the architectural and design issues that must be addressed in order to do so. Zorro has implications not only for the practice of TDD, but also for software engineering “micro-process” definition and recognition through its parent framework, Software Development Stream Analysis.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Similar content being viewed by others

References

  • Beck, K.: Extreme Programming Explained: Embrace Change. Addison Wesley, Massachusetts (2000)

    Google Scholar 

  • Beck, K.: Aim, fire. IEEE Softw. 18(5), 87–89 (2001). doi:10.1109/52.951502

    Article  Google Scholar 

  • Beck, K.: Test-Driven Development by Example. Addison Wesley, Massachusetts (2003)

    Google Scholar 

  • Bhat, T., Nagappan, N.: Evaluating the efficacy of test-driven development: industrial case studies. In: ISESE ’06: Proceedings of the 2006 ACM/IEEE International Symposium on International Symposium on Empirical Software Engineering, pp. 356–363. ACM Press, New York (2006). doi:10.1145/1159733.1159787

    Chapter  Google Scholar 

  • Bill Curtis, M.I.K., Over, J.: Process modeling. Commun. ACM 35(9), 75–90 (1992)

    Article  Google Scholar 

  • Cook, J.E.: Process discovery and validation through event-data analysis. Ph.D. Thesis, University of Colorado (1996)

  • Cook, J.E., Wolf, A.L.: Automating process discovery through event-data analysis. In: ICSE ’95: Proceedings of the 17th International Conference on Software Engineering, pp. 73–82. ACM, New York (1995). doi:10.1145/225014.225021

    Chapter  Google Scholar 

  • Damm, L., Lundberg, L.: Results from introducing component-level test automation and test-driven development. J. Syst. Softw. 79(7), 1001–1014 (2006)

    Article  Google Scholar 

  • Edwards, S.H.: Using software testing to move students from trial-and-error to reflection-in-action. In: Proceedings of the 35th SIGCSE Technical Symposium on Computer Science Education, pp. 26–30. ACM, New York (2004). doi:10.1145/971300.971312

    Chapter  Google Scholar 

  • Erdogmus, H., Morisio, M., Torchiano, M.: On the effectiveness of the test-first approach to programming. IEEE Trans. Softw. Eng. 31(3), 226–237 (2005). doi:10.1109/TSE.2005.37

    Article  Google Scholar 

  • Friedman-Hill, E.: JESS in Action. Mannig, Greenwich (2003)

    Google Scholar 

  • George, B., Williams, L.: An initial investigation of test-driven development in industry. ACM Symp. Appl. Comput. 3(1), 23 (2003)

    Google Scholar 

  • George, B., Williams, L.: A structured experiment of test-driven development. Inf. Softw. Technol. 46(5), 337–342 (2004)

    Article  Google Scholar 

  • Geras, A., Smith, M., Miller, J.: A prototype empirical evaluation of test driven development. In: Software Metrics, 10th International Symposium on (METRICS’04), pp. 405. IEEE Computer Society, Chicago (2004)

    Chapter  Google Scholar 

  • Gupta, A., Jalote, P.: An experimental evaluation of the effectiveness of test-driven development. In: ESEM 2007, 1st International Symposium on Empirical Software Engineering and Measurement, pp. 285–294. IEEE Computer Society, Chicago (2007)

    Chapter  Google Scholar 

  • Janzen, D., Saiedian, H.: Test-driven development:concepts, taxonomy, and future direction. Computer 38(9), 43–50 (2005). doi:10.1109/MC.2005.314

    Article  Google Scholar 

  • Janzen, D., Saiedian, H.: Does test-driven development really improve software design quality. In: IEEE Software, pp. 77–84 (2008)

  • Jeffries, R.: Extreme Programming Installed. Addison-Wesley, Upper Saddle River (2000)

    Google Scholar 

  • Jensen, C., Scacchi, W.: Process modeling across the web information infrastructure. In: Special Issue on ProSim 2004, The Fifth International Workshop on Software Process Simulation and Modeling, Edinburgh, Scotland (2004)

  • Jensen, C., Scacchi, W.: Experience in discovering, modeling, and reenacting open source software development processes. In: Proceedings of the International Software Process Workshop (2005)

  • Johnson, P.M., Paulding, M.G.: Understanding HPCS development through automated process and product measurement with Hackystat. In: Second Workshop on Productivity and Performance in High-End Computing (P-PHEC) (2005). http://csdl.ics.hawaii.edu/techreports/04-22/04-22.pdf

  • Johnson, P.M., Kou, H., Agustin, J.M., Zhang, Q., Kagawa, A., Yamashita, T.: Practical automated process and product metric collection and analysis in a classroom setting: lessons learned from Hackystat-UH. In: Proceedings of the 2004 International Symposium on Empirical Software Engineering, Los Angeles, CA (2004). http://csdl.ics.hawaii.edu/techreports/03-12/03-12.pdf

  • Johnson, P.M., Kou, H., Paulding, M.G., Zhang, Q., Kagawa, A., Yamashita, T.: Improving software development management through software project telemetry. In: IEEE Software (2005). http://csdl.ics.hawaii.edu/techreports/04-11/04-11.pdf

  • Kaufmann, R., Janzen, D.: Implications of test-driven development: a pilot study. In: OOPSLA ’03: Companion of the 18th Annual ACM SIGPLAN Conference on Object-Oriented Programming, Systems, Languages, and Applications, pp. 298–299. ACM, New York (2003). doi:10.1145/949344.949421

    Chapter  Google Scholar 

  • Kellner, M., Feiler, P., Finkelstein, A., Katayama, T., Osterweil, L., Penedo, M., Rombach, H.: Ispw-6 software process example. In: Proceedings of the First International Conference on the Software Process, Pittsburgh, PA (1991)

  • Kou, H.: Eclipse screen recorder (2006). http://csdl.ics.hawaii.edu/Tools/Esr/

  • Kou, H.: Automated inference of software development behaviors: Design, implementation and validation of Zorro for test-driven development. Ph.D. Thesis, University of Hawaii, Department of Information and Computer Sciences (2007). http://csdl.ics.hawaii.edu/techreports/07-04/07-04.pdf

  • Kou, H., Johnson, P.M.: Automated recognition of low-level process: a pilot validation study of Zorro for test-driven development. In: Proceedings of the 2006 International Workshop on Software Process, Shanghai, China (2006). http://csdl.ics.hawaii.edu/techreports/06-02/06-02.pdf

  • Madeyski, L., Szala, L.: The impact of test-driven development on software development productivity—an empirical study. In: Abrahamsson, P., Badoo, N., Margaria, T., Messanarz, R. (eds.) Software Process Improvement, 9th International Conference on Agile Processes in Software Engineering and Extreme Programming. Lecture Notes in Computer Science, vol. 4764, pp. 200–211. Springer, Heidelberg (2007)

    Google Scholar 

  • Maximilien, E.M., Williams, L.: Accessing test-driven development at IBM. In: Proceedings of the 25th International Conference in Software Engineering, p. 564. IEEE Computer Society, Washington (2003)

    Chapter  Google Scholar 

  • Mishali, O., Dubinsky, Y., Katz, S.: The tdd-guide training and guidance tool for test-driven development. In: Abrahamsson, P., Baskerville, R., Conboy, K., Fitzgerald, B., Morgan, L., Wang, X. (eds.) XP 2008: 9th International Conference on Agile Processes in Software Engineering and Extreme Programming. Lecture Notes in Business Information Processing, vol. 9, pp. 63–72. Springer, Heidelberg (2008)

    Chapter  Google Scholar 

  • Muller, M.M., Hagner, O.: Experiment about test-first programming. In: Empirical Assessment in Software Engineering (EASE). IEEE Computer Society, Los Alamitos (2002)

    Google Scholar 

  • Pančur, M., Ciglarič, M., Trampuš, M., Vidmar, T.: Towards empirical evaluation of test-driven development in a university environment. In: Proceedings of EUROCON 2003. IEEE, New York (2003)

    Google Scholar 

  • Sanchez, J., Williams, L., Maximillien, E.: On the sustained use of test-driven development practice at ibm. In: Agile 2007 Conference, pp. 5–14 (2007)

  • Siniaalto, M., Abrahamsson, P.A.: Comparative case study on the impact of test-driven development on program design and test coverage. In: ESEM 2007, 1st International Symposium on Empirical Software Engineering and Measurement, pp. 275–284. IEEE Computer Society, Los Alamitos (2007)

    Chapter  Google Scholar 

  • Sutton, S.M., Heimbigner, D., Osterwell, L.J.: APPL/A: a language for software process programming. ACM Trans. Softw. Eng. Methodol. 4(3), 221–286 (1995)

    Article  Google Scholar 

  • Turnu, I., Melis, M., Cau, A.: Introducing tdd on a free libre open source software project: a simulation experiment. In: QUTE-SWAP Workshop. ACM, New York (2004)

    Google Scholar 

  • Wang, Y., Erdogmus, H.: The role of process measurement in test-driven development. In: XP/Agile Universe, pp. 32–42 (2004)

  • Wege, C.: Automated support for process assessment in test-driven development. Ph.D. Thesis, Eberhard-Karls-Universit at Tubingen (2004)

  • Williams, L., Maximilien, E.M., Vouk, M.: Test-driven development as a defect-reduction practice. In: Proceedings of the 14th International Symposium on Software Reliability Engineering (ISSRE’03), pp. 298–299. ACM, New York (2003)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Collaborative Software Development Laboratory, Department of Information and Computer Sciences, University of Hawaii, Honolulu, HI, 96822, USA

    Hongbing Kou & Philip M. Johnson

  2. Kalemun Research Inc., 4462 Bittersweet Pl., Ottawa, ON, K1V1R9, Canada

    Hakan Erdogmus

Authors
  1. Hongbing Kou
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Philip M. Johnson
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Hakan Erdogmus
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Philip M. Johnson.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Kou, H., Johnson, P.M. & Erdogmus, H. Operational definition and automated inference of test-driven development with Zorro. Autom Softw Eng 17, 57 (2010). https://doi.org/10.1007/s10515-009-0058-8

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s10515-009-0058-8

Search

Navigation