Davide Fucci
Burak Turhan
Markku Oivo
ABSTRACT
Context: One limitation of the empirical studies about test-driven development (TDD) is knowing whether the developers followed the advocated test-code-refactor cycle. Research dealt with the issue of process conformance only in terms of internal validity, while investigating the role of other confounding variables that might explain the controversial effects of TDD. None of the research included process conformance as a fundamental part of the analysis.
Goal: We aim to examine the impact of process conformance on the claimed effects of TDD on external quality, developers' productivity and test quality.
Method: We used data collected during a previous study to create regression models in which the level of process conformance was used to predict external quality, productivity, and tests thoroughness.
Result: Based on our analysis of the available data (n = 22), we observe that neither quality (p -- value = 0.21), productivity (p -- value = 0.80), number of tests (p -- value = 0.39) nor coverage (p -- value = 0.09) was correlated with the level of TDD process conformance.
Conclusion: While based on a small sample, we raise concerns about how TDD is interpreted. We also question whether the cost of strictly following TDD will pay-off in terms of external quality, productivity, and tests thoroughness.
- P. Abrahamsson, A. Hanhineva, and J. Jäälinoja. Improving business agility through technical solutions: A case study on test-driven development in mobile software development. In Business Agility and Information Technology Diffusion, pages 227--243. Springer, 2005.Google Scholar
Cross Ref
- J. G. Adair. The Hawthorne Effect: A Reconsideration Of The Methodological Artefact. Journal of Applied Psychology, 69(2):334, 1984.Google ScholarCross Ref
- D. Astels. Test Driven development: A Practical Guide. Prentice Hall Professional Technical Reference, 2003. Google ScholarDigital Library
- K. Beck. Test-driven Development: by Example. The Addison-Wesley signature series. Addison-Wesley, 2003. Google ScholarDigital Library
- J. Bowyer and J. Hughes. Assessing Undergraduate Experience Of Continuous Integration And Test-Driven Development. In Proceedings Of The 28th International Conference On Software Engineering, pages 691--694. ACM, 2006. Google ScholarDigital Library
- A. Causevic, D. Sundmark, and S. Punnekkat. Factors Limiting Industrial Adoption Of Test Driven Development: A Systematic Review. In Software Testing, Verification and Validation (ICST), 2011 IEEE Fourth International Conference on, pages 337--346. IEEE, 2011. Google ScholarDigital Library
- A. Čaušević, D. Sundmark, and S. Punnekkat. Impact of Test Design Technique Knowledge on Test Driven Development: A Controlled Experiment. In Agile Processes in Software Engineering and Extreme Programming, pages 138--152. Springer, 2012.Google ScholarCross Ref
- J. Cohen and P. Cohen. Applied Multiple Regression/Correlation Analysis For The Behavioral Sciences. Lawrence Erlbaum, 1975.Google Scholar
- J. E. Cook and A. Wolf. Toward metrics for process validation. In Software Process, In Proceedings of the Third International Conference on, pages 33--44, 1994.Google Scholar
- S. H. Edwards. Using Test-Driven Development In The Classroom: Providing Students With Automatic, Concrete Feedback On Performance. In Proceedings of the International Conference on Education and Information Systems: Technologies and Applications EISTA, volume 3, 2003.Google Scholar
- H. Erdogmus, M. Morisio, and M. Torchiano. On the Effectiveness of the Test-First Approach to Programming. IEEE Transactions on Software Engineering, 31(3):226--237, 2005. Google ScholarDigital Library
- M. Fowler. Refactoring: Improving The Design Of Existing Code. Addison-Wesley Professional, 1999. Google ScholarDigital Library
- S. Fraser, K. Beck, B. Caputo, T. Mackinnon, J. Newkirk, and C. Poole. Test driven development. In Extreme Programming and Agile Processes in Software Engineering, pages 459--462. Springer, 2003. Google ScholarDigital Library
- S. Freeman and N. Pryce. Growing Object-Oriented Software, Guided By Tests. Pearson Education, 2009. Google ScholarDigital Library
- D. Fucci and B. Turhan. A Replicated Experiment on the Effectiveness of Test-First Development. In Empirical Software Engineering and Measurement. ACM/IEEE International Symposium on, pages 103--112. IEEE, 2013.Google ScholarCross Ref
- D. Fucci and B. Turhan. On the Role of Tests in Test-driven Development: a Differentiated and Partial Replication. Empirical Software Engineering Journal, pages 1--26, 2013. Google ScholarDigital Library
- B. George and L. Williams. A structured experiment of test-driven development. Information and Software Technology, 46(5):337--342, 2004.Google ScholarCross Ref
- T. Henchy and D. C. Glass. Evaluation Apprehension And The Social Facilitation Of Dominant And Subordinate Responses. Journal of Personality and Social Psychology, 10(4):446, 1968.Google ScholarCross Ref
- J. R. Horgan, S. London, and M. R. Lyu. Achieving Software Quality With Testing Coverage Measures. Computer, 27(9):60--69, 1994. Google ScholarDigital Library
- D. S. Janzen and H. Saiedian. Does Test-Driven Development Really Improve Software Design Quality? Software, IEEE, 25(2):77--84, 2008. Google ScholarDigital Library
- P. Johnson and H. Kou. Automated Recognition of Test-Driven Development with Zorro. In AGILE 2007, pages 15--25. IEEE, 2007. Google ScholarDigital Library
- C. Kaner. Software negligence and testing coverage. Software QA Quarterly, 2(2), 1996.Google Scholar
- M. I. Kellner, P. H. Feiler, A. Finkelstein, T. Katayama, L. J. Osterweil, M. H. Penedo, and H. D. Rombach. Software Process Modeling Example Problem. In Software Process Workshop, Proceedings of the 6th International, pages 19--29. IEEE, 1990.Google ScholarCross Ref
- H. Kou, P. M. Johnson, and H. Erdogmus. Operational Definition And Automated Inference Of Test-Driven Development With Zorro. Automated Software Engineering, 17(1):57--85, 2010. Google ScholarDigital Library
- W. Krebs. Turning The Knobs: A Coaching Pattern For Xp Through Agile Metrics. In Extreme Programming and Agile Methods, pages 60--69. Springer, 2002. Google ScholarDigital Library
- L. Madeyski. The Impact Of Pair Programming And Test-Driven Development On Package Dependencies In Object-Oriented Design --- An Experiment. Product-Focused Software Process Improvement, pages 278--289, 2006. Google ScholarDigital Library
- L. Madeyski and Ł. Szała. The Impact Of Test-Driven Development On Software Development Productivity --- Empirical Study. In Software Process Improvement, pages 200--211. Springer, 2007. Google ScholarDigital Library
- Y. Malaiya, M. Li, J. Bieman, and R. Karcich. Software Reliability Growth With Test Coverage. Reliability, IEEE Transactions on, 51(4):420--426, 2002.Google Scholar
- Y. K. Malaiya, N. Li, J. Bieman, R. Karcich, and B. Skibbe. The Relationship Between Test Coverage And Reliability. In Software Reliability Engineering. Proceedings., 5th International Symposium on, pages 186--195. IEEE, 1994.Google Scholar
- B. Marick. How To Misuse Code Coverage. In Proceedings of the 16th International Conference on Testing Computer Software, pages 16--18, 1999.Google Scholar
- R. C. Martin. Professionalism and Test-driven Development. Software, IEEE, 24(3):32--36, 2007. Google ScholarDigital Library
- A. Montenegro. On Sample Size And Precision In Ordinary Least Squares. Journal of Applied Statistics, 28(5):603--605, 2001.Google ScholarCross Ref
- R. Mugridge. Challenges In Teaching Test Driven Development. In Extreme Programming and Agile Processes in Software Engineering, pages 410--413. Springer, 2003. Google ScholarDigital Library
- M. Müller and A. Höfer. The Effect of Experience on the Test-driven Development Process. Empirical Software Engineering, 12(6):593--615, 2007. Google ScholarDigital Library
- E. A. Pena and E. H. Slate. Global Validation Of Linear Model Assumptions. Journal of the American Statistical Association, 101(473):341--354, 2006.Google ScholarCross Ref
- M. Philipp. Comparison Of The Test-Driven Development Processes Of Novice And Expert Programmer Pairs. PhD thesis, University of Karlsruhe, 2009.Google Scholar
- Y. Rafique and V. B. Misic. The Effects of Test-Driven Development on External Quality and Productivity: A Meta-Analysis. IEEE Transactions on Software Engineering, 39(6):835--856, 2013. Google ScholarDigital Library
- W. R. Shadish, T. D. Cook, and D. T. Campbell. Experimental and Quasi-experimental Designs for Generalized Causal Inference. 2002.Google Scholar
- F. Shull, G. Melnik, B. Turhan, L. Layman, M. Diep, and H. Erdogmus. What Do We Know About Test-driven Development? Software, IEEE, 27(6):16--19, 2010. Google ScholarDigital Library
- L. Silva and G. Travassos. Tool-supported unobtrusive evaluation of software engineering process conformance. In Empirical Software Engineering, International Symposium on, pages 127--135, 2004. Google ScholarDigital Library
- S. Sørumgård. Verification of process conformance in empirical studies of software development. PhD thesis, Ph. D. thesis, Norwegian University of Science and Technology, 1997.Google Scholar
- Y. Wang and H. Erdogmus. The Role Of Process Measurement In Test-Driven Development. In 4th Conference on Extreme Programming and Agile Methods, 2004.Google Scholar
- C. Wohlin. Experimentation in Software Engineering: an Introduction, volume 6. Springer, 2000. Google ScholarDigital Library
- N. Zazworka, V. Basili, and F. Shull. Tool Supported Detection and Judgment of Nonconformance in Process Execution. In Proceedings of the 3rd ACM-IEEE International Symposium on Empirical Software Engineering and Measurement, pages 312--323, 2009. Google ScholarDigital Library
- N. Zazworka, K. Stapel, E. Knauss, F. Shull, V. R. Basili, and K. Schneider. Are Developers Complying with the Process: an XP Study. In Proceedings of the 4th ACM-IEEE International Symposium on Empirical Software Engineering and Measurement, pages 1--10, 2010. Google ScholarDigital Library
- K. Zieliriski and T. Szmuc. Preliminary Analysis Of The Effects Of Pair Programming And Test-Driven Development On The External Code Quality. Software Engineering: Evolution And Emerging Technologies, 130:113, 2006.Google Scholar
Index Terms
- Impact of process conformance on the effects of test-driven development
Recommendations
A longitudinal cohort study on the retainment of test-driven development
ESEM '18: Proceedings of the 12th ACM/IEEE International Symposium on Empirical Software Engineering and MeasurementBackground: Test-Driven Development (TDD) is an agile software development practice, which is claimed to boost both external quality of software products and developers' productivity.
Aims: We want to study: (i) the TDD effects on the external quality ...
An experimental evaluation of test driven development vs. test-last development with industry professionals
EASE '14: Proceedings of the 18th International Conference on Evaluation and Assessment in Software EngineeringTest-Driven Development (TDD) is a software development approach where test cases are written before actual development of the code in iterative cycles. Context: TDD has gained attention of many software practitioners during the last decade since it has ...
Impact of test-driven development on productivity, code and tests: A controlled experiment
Context: Test-driven development is an approach to software development, where automated tests are written before production code in highly iterative cycles. Test-driven development attracts attention as well as followers in professional environment; ...
Comments