Towards an operationalization of test-driven development skills: An industrial empirical study

Abstract

Context: The majority of the empirical studies on Test-driven development (TDD) are concerned with verifying or refuting the effectiveness of the technique over a traditional approach, and they tend to neglect whether the subjects possess the necessary skills to apply TDD, though they argue such skills are necessary.

Objective: We evaluate a set of minimal, a priori and in process skills necessary to apply TDD. We determine whether variations in external quality (i.e., number of defects) and productivity (i.e., number of features implemented) can be associated with different clusters of the TDD skills’ set.

Method: We executed a quasi-experiment involving 30 practitioners from industry. We first grouped the participants according to their TDD skills’ set (consisting of a priori experience on programming and testing as well as in-process TDD conformance) into three levels (Low-Medium-High) using k-means clustering. We then applied ANOVA to compare the clusters in terms of external quality and productivity, and conducted post-hoc pairwise analysis.

Results: We did not observe a statistically significant difference between the clusters either for external software quality ($F(2,27=1.44,$$p=.260$), or productivity ($F(2,27)=3.02,$$p=.065$). However, the analysis of the effect sizes and their confidence intervals shows that the TDD skills’ set is a factor that could account for up to 28% of the external quality, and 38% for productivity.

Conclusion: We have reason to conclude that focusing on the improvement of TDD skills’ set investigated in this study could benefit software developers in improving their baseline productivity and the external quality of the code they produce. However, replications are needed to overcome the issues related with the statistical power of this study. We suggest practical insights for future work to investigate the phenomenon further.

Introduction

Test-driven development (TDD) is a software development technique in which the development is guided by writing unit tests. It was popularized in the late 1990s as part of Extreme Programming [1]. A developer using TDD follows four steps:

• Write a unit test for the functionality she wants to add.

• Run the unit test to make sure it fails.

• Write only enough production code to make the test to pass.

• Refactor both production and test code, and re-run the tests.

TDD is claimed to yield better results than traditional approaches to software development (e.g., when unit tests are written after the intended functionality is considered completed by the development team) in terms of developers’ productivity, external quality (e.g., reduced number of defects), maintainability, and extensibility [2], [3]. However, empirical investigations of the effects of TDD are contrasting [4], [5], arguing that the results are influenced by several variables (e.g., academic vs. industrial settings), including the skills of developers.

Literature reviews on TDD conclude that the application of the technique—and subsequently the manifestation of its postulated benefits—requires some skills [5], [6]; however, these studies do not indicate what these skills are. We started our investigation on skills with students in a previous study [7]. In that context, we looked at their pre-existing knowledge regarding two practical skills: proficiency with programming language and unit testing (UT). When the subjects tackled a small programming task using TDD, we found that such skills had little impact on their productivity—defined as the output (e.g., parts of the task completed) per unit of effort (e.g., time to complete the task). No significant relationship was observed regarding the quality of the software they produced—e.g., the defects found in the parts of the task which were completed by the subjects. In the same study, we acknowledged that other skills must be present in order for TDD developers to achieve the benefits advocated by TDD supporters.

With these motivations based on existing literature and our previous work, we incorporate in this study another practical skill, which we call TDD process conformance, along with programming language and unit test skills. TDD process conformance represents the ability of a developer to follow the TDD cycle. Together, these three skills represent our TDD skill set. Further, we used a more realistic task to overcome the limitations of small programming tasks, and recruited professional developers for the study. Consequently, the research goal of this work is the following:

In our previous studies [7], [8], [9] we have investigated the role that each skill plays individually with student subjects working on toy tasks. We now focus on the impact the skills have, when taken together, on the outcomes of interest, by performing a quasi-experiment involving 43 professional software developers (30 after mortality) without prior working experience in TDD. The developers were trained during a week-long workshop and then asked to implement new features of a legacy system using TDD. Finally, we evaluated the composite effect of their skills on their performance in terms of external quality and productivity. Hence, we contribute to the existing knowledge by:

• Empirically investigating an anecdotal claim: that is, TDD requires skills to manifest benefits, with professional developers.

• Building a model for quality and productivity that takes into account a set of practical skills (Section 3)

• Providing initial empirical evidence that further investigation of the proposed TDD skill set are worth pursuing (Section 5)

The strong points of our study lie in the settings (Section 4) in which it was conducted. In particular, we:

• Analyze data collected from professional software developers.

• Utilize a near real-world, brown-field task, rather than a toy, green-field, task (see Section 4.2 and Appendix B).

• Quantify process conformance analytically, rather than relying on self reports.

The rest of the paper is organized as follows. In Section 2 we present the existing literature related to our research, in Section 3 we define the TDD skill set used in our study. Section 4 explains the details of our empirical study design. Sections 5 and 6 report the results and associated discussions. We address the threats to the validity of our study in Section 7. We conclude the paper in Section 8.