Code churn estimation using organisational and code metrics: An experimental comparison

Abstract

Context

Source code revision control systems contain vast amounts of data that can be exploited for various purposes. For example, the data can be used as a base for estimating future code maintenance effort in order to plan software maintenance activities. Previous work has extensively studied the use of metrics extracted from object-oriented source code to estimate future coding effort. In comparison, the use of other types of metrics for this purpose has received significantly less attention.

Objective

This paper applies machine learning techniques to unveil predictors of yearly cumulative code churn of software projects on the basis of metrics extracted from revision control systems.

Method

The study is based on a collection of object-oriented code metrics, XML code metrics, and organisational metrics. Several models are constructed with different subsets of these metrics. The predictive power of these models is analysed based on a dataset extracted from eight open-source projects.

Results

The study shows that a code churn estimation model built purely with organisational metrics is superior to one built purely with code metrics. However, a combined model provides the highest predictive power.

Conclusion

The results suggest that code metrics in general, and XML metrics in particular, are complementary to organisational metrics for the purpose of estimating code churn.

Introduction

Accurately estimating future code maintenance effort of a software system is one of the keystones of software project planning [1]. Over the past decades, significant attention has been paid to estimating code maintenance effort based on object-oriented code metrics [2], [3], [4]. In the meantime, the eXtensible Markup Language (XML) has grown into an ubiquitous language in contemporary software projects. In a separate study [5], we found that in the context of open-source software development, XML files frequently co-evolve with other types of files – about 20% of changes to non-XML files were accompanied with changes to XML files. Furthermore, the widespread adoption of revision control systems in software projects has made it possible to readily extract a wealth of metrics pertaining to organisational aspects of software projects.

These trends have opened the possibility of building more accurate models for code maintenance prediction by making use of a wider set of metrics than traditional object-oriented code metrics. Accordingly, this paper undertakes to study the relative performance and potential complementarity of different categories of metrics in the context of code maintenance effort estimation.

A common indirect measure of code maintenance effort, which we adopt in this paper, is that of code churn: the sum of number of lines of code added, modified and deleted between two revisions of a software module [6]. In addition to providing an indicator of code maintenance effort, code churn has also been shown to be correlated with software defects [7], [8]. The present study focuses on estimating long-term code churn. Specifically, the dependent variable of our study is the cumulative yearly code churn of a project: the sum of the cumulative yearly code churn of all (non-binary) files in a project, where the cumulative code churn of a file is the sum of the code churn of the file across all its revisions in a given 12-months period.

This paper specifically addresses the following research question: What is the relative performance of models built to estimate the yearly cumulative code churn of a project based on:

• XML/XSLT code metrics.

• Imperative/object-oriented programming language metrics.

• Organisational metrics.

• Organisational metrics and code metrics combined.

Given these questions, we have a choice between hypothesising that certain relations exist between a selected set of input metrics and code churn, or to uncover such relations in an exploratory manner. In the first approach we would start with a set of hypotheses, and we would use statistical conformance testing to validate these hypotheses on the chosen dataset. However, as mentioned above, we are not aware of previous studies on possible relations between XML metrics and code churn. Hence, there is little basis for formulating a priori hypotheses about such relations. On the other hand, there is a wealth of data available from open source repositories that can be leveraged in order to uncover such relations. Accordingly, we adopt a bottom-up approach based on data mining and exploratory data analysis.

The adopted data mining approach comprises the following steps:

• Data pre-processing: choice of prediction targets and proposition of input features (attributes that might influence the value we need to predict), data gathering, normalisation, and cleansing.

• Learning: choice of data mining algorithms and application of these algorithms.

• Results validation: evaluation of models fit using standard statistical techniques.

This data mining approach allows us to identify interference between input features and the prediction target and in doing so they uncover the existence of a predictive model. However, the data mining approach itself does not allow us to explain the cause of the interference. In order to compensate for this shortcoming, exploratory data analysis was used in this study in order to gain an understanding of the models created by the data mining algorithms.

Compared to a conformance testing approach, data mining and exploratory data analysis offer the benefit of not requiring an a priori specific model to test. The aim of exploratory data analysis is to propose models that can then be conformance-tested. Moreover, data mining and exploratory data analysis can uncover non-intuitive relations. In fact, the results of this study show, among other things, that there are no generally-applicable straightforward models to estimate coding effort – that is, linear models based on one or a very small number of interactions between input features. The models with better predictive power uncovered in the study involve a non-trivial number of interactions between input features.

The paper is structured as follows. Sections 2 Dataset, 3 Features and training algorithms introduce the dataset used in the study and the features extracted for training. The training algorithms and process employed are discussed in Section 3. Next, evaluation results are presented in Section 4. In Section 5, related work is discussed. Finally, conclusions and directions for future work are discussed in Section 6.