Teaching Software Metrology: The Science of Measurement for Software Engineering

Abstract

While the methodological rigor of computing research has improved considerably in the past two decades, quantitative software engineering research is hampered by immature measures and inattention to theory. Measurement—the principled assignment of numbers to phenomena—is intrinsically difficult because observation is predicated upon not only theoretical concepts but also the values and perspective of the research. Despite several previous attempts to raise awareness of more sophisticated approaches to measurement and the importance of quantitatively assessing reliability and validity, measurement issues continue to be widely ignored. The reasons are unknown, but differences in typical engineering and computer science graduate training programs (e.g., compared to psychology and management) are likely involved. This chapter therefore reviews key concepts in the science of measurement and applies them to software engineering research. A series of exercises for applying important measurement concepts to the reader’s research are included, and a sample dataset for the reader to try some of the statistical procedures mentioned is provided.

Appendices

Appendix 1: Reliability Analysis

This appendix provides an example of analyzing reliability on software metrics computed by different tools. The metrics were calculated from the source code of Apache Maven.⁶ You can find the data and scripts in the online supplement to this book (see Supplementary Materials).

The purpose of this analysis is to investigate the extent to which metrics calculated by different tools provide consistent measurements.

1.1 Data Preparation

The dataset includes various metrics such as size, cohesion, inheritance, and coupling metrics, which are continuous. The following steps were undertaken to prepare the data for reliability analysis:

• 1. Read the data from the Excel file containing the metrics.

      library(readxl)     data <- read_excel("efaReadyMC.xlsx")

• 2. Select the relevant metrics for analysis. Here we use LOC metrics computed by three different tools—Designite,⁷ JHawk,⁸ and Understand.⁹ The corresponding columns in the dataset are Size.LOC.Designite, Size.LOC.JHawk, and Size.LOC.Understand.

      rel1_data <-         select(rel_data, 'Size.LOC.Designite',         'Size.LOC.JHawk', 'Size.LOC.Understand')

1.2 Calculate a Measure of Reliability

Since lines of code is ratio-level data, we’ll use a measure of reliability rather than agreement (see Sect. 3.3.1). For this example, we will use Cronbach’s alpha because it is simpler to calculate and interpret. Note, however, that if we were looking at reliability after doing factor analysis or a similar technique, more accurate measures of reliability such as McDonald’s omega and composite reliability are available.

We will calculate Cronbach’s alpha using the psych package¹⁰ in R as follows:

• 1. Convert the selected data into a dataframe.

      rel1_data <- as.data.frame(t(rel1_data))

• 2. Calculate alpha.

      library(psych)     alphaResult <- alpha(rel1_data)

1.3 Results and Interpretation

Cronbach’s alpha values range from -1 to 1, with higher values indicating greater reliability and internal consistency among the measured items. Generally, $\alpha >0.7$ is considered acceptable, while $\alpha >0.9$ is considered excellent. Our result $\alpha =0.97$ indicates excellent reliability. This suggests that the three tested tools are measuring basically, if not exactly, the same thing. However, excellent reliability doesn’t mean that the studied metrics reflect the target underlying construct (e.g., class size). To determine that, we need a different kind of analysis (next).

Appendix 2: Exploratory Factor Analysis

This appendix provides an example of an exploratory factor analysis, following established guidelines [18] and using selected metrics calculated from the source code of Apache Maven.¹¹ You can find the data and scripts in the online supplement to this book (see Supplementary Materials on page 138).

1.1 Objective of Factor Analysis

The objective of our exploratory factor analysis is to assess the convergent and discriminant validity of common, object-oriented, class-level software code quality metrics calculated on the source code of Apache Maven where convergent validity refers to how similar the measure is with other measures it should be theoretically similar to and discriminant validity refers to how different the measure is with other measures it should theoretically be different to [62].

1.2 Design the Factor Analysis

The dataset we will use contains measurements from 22 metrics from approximately 1000 classes, well over the 10 observations per variable threshold. We aim to classify these metrics into six factors:

• 1. Cohesion: the degree to which elements of a class belong together.

• 2. In-coupling: the degree to which a class is used by other classes.

• 3. Out-coupling: the degree to which a class depends on other classes.

• 4. Size: how big the class is.

• 5. Sub-inheritance: the degree to which a class has subclasses in an inheritance hierarchy.

• 6. Sup-inheritance: the degree to which a class has superclasses in an inheritance hierarchy.

1.3 Check Assumptions of Factor Analysis

The assumptions of a factor analysis, and how we justify or test them, are as follows:

• Factor analysis should only be used when we theorize that a latent factor structure exists. In this case, we theorize that specific factors (size, coupling, etc.) are latent and do drive changes in metrics.

• Homogeneous sample of measurements. In other words metrics are calculated on the same sample of classes.

• Multicollinearity. If none of the variables are correlated, we cannot perform factor analysis; however, if two or more variables are perfectly or near perfectly correlated, it will cause a “nonpositive definite” matrix, which will prevent the factor analysis from completing. We can assess multicollinearity in three ways:

  • 1. Visually inspecting a correlation matrix. In this case, we can see many correlations $>0.3$, which indicates that a factor analysis is possible [18].

  • 2. The Kaiser-Meyer-Olkin (KMO) test. The KMO test tells us how correlated the variables in a dataset are. A minimum KMO value of 0.5 is acceptable and a value above 0.7 is recommended for a good factor analysis [31]. Our $KMO = 0.71$ is considered “middling” and appropriate for factor analysis [31]. The KMO values for each individual metric are also greater than the acceptable minimum of 0.5 [31] (Fig. 8).

    Fig. 8

    

    Kaiser-Meyer-Olkin (KMO) test

  • 3. Bartlett’s test of sphericity. Bartlett’s test of sphericity analyzes the correlations between variables to see if they are large enough to perform a factor analysis [14]. It was found that Bartlett’s test of sphericity is significant ($p<0.001$) and we can proceed with the factor analysis (Fig. 9).

    Fig. 9

    

    Bartlett’s test of sphericity

1.4 Derive Factors and Assess Fit

Researchers disagree on the best method of determining the number of factors to extract. We recommend using several methods to inform the decision [11].

Parallel analysis is a technique to estimate factors by calculating eigenvalues of random, uncorrelated data with eigenvalues of the actual data. The number of factors to retain is the count of eigenvalues greater than 0 [24]. In our data, 21 factors are retained—21 eigenvalues are greater than 0 (Fig. 10). Parallel analysis is known to overestimate the number of factors extracted from very large datasets like ours [11].

Fig. 10

Parallel analysis

Alternatively, we retain a number of factors equal to the number of eigenvalues greater than 1 (the Kaiser Criterion [30]). This method also loses effectiveness as the size of the dataset increases [63], but not so much. We found five eigenvalues greater than 1 (Fig. 11), suggesting that we retain five factors.

Fig. 11

Kaiser Criterion

We can also estimate the number of factors to retain by counting the eigenvalues before the bend in a scree plot [8]. This technique is a little tricky and requires expertise if the plot is complicated [11]. Figure 12 has multiple bends—at two, four, and seven eigenvalues—which suggests retaining anywhere between two and seven factors.

Fig. 12

Scree plot

In the theory approach, we retain the number of factors that we theorize exist—in this case, six: size, cohesion, sub-inheritance, sup-inheritance, in-coupling, and out-coupling.

From the above discussion, we have the following findings:

• 1. Parallel analysis suggests 21 factors.

• 2. The Kaiser Criteria suggest five factors.

• 3. The scree plot suggests factors between 2 and 7.

• 4. Theory suggests six factors.

Based on this, we tentatively retain seven factors as shown in Table 1. Table 1 shows the variables and their corresponding factor loadings on each of the seven factors. Factor loading refers to the correlation between a variable and a factor. A high loading suggests that the variance explained by a variable is sufficient for it to have a considerable relationship with the factor. Small loadings ($<0.3$) are considered insignificant [11, 18, 25, 51] and are thus suppressed in our model. For example, “Cohesion.LCOM, Cohesion.LCOMModified, Cohesion.YALCOM,” and “Size.CountInstanceVariable” load together on Factor 1, which means that these variables seem to be measuring the same factor.

Table 1 EFA with seven factors

These factors explain 84% of the variance in our dataset, which is good. If the variance explained was less than 60%, we might opt to include additional factors [18]. However, Factor 5 only has two metrics loading on it. The minimum is three, so either we need more metrics or fewer factors.

In this case, we can reduce the number of factors to six, as theorized. Table 2 shows the six-factor solution. This solution explains 78% of the variance and each factor has at least three metrics, so we can move on to iteratively refining and interpreting the factors.

Table 2 EFA with six factors (step 1)

(We included this step to illustrate the realistic complexity of choosing an appropriate number of factors. Sometimes you’re well into the analysis before you figure out how many factors you should have.)

1.5 Interpret and Refine Factors

Rotating the factors helps us interpret them. In a rotated factor solution, the axes are rotated so that variables that load together are plotted closer together causing them to load highly on a single factor.

Factors can be rotated using orthogonal or oblique rotation. An oblique rotation is preferable when factors are assumed to be correlated; an orthogonal rotation is used otherwise [11, 14, 18]. Despite the popularity of orthogonal rotation (specifically varimax), oblique rotation is usually more appropriate because factors are usually correlated. Use orthogonal rotation only if you have a very good reason to believe that the factors are uncorrelated. We use oblimin rotation (a type of oblique rotation) to rotate the axis in our model.

Now we inspect the solution for problems, and remove problems one at a time, beginning with the worst. There is no algorithm for this. “Worst” is subjective. We can only give examples of problems and describe their severity. We are looking for three basic kinds of problems:

• 1. Low communality: Communality (“h2” in Tables 1 and 2) is the amount of variance in a variable that can be explained by the factor solution. Low communality (h2 $<$ 0.5) indicates that less than half of the variance of the variable is taken into account implying that the variable is not closely related to any of the factors and causes unwanted complexity with insufficient explanation [18].

• 2. Cross-loadings: Variables with high loadings on multiple factors.

• 3. Loading on the wrong factor: Variables loading highly (loadings $>$ 0.5) on a factor they shouldn’t be.

Looking at Table 2, Cohesion.LCOM5 has the lowest communality ($h2=0.16$) and loads on the wrong factor (in-coupling), so we remove that one first and rerun the EFA (Table 3). Now Size.CountDeclMethodDefault has the lowest communality (h2 = 0.25) and loads on the wrong factor (in-coupling again). So we remove it and run the EFA again (Table 4). The next variable with lowest communality is Cohesion.YALCOM (h2 = 0.45); however, it loads well on the correct factor, so we’ll retain it for now and move onto cross-loadings. Size.CountInstanceVariable loads higher on the wrong factor (cohesion) than on the correct factor (size). Thus, we remove it. Rerunning the EFA (Table 5), we find that In-Coupling.CBOin also has a cross-loading. However, it loads much higher on the correct factor (in-coupling) than the incorrect factor (out-coupling). Furthermore, the incorrect loading is smaller than the smallest correct loading in the EFA, so we retain Size.CountInstanceVariable for now.

Table 3 EFA with six factors (step 2)

Table 4 EFA with six factors (step 3)

Table 5 EFA with six factors (step 4)

Since there are no more low communalities, cross-loadings, or incorrect loadings, and the solution explains 87% of the total variance in the dataset using six factors, each of which has at least three metrics, our EFA is now complete (Table 6).

Table 6 Final EFA