Implicit and explicit mixture of experts models for software defect prediction

Abstract

Accurately predicting defects in software modules helps the developers and testers to find the defective modules quickly and save their efforts in other software development aspects. Most previous studies have used single machine learning technique-based models to detect defects in software. These models have produced limited results as they perform well in only some parts of the data and fail to capture all the defect-causing patterns. The mixture of experts (MoE) is a combination method that utilizes experts specialized in the given data subspaces. The results of different specialized experts are combined according to their specific expertise for the final prediction governed by a gating network. This paper explores using the MoE method and presents implicit and explicit MoE-based models for software defect prediction. The presented models are evaluated via an experimental study on twenty-two software defect datasets collected from AEEEM, PROMISE, and JIRA repositories. The prediction performance of the presented models is evaluated using accuracy, f1-score, area under the ROC curve (AUC), and Mathew correlation coefficient (MCC) performance metrics. The experimental results showed that the presented MoE-based models outperformed different machine learning and ensemble techniques, such as Bagging and AdaBoost, and produced a state-of-the-art performance for defect prediction. Additionally, we found that the MoE models produced better or at least equal performance than the DNN-based model for most cases. The results are consistent for all the datasets. The results of the Wilcoxon test also showed that the presented models performed significantly better than the other techniques.

Appendices

Appendix A. Comparison with other state-of- the-art works

A comparative analysis of the presented MoE-based prediction models (MIoE and MEoE) has been performed against other previously published SDP-related works. We have not re-produced the results but borrowed them from the studies cited and used them. The works specified in Table 13 for comparison purposes are for the same datasets, with the same performance measures, and a similar experimental environment as used for our presented models. The table shows that except for the work presented by Panday et al. (2020), for all other works, the MIoE and MEoE models produced better performances for different performance measures. The presented work has used twenty-two defect datasets of different domains. For some of the datasets, the presented MoE model performed relatively poorly, reducing the methods’ average performance. In contrast, other previous works have considered fewer fault datasets for the performance evaluation of their methods. The average and highest values of the presented models were greater than their counterparts. Most of the works reported in Table 13 have used static ensemble methods, where the weights of the base learners were decided during training only. Moreover, only a few works have analyzed the base learners’ competitiveness for the ensemble, and the experimental analysis has also been limited to a few datasets.

Table 13 Comparison of the presented MIoE and MEoE models with state-of-the-art works (* showing the average values)

Table 14 Parameter values of different used techniques

Appendix B. Parameter details

In this study, we have used Python’s implementation of used machine learning techniques and ensemble methods. The following parameter values have been set for these techniques/methods as given in Table 14.

Appendix C. Description of the used software metrics

Table 15 provides descriptions of the used software metrics available in different software defect datasets used for experimentation in this work. A detailed description of these metrics can be found in Jureczko and Madeyski (2010), D’Ambros et al. (2010), Zimmermann et al. (2007), and Yatish et al. (2019).

Appendix D. Analysis of MIoE and MEoE models

The working of the presented MIoE and MEoE models (Algorithms 1 and 2) is based on standard algorithms and techniques. The analysis of the models and the steps of the algorithms are discussed as follows. The initial steps of the method involve data cleaning, data balancing, and handling high data dimensionality, which takes some non-constant time. In Algorithm-1 (MIoE model), the training data is initially randomly partitioned into sub-spaces equal to the number of experts. This step takes constant time and is applied only once. After that, experts are trained on their corresponding input sub-spaces. The training time depends on the learning technique; however, it is always less than training a single large model. Finally, the obtained best experts are tested on the testing dataset, and a gating function is then applied to combine the experts’ predictions. It takes some linear computation time. Other computations take constant time and are applied only once. Similarly, in Algorithm-2 (MEoE model), the X-means clustering algorithm is used to partition the training data into sub-spaces. It is applied only once and takes some linear computation time. After that, experts are trained on their corresponding input sub-spaces. Again, the training time depends on the learning technique, but it is always less than training a single large model. Finally, the obtained best experts are tested on the testing dataset, and the results are passed to the gating function for the final prediction. It takes some linear computation time. Other computations take constant time and are applied only once. On the machine with 16 GB of RAM and an Intel i7 processor, it took only 160 s to build the MIoE model and make the prediction and 180 s to build the MEoE model and make the prediction. In their work, Kondratyuk et al. (2020) studied the performance of ensemble or combinational models and found that ensembling can often be more efficient than training larger models. Additionally, the authors concluded that ensembles execute multiple models in parallel and then combine their outputs to make the final prediction. Therefore, the overall time will be better than the large single model.

Table 15 Description of the software metrics (Jureczko & Madeyski, 2010; D’Ambros et al., 2010; Zimmermann et al., 2007; Yatish et al., 2019)