Elsevier

Information and Software Technology

Volume 106, February 2019, Pages 182-200
Information and Software Technology

Software defect prediction based on kernel PCA and weighted extreme learning machine

https://doi.org/10.1016/j.infsof.2018.10.004Get rights and content

Abstract

Context

Software defect prediction strives to detect defect-prone software modules by mining the historical data. Effective prediction enables reasonable testing resource allocation, which eventually leads to a more reliable software.

Objective

The complex structures and the imbalanced class distribution in software defect data make it challenging to obtain suitable data features and learn an effective defect prediction model. In this paper, we propose a method to address these two challenges.

Method

We propose a defect prediction framework called KPWE that combines two techniques, i.e., Kernel Principal Component Analysis (KPCA) and Weighted Extreme Learning Machine (WELM). Our framework consists of two major stages. In the first stage, KPWE aims to extract representative data features. It leverages the KPCA technique to project the original data into a latent feature space by nonlinear mapping. In the second stage, KPWE aims to alleviate the class imbalance. It exploits the WELM technique to learn an effective defect prediction model with a weighting-based scheme.

Results

We have conducted extensive experiments on 34 projects from the PROMISE dataset and 10 projects from the NASA dataset. The experimental results show that KPWE achieves promising performance compared with 41 baseline methods, including seven basic classifiers with KPCA, five variants of KPWE, eight representative feature selection methods with WELM, 21 imbalanced learning methods.

Conclusion

In this paper, we propose KPWE, a new software defect prediction framework that considers the feature extraction and class imbalance issues. The empirical study on 44 software projects indicate that KPWE is superior to the baseline methods in most cases.

Introduction

Software testing is an important part of software development life cycle for software quality assurance [1], [2]. Defect prediction can assist the quality assurance teams to reasonably allocate the limited testing resources by detecting the potentially defective software modules (such as classes, files, components) before releasing the software product. Thus, effective defect prediction can save testing cost and improve software quality [3], [4], [5].

The majority of existing researches leverages various machine learning techniques to build defect prediction methods. In particular, many classification techniques have been used as defect prediction models, such as decision tree [6], Naive Bayes [7], random forest [8], [9], nearest neighbor [10], support vector machine [11], [12], neural network [13], [14], [15], logistic regression [16], and ensemble methods [17], [18]. Since irrelevant and redundant features in the defect data may degrade the performance of the classification models, different feature selection methods have been applied to select an optimal feature subset for defect prediction[19]. These methods can be roughly divided into three categories: the filter-based feature ranking methods, wrapper-based feature subset evaluation methods, and extraction-based feature transformation methods, such as Principal Component Analysis (PCA) [20].

Selecting optimal features that can reveal the intrinsic structures of the defect data is crucial to build effective defect prediction models. The filter-based and wrapper-based feature selection methods only select a subset of the original features without any transformation [21]. However, such raw features may not properly represent the essential structures of raw defect data [22]. Being a linear feature extraction method, PCA has been widely used to transform the raw features to a low-dimensional space where the features are the linear combinations of the raw ones [23], [24], [25], [26]. PCA performs well when the data are linearly separable and follow a Gaussian distribution, whereas the real defect data may have complex structures that can not be simplified in a linear subspace [27], [28]. Therefore, the features extracted by PCA are usually not representative, and cannot gain anticipated performance for defect prediction [19], [29]. To address this issue, we exploit KPCA [30], a nonlinear extension of PCA, to project the original data into a latent high-dimensional feature space in which the mapped features can properly characterize the complex data structures and increase the probability of linear separability of the data. When the original data follow an arbitrary distribution, the mapped data by KPCA obey an approximate Gaussian distribution. Fig. 1 shows the merit of the feature mapping, where the data are linearly inseparable within the low-dimensional space but linearly separable within the high-dimensional space. Existing studies have shown that KPCA outperforms PCA [31], [32].

Although many classifiers have been used for defect prediction, Lessmann et al. [33] suggested that the selection of classifiers for defect prediction needs to consider additional criteria, such as computational efficiency and simplicity, because they found that there are no significant performance differences among most defect prediction classifiers. Moreover, class imbalance is prevalent in defect data in which the non-defective modules usually outnumber the defective ones. It makes most classifiers tend to classify the minority samples (i.e., the defective modules) as the majority samples (i.e., the non-defective modules). However, existing defect prediction methods did not address this problem well, thus leading to unsatisfactory performance. In this work, we exploit Single-hidden Layer Feedforward Neural networks (SLFNs) called Weighted Extreme Learning Machine (WELM) [34] to overcome this challenge. WELM assigns higher weights to defective modules to emphasize their importance. In addition, WELM is efficient and convenient since it only needs to adaptively set the number of hidden nodes while other parameters are randomly generated instead being tuned through iterations like traditional neural networks [35].

In this paper, we propose a new defect prediction framework called KPWE that leverages the two aforementioned techniques: KPCA and WELM. This framework consists of two major stages. First, KPWE exploits KPCA to map original defect data into a latent feature space. The mapped features in the space can well represent the original ones. Second, with the mapped features, KPWE applies WELM to build an efficient and effective defect prediction model that can handle imbalanced defect data.

We conduct extensive experiments on 44 software projects from two datasets (PROMISE dataset and NASA dataset) with four indicators, i.e., F-measure, G-measure, MCC, and AUC. On average, KPWE achieves average F-measure, G-measure, MCC, and AUC values of 0.500, 0.660, 0.374, and 0.764 on PROMISE dataset, of 0.410, 0.611, 0.296 and 0.754 on NASA dataset, and of 0.480, 0.649, 0.356, and 0.761 across 44 projects of the two datasets. We compare KPWE against 41 baseline methods. The experimental results show that KPWE achieves significantly better performance (especially in terms of F-measure, MCC, and AUC) compared with all baseline methods.

The remainder of this paper is organized as follows. Section 2 presents the related work. In Section 3, we describe the proposed method in detail. Section 4 elaborates the experimental setup. In Section 5, we report the experimental results of performance verification. Section 6 discusses the threats to validity. In Section 7, we draw the conclusion.

Section snippets

Feature selection for defect prediction

Some recent studies have investigated the impact of feature selection methods on the performance of defect prediction. Song et al. [4] suggested that feature selection is an indispensable part of a general defect prediction framework. Menzies et al. [7] found that Naive Bayes classifier with Information Gain based feature selection can get good performances over 10 projects from the NASA dataset. Shivaji et al. [36], [37] studied the performance of filter-based and wrapper-based feature

KPWE: The new framework

The new framework consists of two stages: feature extraction and model construction. This section first describes how to project the original data into a latent feature space using the nonlinear feature transformation technique KPCA, and then presents how to build the WELM model with the extracted features by considering the class imbalance issue.

Experimental setup

In this section, we elaborate the experimental setup, including the Research Questions (RQs), benchmark datasets, the performance indicators, and the experimental design.

Answer to RQ1: the efficiency of ELM, WELM and some classic classifiers.

Since many previous defect prediction studies applied classic classifiers as prediction models [33], [44], in this work, we choose seven representative classifiers, including Naive Bayes (NB), Nearest Neighbor (NN), Random Forest (RF), Logistic Regression (LR), Classification and Regression Tree (CART), Back Propagation neural networks (BP) and Support Vector Machine (SVM), and compare their efficiency with ELM and WELM.

The parameter settings of the classifiers are detailed as follows. For NB,

External validity

External validity focuses on whether our experimental conclusions will vary on different projects. We conduct experiments on total 44 projects of two defect datasets to reduce the threat for this kind of validity. In addition, since the features of our benchmark dataset are all static product metrics and the modules are abstracted at class level (for PROMISE dataset) and component level (for NASA dataset), we cannot claim that our experimental conclusions can be generalized to the defect

Conclusion

In this work, we propose a new defect prediction framework KPWE that comprises feature extraction stage and model construction stage. In the first stage, to handle the complex structures in defect data, we learn the representative features by mapping the original data into a latent feature space with a nonlinear feature extraction method KPCA. The mapped features in the new space can better represent the raw data. In the second stage, we construct a class imbalanced classifier on the extracted

Acknowledgments

The authors would like to acknowledge the support provided by the grands of the National Natural Science Foundation of China (61572374, U1636220, 61472423, 61602258), Open Fund of Key Laboratory of Network Assessment Technology from CAS, the Academic Team Building Plan for Young Scholars from Wuhan University (WHU2016012), the National Science Foundation (DGE-1522883), Hong Kong RGC Project (CityU C1008-16G), Hong Kong General Research Fund (PolyU 152279/16E, 152223/17E), and the China

References (97)

  • M.J. Siers et al.

    Software defect prediction using a cost sensitive decision forest and voting, and a potential solution to the class imbalance problem

    Inf. Syst.

    (2015)
  • G. Huang et al.

    Trends in extreme learning machines: a review

    Neural Netw.

    (2015)
  • L. Chen et al.

    Negative samples reduction in cross-company software defects prediction

    Inf. Softw. Technol. (IST)

    (2015)
  • Y. Ma et al.

    Transfer learning for cross-company software defect prediction

    Inf. Softw. Technol. (IST)

    (2012)
  • W. Yu et al.

    Learning deep representations via extreme learning machines

    Neurocomputing

    (2015)
  • H. Qu et al.

    An asymmetric classifier based on partial least squares

    Pattern Recognit.

    (2010)
  • J. Tian

    Software Quality Engineering: Testing, Quality Assurance, and Quantifiable Improvement

    (2005)
  • G.J. Myers et al.

    The Art of Software Testing

    (2011)
  • M. Shepperd et al.

    Researcher bias: the use of machine learning in software defect prediction

    IEEE Trans. Softw. Eng. (TSE)

    (2014)
  • Q. Song et al.

    A general software defect-proneness prediction framework

    IEEE Trans. Softw. Eng. (TSE)

    (2011)
  • X. Yang et al.

    A learning-to-rank approach to software defect prediction

    IEEE Trans. Reliab.

    (2015)
  • P. Knab et al.

    Predicting defect densities in source code files with decision tree learners

    Proceedings of the 3rd International Workshop on Mining Software Repositories (MSR)

    (2006)
  • T. Menzies et al.

    Data mining static code attributes to learn defect predictors

    IEEE Trans. Softw. Eng. (TSE)

    (2007)
  • L. Guo et al.

    Robust prediction of fault-proneness by random forests

    Proceedings of the 15th International Symposium on Software Reliability Engineering (ISSRE)

    (2004)
  • C. Macho et al.

    Predicting build co-changes with source code change and commit categories

    Proceedings of the 23rd International Conference on Software Analysis, Evolution, and Reengineering (SANER)

    (2016)
  • X. Jing et al.

    Heterogeneous cross-company defect prediction by unified metric representation and CCA-based transfer learning

    Proceedings of the 10th Joint Meeting on Foundations of Software Engineering (FSE)

    (2015)
  • Z. Yan et al.

    Software defect prediction using fuzzy support vector regression

    Adv. Neural Netw.

    (2010)
  • T.M. Khoshgoftaar et al.

    Application of neural networks to software quality modeling of a very large telecommunications system

    IEEE Trans. Neural Netw. (TNN)

    (1997)
  • D.E. Neumann

    An enhanced neural network technique for software risk analysis

    IEEE Trans. Soft. Eng. (TSE)

    (2002)
  • A. Panichella et al.

    Cross-project defect prediction models: L’union fait la force

    Proceedings of the 21st Software Evolution Week-IEEE Conference on Software Maintenance, Reengineering and Reverse Engineering (CSMR-WCRE)

    (2014)
  • A. Shanthini

    Effect of ensemble methods for software fault prediction at various metrics level

    Int. J. Appl. Inf. Syst.

    (2014)
  • X. Xia et al.

    Cross-project build co-change prediction

    Proceedings of the 22nd International Conference on Software Analysis, Evolution and Reengineering (SANER)

    (2015)
  • Z. Xu et al.

    The impact of feature selection on defect prediction performance: an empirical comparison

    Proceedings of the 27th International Symposium on Software Reliability Engineering (ISSRE)

    (2016)
  • Q. Song et al.

    A fast clustering-based feature subset selection algorithm for high-dimensional data

    IEEE Trans. Knowl. Data Eng. (TKDE)

    (2013)
  • T. Wang et al.

    Multiple kernel ensemble learning for software defect prediction

    Autom. Softw. Eng. (ASE)

    (2016)
  • F. Liu et al.

    Software defect prediction model based on PCA-isvm

    Comput. Simulat.

    (2014)
  • H. Cao et al.

    A novel PCA-bp fuzzy neural network model for software defect prediction

    Adv. Sci. Lett.

    (2012)
  • C. Zhong

    Software quality prediction method with hybrid applying principal components analysis and wavelet neural network and genetic algorithm

    Int. J. Digital Content Technol. Appl.

    (2011)
  • T.M. Khoshgoftaar et al.

    Improving tree-based models of software quality with principal components analysis

    Proceedings of the 11th International Symposium on Software Reliability Engineering (ISSRE)

    (2000)
  • M. Shepperd et al.

    Data quality: some comments on the nasa software defect datasets

    IEEE Trans. Softw. Eng. (TSE)

    (2013)
  • D. Gray et al.

    The misuse of the nasa metrics data program data sets for automated software defect prediction

    Proceedings of the 15th Annual Conference on Evaluation & Assessment in Software Engineering (EASE)

    (2011)
  • T. Menzies et al.

    How simple is software defect detection

    Submitt. Emprical Softw. Eng. J.

    (2003)
  • B. Schölkopf et al.

    Kernel principal component analysis

    Proceedings of the 7th International Conference on Artificial Neural Networks (ICANN)

    (1997)
  • B. Schölkopf et al.

    Nonlinear component analysis as a kernel eigenvalue problem

    Neural Comput.

    (1998)
  • K.I. Kim et al.

    Iterative kernel principal component analysis for image modeling

    IEEE Trans. Pattern Anal. Mach. Intell. (TPAMI)

    (2005)
  • S. Lessmann et al.

    Benchmarking classification models for software defect prediction: a proposed framework and novel findings

    IEEE Trans. Softw. Eng. (TSE)

    (2008)
  • S. Shivaji et al.

    Reducing features to improve code change-based bug prediction

    IEEE Trans. Softw. Eng. (TSE)

    (2013)
  • S. Shivaji et al.

    Reducing features to improve bug prediction

    Proceedings of the 24th International Conference on Automated Software Engineering (ASE)

    (2009)
  • Cited by (136)

    • Ensemble learning based software defect prediction

      2023, Journal of Engineering Research (Kuwait)
    View all citing articles on Scopus

    Fully documented templates are available in the elsarticle package on CTAN.

    View full text