Skip to main content
SpringerLink
Log in

Evaluating network embedding techniques’ performances in software bug prediction

  • Published:
Empirical Software Engineering Aims and scope Submit manuscript

Abstract

Software bug prediction techniques can be very helpful in testing and code inspection. Over the past decade, network measures have been successfully used in bug prediction. Following the same intuition, recently, researchers started using network embedding techniques in bug prediction. However, existing studies only evaluated the Skip-gram and CBOW models with random walk. Considering network embedding is a fast-developing research direction, it is important to evaluate other network embedding techniques’ performances in bug prediction. Moreover, existing studies have not investigated the application and performance of network embedding in effort-aware bug prediction, which is thought to be a more realistic scenario that evaluates the cost effectiveness of bug prediction models. In this paper, we conduct an extensive empirical study to evaluate network embedding algorithms in bug prediction by utilizing and extending node2defect, a newly proposed bug prediction model that combines the embedded vectors with traditional software engineering metrics through concatenation. Experiments are conducted based on seven network embedding algorithms, two effort-aware models, and 13 open-source Java systems. Experimental results show that node2defect outperforms traditional metrics by + 14.64% in terms of MCC score, and by + 7.51% to + 16.57% in effort-aware bug prediction. More interestingly, when combined with CBS + , the embedded vectors alone can achieve the best performance. Among different network embedding algorithms, the newly proposed algorithm ProNE has the best performance.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

Notes

  1. Throughout the paper, we use the terms “network embedding techniques” and “network embedding algorithms” interchangeably.

  2. http://bug.inf.usi.ch/index.php

  3. https://scikit-learn.org/stable/

  4. https://scikit-learn.org/stable/modules/grid_search.html

  5. https://github.com/thunlp/OpenNE

  6. https://github.com/THUDM/ProNE

  7. https://github.com/benedekrozemberczki/walklets

  8. https://github.com/clowee/OpenSZZ

  9. https://issues.apache.org/jira/

  10. https://pmd.github.io/

  11. https://pmd.github.io/latest/pmd_rules_java.html

References

  • Agrawal A, Menzies T (2018) Is better data better than better data miners?: on the benefits of tuning smote for defect prediction. In: Proceedings of the 40th international conference on software engineering. ACM, pp 1050–1061

  • Agrawal A, Fu W, Chen D, Shen X, Menzies T (2019) How to “dodge” complex software analytics. IEEE Trans Softw Eng:1–1. https://doi.org/10.1109/TSE.2019.2945020

  • Belkin M, Niyogi P (2001) Laplacian eigenmaps and spectral techniques for embedding and clustering. In: Proceedings of the 14th international conference on neural information processing systems: natural and synthetic, pp 585–591

  • Bennin KE, Keung J, Phannachitta P, Monden A, Mensah S (2017) Mahakil: diversity based oversampling approach to alleviate the class imbalance issue in software defect prediction. IEEE Trans Softw Eng 44(6):534–550

    Article  Google Scholar 

  • Bowes D, Hall T, Harman M, Jia Y, Sarro F, Wu F (2016) Mutation-aware fault prediction. In: Proceedings of the 25th international symposium on software testing and analysis, pp 330–341

  • Breiman L (2001) Random forests. Mach Learn 45(1):5–32

    Article  Google Scholar 

  • Cai H, Zheng VW, Chang KCC (2018) A comprehensive survey of graph embedding: Problems, techniques, and applications. IEEE Trans Knowl Data Eng 30(9):1616–1637

    Article  Google Scholar 

  • Cao S, Lu W, Xu Q (2015) Grarep: learning graph representations with global structural information. In: Proceedings of the 24th ACM international on conference on information and knowledge management. ACM, pp 891–900

  • Chawla NV, Bowyer KW, Hall LO, Kegelmeyer WP (2002) Smote: synthetic minority over-sampling technique. J Artif Intell Res 16(1):321–357

    Article  Google Scholar 

  • Chen L, Ma W, Zhou Y, Xu L, Wang Z, Chen Z, Xu B (2016) Empirical analysis of network measures for predicting high severity software faults. Science China Information Sciences 59(12):122901

    Article  Google Scholar 

  • Chidamber SR, Kemerer CF (1994) A metrics suite for object oriented design. IEEE Trans Softw Eng 20(6):476–493

    Article  Google Scholar 

  • Dam HK, Tran T, Pham T, Ng SW, Grundy J, Ghose A (2021) Automatic feature learning for predicting vulnerable software components. IEEE Trans Softw Eng 47(1):67–85. https://doi.org/10.1109/TSE.2018.2881961

    Article  Google Scholar 

  • D’Ambros M, Lanza M, Robbes R (2010) An extensive comparison of bug prediction approaches. In: 2010 7th IEEE working conference on mining software repositories (MSR 2010). IEEE, pp 31–41

  • Elkhail AA, Cerny T (2019) On relating code smells to security vulnerabilities. In: 2019 IEEE 5th intl conference on big data security on cloud (BigDataSecurity), IEEE intl conference on high performance and smart computing, (HPSC) and IEEE intl conference on intelligent data and security (IDS). IEEE, pp 7–12

  • Fu W, Menzies T (2017) Revisiting unsupervised learning for defect prediction. In: Proceedings of the 2017 11th joint meeting on foundations of software engineering. ACM, pp 72–83

  • Goyal P, Ferrara E (2018) Graph embedding techniques, applications, and performance: a survey. Knowl-Based Syst 151:78–94

    Article  Google Scholar 

  • Grover A, Leskovec J (2016) node2vec: scalable feature learning for networks. In: Acm Sigkdd international conference on knowledge discovery & data mining, pp 855–864

  • Guo Y, Shepperd M, Li N (2018) Bridging effort-aware prediction and strong classification: a just-in-time software defect prediction study. In: Proceedings of the 40th international conference on software engineering: companion proceedings. ACM, pp 325–326

  • Hall T, Beecham S, Bowes D, Gray D, Counsell S (2011) A systematic literature review on fault prediction performance in software engineering. IEEE Trans Softw Eng 38(6):1276–1304

    Article  Google Scholar 

  • Harris E (2002) Information gain versus gain ratio: a study of split method biases International symposium on artificial intelligence and mathematics, AI&m 2002, fort lauderdale, florida, USA, January, vol 2-4, p 2002

  • He Z, Shu F, Yang Y, Li M, Wang Q (2012) An investigation on the feasibility of cross-project defect prediction. Autom Softw Eng 19 (2):167–199

    Article  Google Scholar 

  • Herbold S, Trautsch A, Grabowski J (2017) A comparative study to benchmark cross-project defect prediction approaches. IEEE Trans Softw Eng 44 (9):811–833

    Article  Google Scholar 

  • Herzig K, Just S, Rau A, Zeller A (2013) Predicting defects using change genealogies. In: 2013 IEEE 24th international symposium on software reliability engineering (ISSRE). IEEE, pp 118–127

  • Hosmer DW Jr, Lemeshow S, Sturdivant RX (2013) Applied logistic regression, vol 398. Wiley, New York

    Book  Google Scholar 

  • Huang Q, Xia X, Lo D (2017) Supervised vs unsupervised models: a holistic look at effort-aware just-in-time defect prediction. In: 2017 IEEE international conference on software maintenance and evolution (ICSME). IEEE, pp 159–170

  • Huang Q, Xia X, Lo D (2019) Revisiting supervised and unsupervised models for effort-aware just-in-time defect prediction. Empir Softw Eng 24 (5):2823–2862

    Article  Google Scholar 

  • Jimenez M, Rwemalika R, Papadakis M, Sarro F, Le Traon Y, Harman M (2019) The importance of accounting for real-world labelling when predicting software vulnerabilities. In: Proceedings of the 2019 27th ACM joint meeting on European software engineering conference and symposium on the foundations of software engineering. ACM, pp 695–705

  • Jureczko M, Madeyski L (2010) Towards identifying software project clusters with regard to defect prediction. In: Proceedings of the 6th international conference on predictive models in software engineering, pp 1–10

  • Kamei Y, Matsumoto S, Monden A, Matsumoto K, Adams B, Hassan AE (2010) Revisiting common bug prediction findings using effort-aware models. In: 2010 IEEE international conference on software maintenance. IEEE, pp 1–10

  • Kamei Y, Shihab E, Adams B, Hassan AE, Mockus A, Sinha A, Ubayashi N (2013) A large-scale empirical study of just-in-time quality assurance. IEEE Trans Softw Eng 39(6):757–773

    Article  Google Scholar 

  • Kipf TN, Welling M (2017) Semi-supervised classification with graph convolutional networks. In: 5th international conference on learning representations, ICLR 2017, Toulon, France, April 24–26, 2017, Conference Track Proceedings. https://openreview.net/forum?id=SJU4ayYgl

  • Kong YX, Shi GY, Wu RJ, Zhang YC (2019) K-core: theories and applications. Phys Rep 832:1–32

    Article  MathSciNet  Google Scholar 

  • Lee JR, Gharan SO, Trevisan L (2014) Multiway spectral partitioning and higher-order cheeger inequalities. Journal of the ACM (JACM) 61(6):1–30

    Article  MathSciNet  Google Scholar 

  • Lenarduzzi V, Palomba F, Taibi D, Tamburri DA (2020) Openszz: a free, open-source, web-accessible implementation of the szz algorithm. In: Proceedings of the 28th international conference on program comprehension, pp 446–450

  • Lessmann S, Baesens B, Mues C, Pietsch S (2008) Benchmarking classification models for software defect prediction: a proposed framework and novel findings. IEEE Trans Softw Eng 34(4):485–496

    Article  Google Scholar 

  • Loyola P, Matsuo Y (2017a) Learning feature representations from change dependency graphs for defect prediction. In: 2017 IEEE 28th international symposium on software reliability engineering (ISSRE). IEEE, pp 361–372

  • Loyola P, Matsuo Y (2017b) Learning graph representations for defect prediction. In: Proceedings of the 39th international conference on software engineering companion. IEEE Press, pp 265–267

  • Ma W, Chen L, Yang Y, Zhou Y, Xu B (2016) Empirical analysis of network measures for effort-aware fault-proneness prediction. Inf Softw Technol 69:50–70

    Article  Google Scholar 

  • Matthews BW (1975) Comparison of the predicted and observed secondary structure of t4 phage lysozyme. Biochimica et Biophysica Acta (BBA)-Protein Structure 405(2):442–451

    Article  Google Scholar 

  • Mende T, Koschke R (2009) Revisiting the evaluation of defect prediction models. In: Proceedings of the 5th international conference on predictor models in software engineering, pp 1–10

  • Mende T, Koschke R (2010) Effort-aware defect prediction models. In: 2010 14th European conference on software maintenance and reengineering (CSMR). IEEE, pp 107–116

  • Mikolov T, Chen K, Corrado G, Dean J (2013) Efficient estimation of word representations in vector space. In: 1st international conference on learning representations, ICLR 2013, Scottsdale, Arizona, USA, May 2–4, 2013, Workshop Track Proceedings. 1301.3781

  • Monden A, Hayashi T, Shinoda S, Shirai K, Yoshida J, Barker M, Matsumoto K (2013) Assessing the cost effectiveness of fault prediction in acceptance testing. IEEE Trans Softw Eng 39(10):1345–1357

    Article  Google Scholar 

  • Nguyen TH, Adams B, Hassan AE (2010) Studying the impact of dependency network measures on software quality. In: 2010 IEEE international conference on software maintenance (ICSM). IEEE, pp 1–10

  • Osman H, Ghafari M, Nierstrasz O, Lungu M (2017) An extensive analysis of efficient bug prediction configurations. In: Proceedings of the 13th international conference on predictive models and data analytics in software engineering. ACM, pp 107–116

  • Palomba F, Zanoni M, Fontana FA, De Lucia A, Oliveto R (2016) Smells like teen spirit: improving bug prediction performance using the intensity of code smells. In: 2016 IEEE international conference on software maintenance and evolution (ICSME). IEEE, pp 244–255

  • Pascarella L, Palomba F, Bacchelli A (2020) On the performance of method-level bug prediction: a negative result. J Syst Softw 161:110493

    Article  Google Scholar 

  • Perozzi B, Al-Rfou R, Skiena S (2014) Deepwalk: online learning of social representations. In: Proceedings of the 20th ACM SIGKDD international conference on knowledge discovery and data mining. ACM, pp 701–710

  • Perozzi B, Kulkarni V, Chen H, Skiena S (2017) Don’t walk, skip! online learning of multi-scale network embeddings. In: Proceedings of the 2017 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining, vol 2017, pp 258–265

  • Pinzger M, Nagappan N, Murphy B (2008) Can developer-module networks predict failures?. In: Proceedings of the 16th ACM SIGSOFT international symposium on foundations of software engineering. ACM, pp 2–12

  • Premraj R, Herzig K (2011) Network versus code metrics to predict defects: a replication study. In: 2011 international symposium on empirical software engineering and measurement. IEEE, pp 215–224

  • Qu Y, Liu T, Chi J, Jin Y, Cui D, He A, Zheng Q (2018) node2defect: using network embedding to improve software defect prediction. In: Proceedings of the 33rd ACM/IEEE international conference on automated software engineering, ACM. pp 844–849

  • Qu Y, Zheng Q, Chi J, Jin Y, He A, Cui D, Zhang H, Liu T (2021) Using k-core decomposition on class dependency networks to improve bug prediction model’s practical performance. IEEE Trans Softw Eng 47 (2):348–366. https://doi.org/10.1109/TSE.2019.2892959

    Article  Google Scholar 

  • Śliwerski J, Zimmermann T, Zeller A (2005) When do changes induce fixes? ACM Sigsoft Software Engineering Notes 30(4):1–5

    Article  Google Scholar 

  • Šubelj L, Bajec M (2011) Community structure of complex software systems: analysis and applications. Physica A: Statistical Mechanics and its Applications 390(16):2968–2975

    Article  Google Scholar 

  • Suykens JA, Vandewalle J (1999) Least squares support vector machine classifiers. Neural Processing Letters 9(3):293–300

    Article  Google Scholar 

  • Tang J, Qu M, Wang M, Zhang M, Yan J, Mei Q (2015) Line: large-scale information network embedding. In: Proceedings of the 24th international conference on world wide web. International World Wide Web Conferences Steering Committee, pp 1067–1077

  • Tantithamthavorn C, McIntosh S, Hassan AE, Matsumoto K (2019) The impact of automated parameter optimization on defect prediction models. IEEE Trans Softw Eng 45(7):683–711. https://doi.org/10.1109/TSE.2018.2794977

    Article  Google Scholar 

  • Tosun A, Turhan B, Bener A (2009) Validation of network measures as indicators of defective modules in software systems. In: Proceedings of the 5th international conference on predictor models in software engineering, pp 1–9

  • Tu K, Cui P, Wang X, Yu PS, Zhu W (2018) Deep recursive network embedding with regular equivalence. In: Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining, pp 2357–2366

  • Velickovic P, Cucurull G, Casanova A, Romero A, Liò P, Bengio Y (2018) Graph attention networks. In: 6th international conference on learning representations, ICLR 2018, Vancouver, BC, Canada, April 30–May 3, 2018. Conference Track Proceedings. https://openreview.net/forum?id=rJXMpikCZ

  • Wang D, Cui P, Zhu W (2016) Structural deep network embedding. In: Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining. ACM, pp 1225–1234

  • Wattanakriengkrai S, Thongtanunam P, Tantithamthavorn C, Hata H, Matsumoto K (2020) Predicting defective lines using a model-agnostic technique. IEEE Trans Softw Eng:1–1. https://doi.org/10.1109/TSE.2020.3023177

  • Witten IH, Frank E, Hall MA, Pal CJ (2016) Data mining: practical machine learning tools and techniques. Morgan Kaufmann, San Mateo

    Google Scholar 

  • Yang Y, Zhou Y, Lu H, Chen L, Chen Z, Xu B, Leung H, Zhang Z (2015) Are slice-based cohesion metrics actually useful in effort-aware post-release fault-proneness prediction? An empirical study. IEEE Trans Softw Eng 41 (4):331–357

    Article  Google Scholar 

  • Yang Y, Zhou Y, Liu J, Zhao Y, Lu H, Xu L, Xu B, Leung H (2016) Effort-aware just-in-time defect prediction: simple unsupervised models could be better than supervised models. In: Proceedings of the 2016 24th ACM SIGSOFT international symposium on foundations of software engineering. ACM, pp 157–168

  • Zhang J, Dong Y, Wang Y, Tang J, Ding M (2019) Prone: fast and scalable network representation learning. In: Proceedings of the twenty-eighth international joint conference on artificial intelligence, IJCAI-19. International Joint Conferences on Artificial Intelligence Organization, pp 4278–4284. https://doi.org/10.24963/ijcai.2019/594

  • Zhao Y, Yang Y, Lu H, Liu J, Leung H, Wu Y, Zhou Y, Xu B (2017) Understanding the value of considering client usage context in package cohesion for fault-proneness prediction. Autom Softw Eng 24(2):393–453

    Article  Google Scholar 

  • Zimmermann T, Nagappan N (2008) Predicting defects using network analysis on dependency graphs. In: Proceedings of the 30th international conference on Software engineering, pp 531–540

Download references

Acknowledgements

We would like to thank the anonymous reviewers for their insightful comments and valuable suggestions for improving this paper. This work was supported in part by National Science Foundation under Grant No. 1719175, Office of Naval Research under Award No. N00014-17-1-2893, and National Natural Science Foundation of China under Grant No. 61602369. Any opinions, findings, and conclusions or recommendations expressed in this paper are those of the authors and do not necessarily reflect the views of the funding agencies.

Author information

Authors and Affiliations

  1. Department of Computer Science and Engineering, University of California, Riverside, Riverside, CA, 92521, USA

    Yu Qu & Heng Yin

Authors
  1. Yu Qu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Heng Yin
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Yu Qu.

Additional information

Communicated by: Tim Menzies

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Qu, Y., Yin, H. Evaluating network embedding techniques’ performances in software bug prediction. Empir Software Eng 26, 60 (2021). https://doi.org/10.1007/s10664-021-09965-5

Download citation

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s10664-021-09965-5

Keywords

Search

Navigation