skip to main content
10.1145/3609437.3609458acmotherconferencesArticle/Chapter ViewAbstractPublication PagesinternetwareConference Proceedingsconference-collections
research-article

The Impact of the bug number on Effort-Aware Defect Prediction: An Empirical Study

Published: 05 October 2023 Publication History

Abstract

Previous research have utilized public software defect datasets such as NASA, RELINK, and SOFTLAB, which only contain class label information. Almost all Effort-Aware Defect Prediction (EADP) studies are carried out around these datasets. However, EADP studies typically relying on bug density (i.e., the ratio between bug numbers and the lines of code) for ranking software modules. In order to investigate the impact of neglecting bug number information in software defect datasets on the performance of EADP models, we examine the performance degradation of the best-performing learning to rank methods when class labels are utilized instead of bug numbers. The experimental results show that neglecting bug number information in building EADP models results in an increase in the detected bugs. However, it also leads to a significant increase in the initial false alarms, ranging from 45.5% to 90.9% of the datasets, and an significant increase in the modules that need to be inspected, ranging from 5.2% to 70.4%. Therefore, we recommend not only the class labels but also the bug number information should be disclosed when publishing software defect datasets, in order to construct more accurate EADP models.

References

[1]
Hervé Abdi. 2007. Bonferroni and Šidák corrections for multiple comparisons. Encyclopedia of measurement and statistics 3 (2007), 103–107.
[2]
Carina Andersson and Per Runeson. 2007. A replicated quantitative analysis of fault distributions in complex software systems. IEEE transactions on software engineering 33, 5 (2007), 273–286.
[3]
Kwabena Ebo Bennin, Jacky Keung, Akito Monden, Yasutaka Kamei, and Naoyasu Ubayashi. 2016. Investigating the effects of balanced training and testing datasets on effort-aware fault prediction models. In 2016 IEEE 40th annual Computer software and applications conference (COMPSAC), Vol. 1. IEEE, 154–163.
[4]
Kwabena Ebo Bennin, Koji Toda, Yasutaka Kamei, Jacky Keung, Akito Monden, and Naoyasu Ubayashi. 2016. Empirical evaluation of cross-release effort-aware defect prediction models. In 2016 IEEE International Conference on Software Quality, Reliability and Security (QRS). IEEE, 214–221.
[5]
G Boetticher. 2007. The PROMISE repository of empirical software engineering data. http://promisedata. org/repository (2007).
[6]
Gemma Catolino, Dario Di Nucci, and Filomena Ferrucci. 2019. Cross-project just-in-time bug prediction for mobile apps: An empirical assessment. In 2019 IEEE/ACM 6th International Conference on Mobile Software Engineering and Systems (MOBILESoft). IEEE, 99–110.
[7]
Ni Chao, Xin Xia, Lo David, Chen Xiang, and Gu Qing. 2022. Revisiting Supervised and Unsupervised Methods for Effort-Aware Cross-Project Defect Prediction. IEEE Transactions on Software Engineering 48, 3 (2022), 786–802.
[8]
Xiang Chen, Yingquan Zhao, Qiuping Wang, and Zhidan Yuan. 2018. MULTI: Multi-objective effort-aware just-in-time software defect prediction. Information and Software Technology 93 (2018), 1–13.
[9]
Tian Cheng, Kunsong Zhao, Song Sun, Muhammad Mateen, and Junhao Wen. 2022. Effort-aware cross-project just-in-time defect prediction framework for mobile apps. Frontiers of Computer Science 16, 6 (2022), 1–15.
[10]
Marco D’Ambros, Michele Lanza, and Romain Robbes. 2010. An extensive comparison of bug prediction approaches. In 2010 7th IEEE working conference on mining software repositories. IEEE, 31–41.
[11]
Xin Du, Tian Wang, Liuhai Wang, Weifeng Pan, Chunlai Chai, Xinxin Xu, Bo Jiang, and Jiale Wang. 2022. CoreBug: improving effort-aware bug prediction in software systems using generalized k-core decomposition in class dependency networks. Axioms 11, 5 (2022), 205.
[12]
Yuanrui FAN, Xin XIA, Daniel A COSTA, David LO, Ahmed E HASSAN, and Shanping LI. [n. d.]. The impact of changes mislabeled by SZZ on just-in-time defect prediction.(2019). IEEE Transactions on Software Engineering ([n. d.]), 1–26.
[13]
Norman E. Fenton and Niclas Ohlsson. 2000. Quantitative analysis of faults and failures in a complex software system. IEEE Transactions on Software engineering 26, 8 (2000), 797–814.
[14]
Wei Fu and Tim Menzies. 2017. Revisiting unsupervised learning for defect prediction. In Proceedings of the 2017 11th joint meeting on foundations of software engineering. 72–83.
[15]
Lina Gong, Haoxiang Zhang, Jingxuan Zhang, Mingqiang Wei, and Zhiqiu Huang. 2022. A Comprehensive Investigation of the Impact of Class Overlap on Software Defect Prediction. IEEE Transactions on Software Engineering (2022).
[16]
Yuchen Guo, Martin Shepperd, and Ning Li. 2018. Bridging effort-aware prediction and strong classification: a just-in-time software defect prediction study. In Proceedings of the 40th International Conference on Software Engineering: Companion Proceeedings. 325–326.
[17]
Qiao Huang, Xin Xia, and David Lo. 2018. Revisiting supervised and unsupervised models for effort-aware just-in-time defect prediction. Empirical Software Engineering (2018), 1–40.
[18]
Qiao Huang, Xin Xia, and David Lo. 2019. Revisiting supervised and unsupervised models for effort-aware just-in-time defect prediction. Empirical Software Engineering 24, 5 (2019), 2823–2862.
[19]
Yue Jiang, Bojan Cukic, and Yan Ma. 2008. Techniques for evaluating fault prediction models. Empirical Software Engineering 13 (2008), 561–595.
[20]
Yasutaka Kamei, Shinsuke Matsumoto, Akito Monden, Ken-ichi Matsumoto, Bram Adams, and Ahmed E Hassan. 2010. Revisiting common bug prediction findings using effort-aware models. In 2010 IEEE International Conference on Software Maintenance. IEEE, 1–10.
[21]
Yasutaka Kamei, Emad Shihab, Bram Adams, Ahmed E Hassan, Audris Mockus, Anand Sinha, and Naoyasu Ubayashi. 2012. A large-scale empirical study of just-in-time quality assurance. IEEE Transactions on Software Engineering 39, 6 (2012), 757–773.
[22]
Pavneet Singh Kochhar, Xin Xia, David Lo, and Shanping Li. 2016. Practitioners’ expectations on automated fault localization. In Proceedings of the 25th International Symposium on Software Testing and Analysis. 165–176.
[23]
Fuyang Li, Wanpeng Lu, Jacky Wai Keung, Xiao Yu, Lina Gong, and Juan Li. 2023. The impact of feature selection techniques on effort-aware defect prediction: An empirical study. IET Software 17, 2 (2023), 168–193.
[24]
Fuyang Li, Peixin Yang, Jacky Wai Keung, Wenhua Hu, Haoyu Luo, and Xiao Yu. 2023. Revisiting ‘revisiting supervised methods for effort-aware cross-project defect prediction’. IET Software (2023). https://doi.org/10.1049/sfw2.12133
[25]
Jinping Liu, Yuming Zhou, Yibiao Yang, Hongmin Lu, and Baowen Xu. 2017. Code churn: A neglected metric in effort-aware just-in-time defect prediction. In 2017 ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM). IEEE, 11–19.
[26]
Wanwangying Ma, Lin Chen, Yibiao Yang, Yuming Zhou, and Baowen Xu. 2016. Empirical analysis of network measures for effort-aware fault-proneness prediction. Information and Software Technology 69 (2016), 50–70.
[27]
Thilo Mende and Rainer Koschke. 2009. Revisiting the evaluation of defect prediction models. In Proceedings of the 5th International Conference on Predictor Models in Software Engineering. 1–10.
[28]
Thilo Mende and Rainer Koschke. 2010. Effort-aware defect prediction models. In 2010 14th European Conference on Software Maintenance and Reengineering. IEEE, 107–116.
[29]
M Miletić, M Vukušić, G Mauša, and T Galinac Grbac. 2018. Cross-release code churn impact on effort-aware software defect prediction. In 2018 41st International Convention on Information and Communication Technology, Electronics and Microelectronics (MIPRO). IEEE, 1460–1466.
[30]
K Muthukumaran, NL Bhanu Murthy, G Karthik Reddy, and Prateek Talishetti. 2016. Testing and Code Review Based Effort-Aware Bug Prediction Model. Software Engineering, Artificial Intelligence, Networking and Parallel/Distributed Computing (2016), 17–30.
[31]
Tung Thanh Nguyen, Tran Quang An, Vu Thanh Hai, and Tu Minh Phuong. 2014. Similarity-based and rank-based defect prediction. In 2014 International Conference on Advanced Technologies for Communications (ATC 2014). IEEE, 321–325.
[32]
Thomas J Ostrand, Elaine J Weyuker, and Robert M Bell. 2005. Predicting the location and number of faults in large software systems. IEEE Transactions on Software Engineering 31, 4 (2005), 340–355.
[33]
Annibale Panichella, Carol V Alexandru, Sebastiano Panichella, Alberto Bacchelli, and Harald C Gall. 2016. A search-based training algorithm for cost-aware defect prediction. In Proceedings of the Genetic and Evolutionary Computation Conference 2016. 1077–1084.
[34]
Lei Qiao and Yan Wang. 2019. Effort-aware and just-in-time defect prediction with neural network. PloS one 14, 2 (2019), e0211359.
[35]
Yu Qu, Qinghua Zheng, Jianlei Chi, Yangxu Jin, Ancheng He, Di Cui, Hengshan Zhang, and Ting Liu. 2019. Using k-core decomposition on class dependency networks to improve bug prediction model’s practical performance. IEEE Transactions on Software Engineering 47, 2 (2019), 348–366.
[36]
Martin Shepperd, Qinbao Song, Zhongbin Sun, and Carolyn Mair. 2013. Data quality: Some comments on the nasa software defect datasets. IEEE Transactions on Software Engineering 39, 9 (2013), 1208–1215.
[37]
Burak Turhan, Tim Menzies, Ayşe B Bener, and Justin Di Stefano. 2009. On the relative value of cross-company and within-company data for defect prediction. Empirical Software Engineering 14, 5 (2009), 540–578.
[38]
Maria Ulan, Welf Löwe, Morgan Ericsson, and Anna Wingkvist. 2021. Weighted software metrics aggregation and its application to defect prediction. Empirical Software Engineering 26, 5 (2021), 1–34.
[39]
Feng Wang, Jinxiao Huang, and Yutao Ma. 2018. A top-k learning to rank approach to cross-project software defect prediction. In 2018 25th Asia-Pacific Software Engineering Conference (APSEC). IEEE, 335–344.
[40]
Frank Wilcoxon. [n. d.]. Individual Comparisons by Ranking Methods. Biometrics Bulletin 1, 6 ([n. d.]), 80–83.
[41]
Rongxin Wu, Hongyu Zhang, Sunghun Kim, and Shing-Chi Cheung. 2011. Relink: recovering links between bugs and changes. In Proceedings of the 19th ACM SIGSOFT symposium and the 13th European conference on Foundations of software engineering. ACM, 15–25.
[42]
Zhou Xu, Kunsong Zhao, Tao Zhang, Chunlei Fu, Meng Yan, Zhiwen Xie, Xiaohong Zhang, and Gemma Catolino. 2021. Effort-aware just-in-time bug prediction for mobile apps via cross-triplet deep feature embedding. IEEE Transactions on Reliability 71, 1 (2021), 204–220.
[43]
Meng Yan, Yicheng Fang, David Lo, Xin Xia, and Xiaohong Zhang. 2017. File-level defect prediction: Unsupervised vs. supervised models. In 2017 ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM). IEEE, 344–353.
[44]
Meng Yan, Xin Xia, Yuanrui Fan, Ahmed E Hassan, David Lo, and Shanping Li. 2020. Just-in-time defect identification and localization: A two-phase framework. IEEE Transactions on Software Engineering 48, 1 (2020), 82–101.
[45]
Meng Yan, Xin Xia, Yuanrui Fan, David Lo, Ahmed E Hassan, and Xindong Zhang. 2020. Effort-aware just-in-time defect identification in practice: a case study at Alibaba. In Proceedings of the 28th ACM joint meeting on european software engineering conference and symposium on the foundations of software engineering. 1308–1319.
[46]
Xiaoxing Yang, Ke Tang, and Xin Yao. 2014. A learning-to-rank approach to software defect prediction. IEEE Transactions on Reliability 64, 1 (2014), 234–246.
[47]
Xiaoxing Yang and Wushao Wen. 2018. Ridge and lasso regression models for cross-version defect prediction. IEEE Transactions on Reliability 67, 3 (2018), 885–896.
[48]
Xingguang Yang, Huiqun Yu, Guisheng Fan, and Kang Yang. 2021. DEJIT: a differential evolution algorithm for effort-aware just-in-time software defect prediction. International Journal of Software Engineering and Knowledge Engineering 31, 03 (2021), 289–310.
[49]
Yibiao Yang, Mark Harman, Jens Krinke, Syed Islam, David Binkley, Yuming Zhou, and Baowen Xu. 2016. An empirical study on dependence clusters for effort-aware fault-proneness prediction. In Proceedings of the 31st IEEE/ACM International Conference on Automated Software Engineering. 296–307.
[50]
Yibiao Yang, Yuming Zhou, Jinping Liu, Yangyang Zhao, Hongmin Lu, Lei Xu, Baowen Xu, and Hareton Leung. 2016. Effort-aware just-in-time defect prediction: simple unsupervised models could be better than supervised models. In Proceedings of the 2016 24th ACM SIGSOFT International Symposium on Foundations of Software Engineering. ACM, 157–168.
[51]
Yibiao Yang, Yuming Zhou, Hongmin Lu, Lin Chen, Zhenyu Chen, Baowen Xu, Hareton Leung, and Zhenyu Zhang. 2014. Are slice-based cohesion metrics actually useful in effort-aware post-release fault-proneness prediction? An empirical study. IEEE Transactions on Software Engineering 41, 4 (2014), 331–357.
[52]
Guoan You, Feng Wang, and Yutao Ma. 2016. An empirical study of ranking-oriented cross-project software defect prediction. International Journal of Software Engineering and Knowledge Engineering 26, 09n10 (2016), 1511–1538.
[53]
Xiao Yu, Kwabena Ebo Bennin, Jin Liu, Jacky Wai Keung, Xiaofei Yin, and Zhou Xu. 2019. An empirical study of learning to rank techniques for effort-aware defect prediction. In 2019 IEEE 26th International Conference on Software Analysis, Evolution and Reengineering. IEEE, 298–309.
[54]
Xiao Yu, Heng Dai, Li Li, Xiaodong Gu, Jacky Wai Keung, Kwabena Ebo Bennin, Fuyang Li, and Jin Liu. 2023. Finding the best learning to rank algorithms for effort-aware defect prediction. Information and Software Technology (2023), 107165. https://doi.org/10.1016/j.infsof.2023.107165
[55]
Xiao Yu, Jacky Keung, Yan Xiao, Shuo Feng, Fuyang Li, and Heng Dai. 2022. Predicting the precise number of software defects: Are we there yet?Information and Software Technology 146 (2022), 106847.
[56]
Xiao Yu, Jin Liu, Jacky Wai Keung, Qing Li, Kwabena Ebo Bennin, Zhou Xu, Junping Wang, and Xiaohui Cui. 2019. Improving ranking-oriented defect prediction using a cost-sensitive ranking SVM. IEEE Transactions on Reliability 69, 1 (2019), 139–153.
[57]
Xiao Yu, Jiqing Rao, Wenhua Hu, Jacky Keung, Junwei Zhou, and Jianwen Xiang. 2023. Improving effort-aware defect prediction by directly learning to rank software modules. Information and Software Technology (2023), 107250.
[58]
Xiao Yu, Man Wu, Yiheng Jian, Kwabena Ebo Bennin, Mandi Fu, and Chuanxiang Ma. 2018. Cross-company defect prediction via semi-supervised clustering-based data filtering and MSTrA-based transfer learning. Soft Computing 22 (2018), 3461–3472.
[59]
Tao Zhang, Jiachi Chen, Geunseok Yang, Byungjeong Lee, and Xiapu Luo. 2016. Towards more accurate severity prediction and fixer recommendation of software bugs. Journal of Systems and Software 117 (2016), 166–184.
[60]
Wenzhou Zhang, Weiwei Li, and Xiuyi Jia. 2019. Effort-aware tri-training for semi-supervised just-in-time defect prediction. In Advances in Knowledge Discovery and Data Mining: 23rd Pacific-Asia Conference, PAKDD 2019, Macau, China, April 14-17, 2019, Proceedings, Part II 23. Springer, 293–304.
[61]
Kunsong Zhao, Zhou Xu, Meng Yan, Lei Xue, Wei Li, and Gemma Catolino. 2022. A compositional model for effort-aware Just-In-Time defect prediction on android apps. IET Software 16, 3 (2022), 259–278.
[62]
Thomas Zimmermann, Rahul Premraj, and Andreas Zeller. 2007. Predicting defects for eclipse. In Third International Workshop on Predictor Models in Software Engineering (PROMISE’07: ICSE Workshops 2007). IEEE, 9–9.

Cited By

View all
  • (2024)Bug numbers matter: An empirical study of effort‐aware defect prediction using class labels versus bug numbersSoftware: Practice and Experience10.1002/spe.336355:1(49-78)Online publication date: 10-Jul-2024

Index Terms

  1. The Impact of the bug number on Effort-Aware Defect Prediction: An Empirical Study

        Recommendations

        Comments

        Information & Contributors

        Information

        Published In

        cover image ACM Other conferences
        Internetware '23: Proceedings of the 14th Asia-Pacific Symposium on Internetware
        August 2023
        332 pages
        ISBN:9798400708947
        DOI:10.1145/3609437
        Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

        Publisher

        Association for Computing Machinery

        New York, NY, United States

        Publication History

        Published: 05 October 2023

        Permissions

        Request permissions for this article.

        Check for updates

        Author Tags

        1. Bug Number
        2. Effort-Aware
        3. Learning to Rank
        4. Software Defect Prediction

        Qualifiers

        • Research-article
        • Research
        • Refereed limited

        Funding Sources

        Conference

        Internetware 2023

        Acceptance Rates

        Overall Acceptance Rate 55 of 111 submissions, 50%

        Contributors

        Other Metrics

        Bibliometrics & Citations

        Bibliometrics

        Article Metrics

        • Downloads (Last 12 months)46
        • Downloads (Last 6 weeks)4
        Reflects downloads up to 20 Feb 2025

        Other Metrics

        Citations

        Cited By

        View all
        • (2024)Bug numbers matter: An empirical study of effort‐aware defect prediction using class labels versus bug numbersSoftware: Practice and Experience10.1002/spe.336355:1(49-78)Online publication date: 10-Jul-2024

        View Options

        Login options

        View options

        PDF

        View or Download as a PDF file.

        PDF

        eReader

        View online with eReader.

        eReader

        HTML Format

        View this article in HTML Format.

        HTML Format

        Figures

        Tables

        Media

        Share

        Share

        Share this Publication link

        Share on social media