Skip to main content

Advertisement

Springer Nature Link
Log in

An empirical study of data sampling techniques for just-in-time software defect prediction

  • Published:
Automated Software Engineering Aims and scope Submit manuscript

Abstract

Just-in-time software defect prediction (JIT-SDP) is a fine-grained, easy-to-trace, and practical method. Unfortunately, JIT-SDP usually suffers from the class imbalance problem, which affects the performance of the models. Data sampling is one of the commonly used class imbalance techniques to overcome this problem. However, there is a lack of comprehensive empirical studies to compare different data sampling techniques on the performance of JIT-SDP. In this paper, we consider both defect classification and defect ranking, two typical application scenarios. To this end, we performed an empirical comparison of 10 data sampling algorithms on the performance of JIT-SDP. Extensive experiments on 10 open-source projects with 12 performance measures show that the effectiveness of data sampling techniques can indeed vary relying on the specific evaluation measures in both defect classification and defect ranking scenarios. Specifically, the RUM algorithm has demonstrated superior performance overall in the context of defect classification, particularly in F-measure, AUC, and MCC. On the other hand, for defect ranking, the ENN algorithm has emerged as the most favorable option, exhibiting perfect results in \(P_{opt}\), Recall@20%, and F-measure@20%. However, data sampling techniques can lead to an increase in false alarms and require the inspection of a higher number of changes. These findings highlight the importance of carefully selecting the appropriate data sampling technique based on the specific evaluation measures for different scenarios.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

Explore related subjects

Discover the latest articles, news and stories from top researchers in related subjects.

Notes

  1. https://github.com/klainfo/ScottKnottESD.

  2. https://github.com/optuna/optuna.

References

  • Agrawal, A., Menzies, T.: Is “better data” better than “better data miners”? In: 2018 IEEE/ACM 40th International Conference on Software Engineering (ICSE), pp. 1050–1061 (2018)

  • Bennin, K.E., Keung, J.W., Monden, A.: On the relative value of data resampling approaches for software defect prediction. Empir. Softw. Eng. 24, 602–636 (2019)

    Article  Google Scholar 

  • Bennin, K.E., Tahir, A., MacDonell, S.G., et al.: An empirical study on the effectiveness of data resampling approaches for cross-project software defect prediction. IET Softw. 16, 185–199 (2022)

    Article  Google Scholar 

  • Cabral, G.G., Minku, L.L.: Towards reliable online just-in-time software defect prediction. IEEE Trans. Softw. Eng. 49(3), 1342–1358 (2023)

    Article  Google Scholar 

  • Cabral, G.G., Minku, L.L., Shihab, E., et al.: Class imbalance evolution and verification latency in just-in-time software defect prediction. In: 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE), pp. 666–676 (2019)

  • Chen, H., Jing, X.Y., Li, Z., et al.: An empirical study on heterogeneous fefect prediction approaches. IEEE Trans. Softw. Eng. 47(12), 2803–2822 (2021)

    Article  Google Scholar 

  • Chen, X., Zhao, Y., Wang, Q., et al.: Multi: multi-objective effort-aware just-in-time software defect prediction. Inf. Softw. Technol. 93, 1–13 (2018)

    Article  Google Scholar 

  • Fu, W., Menzies, T.: Revisiting unsupervised learning for defect prediction. In: Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering (FSE), pp. 72–83 (2017)

  • Fu, W., Menzies, T., Shen, X.: Tuning for software analytics: is it really necessary? Inf. Softw. Technol. 76, 135–146 (2016)

    Article  Google Scholar 

  • Ghotra, B., McIntosh, S., Hassan, A.E.: Revisiting the impact of classification techniques on the performance of defect prediction models. In: Proceedings of the 2015 IEEE/ACM 37th IEEE International Conference on Software Engineering (ICSE). IEEE, pp. 789–800 (2015)

  • Haixiang, G., Yijing, L., Shang, J., et al.: Learning from class-imbalanced data: review of methods and applications. Expert Syst. Appl. 73, 220–239 (2017)

    Article  Google Scholar 

  • Hall, T., Beecham, S., Bowes, D., et al.: A systematic literature review on fault prediction performance in software engineering. IEEE Trans. Softw. Eng. 38(6), 1276–1304 (2012)

    Article  Google Scholar 

  • Hasanin, T., Khoshgoftaar, T.M., Leevy, J.L., et al.: Severely imbalanced big data challenges: investigating data sampling approaches. J. Big Data 6(1), 1–25 (2019)

    Article  Google Scholar 

  • Hoang, T., Dam, H.K., Kamei, Y., et al.: Deepjit: an end-to-end deep learning framework for just-in-time defect prediction. In: Proceedings of the 2019 IEEE/ACM 16th International Conference on Mining Software Repositories (MSR). IEEE, pp. 34–45 (2019)

  • Hoang, T., Kang, H.J., Lo, D., et al.: Cc2vec: distributed representations of code changes. In: Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering (ICSE), pp. 518–529 (2020)

  • Huang, Q., Xia, X., Lo, D.: Revisiting supervised and unsupervised models for effort-aware just-in-time defect prediction. Empir. Softw. Eng. 24(5), 2823–2862 (2019)

    Article  Google Scholar 

  • Kamei, Y., Shihab, E., Adams, B., et al.: A large-scale empirical study of just-in-time quality assurance. IEEE Trans. Softw. Eng. 39(6), 757–773 (2013)

    Article  Google Scholar 

  • Kamei, Y., Fukushima, T., McIntosh, S., et al.: Studying just-in-time defect prediction using cross-project models. Empir. Softw. Eng. 21(5), 2072–2106 (2016)

    Article  Google Scholar 

  • Kim, S., Whitehead, E.J., Zhang, Y.: Classifying software changes: clean or buggy? IEEE Trans. Softw. Eng. 34(2), 181–196 (2008)

    Article  Google Scholar 

  • Koru, A., Zhang, D., Emam, K.E., et al.: An investigation into the functional form of the size-defect relationship for software modules. IEEE Trans. Softw. Eng. 35(2), 293–304 (2009)

    Article  Google Scholar 

  • Krawczyk, B.: Learning from imbalanced data: open challenges and future directions. Prog. Artif. Intell. 5(4), 221–232 (2016)

    Article  Google Scholar 

  • Lemaître, G., Nogueira, F., Aridas, C.K.: Imbalanced-learn: a python toolbox to tackle the curse of imbalanced datasets in machine learning. J. Mach. Learn. Res. 18(17), 1–5 (2017)

    Google Scholar 

  • Lessmann, S., Baesens, B., Mues, C., et al.: Benchmarking classification models for software defect prediction: a proposed framework and novel findings. IEEE Trans. Softw. Eng. 34(4), 485–496 (2008)

    Article  Google Scholar 

  • Li, W., Zhang, W., Jia, X., et al.: Effort-aware semi-supervised just-in-time defect prediction. Inf. Softw. Technol. 126, 106364 (2020)

    Article  Google Scholar 

  • Li, Z., Jing, X.Y., Wu, F., et al.: Cost-sensitive transfer kernel canonical correlation analysis for heterogeneous defect prediction. Autom. Softw. Eng. 25(2), 201–245 (2018)

    Article  Google Scholar 

  • Li, Z., Jing, X.Y., Zhu, X.: Progress on approaches to software defect prediction. IET Softw. 12(3), 161–175 (2018)

    Article  Google Scholar 

  • Li, Z., Jing, X.Y., Zhu, X., et al.: Heterogeneous defect prediction with two-stage ensemble learning. Autom. Softw. Eng. 26(3), 599–651 (2019)

    Article  Google Scholar 

  • Li, Z., Jing, X.Y., Zhu, X., et al.: On the multiple sources and privacy preservation numbers for heterogeneous defect prediction. IEEE Trans. Softw. Eng. 45(4), 391–411 (2019)

    Article  Google Scholar 

  • Li, Z., Niu, J., Jing, X.Y., et al.: Cross-project defect prediction via landmark selection-based kernelized discriminant subspace alignment. IEEE Trans. Reliab. 70(3), 996–1013 (2021)

    Article  Google Scholar 

  • Li, Z., Du, Q., Zhang, H., et al.: Appendices for “an empirical study of data sampling techniques for just-in-time software defect prediction”. [Online] Available: https://github.com/SoftwareDefect/Sampling (2023a)

  • Li, Z., Zhang, H., Jing, X.Y., et al.: Dssdpp: data selection and sampling based domain programming predictor for cross-project defect prediction. IEEE Trans. Softw. Eng. 49(4), 1941–1963 (2023)

    Article  Google Scholar 

  • Li, Z., Niu, J., Jing, X.Y.: Software defect prediction: future directions and challenges. Autom. Softw. Eng. 31, 1–14 (2024)

    Article  Google Scholar 

  • Liu, J., Zhou, Y., Yang, Y., et al.: Code churn: a neglected metric in effort-aware just-in-time defect prediction. In: Proceedings of the 2017 ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM). IEEE, pp. 11–19 (2017)

  • McIntosh, S., Kamei, Y.: Are fix-inducing changes a moving target? A longitudinal case study of just-in-time defect prediction. IEEE Trans. Softw. Eng. 44(5), 412–428 (2018)

    Article  Google Scholar 

  • Mende, T., Koschke, R.: Effort-aware defect prediction models. In: 2010 14th European Conference on Software Maintenance and Reengineering, pp. 107–116 (2010)

  • Menzies, T., Greenwald, J., Frank, A.: Data mining static code attributes to learn defect predictors. IEEE Trans. Softw. Eng. 33(1), 2–13 (2007)

    Article  Google Scholar 

  • Menzies, T., Milton, Z., Turhan, B., et al.: Defect prediction from static code features: current results, limitations, new approaches. Autom. Softw. Eng. 17(4), 375–407 (2010)

    Article  Google Scholar 

  • Mockus, A., Weiss, D.M.: Predicting risk of software changes. Bell Labs Tech. J. 5(2), 169–180 (2000)

    Article  Google Scholar 

  • Ni, C., Wang, W., Yang, K., et al.: The best of both worlds: integrating semantic features with expert features for defect prediction and localization. In: Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, pp. 672–683 (2022a)

  • Ni, C., Xia, X., Lo, D., et al.: Revisiting supervised and unsupervised methods for effort-aware cross-project defect prediction. IEEE Trans. Softw. Eng. 48(3), 786–802 (2022)

    Article  Google Scholar 

  • Ni, C., Xia, X., Lo, D., et al.: Just-in-time defect prediction on javascript projects: a replication study. ACM Trans. Softw. Eng. Methodol. 31(4), 1–38 (2022)

    Article  Google Scholar 

  • Niu, J., Li, Z., Chen, H., et al.: Data sampling and kernel manifold discriminant alignment for mixed-project heterogeneous defect prediction. Softw. Qual. J. 30(4), 917–951 (2022)

    Article  Google Scholar 

  • Shihab, E., Hassan, A.E., Adams, B., et al.: An industrial study on the risk of software changes. In: Proceedings of the ACM SIGSOFT 20th International Symposium on the Foundations of Software Engineering (FSE). ACM Press, pp. 1–11 (2012)

  • Shivaji, S., Whitehead, E.J., Akella, R., et al.: Reducing features to improve code change-based bug prediction. IEEE Trans. Softw. Eng. 39(4), 552–569 (2013)

    Article  Google Scholar 

  • Song, Q., Guo, Y., Shepperd, M.: A comprehensive investigation of the role of imbalanced learning for software defect prediction. IEEE Trans. Softw. Eng. 45(12), 1253–1269 (2019)

    Article  Google Scholar 

  • Tabassum, S., Minku, L.L., Feng, D., et al.: An investigation of cross-project learning in online just-in-time software defect prediction. In: Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering (ICSE). ACM, pp. 554–565 (2020)

  • Tan, M., Tan, L., Dara, S., et al.: Online defect prediction for imbalanced data. In: 2015 IEEE/ACM 37th IEEE International Conference on Software Engineering (ICSE), pp. 99–108 (2015)

  • Tantithamthavorn, C., McIntosh, S., Hassan, A.E., et al.: An empirical comparison of model validation techniques for defect prediction models. IEEE Trans. Softw. Eng. 43(1), 1–18 (2017)

    Article  Google Scholar 

  • Tantithamthavorn, C., McIntosh, S., Hassan, A.E., et al.: The impact of automated parameter optimization on defect prediction models. IEEE Trans. Softw. Eng. 45(7), 683–711 (2019)

    Article  Google Scholar 

  • Tantithamthavorn, C., Hassan, A.E., Matsumoto, K.: The impact of class rebalancing techniques on the performance and interpretation of defect prediction models. IEEE Trans. Softw. Eng. 46(11), 1200–1219 (2020)

    Article  Google Scholar 

  • Wan, Z., Xia, X., Hassan, A.E., et al.: Perceptions, expectations, and challenges in defect prediction. IEEE Trans. Softw. Eng. 46(11), 1241–1266 (2020)

    Article  Google Scholar 

  • Xu, H., Duan, R., Yang, S., et al.: An empirical study on data sampling for just-in-time defect prediction. In: Proceedings of the 7th International Conference on Artificial Intelligence and Security, pp. 54–69 (2021)

  • Yan, M., Xia, X., Fan, Y., et al.: Effort-aware just-in-time defect identification in practice: a case study at alibaba. In: Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE). ACM, pp. 1308–1319 (2020)

  • Yang, X., Lo, D., Xia, X., et al.: Deep learning for just-in-time defect prediction. In: Proceedings of the 2015 IEEE International Conference on Software Quality, Reliability and Security (QRS). IEEE, pp. 17–26 (2015)

  • Yang, X., Lo, D., Xia, X., et al.: Tlel: a two-layer ensemble learning approach for just-in-time defect prediction. Inf. Softw. Technol. 87, 206–220 (2017)

    Article  Google Scholar 

  • Yang, X., Yu, H., Fan, G., et al.: An empirical study on progressive sampling for just-in-time software defect prediction. In: Proceedings of the 7th International Workshop on Quantitative Approaches to Software Quality co-located with 26th Asia-Pacific Software Engineering Conference (APSEC), pp. 12–18 (2019)

  • Yang, Y., Zhou, Y., Liu, J., et al.: Effort-aware just-in-time defect prediction: simple unsupervised models could be better than supervised models. In: Proceedings of the 2016 24th ACM SIGSOFT International Symposium on Foundations of Software Engineering (FSE), pp. 157-168 (2016)

  • Zeng, Z., Zhang, Y., Zhang, H., et al.: Deep just-in-time defect prediction: how far are we? In: Proceedings of the 30th ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA). ACM, pp. 427–438 (2021)

  • Zhang, T., Yu, Y., Mao, X., et al.: Fense: a feature-based ensemble modeling approach to cross-project just-in-time defect prediction. Empir. Softw. Eng. 27(7), 162 (2022). https://doi.org/10.1007/s10664-022-10185-8

    Article  Google Scholar 

  • Zhou, X., Han, D., Lo, D.: Simple or complex? Together for a more accurate just-in-time defect predictor. In: Proceedings of the 30th IEEE/ACM International Conference on Program Comprehension, pp. 229–240 (2022)

Download references

Acknowledgements

The authors would like to thank the editors and anonymous reviewers for their constructive comments and suggestions. This work is partially funded by the National Natural Science Foundation of China (Grant Nos.: 61902228 and 62176069), Natural Science Basic Research Program of Shaanxi Province (Grant No.: 2024JC-YBMS-497), and funded by the China Scholarship Council.

Author information

Authors and Affiliations

  1. School of Computer Science, Shaanxi Normal University, Xi’an, 710119, China

    Zhiqiang Li & Qiannan Du

  2. School of Big Data and Software Engineering, Chongqing University, Chongqing, 401331, China

    Hongyu Zhang

  3. School of Computer Science, Wuhan University, Wuhan, 430072, China

    Xiao-Yuan Jing

  4. School of Computer, Guangdong University of Petrochemical Technology, Maoming, 525000, China

    Xiao-Yuan Jing

  5. College of Automation and College of Artificial Intelligence, Nanjing University of Posts and Telecommunications, Nanjing, 210023, China

    Fei Wu

Authors
  1. Zhiqiang Li
    View author publications

    You can also search for this author inPubMed Google Scholar

  2. Qiannan Du
    View author publications

    You can also search for this author inPubMed Google Scholar

  3. Hongyu Zhang
    View author publications

    You can also search for this author inPubMed Google Scholar

  4. Xiao-Yuan Jing
    View author publications

    You can also search for this author inPubMed Google Scholar

  5. Fei Wu
    View author publications

    You can also search for this author inPubMed Google Scholar

Contributions

Zhiqiang Li: conceptualization, methodology, writing Qiannan Du: experiment, validation, writing HongyuZhang: review, editing Xiao-Yuan Jing: review, editing Fei Wu: review, editing All authors reviewed the manuscript.

Corresponding author

Correspondence to Zhiqiang Li.

Ethics declarations

Conflict of interest

The authors declare no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Li, Z., Du, Q., Zhang, H. et al. An empirical study of data sampling techniques for just-in-time software defect prediction. Autom Softw Eng 31, 56 (2024). https://doi.org/10.1007/s10515-024-00455-8

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s10515-024-00455-8

Keywords