Skip to main content
SpringerLink
Log in

Automated classification of software issue reports using machine learning techniques: an empirical study

  • S.I. : ICACNI-2016
  • Published:
Innovations in Systems and Software Engineering Aims and scope Submit manuscript

Abstract

Software developers, testers and customers routinely submit issue reports to software issue trackers to record the problems they face in using a software. The issues are then directed to appropriate experts for analysis and fixing. However, submitters often misclassify an improvement request as a bug and vice versa. This costs valuable developer time. Hence automated classification of the submitted reports would be of great practical utility. In this paper, we analyze how machine learning techniques may be used to perform this task. We apply different classification algorithms, namely naive Bayes, linear discriminant analysis, k-nearest neighbors, support vector machine (SVM) with various kernels, decision tree and random forest separately to classify the reports from three open-source projects. We evaluate their performance in terms of F-measure, average accuracy and weighted average F-measure. Our experiments show that random forests perform best, while SVM with certain kernels also achieve high performance.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16

Similar content being viewed by others

Notes

  1. https://www.bugzilla.org/.

  2. https://www.gnu.org/software/gnats/.

  3. https://www.atlassian.com/software/jira/.

  4. hc.apache.org/httpclient-3.x/.

  5. https://lucene.apache.org/.

  6. http://jackrabbit.apache.org/jcr/index.html.

  7. https://www.r-project.org/about.html.

  8. Available at https://www.st.cs.uni-saarland.de/softevo//bugclassify/.

  9. http://hc.apache.org/.

  10. Denotes \(P(C=c_k|\mathbf {X}=\mathbf {x})\).

  11. \(\mathcal {N}(\mu , \sigma ^2)\) denotes normal distribution with mean \(\mu \) and variance \(\sigma ^2\).

  12. Recollect we abbreviate linear discriminant analysis as LDA. So we use LDAL here.

References

  1. Aggarwal K, Timbers F, Rutgers T, Hindle A, Stroulia E, Greiner R (2017) Detecting duplicate bug reports with software engineering domain knowledge. J Softw Evol Process. doi:10.1002/smr.1821

  2. Antoniol G, Ayari K, Di Penta M, Khomh F, Guéhéneuc YG (2008) Is it a bug or an enhancement? A text-based approach to classify change requests. In: Proceedings of the 2008 conference of the center for advanced studies on collaborative research: meeting of minds (CASCON’08), ACM, pp 23:304–23:318

  3. Anvik J, Murphy GC (2011) Reducing the effort of bug report triage: recommenders for development-oriented decisions. ACM Trans Softw Eng Methodol 20(3):10

    Article  Google Scholar 

  4. Anvik J, Hiew L, Murphy GC (2006) Who should fix this bug? In: Proceedings of the 28th international conference on software engineering (ICSE’06). ACM, pp 361–370

  5. Bhattacharya P, Neamtiu I, Shelton CR (2012) Automated, highly-accurate, bug assignment using machine learning and tossing graphs. J Syst Softw 85(10):2275–2292

    Article  Google Scholar 

  6. Breiman L (2001) Random forests. Mach Learn 45(1):5–32

    Article  MATH  Google Scholar 

  7. Cavalcanti YC, Mota Silveira Neto PA, Machado IdC, Vale TF, Almeida ES, Meira SRdL (2014) Challenges and opportunities for software change request repositories: a systematic mapping study. J Softw Evol Process 26(7):620–653

    Article  Google Scholar 

  8. Chawla I, Singh SK (2014) Automatic bug labeling using semantic information from lSI. In: Proceedings of the 2014 7th international conference on contemporary computing (IC3’14). IEEE, pp 376–381

  9. Chawla I, Singh SK (2015) An automated approach for bug categorization using fuzzy logic. In: Proceedings of the 8th India software engineering conference (ISEC’15). ACM, pp 90–99

  10. Chen TH, Thomas SW, Hassan AE (2016) A survey on the use of topic models when mining software repositories. Empir Softw Eng 21(5):1843–1919

    Article  Google Scholar 

  11. Forman G, Scholz M (2010) Apples-to-apples in cross-validation studies: pitfalls in classifier performance measurement. ACM SIGKDD Explor Newsl 12(1):49–57

    Article  Google Scholar 

  12. Hemmati H, Nadi S, Baysal O, Kononenko O, Wang W, Holmes R, Godfrey MW (2013) The MSR cookbook: mining a decade of research. In: Proceedings of the 2013 10th IEEE working conference on mining software repositories (MSR’13). IEEE, pp 343–352

  13. Herzig K, Zeller A (2014) Mining bug data. In: Recommendation systems in software engineering. Springer, pp 131–171

  14. Herzig K, Just S, Zeller A (2013) It’s not a bug, it’s a feature: how misclassification impacts bug prediction. In: Proceedings of the 2013 international conference on software engineering (ICSE’13). IEEE, pp 392–401

  15. Jeong G, Kim S, Zimmermann T (2009) Improving bug triage with bug tossing graphs. In: Proceedings of the the 7th joint meeting of the european software engineering conference and the ACM SIGSOFT symposium on the foundations of software engineering. ACM, pp 111–120

  16. Jonsson L, Borg M, Broman D, Sandahl K, Eldh S, Runeson P (2016) Automated bug assignment: ensemble-based machine learning in large scale industrial contexts. Empir Softw Eng 21(4):1533–1578

    Article  Google Scholar 

  17. Ko A, Myers B, Chau D (2006) A linguistic analysis of how people describe software problems. In: Proceedings of the 2006 IEEE symposium on visual languages and human-centric computing (VL/HCC’06). IEEE, pp 127–134

  18. Kochhar PS, Thung F, Lo D (2014) Automatic fine-grained issue report reclassification. In: Proceedings of the 2014 19th international conference on engineering of complex computer systems (ICECCS’14). IEEE, pp 126–135

  19. Lamkanfi A, Demeyer S, Giger E, Goethals B (2010) Predicting the severity of a reported bug. In: Proceedings of the 2010 7th IEEE working conference on mining software repositories (MSR’10). IEEE, pp 1–10

  20. Layman L, Nikora AP, Meek J, Menzies T (2016) Topic modeling of NASA space system problem reports: research in practice. In: Proceedings of the 13th international conference on mining software repositories (MSR’16). ACM, pp 303–314

  21. Limsettho N, Hata H, Matsumoto K (2014) Comparing hierarchical Dirichlet process with latent Dirichlet allocation in bug report multiclass classification. In: Proceedings of the 2014 15th IEEE/ACIS international conference on software engineering, artificial intelligence, networking and parallel/distributed computing (SNPD’14). IEEE, pp 1–6

  22. Limsettho N, Hata H, Monden A, Matsumoto K (2014) Automatic unsupervised bug report categorization. In: Proceedings of the 2014 6th international workshop on empirical software engineering in practice (IWESEP’14). IEEE, pp 7–12

  23. Maalej W, Kurtanović Z, Nabil H, Stanik C (2016) On the automatic classification of app reviews. Requir Eng 21(3):311–331

    Article  Google Scholar 

  24. Murphy G, C̆ubranić D (2004) Automatic bug triage using text categorization. In: Proceedings of the 16th international conference on software engineering and knowledge engineering (SEKE’04)

  25. Nagwani N, Verma S, Mehta KK (2013) Generating taxonomic terms for software bug classification by utilizing topic models based on latent Dirichlet allocation. In: Proceedings of the 2013 11th international conference on ICT and knowledge engineering (ICT&KE’13). IEEE, pp 1–5

  26. Ohira M, Kashiwa Y, Yamatani Y, Yoshiyuki H, Maeda Y, Limsettho N, Fujino K, Hata H, Ihara A, Matsumoto K (2015) A dataset of high impact bugs: manually-classified issue reports. In: Proceedings of the 2015 IEEE/ACM 12th working conference on mining software repositories (MSR’15). IEEE, pp 518–521

  27. Pandey N, Hudait A, Sanyal DK, Sen A (2016) Automated classification of issue reports from a software issue tracker. In: Proceedings of the 4th international conference on advanced computing, networking, and informatics (ICACNI’16). Springer

  28. Pingclasai N, Hata H, Matsumoto Ki (2013) Classifying bug reports to bugs and other requests using topic modeling. In: Proceedings of the 2013 20th Asia-Pacific software engineering conference (APSEC’13). IEEE, vol 2, pp 13–18

  29. Salton G, Buckley C (1988) Term-weighting approaches in automatic text retrieval. Inf Process Manag 24(5):513–523

    Article  Google Scholar 

  30. Sebastiani F (2002) Machine learning in automated text categorization. ACM Comput Surv 34(1):1–47

    Article  Google Scholar 

  31. Somasundaram K, Murphy GC (2012) Automatic categorization of bug reports using latent Dirichlet allocation. In: Proceedings of the 5th India software engineering conference (ISEC’15). ACM, pp 125–130

  32. Strate JD, Laplante PA (2013) A literature review of research in software defect reporting. IEEE Trans Reliab 62(2):444–454

  33. Tamrawi A, Nguyen TT, Al-Kofahi J, Nguyen TN (2011) Fuzzy set-based automatic bug triaging. In: Proceedings of the 2011 33rd international conference on software engineering (ICSE’13). IEEE, pp 884–887

  34. Thung F, Lo D, Jiang L (2012) Automatic defect categorization. In: Proceedings of the 2012 19th working conference on reverse engineering (WCRE’12). IEEE, pp 205–214

  35. Tian Y, Lo D, Xia X, Sun C (2015) Automated prediction of bug report priority using multi-factor analysis. Empir Softw Eng 20(5):1354–1383

    Article  Google Scholar 

  36. Uddin J, Ghazali R, Deris MM, Naseem R, Shah H (2016) A survey on bug prioritization. Artif Intell Rev 47:1–36

    Google Scholar 

  37. Wang X, Zhang L, Xie T, Anvik J, Sun J (2008) An approach to detecting duplicate bug reports using natural language and execution information. In: Proceedings of the 30th international conference on software engineering (ICSE’08). ACM, pp 461–470

  38. Wu L, Boyi X, Kaiser G, Passonneau R (2011) Bugminer: software reliability analysis via data mining of bug reports. In: Proceedings of the 23rd internal conference on software engineering and knowledge engineering (SEKE’11), pp 95–100

  39. Zhang H, Gong L, Versteeg S (2013) Predicting bug-fixing time: an empirical study of commercial software projects. In: Proceedings of the 2013 international conference on software engineering (ICSE’13). IEEE, pp 1042–1051

  40. Zhou Y, Tong Y, Gu R, Gall H (2016) Combining text mining and data mining for bug report classification. J Softw Evol Process 28(3):150–176

    Article  Google Scholar 

  41. Zibran MF (2016) On the effectiveness of labeled latent Dirichlet allocation in automatic bug-report categorization. In: Proceedings of the 38th international conference on software engineering (ICSE’16) companion. ACM, pp 713–715

  42. Zimmermann T, Premraj R, Bettenburg N, Just S, Schroter A, Weiss C (2010) What makes a good bug report? IEEE Trans Softw Eng 36(5):618–643

    Article  Google Scholar 

Download references

Acknowledgements

We are immensely thankful to Professor Hideaki Hata, Nara Institute of Science and Technology, Nara, Japan, for providing us the datasets used in [28] (and based on [14]) which helped us a lot in generating the datasets used in this paper. We also express our thanks to Ananda Das, Research Scholar, Department of Computer Science and Engineering, Indian Institute of Technology Kharagpur for helping us set up the R software environment.

Author information

Authors and Affiliations

  1. School of Computer Engineering, KIIT University, Bhubaneswar, Odisha, 751024, India

    Nitish Pandey, Debarshi Kumar Sanyal & Abir Hudait

  2. Indian Institute of Technology Kharagpur, Kharagpur, West Bengal, 721302, India

    Debarshi Kumar Sanyal

  3. Department of Computer Science and Engineering, JIS University, Kolkata, West Bengal, 700109, India

    Amitava Sen

Authors
  1. Nitish Pandey
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Debarshi Kumar Sanyal
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Abir Hudait
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Amitava Sen
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Debarshi Kumar Sanyal.

A Tables on classifier performance

A Tables on classifier performance

This appendix shows the tables used to plot the graphs in Sect. 6 of the paper. The highest performance value in a row is highlighted with italics in Tables 5, 6, 7, 8, 9, 10, 11, 12, 13 and 14. The highest performance value for every project in a row is highlighted with italics in Table 15.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Pandey, N., Sanyal, D.K., Hudait, A. et al. Automated classification of software issue reports using machine learning techniques: an empirical study. Innovations Syst Softw Eng 13, 279–297 (2017). https://doi.org/10.1007/s11334-017-0294-1

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11334-017-0294-1

Keywords

Search

Navigation