skip to main content
10.1145/2597073.2597098acmconferencesArticle/Chapter ViewAbstractPublication PagesicseConference Proceedingsconference-collections
Article

Works for me! characterizing non-reproducible bug reports

Published: 31 May 2014 Publication History

Abstract

Bug repository systems have become an integral component of software development activities. Ideally, each bug report should help developers to find and fix a software fault. However, there is a subset of reported bugs that is not (easily) reproducible, on which developers spend considerable amounts of time and effort. We present an empirical analysis of non-reproducible bug reports to characterize their rate, nature, and root causes. We mine one industrial and five open-source bug repositories, resulting in 32K non-reproducible bug reports. We (1) compare properties of non-reproducible reports with their counterparts such as active time and number of authors, (2) investigate their life-cycle patterns, and (3) examine 120 Fixed non-reproducible reports. In addition, we qualitatively classify a set of randomly selected non-reproducible bug reports (1,643) into six common categories. Our results show that, on average, non-reproducible bug reports pertain to 17% of all bug reports, remain active three months longer than their counterparts, can be mainly (45%) classified as "Interbug Dependencies'', and 66% of Fixed non-reproducible reports were indeed reproduced and fixed.

References

[1]
Bugzilla. http://www.bugzilla.org/docs/.
[2]
Bugzilla: Eclipse Bug #106396. https: //bugs .eclipse.org/bugs/show_bug.cgi?id=106396.
[3]
Bugzilla@Mozilla. https://bugzilla.mozilla.org.
[4]
Eclipse Bugzilla. https://bugs.eclipse.org/bugs/.
[5]
JIRA. https://confluence.atlassian.com/display/ JIRA050/JIRA+Documentation.
[6]
MediaWiki Bugzilla. https://bugzilla .wikimedia.org/.
[7]
Moodle Tracker! https://tracker .moodle.org/issues/?jql=.
[8]
Non-reproducible bug report analyser and empirical data. https://github .com/saltlab/NR-bug-analyzer.
[9]
Works on my machine - How to fix non-reproducible bugs? http: //stackoverflow .com/questions/1102716/works-onmy-machine-how-to-fix-non-reproducible-bugs.
[10]
Bug fields. https: //bugzilla .mozilla.org/page.cgi?id=fields.html, Sep 2013.
[11]
What is an issue? https://confluence .atlassian.com/display/JIRA/ What+is+an+Issue, Sep 2013.
[12]
“works on my machine” - how to fix non-reproducible bugs? http://stackoverflow.com/q/1102716, Sep 2013.
[13]
J. Aranda and G. Venolia. The secret life of bugs: Going past the errors and omissions in software repositories. In Proceedings of the International Conference on Software Engineering (ICSE), pages 298–308. IEEE Computer Society, 2009.
[14]
J. Bell, N. Sarda, and G. Kaiser. Chronicler: Lightweight recording to reproduce field failures. In Proceedings of the 2013 International Conference on Software Engineering, ICSE ’13, pages 362–371. IEEE Press, 2013.
[15]
N. Bettenburg, S. Just, A. Schröter, C. Weiss, R. Premraj, and T. Zimmermann. What makes a good bug report? In Proceedings of the 16th ACM SIGSOFT International Symposium on Foundations of software engineering, pages 308–318. ACM, 2008.
[16]
P. Bhattacharya, L. Ulanova, I. Neamtiu, and S. C. Koduru. An empirical analysis of bug reports and bug fixing in open source Android apps. In Proceedings of the European Conference on Software Maintenance and Reengineering (CSMR), pages 133–143. IEEE Computer Society, 2013.
[17]
CNET. Researcher posts Facebook bug report to Mark Zuckerberg’s wall, 2013. http://news.cnet.com/8301- 1023_3-57599043-93/researcher-posts-facebookbug-report-to-mark-zuckerbergs-wall/.
[18]
J. W. Creswell. Research design: Qualitative, quantitative, and mixed methods approaches. Sage Publications, Incorporated, 2013.
[19]
E. Giger, M. D’Ambros, M. Pinzger, and H. C. Gall. Method-level bug prediction. In Proceedings of the International Symposium on Empirical Software Engineering and Measurement, ESEM, pages 171–180. ACM, 2012.
[20]
P. J. Guo, T. Zimmermann, N. Nagappan, and B. Murphy. Characterizing and predicting which bugs get fixed: an empirical study of Microsoft Windows. In Proceedings of the International Conference on Software Engineering (ICSE), pages 495–504. ACM, 2010.
[21]
P. J. Guo, T. Zimmermann, N. Nagappan, and B. Murphy. ‘not my bug!’ and other reasons for software bug report reassignments. In Proceedings of the Conference on Computer Supported Cooperative Work, CSCW, pages 395–404. ACM, 2011.
[22]
S. Herbold, J. Grabowski, S. Waack, and U. Bünting. Improved bug reporting and reproduction through non-intrusive gui usage monitoring and automated replaying. In Proceedings of the 2011 IEEE Fourth International Conference on Software Testing, Verification and Validation Workshops, ICSTW ’11, pages 232–241. IEEE Computer Society, 2011.
[23]
K. Herzig, S. Just, and A. Zeller. It’s not a bug, it’s a feature: how misclassification impacts bug prediction. In Proceedings of the International Conference on Software Engineering (ICSE), pages 392–401. IEEE Computer Society, 2013.
[24]
P. Hooimeijer and W. Weimer. Modeling bug report quality. In Proceedings of the International Conference on Automated Software Engineering (ASE), pages 34–43. ACM, 2007.
[25]
A. Kumar Maji, K. Hao, S. Sultana, and S. Bagchi. Characterizing failures in mobile OSes: A case study with Android and Symbian. In Proceedings of the International Symposium on Software Reliability Engineering (ISSRE), pages 249–258. IEEE Computer Society, 2010.
[26]
C. Lewis, Z. Lin, C. Sadowski, X. Zhu, R. Ou, and E. J. Whitehead Jr. Does bug prediction support human developers? findings from a Google case study. In Proceedings of the International Conference on Software Engineering, ICSE, pages 372–381. IEEE Computer Society, 2013.
[27]
T. Roehm, N. Gurbanova, B. Bruegge, C. Joubert, and W. Maalej. Monitoring user interactions for supporting failure reproduction. In International Conference on Program Comprehension (ICPC), pages 73–82. IEEE, 2013.
[28]
H. Seo and S. Kim. Predicting recurring crash stacks. In Proceedings of the International Conference on Automated Software Engineering (ASE), pages 180–189. ACM, 2012.
[29]
E. Shihab, A. Ihara, Y. Kamei, W. Ibrahim, M. Ohira, B. Adams, A. Hassan, and K.-i. Matsumoto. Predicting re-opened bugs: A case study on the eclipse project. In Proceedings of the Working Conference on Reverse Engineering (WCRE), pages 249–258. IEEE Computer Society, 2010.
[30]
Z. Yin, D. Yuan, Y. Zhou, S. Pasupathy, and L. Bairavasundaram. How do fixes become bugs? In Proceedings of ESEC/FSE, pages 26–36. ACM, 2011.
[31]
T. Zimmermann, N. Nagappan, P. J. Guo, and B. Murphy. Characterizing and predicting which bugs get reopened. In Proceedings of the International Conference on Software Engineering (ICSE), pages 1074–1083. IEEE Computer Society, 2012.

Cited By

View all
  • (2025)Information needs in bug reports for web applicationsJournal of Systems and Software10.1016/j.jss.2024.112230219:COnline publication date: 1-Jan-2025
  • (2024)Inside Bug Report Templates: An Empirical Study on Bug Report Templates in Open-Source SoftwareProceedings of the 15th Asia-Pacific Symposium on Internetware10.1145/3671016.3671401(125-134)Online publication date: 24-Jul-2024
  • (2024)Mobile Bug Report Reproduction via Global Search on the App UI ModelProceedings of the ACM on Software Engineering10.1145/36608241:FSE(2656-2676)Online publication date: 12-Jul-2024
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
MSR 2014: Proceedings of the 11th Working Conference on Mining Software Repositories
May 2014
427 pages
ISBN:9781450328630
DOI:10.1145/2597073
  • General Chair:
  • Premkumar Devanbu,
  • Program Chairs:
  • Sung Kim,
  • Martin Pinzger
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

In-Cooperation

  • TCSE: IEEE Computer Society's Tech. Council on Software Engin.

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 31 May 2014

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. Non-reproducible bugs
  2. bug tracking systems
  3. mining bug reports

Qualifiers

  • Article

Conference

ICSE '14
Sponsor:

Upcoming Conference

ICSE 2025

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)27
  • Downloads (Last 6 weeks)4
Reflects downloads up to 14 Feb 2025

Other Metrics

Citations

Cited By

View all
  • (2025)Information needs in bug reports for web applicationsJournal of Systems and Software10.1016/j.jss.2024.112230219:COnline publication date: 1-Jan-2025
  • (2024)Inside Bug Report Templates: An Empirical Study on Bug Report Templates in Open-Source SoftwareProceedings of the 15th Asia-Pacific Symposium on Internetware10.1145/3671016.3671401(125-134)Online publication date: 24-Jul-2024
  • (2024)Mobile Bug Report Reproduction via Global Search on the App UI ModelProceedings of the ACM on Software Engineering10.1145/36608241:FSE(2656-2676)Online publication date: 12-Jul-2024
  • (2024)The Impact Of Bug Localization Based on Crash Report Mining: A Developers' PerspectiveProceedings of the 46th International Conference on Software Engineering: Software Engineering in Practice10.1145/3639477.3639730(13-24)Online publication date: 14-Apr-2024
  • (2024)Prompting Is All You Need: Automated Android Bug Replay with Large Language ModelsProceedings of the IEEE/ACM 46th International Conference on Software Engineering10.1145/3597503.3608137(1-13)Online publication date: 20-May-2024
  • (2024)Reproducibility of issues reported in stack overflow questions: Challenges, impact & estimationJournal of Systems and Software10.1016/j.jss.2024.112158217(112158)Online publication date: Nov-2024
  • (2024)Application Monitoring for bug reproduction in web-based applicationsJournal of Systems and Software10.1016/j.jss.2023.111834207:COnline publication date: 1-Jan-2024
  • (2024)A systematic mapping study of bug reproduction and localizationInformation and Software Technology10.1016/j.infsof.2023.107338165(107338)Online publication date: Jan-2024
  • (2024)Understanding the characteristics and the role of visual issue reportsEmpirical Software Engineering10.1007/s10664-024-10459-329:4Online publication date: 10-Jun-2024
  • (2023)R3: Record-Replay-Retroaction for Database-Backed ApplicationsProceedings of the VLDB Endowment10.14778/3611479.361151016:11(3085-3097)Online publication date: 24-Aug-2023
  • Show More Cited By

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media