skip to main content
10.1145/2901739.2903506acmconferencesArticle/Chapter ViewAbstractPublication PagesicseConference Proceedingsconference-collections
short-paper

MUBench: a benchmark for API-misuse detectors

Published: 14 May 2016 Publication History

Abstract

Over the last few years, researchers proposed a multitude of automated bug-detection approaches that mine a class of bugs that we call API misuses. Evaluations on a variety of software products show both the omnipresence of such misuses and the ability of the approaches to detect them.
This work presents MuBench, a dataset of 89 API misuses that we collected from 33 real-world projects and a survey. With the dataset we empirically analyze the prevalence of API misuses compared to other types of bugs, finding that they are rare, but almost always cause crashes. Furthermore, we discuss how to use it to benchmark and compare API-misuse detectors.

References

[1]
C. Cifuentes, C. Hoermann, N. Keynes, L. Li, S. Long, E. Mealy, M. Mounteney, and B. Scholz. BegBunch: Benchmarking for C Bug Detection Tools. DEFECTS'09, pages 16--20. ACM, 2009.
[2]
V. Dallmeier and T. Zimmermann. Extraction of Bug Localization Benchmarks from History. ASE'07, pages 433--436. ACM, 2007.
[3]
M. Egele, D. Brumley, Y. Fratantonio, and C. Kruegel. An Empirical Study of Cryptographic Misuse in Android Applications. CCS'13, pages 73--84. ACM, 2013.
[4]
Q. Gao, H. Zhang, J. Wang, Y. Xiong, L. Zhang, and H. Mei. Fixing Recurring Crash Bugs via Analyzing Q&A Sites. ASE'15, pages 307--318, 2015.
[5]
M. Georgiev, S. Iyengar, S. Jana, R. Anubhai, D. Boneh, and V. Shmatikov. The Most Dangerous Code in the World: Validating SSL Certificates in Non-browser Software. CCS'12, pages 38--49. ACM, 2012.
[6]
K. Herzig, S. Just, and A. Zeller. It's Not a Bug, It's a Feature: How Misclassification Impacts Bug Prediction. ICSE'13, pages 392--401. IEEE Press, 2013.
[7]
R. Just, D. Jalali, and M. D. Ernst. Defects4J: A Database of Existing Faults to Enable Controlled Testing Studies for Java Programs. ISSTA'14, pages 437--440. ACM, 2014.
[8]
D. Lazar, H. Chen, X. Wang, and N. Zeldovich. Why Does Cryptographic Software Fail?: A Case Study and Open Problems. APSys'14, pages 7:1--7:7. ACM, 2014.
[9]
Z. Li and Y. Zhou. PR-Miner: Automatically Extracting Implicit Programming Rules and Detecting Violations in Large Software Code. ESEC/FSE'13, pages 306--315. ACM, 2005.
[10]
S. Nadi, S. Krüger, M. Mezini, and E. Bodden. "Jumping Through Hoops": Why do Developers Struggle with Cryptography APIs? ICSE'16, 2016.
[11]
H. A. Nguyen, T. T. Nguyen, N. H. Pham, J. Al-Kofahi, and T. N. Nguyen. Clone Management for Evolving Software. IEEE Trans. Softw. Eng., 38(5): 1008--1026, 2012.
[12]
H. A. Nguyen, T. T. Nguyen, G. Wilson, Jr., A. T. Nguyen, M. Kim, and T. N. Nguyen. A Graph-based Approach to API Usage Adaptation. OOPSLA'10, pages 302--321. ACM, 2010.
[13]
M. P. Robillard, E. Bodden, D. Kawrykow, M. Mezini, and T. Ratchford. Automated API Property Inference Techniques. IEEE Trans. Soft. Eng., 39:613--637, 2013.
[14]
A. Wasylkowski and A. Zeller. Mining Temporal Specifications from Object Usage. ASE, 18(3):263--292, 2011.

Cited By

View all
  • (2024)ChatGPT’s Potential in Cryptography Misuse Detection: A Comparative Analysis with Static Analysis ToolsProceedings of the 18th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement10.1145/3674805.3695408(582-588)Online publication date: 24-Oct-2024
  • (2024)An Empirical Study of API Misuses of Data-Centric LibrariesProceedings of the 18th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement10.1145/3674805.3686685(245-256)Online publication date: 24-Oct-2024
  • (2024)API Misuse Detection via Probabilistic Graphical ModelProceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis10.1145/3650212.3652112(88-99)Online publication date: 11-Sep-2024
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
MSR '16: Proceedings of the 13th International Conference on Mining Software Repositories
May 2016
544 pages
ISBN:9781450341868
DOI:10.1145/2901739
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 14 May 2016

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. API-misuse detection
  2. benchmark
  3. bug detection

Qualifiers

  • Short-paper

Funding Sources

Conference

ICSE '16
Sponsor:

Upcoming Conference

ICSE 2025

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)58
  • Downloads (Last 6 weeks)6
Reflects downloads up to 20 Jan 2025

Other Metrics

Citations

Cited By

View all
  • (2024)ChatGPT’s Potential in Cryptography Misuse Detection: A Comparative Analysis with Static Analysis ToolsProceedings of the 18th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement10.1145/3674805.3695408(582-588)Online publication date: 24-Oct-2024
  • (2024)An Empirical Study of API Misuses of Data-Centric LibrariesProceedings of the 18th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement10.1145/3674805.3686685(245-256)Online publication date: 24-Oct-2024
  • (2024)API Misuse Detection via Probabilistic Graphical ModelProceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis10.1145/3650212.3652112(88-99)Online publication date: 11-Sep-2024
  • (2024)Boosting API Misuse Detection via Integrating API Constraints from Multiple SourcesProceedings of the 21st International Conference on Mining Software Repositories10.1145/3643991.3644904(14-26)Online publication date: 15-Apr-2024
  • (2024)ASAP-Repair: API-Specific Automated Program Repair Based on API Usage GraphsProceedings of the 5th ACM/IEEE International Workshop on Automated Program Repair10.1145/3643788.3648011(1-4)Online publication date: 20-Apr-2024
  • (2024)Mutation Testing of Java Bytecode: A Model-Driven ApproachProceedings of the ACM/IEEE 27th International Conference on Model Driven Engineering Languages and Systems10.1145/3640310.3674103(237-248)Online publication date: 22-Sep-2024
  • (2024)Programming Assistant for Exception Handling with CodeBERTProceedings of the IEEE/ACM 46th International Conference on Software Engineering10.1145/3597503.3639188(1-13)Online publication date: 20-May-2024
  • (2024)Demystifying and Detecting Misuses of Deep Learning APIsProceedings of the IEEE/ACM 46th International Conference on Software Engineering10.1145/3597503.3639177(1-12)Online publication date: 20-May-2024
  • (2024)Demystifying API misuses in deep learning applicationsEmpirical Software Engineering10.1007/s10664-023-10413-929:2Online publication date: 16-Feb-2024
  • (2024)CryptoLLM: Harnessing the Power of LLMs to Detect Cryptographic API MisuseComputer Security – ESORICS 202410.1007/978-3-031-70879-4_18(353-373)Online publication date: 5-Sep-2024
  • Show More Cited By

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Media

Figures

Other

Tables

Share

Share

Share this Publication link

Share on social media