research-article

Vulnerability discovery strategies used in software projects

Authors:

Farzana Ahamed Bhuiyan,

Patrick MorrisonAuthors Info & Claims

ASE '20: Proceedings of the 35th IEEE/ACM International Conference on Automated Software Engineering

Pages 13 - 18

https://doi.org/10.1145/3417113.3422153

Published: 22 January 2021 Publication History

Abstract

Malicious users can exploit undiscovered software vulnerabilities i.e., undiscovered weaknesses in software, to cause serious consequences, such as large-scale data breaches. A systematic approach that synthesizes strategies used by security testers can aid practitioners to identify latent vulnerabilities. The goal of this paper is to help practitioners identify software vulnerabilities by categorizing vulnerability discovery strategies using open source software bug reports. We categorize vulnerability discovery strategies by applying qualitative analysis on 312 OSS bug reports. Next, we quantify the frequency and evolution of the identified strategies by analyzing 1,632 OSS bug reports collected from five software projects spanning across 2009 to 2019. The five software projects are Chrome, Eclipse, Mozilla, OpenStack, and PHP.

We identify four vulnerability discovery strategies: diagnostics, malicious payload construction, misconfiguration, and pernicious execution. For Eclipse and OpenStack, the most frequently used strategy is diagnostics, where security testers inspect source code and build/debug logs. For three web-related software projects namely, Chrome, Mozilla, and PHP, the most frequently occurring strategy is malicious payload construction i.e., creating malicious files, such as malicious certificates and malicious videos.

References

[1]

Paul Ammann and Jeff Offutt. 2016. Introduction to software testing. Cambridge University Press.

Digital Library

[2]

Mauricio Aniche, Christoph Treude, Igor Steinmacher, Igor Wiese, Gustavo Pinto, Margaret-Anne Storey, and Marco Aurelio Gerosa. 2018. How Modern News Aggregators Help Development Communities Shape and Share Knowledge. In Proceedings of the 40th International Conference on Software Engineering (Gothenburg, Sweden) (ICSE '18). Association for Computing Machinery, New York, NY, USA, 499--510.

Digital Library

[3]

A. Austin and L. Williams. 2011. One Technique is Not Enough: A Comparison of Vulnerability Discovery Techniques. In 2011 International Symposium on Empirical Software Engineering and Measurement. 97--106.

Digital Library

[4]

Farzana Ahamed Bhuiyan, Akond Rahman, and Patrick Morrison. 2020. Verifiability Package for Paper. https://figshare.com/s/eaac5aeea283239f2c56. [Online; accessed 05-Sep-2020].

[5]

Matthew A. Bishop. 2002. The Art and Science of Computer Security. Addison-Wesley Longman Publishing Co., Inc., USA.

Digital Library

[6]

Brandon Perry. 2016. Bug 1275400 (CVE-2016-5824). https://bugzilla.mozilla.org/show_bug.cgi?id=1275400. [Online; accessed 16-May-2020].

[7]

C. Evans and C. Palmer. 2011. Public Key Pinning Extension for HTTP. https://tools.ietf.org/html/draft-ietf-websec-key-pinning-01. [Online; accessed 15-Feb-2020].

[8]

Oscar Chaparro, Carlos Bernal-Cardenas, Jing Lu, Kevin Moran, Andrian Marcus, Massimiliano Di Penta, Denys Poshyvanyk, and Vincent Ng. 2019. Assessing the Quality of the Steps to Reproduce in Bug Reports. In Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering (Tallinn, Estonia) (ESEC/FSE 2019). ACM, New York, NY, USA, 86--96.

Digital Library

[9]

Oscar Chaparro, Jing Lu, Fiorella Zampetti, Laura Moreno, Massimiliano Di Penta, Andrian Marcus, Gabriele Bavota, and Vincent Ng. 2017. Detecting Missing Information in Bug Descriptions. In Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering (Paderborn, Germany) (ESEC/FSE 2017). ACM, New York, NY, USA, 396--407.

Digital Library

[10]

Chrome. 2020. Chrome infrastructure. https://chromium.googlesource.com/infra/infra/+/master/appengine/monorail/doc/api.md. [Online; accessed 26-Jan-2020].

[11]

Jacob Cohen. 1960. A Coefficient of Agreement for Nominal Scales. Educational and Psychological Measurement 20, 1 (1960), 37--46.

[12]

Benjamin F Crabtree and William L Miller. 1999. Doing qualitative research. sage publications.

[13]

Dan Gohman. 2013. Bug 916580 (CVE-2013-5595). https://bugzilla.mozilla.org/show_bug.cgi?id=916580. [Online; accessed 09-Feb-2020].

[14]

Danelle Au. 2015. The Importance of Learning From Hackers. https://www.securityweek.com/importance-learning-hackers. [Online; accessed 09-Feb-2020].

[15]

National Vulnerability Database. 2015. NVD-CVE-2015-0829. https://nvd.nist.gov/vuln/detail/CVE-2015-0829. [Online; accessed 19-Feb-2020].

[16]

National Vulnerability Database. 2016. NVD-CVE-2016-5262. https://nvd.nist.gov/vuln/detail/CVE-2016-5262. [Online; accessed 27-Feb-2020].

[17]

National Vulnerability Database. 2016. NVD-CVE-2016-5824. https://nvd.nist.gov/vuln/detail/CVE-2016-5824. [Online; accessed 19-May-2020].

[18]

National Vulnerability Database. 2017. NVD-CVE-2017-7761. https://nvd.nist.gov/vuln/detail/CVE-2017-7761. [Online; accessed 17-Feb-2020].

[19]

National Vulnerability Database. 2017. NVD-CVE-2017-7792. https://nvd.nist.gov/vuln/detail/CVE-2017-7792. [Online; accessed 18-Jan-2020].

[20]

National Vulnerability Database. 2018. NVD-CVE-2018-5158. https://nvd.nist.gov/vuln/detail/CVE-2018-5158. [Online; accessed 30-Jan-2020].

[21]

Eclipse. 2020. The Platform for Open Innovation and Collaboration. https://www.eclipse.org/. [Online; accessed 27-Jan-2020].

[22]

Ehsan Akhgari. 2014. Bug 961512 (CVE-2014-1499). https://bugzilla.mozilla.org/show_bug.cgi?id=961512. [Online; accessed 01-Feb-2020].

[23]

eusian. 2020. Strategies to find software vulnerabilities: what are the categories? https://www.reddit.com/r/cybersecurity/comments/egflzj/strategies_to_find_software_vulnerabilities_what/. [Online; accessed 23-Jan-2020].

[24]

M. Gegick, P. Rotella, and T. Xie. 2010. Identifying security bug reports via text mining: An industrial case study. In 2010 7th IEEE Working Conference on Mining Software Repositories (MSR 2010). 11--20.

[25]

Giovanni Vigna. 2019. How to Think Like a Hacker. https://www.darkreading.com/vulnerabilities---threats/how-to-think-like-a-hacker/a/d-id/1335989. [Online; accessed 10-Feb-2020].

[26]

K. Herzig, S. Just, and A. Zeller. 2013. It's not a bug, it's a feature: How misclassification impacts bug prediction. In 2013 35th International Conference on Software Engineering (ICSE). 392--401.

[27]

Yuan Jiang, Pengcheng Lu, Xiaohong Su, and Tiantian Wang. 2020. LTRWES: A new framework for security bug report detection. Information and Software Technology 124 (2020), 106314.

[28]

Matthieu Jimenez, Renaud Rwemalika, Mike Papadakis, Federica Sarro, Yves Le Traon, and Mark Harman. 2019. The Importance of Accounting for Real-World Labelling When Predicting Software Vulnerabilities. In Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering (Tallinn, Estonia) (ESEC/FSE 2019). Association for Computing Machinery, New York, NY, USA, 695--705.

Digital Library

[29]

J. Richard Landis and Gary G. Koch. 1977. The Measurement of Observer Agreement for Categorical Data. Biometrics 33, 1 (1977), 159--174. http://www.jstor.org/stable/2529310

[30]

Gary McGraw. 2018. Software Security: Building Security In. Addison-Wesley Professional.

[31]

L. Moreno, W. Bandara, S. Haiduc, and A. Marcus. 2013. On the Relationship between the Vocabulary of Bug Reports and Source Code. In 2013 IEEE International Conference on Software Maintenance. 452--455.

Digital Library

[32]

Mozilla. 2014. Mozilla Foundation Security Advisory 2014-19. https://www.mozilla.org/en-US/security/advisories/mfsa2014-19/. [Online; accessed 11-Jan-2020].

[33]

Mozilla. 2015. Mozilla Foundation Security Advisory 2015-13. https://www.mozilla.org/en-US/security/advisories/mfsa2015-13/. [Online; accessed 17-Jan-2020].

[34]

Mozilla. 2020. Bugzilla Main Page. https://bugzilla.mozilla.org/home. [Online; accessed 03-Feb-2020].

[35]

Mozilla Developer Network. 2020. WebRTC API. https://developer.mozilla.org/en-US/docs/Web/API/WebRTC_API. [Online; accessed 16-Feb-2020].

[36]

Muneaki Nishimura. 2014. Bug 1065909 (CVE-2015-0832). https://bugzilla.mozilla.org/show_bug.cgi?id=1065909. [Online; accessed 22-Feb-2020].

[37]

Nikita. 2016. Bug 1277475 (CVE-2016-5262). https://bugzilla.mozilla.org/show_bug.cgi?id=1277475. [Online; accessed 13-Feb-2020].

[38]

Openstack. 2020. Openstack - Build the future of Open Infrastructure. https://www.openstack.org/. [Online; accessed 21-Feb-2020].

[39]

pantrombka. 2015. Bug 1128939 (CVE-2015-0829). https://bugzilla.mozilla.org/show_bug.cgi?id=1128939. [Online; accessed 10-Feb-2020].

[40]

PHP. 2020. PHP Bug Tracking System. https://bugs.php.net/. [Online; accessed 28-Jan-2020].

[41]

T. Pornin. 2013. Deterministic Usage of the Digital Signature Algorithm (DSA) and Elliptic Curve Digital Signature Algorithm (ECDSA). https://tools.ietf.org/html/rfc6979. [Online; accessed 28-Jan-2020].

[42]

Akond Rahman and Laurie Williams. 2019. A Bird's Eye View of Knowledge Needs Related to Penetration Testing. In Proceedings of the 6th Annual Symposium on Hot Topics in the Science of Security (Nashville, Tennessee, USA) (HotSoS '19). Association for Computing Machinery, New York, NY, USA, Article 9, 2 pages.

Digital Library

[43]

Johnny Saldana. 2015. The coding manual for qualitative researchers. Sage.

[44]

Yonghee Shin and Laurie Williams. 2013. Can traditional fault prediction models be used for vulnerability prediction? Empirical Software Engineering 18, 1 (01 Feb 2013), 25--59.

[45]

Tyson Smith. 2017. Bug 1368652 (CVE-2017-7792). https://bugzilla.mozilla.org/show_bug.cgi?id=1368652. [Online; accessed 14-Feb-2020].

[46]

Valgrind. 2019. Valgrind Home. http://www.valgrind.org/. [Online; accessed 22-Feb-2020].

[47]

w3docs. 2020. HTML <marquee> Tag. https://www.w3docs.com/learn-html/html-marquee-tag.html. [Online; accessed 13-Feb-2020].

[48]

watsonbladd. 2015. Bug 1125025 (CVE-2015-2730). https://bugzilla.mozilla.org/show_bug.cgi?id=1125025. [Online; accessed 10-Feb-2020].

[49]

D. Wijayasekara, M. Manic, and M. McQueen. 2014. Vulnerability identification and classification via text mining bug databases. In IECON 2014 - 40th Annual Conference of the IEEE Industrial Electronics Society. 3612--3618.

[50]

Mozilla Wiki. 2019. SecurityEngineering/Public Key Pinning. https://wiki.mozilla.org/SecurityEngineering/Public_Key_Pinning. [Online; accessed 29-Jan-2020].

[51]

Wladimir Palant. 2017. Bug 1215648 (CVE-2017-7761). https://bugzilla.mozilla.org/show_bug.cgi?id=1215648. [Online; accessed 10-Feb-2020].

[52]

Wladimir Palant. 2018. Bug 1452075 (CVE-2018-5158). https://bugzilla.mozilla.org/show_bug.cgi?id=1452075. [Online; accessed 10-Feb-2020].

[53]

Tianyin Xu, Long Jin, Xuepeng Fan, Yuanyuan Zhou, Shankar Pasupathy, and Rukma Talwadker. 2015. Hey, You Have given Me Too Many Knobs!: Understanding and Dealing with over-Designed Configuration in System Software. In Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering (Bergamo, Italy) (ESEC/FSE 2015). Association for Computing Machinery, New York, NY, USA, 307--319.

Digital Library

[54]

Z. Yu, C. Theisen, L. Williams, and T. Menzies. 2019. Improving Vulnerability Inspection Efficiency Using Active Learning. IEEE Transactions on Software Engineering (2019), 1--1.

[55]

T. Zimmermann, N. Nagappan, and L. Williams. 2010. Searching for a Needle in a Haystack: Predicting Security Vulnerabilities for Windows Vista. In 2010 Third International Conference on Software Testing, Verification and Validation. 421--428.

Digital Library

Cited By

Happe ACito JChandra SBlincoe KTonella P(2023)Understanding Hackers’ Work: An Empirical Study of Offensive Security PractitionersProceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering10.1145/3611643.3613900(1669-1680)Online publication date: 30-Nov-2023
https://dl.acm.org/doi/10.1145/3611643.3613900
Bhuiyan FMurphy JMorrison PRahman A(2021)Practitioner Perception of Vulnerability Discovery Strategies2021 IEEE/ACM 2nd International Workshop on Engineering and Cybersecurity of Critical Systems (EnCyCriS)10.1109/EnCyCriS52570.2021.00014(41-44)Online publication date: Jun-2021
https://doi.org/10.1109/EnCyCriS52570.2021.00014
Cottrell KBose DShahriar HRahman A(2021)An Empirical Study of Vulnerabilities in Robotics2021 IEEE 45th Annual Computers, Software, and Applications Conference (COMPSAC)10.1109/COMPSAC51774.2021.00105(735-744)Online publication date: Jul-2021
https://doi.org/10.1109/COMPSAC51774.2021.00105

Index Terms

Vulnerability discovery strategies used in software projects
1. Security and privacy
  1. Software and application security
    1. Software security engineering

Recommendations

Can we use software bug reports to identify vulnerability discovery strategies?
HotSoS '20: Proceedings of the 7th Symposium on Hot Topics in the Science of Security

Daily horror stories related to software vulnerabilities necessitates the understanding of how vulnerabilities are discovered. Identification of data sources that can be leveraged to understand how vulnerabilities are discovered could aid cybersecurity ...
Software Vulnerability Discovery Techniques: A Survey
MINES '12: Proceedings of the 2012 Fourth International Conference on Multimedia Information Networking and Security

Software vulnerabilities are the root cause of computer security problem. How people can quickly discover vulnerabilities existing in a certain software has always been the focus of information security field. This paper has done research on software ...
Fuzzing vulnerability discovery techniques: Survey, challenges and future directions
Abstract
Fuzzing is a powerful tool for vulnerability discovery in software, with much progress being made in the field in recent years. There is limited literature available on the fuzzing vulnerability discovery approaches. Hence, in this ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

ASE '20: Proceedings of the 35th IEEE/ACM International Conference on Automated Software Engineering

September 2020

195 pages

ISBN:9781450381284

DOI:10.1145/3417113

General Chair:
John Grundy,
Program Chairs:
Claire Le Goues,
David Lo

Copyright © 2020 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

In-Cooperation

IEEE CS

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 22 January 2021

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

Cybersecurity Education, Research and Outreach Center (CEROC) at Tennessee Tech. University

Conference

ASE '20

Sponsor:

ASE '20: 35th IEEE/ACM International Conference on Automated Software Engineering

September 21 - 25, 2020

Virtual Event, Australia

Acceptance Rates

Overall Acceptance Rate 82 of 337 submissions, 24%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

3
Total Citations
View Citations
165
Total Downloads

Downloads (Last 12 months)31
Downloads (Last 6 weeks)3

Reflects downloads up to 03 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Happe ACito JChandra SBlincoe KTonella P(2023)Understanding Hackers’ Work: An Empirical Study of Offensive Security PractitionersProceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering10.1145/3611643.3613900(1669-1680)Online publication date: 30-Nov-2023
https://dl.acm.org/doi/10.1145/3611643.3613900
Bhuiyan FMurphy JMorrison PRahman A(2021)Practitioner Perception of Vulnerability Discovery Strategies2021 IEEE/ACM 2nd International Workshop on Engineering and Cybersecurity of Critical Systems (EnCyCriS)10.1109/EnCyCriS52570.2021.00014(41-44)Online publication date: Jun-2021
https://doi.org/10.1109/EnCyCriS52570.2021.00014
Cottrell KBose DShahriar HRahman A(2021)An Empirical Study of Vulnerabilities in Robotics2021 IEEE 45th Annual Computers, Software, and Applications Conference (COMPSAC)10.1109/COMPSAC51774.2021.00105(735-744)Online publication date: Jul-2021
https://doi.org/10.1109/COMPSAC51774.2021.00105

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten