Skip to main content
SpringerLink
Log in

Game of detections: how are security vulnerabilities discovered in the wild?

  • Published:
Empirical Software Engineering Aims and scope Submit manuscript

Abstract

There is little or no information available on what actually happens when a software vulnerability is detected. We performed an empirical study on reporters of the three most prominent security vulnerabilities: buffer overflow, SQL injection, and cross site scripting vulnerabilities. The goal was to understand the methods and tools used during the discovery and whether the community of developers exploring one security vulnerability differs—in their approach—from another community of developers exploring a different vulnerability. The reporters were featured in the SecurityFocus repository for twelve month periods for each vulnerability. We collected 127 responses. We found that the communities differ based on the security vulnerability they target; but within a specific community, reporters follow similar approaches. We also found a serious problem in the vulnerability reporting process that is common for all communities. Most reporters, especially the experienced ones, favor full-disclosure and do not collaborate with the vendors of vulnerable software. They think that the public disclosure, sometimes supported by a detailed exploit, will put pressure on vendors to fix the vulnerabilities. But, in practice, the vulnerabilities not reported to vendors are less likely to be fixed. Ours is the first study on vulnerability repositories that targets the reporters of the most common security vulnerabilities, thus concentrating on the people involved in the process; previous works have overlooked this rich information source. The results are valuable for beginners exploring how to detect and report security vulnerabilities and for tool vendors and researchers exploring how to automate and fix the process.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18
Fig. 19
Fig. 20
Fig. 21
Fig. 22
Fig. 23

Similar content being viewed by others

Notes

  1. Krippendorff’s alpha coefficient (Krippendorff 2004) is a statistical measure of the inter-rater agreement when coding a set of units of analysis. We applied this measurement to the codes assigned by the two coders. An alpha value close to 1.0 means good inter-rater agreement.

  2. A code review is an examination of source code that can be done informally or in a formal, systematic setting.

  3. From Jan 2013, Mona is also available with WinDbg.

  4. Common Vulnerability Scoring System (CVSS) is an open industry standard for assessing the severity of computer system security vulnerabilities. There are several aspects that make up the CVSS score, but the base subscore, the exploitability subscore, and the impact subscore are the most common. The scores range from 0 to 10. A vulnerability is considered to have “high severity” if it has a base subscore of 7.0–10.0.

References

  • Alhazmi O, Malaiya Y (2006) Prediction capabilities of vulnerability discovery models. In: RAMS’06. IEEE Computer Society

  • Anbalagan P, Vouk M (2009) Towards a unifying approach in understanding security problems. In: ISSRE’09. IEEE Press

  • Aranda J, Venolia G (2009) The secret life of bugs: Going past the errors and omissions in software repositories. In: ICSE ’09. IEEE Computer Society

  • Arbaugh W, Fithen W, McHugh J (2000) Windows of vulnerability: A case study analysis. Computer 33(12):52–59

    Article  Google Scholar 

  • Arora A, Krishnan R, Telang R, Yang Y (2004) Impact of vulnerability disclosure and patch availability - An empirical analysis. In: WEIS ’04

  • Austin A, Williams L (2011) One technique is not enough: A comparison of vulnerability discovery techniques. In: ESEM ’11

  • Baca D, Carlsson B, Petersen K, Lundberg L (2013) Improving software security with static automated code analysis in an industry setting. Software—Practice and Experience 43(3):259–279

    Article  Google Scholar 

  • Bessey A, Block K, Chelf B, Chou A, Fulton B, Hallem S, Henri-Gros C, Kamsky A, McPeak S, Engler D (2010) A few billion lines of code later: Using static analysis to find bugs in the real world. Commun ACM 53(2):66–75

    Article  Google Scholar 

  • Bosu A, Carver JC, Hafiz M, Hilley P, Janni D (2014) Identifying the characteristics of vulnerable code changes: An empirical study. In: 22nd ACM SIGSOFT international symposium on the foundations of software engineering, p To appear

  • Browne H, Arbaugh W, McHugh J, Fithen W (2001) A trend analysis of exploitations. In: IEEE S&P ’01. IEEE Computer Society

  • Cheswick B (1992) An evening with berferd in which a cracker is lured, endured, and studied. In: Proc. Winter USENIX Conference, pp 163–174

  • Cova M, Leita C, Thonnard O, Keromytis A, Dacier M (2010) An analysis of rogue AV campaigns. In: Proceedings of the 13th international conference on Recent advances in intrusion detection, RAID’10. Springer-Verlag, Berlin, Heidelberg, pp 442–463

  • Denzin N (1978) The Research Act: A Theoretical Introduction to Sociological Methods. McGraw-Hill, New York

  • Doupé A, Cova M, Vigna G (2010) Why Johnny can’t Pentest: An analysis of black-box web vulnerability scanners. In: DIMVA ’10. Springer

  • Edmundson A, Holtkamp B, Rivera E, Finifter M, Mettler A, Wagner D (2013) An empirical study on the effectiveness of security code review. In: ESSoS ’13, Lecture Notes in Computer Science, vol 7781, pp 197–212. Springer Berlin Heidelberg

  • Fang M, Hafiz M (2014) Discovering buffer overflow vulnerabilities in the wild: an empirical study. In: Morisio M, Dybå T, Torchiano M (eds) 2014 ACM-IEEE International Symposium on Empirical Software Engineering and Measurement, ESEM ’14, Torino, Italy, September 18–19, 2014, p 23. ACM

  • Finifter M, Akhawe D, Wagner D (2013) An empirical study of vulnerability rewards programs. In: USENIX Security’ 13. USENIX Association

  • Franklin J, Perrig A, Paxson V, Savage S (2007) An inquiry into the nature and causes of the wealth of internet miscreants. In: Ning P, di Vimercati SDC, Syverson PF (eds) Proceedings of the 2007 ACM Conference on Computer and Communications Security, CCS 2007, pp 375–388. ACM

  • Frei S, Schatzmann D, Plattner B, Trammell B (2009) Modelling the security ecosystem- The dynamics of (in)security. In: WEIS ’09

  • Frei S, Tellenbach B, Plattner B (2008) 0-day patch - Exposing vendors’ (In)security performance. BlackHat Europe

  • Gopalakrishna R, Spafford E (2005) A trend analysis of vulnerabilities. Tech. rep., CERIAS

  • Johnson B, Song Y, Murphy-Hill E, Bowdidge R (2013) Why don’t software developers use static analysis tools to find bugs?. In: ICSE ’13. ACM

  • Kanich C, Kreibich C, Levchenko K, Enright B, Voelker GM, Paxson V, Savage S (2008) Spamalytics: An empirical analysis of spam marketing conversion. In: Proceedings of the 15th ACM conference on computer and communications security, CCS ’08. ACM, New York, NY, USA, pp 3–14

  • Krippendorff K (2004) Content Analysis: An Introduction to Its Methodology. Sage Publications Ltd., Singapore

  • Layman L, Williams L, Amant R (2007) Toward reducing fault fix time: Understanding developer behavior for the design of automated fault detection tools. In: ESEM ’07. IEEE Computer Society

  • Massacci F, Nguyen V (2010) Which is the right source for vulnerability studies?: An empirical analysis on mozilla firefox. In: MetriSec ’10. ACM

  • McGraw G, Steven J (2011) Software [In]security: Comparing apples, oranges, and aardvarks (or, all static analysis tools are not created equal). http://www.informit.com/articles/article.aspx?p=1680863

  • McQueen M, McQueen T, Boyer W, Chaffin M (2009) Empirical estimates and observations of 0-day vulnerabilities. In: HICSS ’09, pp 1 –12

  • Meiklejohn S, Pomarole M, Jordan G, Levchenko K, McCoy D, Voelker GM, Savage S (2013) A fistful of bitcoins: Characterizing payments among men with no names. In: Proceedings of the 2013 conference on internet measurement conference, IMC ’13. ACM, New York, NY, USA, pp 127–140

  • Mell P, Scarfone K, Romanosky S (2007) CVSS: A complete guide to the Common Vulnerability Scoring System Version 2.0. Tech. rep., FIRST.org

  • Okhravi H, Nicol D (2008) Evaluation of patch management strategies. Int J Comput Intell Theory Pract 3:109–117

    Google Scholar 

  • Open Web Application Security Project (OWASP) (2014) Owasp top ten 2013 project. https://www.owasp.org/index.php/Top_10_2013-Table_of_Contents

  • Patton M (2001) Qualitative Research & Evaluation Methods, 3 edn. Sage Publications Ltd., Singapore

  • Rutar N, Almazan C, Foster J (2004) A comparison of bug finding tools for Java. In: ISSRE ’04. IEEE Computer Society

  • Saldana J (2009) The Coding Manual for Qualitative Researchers. Sage Publications Ltd, Singapore

  • Scandariato R, Walden J, Joosen W (2013) Static analysis versus penetration testing: A controlled experiment. In: 2013 IEEE 24th international symposium on software reliability engineering (ISSRE), pp 451–460

  • Schneier B (2000) Full disclosure and the window of exposure. Crypto-Gram Newsletter

  • Scholte T, Balzarotti D, Kirda E (2012) Quo vadis? A study of the evolution of input validation vulnerabilities in web applications. In: FC’11. Springer-Verlag

  • Schryen G (2009) A comprehensive and comparative analysis of the patching behavior of open source and closed source software vendors. In: IMF

  • SecurityFocus Bugtraq vulnerability list. http://www.securityfocus.com/

  • Shahzad M, Shafiq M, Liu A (2012) A large scale exploratory analysis of software vulnerability life cycles. In: ICSE ’12. IEEE Press

  • Suto L (2007) Analyzing the effectiveness and coverage of Web application security scanners. Tech. rep., eEye Digital Security

  • TippingPoint Zero Day Initiative (ZDI). http://www.zerodayinitiative.com/

  • Verisign iDefense security intelligence services. http://www.verisigninc.com/en_US/products-and-services/network-intelligence-availability/idefense/index.xhtml

  • Weinstein M (2012) TAMS Analyzer for Macintosh OS X: The native open source, Macintosh qualitative research tool

  • Wilander J, Kamkar M (2003) A comparison of publicly available tools for dynamic buffer overflow prevention. In: NDSS ’03. The Internet Society

  • Wu Y, Gandhi R, Siy H (2010) Using semantic templates to study vulnerabilities recorded in large software repositories. In: SESS ’10. ACM

  • Xiao S, Witschey J, Murphy-Hill E (2014) Social influences on secure development tool adoption: Why security tools spread. In: CSCW ’14. ACM, New York, NY, USA, pp 1095–1106

  • Yin R (2011) Case Study Research: Design and Methods. Sage Publications Ltd, Singapore

  • Zhang S, Caragea D, Ou X (2011) An empirical study on using the national vulnerability database to predict software vulnerabilities. In: DEXA ’11. Springer

Download references

Author information

Authors and Affiliations

  1. Department of Computer Science and Software Engineering, Auburn University, Auburn, AL, USA

    Munawar Hafiz & Ming Fang

Authors
  1. Munawar Hafiz
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ming Fang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Munawar Hafiz.

Additional information

Communicated by: Mark Grechanik

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Hafiz, M., Fang, M. Game of detections: how are security vulnerabilities discovered in the wild?. Empir Software Eng 21, 1920–1959 (2016). https://doi.org/10.1007/s10664-015-9403-7

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10664-015-9403-7

Keywords

Search

Navigation