Skip to main content
SpringerLink
Log in

Are vulnerabilities discovered and resolved like other defects?

  • Published:
Empirical Software Engineering Aims and scope Submit manuscript

Abstract

Software defect data has long been used to drive software development process improvement. If security defects (vulnerabilities) are discovered and resolved by different software development practices than non-security defects, the knowledge of that distinction could be applied to drive process improvement. The goal of this research is to support technical leaders in making security-specific software development process improvements by analyzing the differences between the discovery and resolution of defects versus that of vulnerabilities. We extend Orthogonal Defect Classification (ODC), a scheme for classifying software defects to support software development process improvement, to study process-related differences between vulnerabilities and defects, creating ODC + Vulnerabilities (ODC + V). We applied ODC + V to classify 583 vulnerabilities and 583 defects across 133 releases of three open-source projects (Firefox, phpMyAdmin, and Chrome). Compared with defects, vulnerabilities are found later in the development cycle and are more likely to be resolved through changes to conditional logic. In Firefox, vulnerabilities are resolved 33% more quickly than defects. From a process improvement perspective, these results indicate opportunities may exist for more efficient vulnerability detection and resolution.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

Similar content being viewed by others

Notes

  1. https://www.wired.com/story/ether-cryptocurrency-theft/

  2. https://www.mozilla.org/en-US/firefox

  3. http://www.phpmyadmin.net/

  4. https://www.google.com/chrome

  5. https://cwe.mitre.org

  6. http://www.nvd.com

  7. https://sites.google.com/a/ncsu.edu/odc-v/

  8. https://bugzilla.mozilla.org/show_bug.cgi?id=394610

  9. https://bugzilla.mozilla.org/show_bug.cgi?id=545080

  10. https://www.mozilla.org/security/announce/2011/mfsa2011-10.html

  11. https://bugzilla.mozilla.org/show_bug.cgi?id=643051

  12. https://bugzilla.mozilla.org/show_bug.cgi?id=643927

  13. http://hg.mozilla.org/mozilla-central/rev/1ecbcf5cf362

  14. https://bugzilla.mozilla.org/show_bug.cgi?id=547608

  15. http://hg.mozilla.org/releases/mozilla-1.9.2/rev/1878344aded9

  16. http://www.phpmyadmin.net/home_page/security/PMASA-2007-3.php

  17. http://hg.mozilla.org/releases/mozilla-1.9.2/rev/6ccaf1dd6553

  18. http://www.phpmyadmin.net/home_page/security/PMASA-2007-3.php

  19. http://www.bugzilla.org/

  20. http://www.mozilla.org/security/announce/

  21. https://developer.mozilla.org/en-US/docs/Mozilla/Developer_guide/Development_process_overview

  22. https://wiki.mozilla.org/images/e/ed/Analyst_report_Q1_2010.eps

  23. https://www.openhub.net/p/firefox/factoids#FactoidTeamSizeVeryLarge

  24. http://www.phpmyadmin.net/home_page/docs.php

  25. https://www.phpmyadmin.net/contribute/

  26. https://www.phpmyadmin.net/15-years/

  27. https://bugs.chromium.org/p/chromium/issues/list

  28. https://googleprojectzero.blogspot.com/2015/02/feedback-and-data-driven-updates-to.html

  29. https://www.netmarketshare.com/browser-market-share.aspx

  30. https://www.openhub.net/p/chrome/factoids#FactoidTeamSizeVeryLarge

  31. We use a significance level of .05 for applicable statistical tests.

  32. https://cran.r-project.org/web/packages/coin/index.html

  33. For example, we ran out of memory on an 8G laptop while attempting to compute Fisher’s exact test for Activity by Security Impact for Firefox

  34. Computed using assocstats from R package vsd.

  35. When the Cramer’s V column is blank, a computed value was not available.

  36. https://www.facebook.com/realsearchNCSU

References

  • Agresti A (2007) An introduction to categorical data analysis, vol 135. Wiley, New York

    Book  Google Scholar 

  • Alhazmi O H, Malaiya Y K (2005) Modeling the vulnerability discovery process. In: 16th IEEE international symposium on software reliability engineering, 2005. ISSRE 2005. IEEE, p 10

  • Anbalagan P (2011) A study of software security problem disclosure, correction and patching processes. PhD thesis, North Carolina State University

  • Basili V R, Rombach H D (1987) Tailoring the software process to project goals and environments. In: Proceedings of ICSE. IEEE, pp 345–357

  • Bhandari I, Halliday M J, Chaar J, Chillarege R, Jones K, Atkinson J, Lepori-Costello C, Jasper P, Tarver E, Lewis C et al (1994) In-process improvement through defect data interpretation. IBM Syst J 33(1):182–214

    Article  Google Scholar 

  • Boehm B (1981) Software engineering economics. Prentice Hall PTR, Upper Saddle River

    MATH  Google Scholar 

  • Bridge N, Miller C (1998) Orthogonal defect classification using defect data to improve software development. Softw Qual 3(1):1–8

    Google Scholar 

  • Butcher M, Munro H, Kratschmer T (2002) Improving software testing via odc: three case studies. IBM Syst J 41(1):31–44

    Article  Google Scholar 

  • Camilo F, Meneely A, Nagappan M (2015) Do bugs foreshadow vulnerabilities?: A study of the chromium project. In: Proceedings of the 12th working conference on mining software repositories, MSR ’15. IEEE Press, Piscataway, pp 269–279

  • Chillarege R (2006) ODC-a 10x for root cause analysis. Available online at: http://www.chillarege.com/articles/odc-10x-root-cause-analysis.html

  • Chillarege R, Bhandari I S, Chaar J K, Halliday M J, Moebus D S, Ray B K, Wong M-Y (1992) Orthogonal defect classification-a concept for in-process measurements. IEEE Trans Softw Eng 18(11):943–956

    Article  Google Scholar 

  • Chowdhury I, Zulkernine M (2011) Using complexity, coupling, and cohesion metrics as early indicators of vulnerabilities. J Syst Archit 57(3):294–313

    Article  Google Scholar 

  • Cochran W G (1954) Some methods for strengthening the common chi-squared tests. Biometrics 10(4):417–451

    Article  MathSciNet  Google Scholar 

  • Deming W E (1986) Out of the crisis. MIT Press, Cambridge

    Google Scholar 

  • Gegick M, Williams L, Osborne J, Vouk M (2008) Prioritizing software security fortification throughcode-level metrics. In: Proceedings of the 4th ACM workshop on quality of protection, QoP ’08. ACM. New York, pp 31–38

  • Howard M, Lipner S (2009) The security development lifecycle. O’Reilly Media, Incorporated, New York

    Google Scholar 

  • Hunny U, Zulkernine M, Weldemariam K (2013) Osdc: adapting odc for developing more secure software. In: Proceedings of the 28th SAC. ACM, pp 1131–1136

  • Landis J R, Koch G G (1977) The measurement of observer agreement for categorical data. Biometrics 33(1):159–174

    Article  Google Scholar 

  • Massacci F, Nguyen V H (2014) An empirical methodology to evaluate vulnerability discovery models. IEEE Trans Softw Eng 40(12):1147–1162

    Article  Google Scholar 

  • Massacci F, Neuhaus S, Nguyen V H (2011) After-life vulnerabilities: a study on firefox evolution, its vulnerabilities, and fixes. In: Engineering secure software and systems, pp 195–208. Springer, Berlin

    Google Scholar 

  • Mays R, Jones C, Holloway G, Studinski D (1990) Experiences with defect prevention. IBM Syst J 29(1):4–32

    Article  Google Scholar 

  • McGraw G (2006) Software security: building security in, volume 1. Addison-Wesley Professional

  • Neuhaus S, Zimmermann T, Holler C, Zeller A (2007) Predicting vulnerable software components. In: Proceedings of the 14th ACM conference on computer and communications security, CCS ’07. ACM, New York, pp 529–540

  • Nguyen V H, Massacci F (2013) The (un)reliability of nvd vulnerable versions data: an empirical experiment on google chrome vulnerabilities. In: Proceedings of the 8th ACM SIGSAC symposium on information, computer and communications security, ASIA CCS ’13. ACM, New York, pp 493–498

  • Ott L (1988) An introduction to statistical methods and data analysis. Duxbury Press

  • Ozment J A (2007) Vulnerability discovery & software security. PhD thesis, Citeseer

  • Paulk M C, Weber C V, Curtis B, Chrissis M B (1995) The capability maturity model: guidelines for improving the software process. Addison-Wesley, Reading

    Google Scholar 

  • Ray B, Posnett D, Filkov V, Devanbu P (2014) A large scale study of programming languages and code quality in github. In: Proceedings of the 22Nd ACM SIGSOFT international symposium on foundations of software engineering, FSE 2014. ACM, New York, pp 155–165

  • Riaz M, King J, Slankas J, Williams L (2014) Hidden in plain sight: automatically identifying security requirements from natural language artifacts. In: Proceedings of the 22nd RE. IEEE, pp 183–192

  • Robinson B, Francis P, Ekdahl F (2008) A defect-driven process for software quality improvement. In: Proceedings of the 2nd ESEM. ACM, pp 333–335

  • Shewhart W (1930) Economic quality control of manufactured product. Bell Syst Tech J 9(2):364–389

    Article  Google Scholar 

  • Shin Y, Meneely A, Williams L, Osborne J A (2011) Evaluating complexity, code churn, and developer activity metrics as indicators of software vulnerabilities. IEEE Trans Softw Eng 37(6):772–787

    Article  Google Scholar 

  • Shostack A (2014) Threat modeling: designing for security. Wiley, New York

    Google Scholar 

  • Souza R, Silva B (2017) Sentiment analysis of travis ci builds. In: Proceedings of the 14th international conference on mining software repositories, MSR ’17. IEEE Press, Piscataway, pp 459–462

  • Syed-Mohamad S M, McBride T (2008) A comparison of the reliability growth of open source and in-house software. In: Proceedings of the 15th APSEC. IEEE, pp 229–236

  • Theisen C, Herzig K, Morrison P, Murphy B, Williams L A (2015) Approximating attack surfaces with stack traces. In: 37th IEEE/ACM international conference on software engineering, ICSE 2015, Florence, Italy, May 16–24, vol 2. IEEE, pp 199–208

  • Walden J, Stuckman J, Scandariato R (2014) Predicting vulnerable components: software metrics vs text mining. In: 2014 IEEE 25th international symposium on software reliability engineering, pp 23–33

  • Zaman S, Adams B, Hassan A E (2011) Security versus performance bugs: a case study on firefox. In: Proceedings of the 8th working conference on mining software repositories, MSR ’11. ACM, New York, pp 93–102

  • Zheng J, Williams L, Nagappan N, Snipes W, Hudepohl J P, Vouk M A (2006) On the value of static analysis for fault detection in software. IEEE Trans Softw Eng 32(4):240–253

    Article  Google Scholar 

Download references

Acknowledgments

This work is supported, in part, by IBM and by the USA National Security Agency (NSA) Science of Security Lablet at NCSU. Any opinions expressed in this report are those of the author(s) and do not necessarily reflect the views of IBM or the NSA. We thank Marc Delisle of the phpMyadmin for providing us with the snapshot of defect repostitories for this study, and for kindly answering many questions and offering his perspective. We also thank Dr. Alyson Wilson for providing helpful feedback on designing the classification assignments for the raters. We are grateful to Dr. Andy Meneely for providing the Chrome database snapshot, and to Dr. Fabio Massacci and the University of Trento for granting access to their curated Chrome vulnerability list. Finally, we thank the RealSearchFootnote 36 research group for providing helpful feedback on this work.

Author information

Authors and Affiliations

  1. Department of Computer Science, North Carolina State University, Raleigh, NC, USA

    Patrick J. Morrison, Rahul Pandita & Laurie Williams

  2. Phase Change Software LLC, Denver, CO, USA

    Xusheng Xiao

  3. Chillarege Inc., Raleigh, NC, USA

    Ram Chillarege

Authors
  1. Patrick J. Morrison
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Rahul Pandita
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Xusheng Xiao
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Ram Chillarege
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Laurie Williams
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Patrick J. Morrison.

Additional information

Communicated by: Mark Grechanik

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Morrison, P.J., Pandita, R., Xiao, X. et al. Are vulnerabilities discovered and resolved like other defects?. Empir Software Eng 23, 1383–1421 (2018). https://doi.org/10.1007/s10664-017-9541-1

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10664-017-9541-1

Keywords

Search

Navigation