Skip to main content
Springer Nature Link
Log in

Insights from Running 24 Static Analysis Tools on Open Source Software Repositories

  • Conference paper
  • First Online:
Information Systems Security (ICISS 2024)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 15416))

Included in the following conference series:

  • 384 Accesses

Abstract

OSS is important and useful. We want to ensure that it is of high quality and has no security issues. Static analysis tools provide easy-to-use and application-independent mechanisms to assess various aspects of a given code. Many effective open-source static analysis tools exist. In this paper, we perform the first comprehensive analysis using 24 open-source static analysis tools (through Omega Analyzer) on 4,947 repositories. Our study identified several interesting findings, such as the distribution of errors in relation to the criticality score of repositories shows that repositories with a criticality score have the highest percentage of errors. We envision that our findings provide insights into the effectiveness of static analysis tools on OSS and future research directions in securing OSS repositories.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. Alpha Omega – Linux Foundation Project. https://alpha-omega.dev/

  2. CodeQL. https://codeql.github.com/

  3. Open Source Security Foundation – Linux Foundation Projects. https://openssf.org/

  4. Understanding GitHub Actions. https://docs.github.com/_next/data/0DKyBPMqZhPYD1Lsg3qKt/en/free-pro-team@latest/actions/learn-github-actions/understanding-github-actions.json?versionId=free-pro-team%40latest&productId=actions &restPage=learn-github-actions &restPage=understanding-github-actions

  5. Node.js (2023). https://github.com/nodejs/node. Original-date 26 Nov 2014

  6. SwiftSyntax (2023). https://github.com/apple/swift-syntax. Original-date 31 July 2018

  7. Fact Sheet: Biden-Harris Administration Releases End of Year Report on Open-Source Software Security Initiative - ONCD (2024). https://www.whitehouse.gov/oncd/briefing-room/2024/01/30/fact-sheet-biden-harris-administration-releases-end-of-year-report-on-open-source-software-security-initiative/

  8. nodejs/node (2024). https://github.com/nodejs/node. Original-date 26 Nov 2014

  9. Abraham, A.: Nodejsscan (2023). https://github.com/ajinabraham/NodeJsScan. Accessed 18 May 2024

  10. Al Kassar, F., Clerici, G., Compagna, L., Balzarotti, D., Yamaguchi, F.: Testability tarpits: the impact of code patterns on the security testing of web applications. In: NDSS Symposium 2022. Internet Society, San Diego (2022)

    Google Scholar 

  11. Aloraini, B., Nagappan, M., German, D.M., Hayashi, S., Higo, Y.: An empirical study of security warnings from static application security testing tools. J. Syst. Softw. 158, 110427 (2019)

    Article  Google Scholar 

  12. Alvarez, S.: Radare2 (2006). https://www.radare.org/. Accessed 18 May 2024

  13. Alvarez, V.: Yara (2024). https://virustotal.github.io/yara/. Accessed 18 May 2024

  14. Arusoaie, A., Ciobâca, S., Craciun, V., Gavrilut, D., Lucanu, D.: A comparison of open-source static analysis tools for vulnerability detection in C/C++ code. In: 2017 19th International Symposium on Symbolic and Numeric Algorithms for Scientific Computing (SYNASC), pp. 161–168 (2017). https://doi.org/10.1109/SYNASC.2017.00035

  15. Arya, A., Brown, C., Pike, R., The open source security foundation: open source project criticality score (2023). https://github.com/ossf/criticality_score. Original-date 17 Nov 2020

  16. Bakhshandeh, A., Keramatfar, A., Norouzi, A., Chekidehkhoun, M.M.: Using ChatGPT as a static application security testing tool. arXiv preprint arXiv:2308.14434 (2023)

  17. Bhandari, G., Naseer, A., Moonen, L.: CVEfixes: automated collection of vulnerabilities and their fixes from open-source software. In: Proceedings of the 17th International Conference on Predictive Models and Data Analytics in Software Engineering, pp. 30–39. Association for Computing Machinery, Athens, Greece (2021)

    Google Scholar 

  18. Bonaccorsi, A., Rossi, C.: Why open source software can succeed. Res. Policy 32(7), 1243–1258 (2003)

    Article  Google Scholar 

  19. Boyter, B.: SCC (2018). https://github.com/boyter/scc. Accessed 18 May 2024

  20. Casanova Páez, M.M.: Application security testing tools study and proposal (2021)

    Google Scholar 

  21. Collins, J.: Brakeman (2010). https://brakemanscanner.org/. Accessed 18 May 2024

  22. Cybersecurity and Infrastructure Security Agency (CISA): Government and industry partners publish fact sheet for organizations using open source software (2023). https://www.cisa.gov/news-events/news/government-and-industry-partners-publish-fact-sheet-organizations-using-open-source-software. Accessed 13 June 2024

  23. Deepfence: Secretscanner (2020). https://github.com/deepfence/SecretScanner. Accessed 18 May 2024

  24. Dencheva, L.: Comparative analysis of Static application security testing (SAST) and Dynamic application security testing (DAST) by using open-source web application penetration testing tools. Ph.D. thesis, Dublin, National College of Ireland (2022)

    Google Scholar 

  25. Devine, T.R., Campbell, M., Anderson, M., Dzielski, D.: SREP+ SAST: a comparison of tools for reverse engineering machine code to detect cybersecurity vulnerabilities in binary executables. In: 2022 International Conference on Computational Science and Computational Intelligence (CSCI), pp. 862–869. IEEE, Las Vegas (2022)

    Google Scholar 

  26. Esposito, M., Falaschi, V., Falessi, D.: An extensive comparison of static application security testing tools (2024)

    Google Scholar 

  27. Facebook: Hermes JavaScript engine. https://github.com/facebook/hermes (2024). Accessed 30 Apr 2024

  28. Felderer, M., Büchler, M., Johns, M., Brucker, A.D., Breu, R., Pretschner, A.: Security testing: a survey. In: Advances in Computers, vol. 101, pp. 1–51. Elsevier (2016)

    Google Scholar 

  29. Gentsch, C.: Evaluation of open source static analysis security testing (SAST) tools for C (2020)

    Google Scholar 

  30. Gentsch, C., Krishnamurthy, R., Heinze, T.S.: Benchmarking open-source static analyzers for security testing for C. In: ISoLA 2020, Part IV, pp. 182–198. Springer, Cham (2021)

    Google Scholar 

  31. Ghazaly, N.M.: Learning the idea behind SAST (static application security testing) and how it functions. Int. J. Manag. Eng. Res. 1(1), 01–04 (2021)

    Google Scholar 

  32. GitHub: CodeQL (2019). https://securitylab.github.com/tools/codeql. Accessed 18 May 2024

  33. GitHub: Potential use after free (2024). https://codeql.github.com/codeql-query-help/cpp/cpp-use-after-free/. Accessed 24 June 2024

  34. Gkortzis, A., Mitropoulos, D., Spinellis, D.: VulinOSS: a dataset of security vulnerabilities in open-source systems. In: Proceedings of the 15th International conference on mining software repositories, pp. 18–21. ACM, Gothenburg (2018)

    Google Scholar 

  35. Goseva-Popstojanova, K., Perhinschi, A.: On the capability of static code analysis to detect security vulnerabilities. Inf. Softw. Technol. 68, 18–33 (2015)

    Article  Google Scholar 

  36. Hauge, Ø., Ayala, C., Conradi, R.: Adoption of open source software in software-intensive organizations-a systematic literature review. Inf. Softw. Technol. 52(11), 1133–1154 (2010)

    Article  Google Scholar 

  37. Heffner, C.: binwalk (2010). https://github.com/ReFirmLabs/binwalk. Accessed 18 May 2024

  38. ICSharpCode: Ilspy (2011). https://github.com/icsharpcode/ILSpy. Accessed 18 May 2024

  39. Imtiaz, N., Khanom, A., Williams, L.: Open or sneaky? Fast or slow? light or heavy?: investigating security releases of open source packages. IEEE Trans. Software Eng. 49(4), 1540–1560 (2022)

    Article  Google Scholar 

  40. Jiang, Y., Jeusfeld, M., Ding, J.: Evaluating the data inconsistency of open-source vulnerability repositories. In: Proceedings of the 16th International Conference on Availability, Reliability and Security, pp. 1–10. ACM, Vienna (2021)

    Google Scholar 

  41. JusticeRage: Manalyze (2010). https://github.com/JusticeRage/Manalyze. Accessed 18 May 2024

  42. Konves, S.: TBV (2019). https://github.com/verifynpm/tbv. Accessed 18 May 2024

  43. Levin, D.V.: Strace (1992). https://strace.io/. Accessed 18 May 2024

  44. Li, K., et al.: Comparison and evaluation on static application security testing (SAST) tools for java. In: Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, pp. 921–933. ACM, San Francisco (2023)

    Google Scholar 

  45. Marjamäki, D.: Cppcheck (2007). http://cppcheck.sourceforge.net/. Accessed 18 May 2024

  46. Mateo Tudela, F., Bermejo Higuera, J.R., Bermejo Higuera, J., Sicilia Montalvo, J.A., Argyros, M.I.: On combining static, dynamic and interactive analysis security testing tools to improve OWASP top ten security vulnerability detection in web applications. Appl. Sci. 10(24), 9119 (2020)

    Article  Google Scholar 

  47. Microsoft: Devskim (2017). https://github.com/microsoft/DevSkim. Accessed 18 May 2024

  48. Microsoft: ApplicationInspector (2019). https://github.com/microsoft/ApplicationInspector. Accessed 18 May 2024

  49. Microsoft: OSSGadget (2020). https://github.com/microsoft/OSSGadget. Accessed 18 May 2024

  50. Microsoft: Workflow configuration for similar issues in Microsoft terminal (2024). https://github.com/microsoft/terminal/blob/main/.github/workflows/similarIssues.yml. Accessed 30 Apr 2024

  51. Nguyen-Duc, A., Do, M.V., Hong, Q.L., Khac, K.N., Quang, A.N.: On the adoption of static analysis for software security assessment-a case study of an open-source e-government project. Comput. Secur. 111, 102470 (2021)

    Article  Google Scholar 

  52. npm, I.: npm audit (2018). https://docs.npmjs.com/cli/v7/commands/npm-audit. Accessed 27 May 2024

  53. Onarcan, M.O., Fu, Y., et al.: A case study on design patterns and software defects in open source software. J. Softw. Eng. Appl. 11(05), 249 (2018)

    Article  Google Scholar 

  54. Oyetoyan, T.D., Milosheska, B., Grini, M., Soares Cruzes, D.: Myths and facts about static application security testing tools: an action research at Telenor digital. In: XP 2018, pp. 86–103. Springer, Cham (2018)

    Google Scholar 

  55. Piantadosi, V., Scalabrino, S., Oliveto, R.: Fixing of security vulnerabilities in open source projects: a case study of apache http server and apache tomcat. In: 2019 12th IEEE Conference on Software Testing, Validation and Verification (ICST), pp. 68–78. IEEE, Xi’an (2019)

    Google Scholar 

  56. PlatformIO: Project helpers for platformio (2024). https://github.com/platformio/platformio-core/blob/develop/platformio/project/helpers.py. Accessed 30 Apr 2024

  57. Ponta, S.E., Plate, H., Sabetta, A., Bezzi, M., Dangremont, C.: A manually-curated dataset of fixes to vulnerabilities of open-source software. In: 2019 IEEE/ACM 16th International Conference on Mining Software Repositories (MSR), pp. 383–387. IEEE, Montreal (2019)

    Google Scholar 

  58. Price, P.: Shhgit (2018). https://github.com/eth0izzle/shhgit. Accessed 18 May 2024

  59. PyCQA: Bandit (2013). https://github.com/PyCQA/bandit. Accessed 18 May 2024

  60. R2C: Semgrep (2020). https://semgrep.dev/. Accessed 18 May 2024

  61. Raducu, R., Esteban, G., Rodriguez Lera, F.J., Fernández, C.: Collecting vulnerable source code from open-source repositories for dataset generation. Appl. Sci. 10(4), 1270 (2020)

    Article  Google Scholar 

  62. Ramsauer, R., Bulwahn, L., Lohmann, D., Mauerer, W.: The sound of silence: mining security vulnerabilities from secret integration channels in open-source projects. In: Proceedings of the 2020 ACM SIGSAC Conference on Cloud Computing Security Workshop, pp. 147–157. ACM, Virtual Event (2020)

    Google Scholar 

  63. SageMath: SageMath mathematical software system (2024). https://github.com/sagemath/sage. Accessed 30 Apr 2024

  64. Shen, M., Pillai, A., Yuan, B.A., Davis, J.C., Machiry, A.: An empirical study on the use of static analysis tools in open source embedded software. arXiv preprint arXiv:2310.00205, pp. 1–14 (2023)

  65. Smith, L.J.: Estimating security risk in open source package repositories: an empirical analysis and predictive model of software vulnerabilities. Ph.D. thesis, Capella University (2019)

    Google Scholar 

  66. Snyk: Snyk code (2020). https://snyk.io/product/snyk-code/. Accessed 27 May 2024

  67. Talos, C.: ClamAV (2024). https://www.clamav.net/. Accessed 18 May 2024

  68. Torvalds, L.: torvalds/linux (2023). https://github.com/torvalds/linux. Original-date 04 Sept 2011

  69. Torvalds, L.: torvalds/linux (2024). https://github.com/torvalds/linux. Original-date 94 Sept 2011

  70. Ven, K., Verelst, J., Mannaert, H.: Should you adopt open source software? IEEE Softw. 25(3), 54–59 (2008)

    Article  Google Scholar 

  71. wireghoul: Graudit (2010). https://github.com/wireghoul/graudit. Accessed 18 May 2024

  72. Xu, R., Tang, Z., Ye, G., Wang, H., Ke, X., Fang, D., Wang, Z.: Detecting code vulnerabilities by learning from large-scale open source repositories. J. Inf. Secur. Appl. 69, 103293 (2022)

    Google Scholar 

  73. Yelp: Detectsecrets (2017). https://github.com/Yelp/detect-secrets. Accessed 18 May 2024

  74. Yin, T.: Lizard (2014). https://github.com/terryyin/lizard. Accessed 18 May 2024

  75. Yoast: Components directory of the yoast seo wordpress plugin (2024). https://github.com/Yoast/wordpress-seo/tree/trunk/apps/components. Accessed 30 Apr 2024

  76. Zahedi, M., Ali Babar, M., Treude, C.: An empirical study of security issues posted in open source projects. In: Proceedings of the 51st Hawaii International Conference on System Sciences (HICSS), pp. 5504–5513. IEEE, Hawaii (2018)

    Google Scholar 

Download references

Acknowledgements

This research was partly supported by the National Science Foundation (NSF) under Grant CNS-2340548. Any opinions, findings, conclusions, or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the NSF.

Author information

Authors and Affiliations

  1. Purdue University, West Lafayette, IN, USA

    Fabiha Hashmat, Zeyad Alwaleed Aljaali, Mingjie Shen & Aravind Machiry

Authors
  1. Fabiha Hashmat
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Zeyad Alwaleed Aljaali
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Mingjie Shen
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Aravind Machiry
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Fabiha Hashmat .

Editor information

Editors and Affiliations

  1. Indian Institute of Technology Bombay, Mumbai, Maharashtra, India

    Vishwas T. Patil

  2. The University of Texas at San Antonio, San Antonio, TX, USA

    Ram Krishnan

  3. Indian Institute of Technology Bombay, Mumbai, Maharashtra, India

    Rudrapatna K. Shyamasundar

Rights and permissions

Reprints and permissions

Copyright information

© 2025 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Hashmat, F., Aljaali, Z.A., Shen, M., Machiry, A. (2025). Insights from Running 24 Static Analysis Tools on Open Source Software Repositories. In: Patil, V.T., Krishnan, R., Shyamasundar, R.K. (eds) Information Systems Security. ICISS 2024. Lecture Notes in Computer Science, vol 15416. Springer, Cham. https://doi.org/10.1007/978-3-031-80020-7_13

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-80020-7_13

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-80019-1

  • Online ISBN: 978-3-031-80020-7

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics