research-article

Open access

Semgrep*: Improving the Limited Performance of Static Application Security Testing (SAST) Tools

Authors:

Gareth Bennett,

Steve CounsellAuthors Info & Claims

EASE '24: Proceedings of the 28th International Conference on Evaluation and Assessment in Software Engineering

Pages 614 - 623

https://doi.org/10.1145/3661167.3661262

Published: 18 June 2024 Publication History

All formats PDF

Abstract

Vulnerabilities in code should be detected and patched quickly to reduce the time in which they can be exploited. There are many automated approaches to assist developers in detecting vulnerabilities, most notably Static Application Security Testing (SAST) tools. However, no single tool detects all vulnerabilities and so relying on any one tool may leave vulnerabilities dormant in code. In this study, we use a manually curated dataset to evaluate four SAST tools on production code with known vulnerabilities. Our results show that the vulnerability detection rates of individual tools range from 11.2% to 26.5%, but combining these four tools can detect 38.8% of vulnerabilities. We investigate why SAST tools are unable to detect 61.2% of vulnerabilities and identify missing vulnerable code patterns from tool rule sets. Based on our findings, we create new rules for Semgrep, a popular configurable SAST tool. Our newly configured Semgrep tool detects 44.7% of vulnerabilities, more than using a combination of tools, and a 181% improvement in Semgrep’s detection rate.

References

[1]

Omar Alhazmi, Yashwant Malaiya, and Indrajit Ray. 2005. Security vulnerabilities in software systems: A quantitative perspective. In IFIP Annual Conference on Data and Applications Security and Privacy. Springer, 281–294.

Digital Library

[2]

Bushra Aloraini, Meiyappan Nagappan, Daniel M German, Shinpei Hayashi, and Yoshiki Higo. 2019. An empirical study of security warnings from static application security testing tools. Journal of Systems and Software 158 (2019), 110427.

Digital Library

[3]

Richard Amankwah, Patrick Kwaku Kudjo, and Samuel Yeboah Antwi. 2017. Evaluation of software vulnerability detection methods and tools: a review. International Journal of Computer Applications 169, 8 (2017), 22–27.

[4]

Amit Seal Ami, Kevin Moran, Denys Poshyvanyk, and Adwait Nadkarni. 2023. " False negative–that one is going to kill you": Understanding Industry Perspectives of Static Analysis based Security Testing. arXiv preprint arXiv:2307.16325 (2023).

[5]

Ross Anderson. 2002. Security in open versus closed systems—the dance of Boltzmann, Coase and Moore. (2002).

[6]

Aman Anupam, Prathika Gonchigar, Shashank Sharma, Prapulla SB, and Anala MR. 2020. Analysis of Open Source Node. js Vulnerability Scanners. (2020).

[7]

Andrei Arusoaie, Stefan Ciobâca, Vlad Craciun, Dragos Gavrilut, and Dorel Lucanu. 2017. A comparison of open-source static analysis tools for vulnerability detection in c/c++ code. In 2017 19th International Symposium on Symbolic and Numeric Algorithms for Scientific Computing (SYNASC). IEEE, 161–168.

[8]

Dejan Baca, Kai Petersen, Bengt Carlsson, and Lars Lundberg. 2009. Static code analysis to detect software security vulnerabilities-does experience matter?. In 2009 International Conference on Availability, Reliability and Security. IEEE, 804–810.

[9]

Moritz Beller, Radjino Bholanath, Shane McIntosh, and Andy Zaidman. 2016. Analyzing the state of static analysis: A large-scale evaluation in open source software. In 2016 IEEE 23rd International Conference on Software Analysis, Evolution, and Reengineering (SANER), Vol. 1. IEEE, 470–481.

[10]

Paul E Black, Damien Cupif, Guillaume Haben, Alex-Kevin Loembe, Vadim Okun, and Yann Prono. 2023. SATE VI Report. (2023).

[11]

Saikat Chakraborty, Rahul Krishna, Yangruibo Ding, and Baishakhi Ray. 2022. Deep Learning Based Vulnerability Detection: Are We There Yet?IEEE Transactions on Software Engineering 48, 9 (2022), 3280–3296. https://doi.org/10.1109/TSE.2021.3087402

[12]

Roland Croft, Dominic Newlands, Ziyu Chen, and M Ali Babar. 2021. An empirical study of rule-based and learning-based approaches for static application security testing. In Proceedings of the 15th ACM/IEEE international symposium on empirical software engineering and measurement (ESEM). 1–12.

Digital Library

[13]

cve. 1999. CVE Program Mission. https://www.cve.org/. Accessed: 2023-08-04.

[14]

cwe. 2022. 2022 CWE Top 25 Most Dangerous Software Weaknesses. https://cwe.mitre.org/top25/archive/2022/2022_cwe_top25.html. Accessed: 2023-07-19.

[15]

Aurelien Delaitre, Bertrand Stivalet, Paul Black, Vadim Okun, Terry Cohen, and Athos Ribeiro. 2018. SATE V Report: Ten Years of Static Analysis Tool Expositions. (2018-10-23 2018). https://doi.org/10.6028/NIST.SP.500-326

[16]

Sarah Elder, Nusrat Zahan, Rui Shu, Monica Metro, Valeri Kozarev, Tim Menzies, and Laurie Williams. 2022. Do I really need all this work to find vulnerabilities? An empirical case study comparing vulnerability detection techniques on a Java application. Empirical Software Engineering 27, 6 (2022), 154.

Digital Library

[17]

Christoph Gentsch. 2020. Evaluation of open source static analysis security testing (SAST) tools for c. (2020).

[18]

Brittany Johnson, Yoonki Song, Emerson Murphy-Hill, and Robert Bowdidge. 2013. Why don’t software developers use static analysis tools to find bugs?. In 2013 35th International Conference on Software Engineering (ICSE). 672–681. https://doi.org/10.1109/ICSE.2013.6606613

[19]

Kaixuan Li, Sen Chen, Lingling Fan, Ruitao Feng, Han Liu, Chengwei Liu, Yang Liu, and Yixiang Chen. 2023. Comparison and Evaluation on Static Application Security Testing (SAST) Tools for Java. (2023).

[20]

Rahma Mahmood and Qusay H Mahmoud. 2018. Evaluation of static analysis tools for finding vulnerabilities in Java and C/C++ source code. arXiv preprint arXiv:1805.09040 (2018).

[21]

Francesc Mateo Tudela, Juan-Ramon Bermejo Higuera, Javier Bermejo Higuera, Juan-Antonio Sicilia Montalvo, and Michael I Argyros. 2020. On Combining Static, Dynamic and Interactive Analysis Security Testing Tools to Improve OWASP Top Ten Security Vulnerability Detection in Web Applications. Applied Sciences 10, 24 (2020), 9119.

[22]

Gary McGraw. 2003. From the ground up: The DIMACS software security workshop. IEEE Security & Privacy 1, 2 (2003), 59–66.

Digital Library

[23]

P.J. Morrison, R. Pandita, and X. Xiao. 2018. Are vulnerabilities discovered and resolved like other defects?. In Empir Software Eng 23. 1383–1421.

[24]

Nuthan Munaiah, Felivel Camilo, Wesley Wigham, Andrew Meneely, and Meiyappan Nagappan. 2017. Do bugs foreshadow vulnerabilities? An in-depth study of the chromium project. Empirical Software Engineering 22 (2017), 1305–1347.

Digital Library

[25]

Owasp NodeGoat. 2013. OWASP NodeGoat. https://github.com/OWASP/NodeGoat. Accessed: 2023-07-19.

[26]

Paulo Nunes, Ibéria Medeiros, José Fonseca, Nuno Neves, Miguel Correia, and Marco Vieira. 2019. An empirical study on combining diverse static analysis tools for web security vulnerabilities based on development scenarios. Computing 101 (2019), 161–185.

Digital Library

[27]

owasp. 2023. Source Code Analysis Tools. https://owasp.org/www-community/Source_Code_Analysis_Tools. Accessed: 2023-07-18.

[28]

Serena Elisa Ponta, Henrik Plate, Antonino Sabetta, Michele Bezzi, and Cédric Dangremont. 2019. A manually-curated dataset of fixes to vulnerabilities of open-source software. In 2019 IEEE/ACM 16th International Conference on Mining Software Repositories (MSR). IEEE, 383–387.

Digital Library

[29]

SARD. 2000. NIST Software Assurance Reference Dataset. https://samate.nist.gov/SARD/. Accessed: 2023-07-19.

[30]

sate. 2013. Static Analysis Tool Exposition (SATE). https://www.nist.gov/itl/ssd/software-quality-group/samate/static-analysis-tool-exposition-sate. Accessed: 2023-07-18.

[31]

Carmine Vassallo, Sebastiano Panichella, Fabio Palomba, Sebastian Proksch, Harald C Gall, and Andy Zaidman. 2020. How developers engage with static analysis tools in different contexts. Empirical Software Engineering 25 (2020), 1419–1457.

Cited By

Thompson MKyer M(2025)Securing the Containerized Environment Along the CI/CD Pipeline2025 IEEE 15th Annual Computing and Communication Workshop and Conference (CCWC)10.1109/CCWC62904.2025.10903704(00250-00256)Online publication date: 6-Jan-2025
https://doi.org/10.1109/CCWC62904.2025.10903704
Bennett GHall TCounsell SWinter EShippey T(2024)Do Developers Use Static Application Security Testing (SAST) Tools Straight Out of the Box? A large-scale Empirical StudyProceedings of the 18th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement10.1145/3674805.3690750(454-460)Online publication date: 24-Oct-2024
https://dl.acm.org/doi/10.1145/3674805.3690750
Jerónimo AMoreno PCamacho JVega G(2024)Techniques of SAST Tools in the Early Stages of Secure Software Development: A Systematic Literature Review2024 IEEE International Conference on Engineering Veracruz (ICEV)10.1109/ICEV63254.2024.10766004(1-8)Online publication date: 21-Oct-2024
https://doi.org/10.1109/ICEV63254.2024.10766004

Recommendations

Do Developers Use Static Application Security Testing (SAST) Tools Straight Out of the Box? A large-scale Empirical Study
ESEM '24: Proceedings of the 18th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement

Static application Security Testing (SAST) tools are an established means of detecting vulnerabilities early in development. Previous studies have reported low detection rates from SAST tools and recommend either combining SAST tools or configuring rule ...
Static Application Security Testing (SAST) Tools for Smart Contracts: How Far Are We?

In recent years, the importance of smart contract security has been heightened by the increasing number of attacks against them. To address this issue, a multitude of static application security testing (SAST) tools have been proposed for detecting ...
Comparison and Evaluation on Static Application Security Testing (SAST) Tools for Java
ESEC/FSE 2023: Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering

Static application security testing (SAST) takes a significant role in the software development life cycle (SDLC). However, it is challenging to comprehensively evaluate the effectiveness of SAST tools to determine which is the better one for detecting ...

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences

EASE '24: Proceedings of the 28th International Conference on Evaluation and Assessment in Software Engineering

June 2024

728 pages

ISBN:9798400717017

DOI:10.1145/3661167

Copyright © 2024 Owner/Author.

This work is licensed under a Creative Commons Attribution International 4.0 License.

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 18 June 2024

Check for updates

Qualifiers

Research-article
Research
Refereed limited

Conference

EASE 2024

EASE 2024: 28th International Conference on Evaluation and Assessment in Software Engineering

June 18 - 21, 2024

Salerno, Italy

Acceptance Rates

Overall Acceptance Rate 71 of 232 submissions, 31%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

3
Total Citations
View Citations
991
Total Downloads

Downloads (Last 12 months)991
Downloads (Last 6 weeks)192

Reflects downloads up to 05 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Thompson MKyer M(2025)Securing the Containerized Environment Along the CI/CD Pipeline2025 IEEE 15th Annual Computing and Communication Workshop and Conference (CCWC)10.1109/CCWC62904.2025.10903704(00250-00256)Online publication date: 6-Jan-2025
https://doi.org/10.1109/CCWC62904.2025.10903704
Bennett GHall TCounsell SWinter EShippey T(2024)Do Developers Use Static Application Security Testing (SAST) Tools Straight Out of the Box? A large-scale Empirical StudyProceedings of the 18th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement10.1145/3674805.3690750(454-460)Online publication date: 24-Oct-2024
https://dl.acm.org/doi/10.1145/3674805.3690750
Jerónimo AMoreno PCamacho JVega G(2024)Techniques of SAST Tools in the Early Stages of Secure Software Development: A Systematic Literature Review2024 IEEE International Conference on Engineering Veracruz (ICEV)10.1109/ICEV63254.2024.10766004(1-8)Online publication date: 21-Oct-2024
https://doi.org/10.1109/ICEV63254.2024.10766004

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Figures

Tables

Media

View Table of Conten