Total Recall? How Good Are Static Call Graphs Really?
Authors: 
Dominik Helm, Sven Keidel, Anemone Kampkötter, Johannes Düsing, Tobias Roth, 
Ben Hermann, 
Mira MeziniAuthors Info & Claims
Dominik Helm, Sven Keidel, Anemone Kampkötter, Johannes Düsing, Tobias Roth, Ben Hermann, Mira MeziniAuthors Info & ClaimsPages 112 - 123
Abstract
Static call graphs are a fundamental building block of program analysis. However, differences in call-graph construction and the use of specific language features can yield unsoundness and imprecision. Call-graph analyses are evaluated using measures of precision and recall, but this is hard when a ground truth for real-world programs is generally unobtainable. In this work, we propose to use carefully constructed dynamic baselines based on fixed entry points and input corpora. The creation of this dynamic baseline is posed as an approximation of the ground truth---an optimization problem. We use manual extension and coverage-guided fuzzing for creating suitable input corpora. With these dynamic baselines, we study call-graph quality of multiple algorithms and implementations using four real-world Java programs. We find that our methodology provides valuable insights into call-graph quality and how to measure it. With this work, we provide a novel methodology to advance the field of static program analysis as we assess the computation of one of its core data structures---the call graph.
References
[1]
Karim Ali and Ondrej Lhoták. 2012. Application-Only Call Graph Construction. In ECOOP 2012 - Object-Oriented Programming (ECOOP’12). Springer, 688–712. https://doi.org/10.1007/978-3-642-31057-7_30
[2]
Gábor Antal, Péter Hegedűs, Zoltán Herczeg, Gábor Lóki, and Rudolf Ferenc. 2023. Is JavaScript Call Graph Extraction Solved Yet? A Comparative Study of Static and Dynamic Tools. IEEE Access, 11 (2023), 25266–25284. https://doi.org/10.1109/ACCESS.2023.3255984
[3]
David F. Bacon and Peter F. Sweeney. 1996. Fast Static Analysis of C++ Virtual Function Calls. In Proceedings of the 11th ACM SIGPLAN Conference on Object-Oriented Programming, Systems, Languages, and Applications (OOPSLA’96). ACM, 324–341. https://doi.org/10.1145/236337.236371
[4]
Eric Bodden. 2018. The Secret Sauce in Efficient and Precise Static Analysis: The Beauty of Distributive, Summary-Based Static Analyses (and How to Master Them). In Companion Proceedings for the ISSTA/ECOOP 2018 Workshops (SOAP’18). ACM, 85–93. https://doi.org/10.1145/3236454.3236500
[5]
Martin Bravenboer and Yannis Smaragdakis. 2009. Strictly Declarative Specification of Sophisticated Points-to Analyses. In Proceedings of the 24th ACM SIGPLAN Conference on Object Oriented Programming Systems Languages and Applications (OOPSLA’09). ACM, 243–262. https://doi.org/10.1145/1640089.1640108
[6]
Marcel Böhme, Van-Thuan Pham, Manh-Dung Nguyen, and Abhik Roychoudhury. 2017. Directed Greybox Fuzzing. In Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security (CCS’17). ACM, 2329–2344. https://doi.org/10.1145/3133956.3134020
[7]
Madhurima Chakraborty, Renzo Olivares, Manu Sridharan, and Behnaz Hassanshahi. 2022. Automatic Root Cause Quantification for Missing Edges in JavaScript Call Graphs. In 36th European Conference on Object-Oriented Programming (ECOOP’22). Schloss Dagstuhl - Leibniz-Zentrum für Informatik, 3:1–3:28. https://doi.org/10.4230/LIPIcs.ECOOP.2022.3
[8]
Jeffrey Dean, David Grove, and Craig Chambers. 1995. Optimization of Object-Oriented Programs Using Static Class Hierarchy Analysis. In European Conference on Object-Oriented Programming (ECOOP’95). Springer, 77–101. https://doi.org/10.1007/3-540-49538-X_5
[9]
Jens Dietrich, Henrik Schole, Li Sui, and Ewan Tempero. 2017. XCorpus–An executable Corpus of Java Programs. Journal of Object Technology, 16, 4 (2017), 1:1–1:24. https://doi.org/10.5381/jot.2017.16.4.a1
[10]
Michael Eichberg and Ben Hermann. 2014. A software product line for static analyses: the OPAL framework. In Proceedings of the 3rd ACM SIGPLAN International Workshop on the State of the Art in Java Program Analysis (SOAP’14). ACM, 1–6. https://doi.org/10.1145/2614628.2614630
[11]
David Grove and Craig Chambers. 2001. A Framework for Call Graph Construction Algorithms. ACM Transactions on Programming Languages and Systems, 23, 6 (2001), 685–746. https://doi.org/10.1145/506315.506316
[12]
Tobias Gutzmann, Antonina Khairova, Jonas Lundberg, and Welf Löwe. 2009. Towards Comparing and Combining Points-to Analyses. In 2009 Ninth IEEE International Working Conference on Source Code Analysis and Manipulation (SCAM’09). IEEE, 45–54. https://doi.org/10.1109/SCAM.2009.14
[13]
Dongjie He, Jingbo Lu, and Jingling Xue. 2022. Qilin: A New Framework For Supporting Fine-Grained Context-Sensitivity in Java Pointer Analysis. In 36th European Conference on Object-Oriented Programming (ECOOP’22, Vol. 222). Schloss Dagstuhl – Leibniz-Zentrum für Informatik, 30:1–30:29. https://doi.org/10.4230/LIPIcs.ECOOP.2022.30
[14]
Dominik Helm, Florian Kübler, Michael Reif, Michael Eichberg, and Mira Mezini. 2020. Modular Collaborative Program Analysis in OPAL. In Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE’20). ACM, 184–196. https://doi.org/10.1145/3368089.3409765
[15]
IBM. 2024. WALA - Static Analysis Framework for Java. http://wala.sourceforge.net/ [Online; accessed 11-March-2024]
[16]
Ondvrej Lhoták. 2007. Comparing Call Graphs. In Proceedings of the 7th ACM SIGPLAN-SIGSOFT Workshop on Program Analysis for Software Tools and Engineering (PASTE’07). ACM, 37–42. https://doi.org/10.1145/1251535.1251542
[17]
Linghui Luo, Goran Piskachev, Ranjith Krishnamurthy, Julian Dolby, Eric Bodden, and Martin Schäf. 2023. Model Generation For Java Frameworks. In 2023 IEEE Conference on Software Testing, Verification and Validation (ICST’23). IEEE, 165–175. https://doi.org/10.1109/ICST57152.2023.00024
[18]
Oracle. 2024. JVM(TM) Tool Interface. https://docs.oracle.com/en/java/javase/20/docs/specs/jvmti.html [Online; accessed 12-March-2024]
[19]
Michael Reif, Michael Eichberg, Ben Hermann, Johannes Lerch, and Mira Mezini. 2016. Call Graph Construction for Java Libraries. In Proceedings of the 2016 24th ACM SIGSOFT International Symposium on Foundations of Software Engineering (FSE’16). ACM, 474–486. https://doi.org/10.1145/2950290.2950312
[20]
Michael Reif, Florian Kübler, Michael Eichberg, Dominik Helm, and Mira Mezini. 2019. Judge: Identifying, Understanding, and Evaluating Sources of Unsoundness in Call Graphs. In Proceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA’19). ACM, 251–261. https://doi.org/10.1145/3293882.3330555
[21]
Michael Reif, Florian Kübler, Michael Eichberg, and Mira Mezini. 2018. Systematic Evaluation of the Unsoundness of Call Graph Construction Algorithms for Java. In Companion Proceedings for the ISSTA/ECOOP 2018 Workshops (SOAP’18). ACM, 107–112. https://doi.org/10.1145/3236454.3236503
[22]
Henry Gordon Rice. 1953. Classes of recursively enumerable sets and their decision problems. Transactions of the American Mathematical society, 74, 2 (1953), 358–366. https://doi.org/10.2307/1990888
[23]
Vitalis Salis, Thodoris Sotiropoulos, Panos Louridas, Diomidis Spinellis, and Dimitris Mitropoulos. 2021. PyCG: Practical Call Graph Generation in Python. In 43rd IEEE/ACM International Conference on Software Engineering (ICSE’21). IEEE, 1646–1657. https://doi.org/10.1109/ICSE43902.2021.00146
[24]
Olin Shivers. 1988. Control Flow Analysis in Scheme. In Proceedings of the ACM SIGPLAN 1988 Conference on Programming Language Design and Implementation (PLDI’88). ACM, 164–174. https://doi.org/10.1145/53990.54007
[25]
Yannis Smaragdakis, Martin Bravenboer, and Ondřej Lhoták. 2011. Pick Your Contexts Well: Understanding Object-Sensitivity. In Proceedings of the 38th Annual ACM SIGPLAN-SIGACT Symposium on Principles of Programming Languages (POPL’11). ACM, 17–30. https://doi.org/10.1145/1926385.1926390
[26]
Li Sui, Jens Dietrich, Michael Emery, Shawn Rasheed, and Amjed Tahir. 2018. On the Soundness of Call Graph Construction in the Presence of Dynamic Language Features - A Benchmark and Tool Evaluation. In Programming Languages and Systems (APLAS’18). Springer, 69–88. https://doi.org/10.1007/978-3-030-02768-1_4
[27]
Li Sui, Jens Dietrich, Amjed Tahir, and George Fourtounis. 2020. On the Recall of Static Call Graph Construction in Practice. In Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering (ICSE’20). ACM, 1049–1060. https://doi.org/10.1145/3377811.3380441
[28]
Zoltán Ságodi, Edit Pengő, Judit Jász, István Siket, and Rudolf Ferenc. 2022. Static Call Graph Combination to Simulate Dynamic Call Graph Behavior. IEEE Access, 10 (2022), 131829–131840. https://doi.org/10.1109/ACCESS.2022.3229182
[29]
Frank Tip and Jens Palsberg. 2000. Scalable Propagation-Based Call Graph Construction Algorithms. In Proceedings of the 15th ACM SIGPLAN Conference on Object-Oriented Programming, Systems, Languages, and Applications (OOPSLA’00). ACM, 281–293. https://doi.org/10.1145/353171.353190
[30]
Raja Vallée-Rai, Phong Co, Etienne Gagnon, Laurie Hendren, Patrick Lam, and Vijay Sundaresan. 1999. Soot - a Java Bytecode Optimization Framework. In Proceedings of the 1999 Conference of the Centre for Advanced Studies on Collaborative Research (CASCON’99). IBM Press, 13.
[31]
Yanhao Wang, Xiangkun Jia, Yuwei Liu, Kyle Zeng, Tiffany Bao, Dinghao Wu, and Purui Su. 2020. Not All Coverage Measurements Are Equal: Fuzzing by Coverage Accounting for Input Prioritization. In Network and Distributed Systems Security Symposium (NDSS’20). Internet Society, 2329–2344. https://doi.org/10.14722/ndss.2020.24422
[32]
Thomas Wetzlmaier, Rudolf Ramler, and Werner Putschögl. 2016. A Framework for Monkey GUI Testing. In 2016 IEEE International Conference on Software Testing, Verification and Validation (ICST’16). IEEE, 416–423. https://doi.org/10.1109/ICST.2016.51
Index Terms
- Total Recall? How Good Are Static Call Graphs Really?
Recommendations
Call Graph Soundness in Android Static Analysis
ISSTA 2024: Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and AnalysisStatic analysis is sound in theory, but an implementation may unsoundly fail to analyze all of a program's code. Any such omission is a serious threat to the validity of the tool's output. Our work is the first to measure the prevalence of these ...
Precise Call Graphs for C Programs with Function Pointers
The use of pointers presents serious problems for software productivity tools for software understanding, restructuring, and testing. Pointers enable indirect memory accesses through pointer dereferences, as well as indirect procedure calls (e.g., ...
The secret sauce in efficient and precise static analysis: the beauty of distributive, summary-based static analyses (and how to master them)
ISSTA '18: Companion Proceedings for the ISSTA/ECOOP 2018 WorkshopsIn this paper I report on experiences gained from more than five years of extensively designing static code analysis tools- in particular such ones with a focus on security- to scale to real-world projects within an industrial context. Within this time ...
Comments
Information & Contributors
Information
Published In
September 2024
1928 pages
Copyright © 2024 Owner/Author.
This work is licensed under a Creative Commons Attribution International 4.0 License.
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 11 September 2024
Check for updates
Badges
Author Tags
Qualifiers
- Research-article
Conference
ISSTA '24
Sponsor:
ISSTA '24: 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis
September 16 - 20, 2024
Vienna, Austria
Acceptance Rates
Overall Acceptance Rate 58 of 213 submissions, 27%
Upcoming Conference
ISSTA '25
- Sponsor:
- sigsoft
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 308Total Downloads
- Downloads (Last 12 months)308
- Downloads (Last 6 weeks)64
Reflects downloads up to 27 Feb 2025
Other Metrics
Citations
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in