skip to main content
10.1145/2635868.2635889acmconferencesArticle/Chapter ViewAbstractPublication PagesfseConference Proceedingsconference-collections
research-article

Solving complex path conditions through heuristic search on induced polytopes

Published: 11 November 2014 Publication History

Abstract

Test input generators using symbolic and concolic execution must solve path conditions to systematically explore a program and generate high coverage tests. However, path conditions may contain complicated arithmetic constraints that are infeasible to solve: a solver may be unavailable, solving may be computationally intractable, or the constraints may be undecidable. Existing test generators either simplify such constraints with concrete values to make them decidable, or rely on strong but incomplete constraint solvers. Unfortunately, simplification yields coarse approximations whose solutions rarely satisfy the original constraint. Moreover, constraint solvers cannot handle calls to native library methods. We show how a simple combination of linear constraint solving and heuristic search can overcome these limitations. We call this technique Concolic Walk. On a corpus of 11 programs, an instance of our Concolic Walk algorithm using tabu search generates tests with two- to three-times higher coverage than simplification-based tools while being up to five-times as efficient. Furthermore, our algorithm improves the coverage of two state-of-the-art test generators by 21% and 32%. Other concolic and symbolic testing tools could integrate our algorithm to solve complex path conditions without having to sacrifice any of their own capabilities, leading to higher overall coverage.

References

[1]
The economic impacts of inadequate infrastructure for software testing. Planning report 02-03, National Institute of Standards, May 2002.
[2]
A. I. Baars, M. Harman, Y. Hassoun, K. Lakhotia, P. McMinn, P. Tonella, and T. E. J. Vos. Symbolic search-based testing. In ASE 2011, pages 53–62. IEEE, 2011.
[3]
N. E. Beckman, A. V. Nori, S. K. Rajamani, and R. J. Simmons. Proofs from tests. In ISSTA 2008, pages 3–14. ACM, 2008.
[4]
N. E. Beckman, A. V. Nori, S. K. Rajamani, R. J. Simmons, S. Tetali, and A. V. Thakur. Proofs from tests. IEEE Trans. Software Eng., 36(4):495–508, 2010.
[5]
J. Bloch. Effective Java. Addison-Wesley, second edition, 2008.
[6]
M. Borges, M. d’Amorim, S. Anand, D. H. Bushnell, and C. S. Păsăreanu. Symbolic execution with interval solving and meta-heuristic search. In ICST 2012, pages 111–120. IEEE, 2012.
[7]
C. Cadar, D. Dunbar, and D. R. Engler. KLEE: Unassisted and automatic generation of high-coverage tests for complex systems programs. In OSDI 2008, pages 209–224. USENIX Association, 2008.
[8]
C. Cadar, V. Ganesh, P. M. Pawlowski, D. L. Dill, and D. R. Engler. EXE: Automatically generating inputs of death. In CCS 2006, pages 322–335. ACM, 2006.
[9]
P. Codognet and D. Diaz. Yet another local search method for constraint solving. In SAGA 2001, volume 2264 of LNCS, pages 73–90. Springer, 2001.
[10]
V. Dallmeier, N. Knopp, C. Mallon, S. Hack, and A. Zeller. Generating test cases for specification mining. In ISSTA 2010, pages 85–96. ACM, 2010.
[11]
M. Davis. Hilbert’s tenth problem is unsolvable. American Mathematical Monthly, 80:233–269, 1973.
[12]
X. Deng, Robby, and J. Hatcliff. Kiasan: A verification and test-case generation framework for java based on symbolic execution. In ISoLA 2006, page 137. IEEE, 2006.
[13]
G. Fraser and A. Arcuri. Evosuite: automatic test suite generation for object-oriented software. In SIGSOFT FSE 2011, pages 416–419. ACM, 2011.
[14]
J. P. Galeotti, G. Fraser, and A. Arcuri. Improving search-based test suite generation with dynamic symbolic execution. In ISSRE 2013, pages 360–369. IEEE, 2013.
[15]
M. Gligoric, A. Groce, C. Zhang, R. Sharma, M. A. Alipour, and D. Marinov. Comparing non-adequate test suites using coverage criteria. In ISSTA 2013, pages 302–313. ACM, 2013.
[16]
F. Glover. Tabu search - part i. INFORMS Journal on Computing, 1(3):190–206, 1989.
[17]
F. Glover. Tabu search - part ii. INFORMS Journal on Computing, 2(1):4–32, 1990.
[18]
P. Godefroid. Higher-order test generation. In PLDI 2011, pages 258–269. ACM, 2011.
[19]
P. Godefroid, N. Klarlund, and K. Sen. DART: directed automated random testing. In PLDI 2005, pages 213–223. ACM, 2005.
[20]
A. Gotlieb and M. Petit. Path-oriented random testing. In Proceedings of the First International Workshop on Random Testing, 2006.
[21]
A. Gotlieb and M. Petit. A uniform random test data generator for path testing. Journal of Systems and Software, 83(12):2618–2626, 2010.
[22]
B. S. Gulavani, T. A. Henzinger, Y. Kannan, A. V. Nori, and S. K. Rajamani. Synergy: a new algorithm for property checking. In SIGSOFT FSE 2006, pages 117–127. ACM, 2006.
[23]
K. Inkumsah and T. Xie. Improving structural testing of object-oriented programs via integrating evolutionary testing and symbolic execution. In ASE 2008, pages 297–306. IEEE, 2008.
[24]
W. Jin, A. Orso, and T. Xie. Automated behavioral regression testing. In ICST 2010, pages 137–146. IEEE Computer Society, 2010.
[25]
J. Kennedy and R. Eberhart. Particle swarm optimization. In IEEE Internat. Conf. on Neural Networks, volume 4, pages 1942–1948. IEEE, 1995.
[26]
S. Khurshid, C. S. Păsăreanu, and W. Visser. Generalized symbolic execution for model checking and testing. In TACAS 2003, volume 2619 of LNCS, pages 553–568. Springer, 2003.
[27]
B. Korel. Automated software test data generation. IEEE Trans. Software Eng., 16(8):870–879, 1990.
[28]
K. Lakhotia, N. Tillmann, M. Harman, and J. de Halleux. FloPSy - search-based floating point constraint solving for symbolic execution. In ICTSS 2010, volume 6435 of LNCS, pages 142–157. Springer, 2010.
[29]
J. Malburg and G. Fraser. Combining search-based and constraint-based testing. In ASE 2011, pages 436–439. IEEE, 2011.
[30]
P. McMinn. Search-based software test data generation: a survey. Softw. Test., Verif. Reliab., 14(2):105–156, 2004.
[31]
L. M. de Moura and N. Bjørner. Z3: An efficient SMT solver. In TACAS, volume 4963 of LNCS. Springer, 2008.
[32]
C. Pacheco and M. D. Ernst. Randoop: feedback-directed random testing for java. In OOPSLA 2007 Companion, pages 815–816. ACM, 2007.
[33]
C. S. Păsăreanu and N. Rungta. Symbolic PathFinder: symbolic execution of java bytecode. In ASE 2010, pages 179–180. ACM, 2010.
[34]
C. S. Păsăreanu, N. Rungta, and W. Visser. Symbolic execution with mixed concrete-symbolic solving. In ISSTA 2011, pages 34–44. ACM, 2011.
[35]
C. S. Păsăreanu, W. Visser, D. H. Bushnell, J. Geldenhuys, P. C. Mehlitz, and N. Rungta. Symbolic PathFinder: integrating symbolic execution with model checking for Java bytecode analysis. Autom. Softw. Eng., 20(3):391–425, 2013.
[36]
K. Sen and G. Agha. CUTE and jCUTE: Concolic unit testing and explicit path model-checking tools. In CAV 2006, volume 4144 of LNCS, pages 419–423. Springer, 2006.
[37]
M. Souza, M. Borges, M. d’Amorim, and C. S. Păsăreanu. CORAL: Solving complex constraints for Symbolic PathFinder. In NASA Formal Methods, pages 359–374, 2011.
[38]
M. Takaki, D. Cavalcanti, R. Gheyi, J. Iyoda, M. d’Amorim, and R. B. C. Prudˆ encio. A comparative study of randomized constraint solvers for random-symbolic testing. In NASA Formal Methods, pages 56–65, 2009.
[39]
M. Takaki, D. Cavalcanti, R. Gheyi, J. Iyoda, M. d’Amorim, and R. B. C. Prudˆ encio. Randomized constraint solvers: a comparative study. ISSE, 6(3):243–253, 2010.
[40]
K. Taneja, T. Xie, N. Tillmann, and J. de Halleux. eXpress: guided path exploration for efficient regression test generation. In ISSTA 2011, pages 1–11. ACM, 2011.
[41]
N. Tillmann and J. de Halleux. Pex-white box test generation for .NET. In TAP 2008, volume 4966 of LNCS, pages 134–153. Springer, 2008.
[42]
W. Visser, J. Geldenhuys, and M. B. Dwyer. Green: reducing, reusing and recycling constraints in program analysis. In SIGSOFT FSE 2012, page 58. ACM, 2012.
[43]
W. Visser, C. S. Păsăreanu, and S. Khurshid. Test input generation with Java PathFinder. In ISSTA 2004, pages 97–107. ACM, 2004.
[44]
X. Xiao, T. Xie, N. Tillmann, and J. de Halleux. Precise identification of problems for structural test generation. In ICSE 2011, pages 611–620. ACM, 2011.
[45]
T. Xie, N. Tillmann, J. de Halleux, and W. Schulte. Fitness-guided path exploration in dynamic symbolic execution. In DSN 2009, pages 359–368. IEEE, 2009.
[46]
G. Yang, S. Khurshid, and C. S. Păsăreanu. Memoise: a tool for memoized symbolic execution. In ICSE 2013, pages 1343–1346. IEEE / ACM, 2013. Introduction Motivation Algorithm Synopsis Terms and Definitions Concolic Walk Algorithm Discussion Implementation Evaluation Effectiveness and Efficiency Influence of Algorithm Parameters Limitations and Future Work Related Work Conclusions Acknowledgments References

Cited By

View all
  • (2024)Evolutionary Analysis of Alloy Specifications with an Adaptive Fitness FunctionSearch-Based Software Engineering10.1007/978-3-031-64573-0_1(1-17)Online publication date: 2-Jul-2024
  • (2023)Quantum symbolic executionQuantum Information Processing10.1007/s11128-023-04144-522:10Online publication date: 20-Oct-2023
  • (2022)Satisfiability modulo fuzzing: a synergistic combination of SMT solving and fuzzingProceedings of the ACM on Programming Languages10.1145/35633326:OOPSLA2(1236-1263)Online publication date: 31-Oct-2022
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
FSE 2014: Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering
November 2014
856 pages
ISBN:9781450330565
DOI:10.1145/2635868
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 11 November 2014

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. Concolic Testing
  2. Local Search
  3. Non-Linear Constraints

Qualifiers

  • Research-article

Conference

SIGSOFT/FSE'14
Sponsor:

Acceptance Rates

Overall Acceptance Rate 17 of 128 submissions, 13%

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)7
  • Downloads (Last 6 weeks)0
Reflects downloads up to 06 Jan 2025

Other Metrics

Citations

Cited By

View all
  • (2024)Evolutionary Analysis of Alloy Specifications with an Adaptive Fitness FunctionSearch-Based Software Engineering10.1007/978-3-031-64573-0_1(1-17)Online publication date: 2-Jul-2024
  • (2023)Quantum symbolic executionQuantum Information Processing10.1007/s11128-023-04144-522:10Online publication date: 20-Oct-2023
  • (2022)Satisfiability modulo fuzzing: a synergistic combination of SMT solving and fuzzingProceedings of the ACM on Programming Languages10.1145/35633326:OOPSLA2(1236-1263)Online publication date: 31-Oct-2022
  • (2022)Enhancing Dynamic Symbolic Execution by Automatically Learning Search HeuristicsIEEE Transactions on Software Engineering10.1109/TSE.2021.310187048:9(3640-3663)Online publication date: 1-Sep-2022
  • (2021)EqBench: A Dataset of Equivalent and Non-equivalent Program Pairs2021 IEEE/ACM 18th International Conference on Mining Software Repositories (MSR)10.1109/MSR52588.2021.00084(610-614)Online publication date: May-2021
  • (2020)ARDiff: scaling program equivalence checking via iterative abstraction and refinement of common codeProceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering10.1145/3368089.3409757(13-24)Online publication date: 8-Nov-2020
  • (2020)An Integrated Approach for Effective Injection Vulnerability Analysis of Web Applications Through Security Slicing and Hybrid Constraint SolvingIEEE Transactions on Software Engineering10.1109/TSE.2018.284434346:2(163-195)Online publication date: 1-Feb-2020
  • (2019)Deferred concretization in symbolic execution via fuzzingProceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis10.1145/3293882.3330554(228-238)Online publication date: 10-Jul-2019
  • (2019)InSPeCT: Iterated Local Search for Solving Path Conditions2019 IEEE 15th International Conference on Automation Science and Engineering (CASE)10.1109/COASE.2019.8843039(1724-1729)Online publication date: Aug-2019
  • (2019)JConstraints: A Library for Working with Logic Expressions in JavaModels, Mindsets, Meta: The What, the How, and the Why Not?10.1007/978-3-030-22348-9_19(310-325)Online publication date: 26-Jun-2019
  • Show More Cited By

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Media

Figures

Other

Tables

Share

Share

Share this Publication link

Share on social media