Skip to main content
SpringerLink
Log in

Randomized constraint solvers: a comparative study

  • Original Paper
  • Published:
Innovations in Systems and Software Engineering Aims and scope Submit manuscript

Abstract

The complexity of constraints is a major obstacle for constraint-based software verification. Automatic constraint solvers are fundamentally incomplete: input constraints often build on some undecidable theory or some theory the solver does not support. This paper proposes and evaluates several randomized solvers to address this issue. We compared the effectiveness of a symbolic solver (CVC3), a random solver, two heuristic search solvers, and seven hybrid solvers (i.e. mix of random, symbolic, and heuristic solvers). We evaluated the solvers on a benchmark generated with a concolic execution of 9 subjects. The performance of each solver was measured by its precision, which is the fraction of constraints that the solver can find solution out of the total number of constraints that some solver can find solution. As expected, symbolic solving subsumes the other approaches for the 4 subjects that only generate decidable constraints. For the remaining 5 subjects, which contain undecidable constraints, the hybrid solvers achieved the highest precision (fraction of constraints that a solver can find a solution out of the total number of satisfiable constraints). We also observed that the solvers were complementary, which suggests that one should alternate their use in iterations of a concolic execution driver.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Abdelbar A, Abdelshahid S (2004) Instinct-based PSO with local search applied to satisfiability. In: IEEE International Joint Conference on Neural Networks, pp 2291–2295

  2. Anand S, Pasareanu CS, Visser W (2007) JPF-SE: a symbolic execution extension to Java PathFinder. In: Tools and algorithms for the construction and analysis of systems (TACAS), pp 134–138

  3. Bäck T, Eiben AE, Vink ME (1998) A superior evolutionary algorithm for 3-SAT. In: 7th conference on evolutionary programming. Springer, UK, pp 125–136

  4. Bauer A, Pister M, Tautschnig M (2007) Tool-support for the analysis of hybrid systems and models. In: Lauwereins R, Madsen J (eds) Design, automation and test in Europe conference and exposition (DATE). ACM, Nice, pp 924–929. doi:10.1145/1266366.1266565

  5. Beizer B (1990) Software testing techniques. International Thomson Computer Press

  6. Benhamou F, Granvilliers L (2006) Continuous and interval constraints. In: Rossi F, van Beek P, Walsh T (eds) Handbook of constraint programming, foundations of artificial intelligence, Chap. 16. Elsevier Science Publishers, Amsterdam. URL http://www.lina.sciences.univ-nantes.fr/Publications/2006/BG06

  7. Bird DL, Munoz CU (1983) Automatic generation of random self-checking test cases. IBM Syst J 23(3): 228–245

    Google Scholar 

  8. Boyapati C, Khurshid S, Marinov D (2002) Korat: automated testing based on Java predicates. In: Proceedings of the international symposium on software testing and analysis (ISSTA), pp 123–133. doi:10.1145/566172.566191

  9. Bozzano M, Bruttomesso R, Cimatti A, Junttila T, van Rossum P, Schulz S, Sebastiani R (2005) An incremental and layered procedure for the satisfiability of linear arithmetic logic. In: Proceedings of the international conference on tools and algorithms for the construction and analysis of systems (TACAS), pp 317–333

  10. Csallner C, Smaragdakis Y (2004) JCrasher: an automatic robustness tester for Java. Softw Pract Exp 34: 1025–1050

    Article  Google Scholar 

  11. Davis M, Logemann G, Loveland D (1962) A machine program for theorem-proving. Communications of ACM 5(7): 394–397. doi:10.1145/368273.368557

    Article  MATH  MathSciNet  Google Scholar 

  12. Dwyer MB, Elbaum S, Person S, Purandare R (2007) Parallel randomized state-space search. In: ICSE ’07: Proceedings of the 29th international conference on Software Engineering. IEEE Computer Society, Washington, DC, pp 3–12

  13. Eiben A, van der Hauw J (1997) Solving 3-SAT by GAs adapting constraint weights. Evolutionary computation, pp 81–86. doi:10.1109/ICEC.1997.592273

  14. Folino G, Pizzuti C, Spezzano O (1998) Combining cellular genetic algorithms and local search for solving satisfiability problems. In: IEEE conference on tools with artificial intelligence, pp 192–198

  15. Forrester J, Miller B (2000) An empirical study of the robustness of windows NT applications using random testing. In: USENIX Windows Systems Symposium, pp 59–68

  16. Franzle M, Herde C, Teige T, Ratschan S, Schubert T (2007) Efficient solving of large non-linear arithmetic constraint systems with complex boolean structure. J Satisf Boolean Model Comput 1: 209–236

    Google Scholar 

  17. Ganzinger H, Hagen G, Nieuwenhuis R, Oliveras A, Tinelli C (2004) DPLL(T): fast decision procedures. In: Alur R, Peled D (eds) Computer aided verification : 16th international conference, CAV 2004. Lecture Notes in Computer Science, vol 3114. Springer, Boston, pp 175–188

  18. Godefroid P (2007) Compositional dynamic test generation. In: 34th symposium on principles of programming languages (POPL). ACM, New York, pp 47–54. doi:10.1145/1190216.1190226

  19. Godefroid P, Klarlund N, Sen K (2005) DART: directed automated random testing. In: Proceedings of the ACM SIGPLAN conference on programming language design and implementation (PLDI), vol 40. ACM Press, New York, pp 213–223. doi:10.1145/1064978.1065036

  20. Goldberg DE (1989) Genetic Algorithms in search, optimization and machine learning. Addison-Wesley Longman Publishing Co., Inc., Boston

  21. Harman M (2007) Automated test data generation using search based software engineering. In: AST ’07: proceedings of the second international workshop on automation of software test, p. 2. IEEE Computer Society, Washington. doi:10.1109/AST.2007.4

  22. Holzmann GJ, Joshi R, Groce A (2008) Swarm verification. In: Proceedings of the international conference on automated software engineering (ASE). IEEE, pp 1–6

  23. Kennedy J, Eberhart R (1995) Particle swarm optimization. In: IEEE neural networks, pp 1942–1948

  24. King JC (1976) Symbolic execution and program testing. Commun ACM 19(7): 385–394

    Article  MATH  Google Scholar 

  25. Majumdar R, Sen K (2007) Hybrid concolic testing. In: Proceedings of the International Conference on Software Engineering (ICSE), pp. 416–426. doi:10.1109/ICSE.2007.41

  26. Marchiori E, Rossi C (1999) A flipping genetic algorithm for hard 3-SAT problems. In: Genetic and evolutionary computation conference, vol 1. Morgan Kaufmann, Orlando, pp 393–400

  27. Matsumoto M, Nishimura T (1998) Mersenne Twister: a 623-dimensionally equidistributed uniform pseudo-random number generator. ACM Trans Model Comput Simul 8(1): 3–30. doi:10.1145/272991.272995

    Article  MATH  Google Scholar 

  28. National Institute of Standards and Technology (2002) The economic impacts of inadequate infrastructure for software testing. Planning Report 02-3

  29. Pacheco C, Ernst M (2007) Randoop: feedback-directed random testing for Java. In: Companion to the 22nd Annual ACM SIGPLAN conference on object-oriented programming, systems, languages, and applications, OOPSLA, pp 815–816

  30. Pacheco C, Ernst MD (2004) Eclat documents. Online manual. http://people.csail.mit.edu/people/cpacheco/eclat/

  31. Pacheco C, Lahiri SK, Ernst MD, Ball T (2007) Feedback-directed random test generation. In: Proceedings of the international conference on software engineering (ICSE). IEEE Computer Society, Washington, DC, pp 75–84. doi:10.1109/ICSE.2007.37

  32. Qadeer S (2004) Daisy file system. Joint CAV/ISSTA special event on specification, verification, and testing of concurrent software

  33. Ranise S, Tinelli C (2006) The SMT-LIB standard: Version 1.2. Tech. rep.

  34. Rossi C, Marchiori E, Kok JN (2000) An adaptive evolutionary algorithm for the satisfiability problem. In: Symposium on applied computing (SAC). Como, Italy, pp 463–469

  35. Ru N, Jianhua Y (2008) A GA and particle swarm optimization based hybrid algorithm. In: IEEE world congress on computational intelligence. Hong Kong

  36. Santhanam P, Hailpern B (2002) Software debugging, testing, and verification. IBM Syst J 41: 4–12

    Article  Google Scholar 

  37. Sen K, Agha G (2006) CUTE and jCUTE: Concolic unit testing and explicit path model-checking tools. In: CAV, pp 419–423

  38. Tinelli C (2002) A DPLL-based calculus for ground satisfiability modulo theories. In: Proceedings of the European conference on logics in artificial intelligence. Springer, London, pp 308–319

  39. Windisch A, Wappler S, Wegener J (2007) Applying particle swarm optimization to software testing. In: Lipson H (ed) Genetic and evolutionary computation conference, pp 1121–1128

  40. Wintersteiger C, Hamadi Y, de Moura L (2009) A concurrent portfolio approach to SMT solving, pp 715–720

  41. Ypma TJ (1995) Historical development of the Newton–Raphson method. SIAM Rev 37(4): 531–551

    Article  MATH  MathSciNet  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Federal University of Pernambuco, Recife, Brazil

    Mitsuo Takaki, Juliano Iyoda, Marcelo d’Amorim & Ricardo B. C. Prudêncio

  2. Federal University of Campina Grande, Campina Grande, Brazil

    Diego Cavalcanti & Rohit Gheyi

Authors
  1. Mitsuo Takaki
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Diego Cavalcanti
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Rohit Gheyi
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Juliano Iyoda
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Marcelo d’Amorim
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Ricardo B. C. Prudêncio
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Mitsuo Takaki.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Takaki, M., Cavalcanti, D., Gheyi, R. et al. Randomized constraint solvers: a comparative study. Innovations Syst Softw Eng 6, 243–253 (2010). https://doi.org/10.1007/s11334-010-0124-1

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11334-010-0124-1

Keywords

Search

Navigation