Skip to main content
SpringerLink
Log in
Book cover

International Conference on Principles and Practice of Constraint Programming

CP 2019: Principles and Practice of Constraint Programming pp 565–582Cite as

SolverCheck: Declarative Testing of Constraints

  • Conference paper
  • First Online:

Part of the book series: Lecture Notes in Computer Science ((LNPSE,volume 11802))

Abstract

This paper introduces SolverCheck, a property-based testing (PBT) library specifically designed to test CP solvers. In particular, SolverCheck provides a declarative language to express a propagator’s expected behavior and test it automatically. That language is easily extended with new constraints and flexible enough to precisely describe a propagator’s consistency. Experiments carried out using Choco [41], JaCoP [27] and MiniCP [35] revealed the presence of numerous non-trivial bugs, no matter how carefully the test suites of these solvers have been engineered. Beyond the remarkable effectiveness of our technique to assess the correctness and robustness of a solver, our experiments also demonstrated the practical usability of SolverCheck to test actual CP-solvers.

Massart and Rombouts have worked on a preliminary version of this work for their MSc. thesis which we supervised. They presented it at the CP-2018 Doctoral Programme [33].

This is a preview of subscription content, log in via an institution.

Chapter
USD   29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD   39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD   54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Learn about institutional subscriptions

Notes

  1. 1.

    Gecode, and likewise Choco for some of its propagators, are a notable exception which is covered in the related work section.

  2. 2.

    Actually, both solvers adopt a slightly different approach, but this is not relevant for our matter as they are based on the same idea. For the full details, see http://bit.ly/cst-gecode and http://bit.ly/cst-choco.

  3. 3.

    sets of pseudo-random int.

  4. 4.

    SolverCheck comes with built-in checkers for the usual constraints alldiff, element, gcc, etc.

  5. 5.

    Experiments were also realized using AbsCon [30]. However, even though we highlighted some defects in this solver, we chose not to report on the outcome of these experiments because we are still discussing some of our findings with the maintainer of that solver.

  6. 6.

    The plots and observations to be made when it is the domain size that varies for a fixed number of variables are substantially the same as the case presented here (varying number of variables, fixed domain size). These are therefore omitted.

References

  1. Ahrendt, W., Beckert, B., Bubel, R., Hähnle, R., Schmitt, P.H., Ulbrich, M. (eds.): Deductive Software Verification - The KeY Book - From Theory to Practice. Lecture Notes in Computer Science, vol. 10001. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-49812-6

    Book  Google Scholar 

  2. Akgün, Ö., Gent, I.P., Jefferson, C., Miguel, I., Nightingale, P.: Metamorphic testing of constraint solvers. In: Hooker, J. (ed.) CP 2018. LNCS, vol. 11008, pp. 727–736. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-98334-9_46

    Chapter  Google Scholar 

  3. Artho, C., Biere, A., Seidl, M.: Model-based testing for verification back-ends. In: Veanes, M., Viganò, L. (eds.) TAP 2013. LNCS, vol. 7942, pp. 39–55. Springer, Heidelberg (2013). https://doi.org/10.1007/978-3-642-38916-0_3

    Chapter  Google Scholar 

  4. Barnes, J.: SPARK: The Proven Approach to High Integrity Software. Altran Praxis, UK (2012). http://www.altran.co.uk

  5. Beldiceanu, N., Carlsson, M., Rampon, J.X.: Global constraint catalog, 2nd edn, July 2010. Tech. Rep. Swedish Institute of Computer Science (2010)

    Google Scholar 

  6. Bessiere, C.: Constraint propagation. In: Rossi, F., van Beek, P., Walsh, T. (eds.) Handbook of Constraint Programming, Foundations of Artificial Intelligence, vol. 2, pp. 29–83. Elsevier, Amsterdam (2006)

    Chapter  Google Scholar 

  7. Boussemart, F., Lecoutre, C., Piette, C.: XCSP3: An integrated format for benchmarking combinatorial constrained problems. arXiv preprint arXiv:1611.03398 (2016)

  8. Brummayer, R., Järvisalo, M.: Testing and debugging techniques for answer set solver development. TPLP 10(4–6), 741–758 (2010)

    MathSciNet  MATH  Google Scholar 

  9. Brummayer, R., Lonsing, F., Biere, A.: Automated testing and debugging of SAT and QBF solvers. In: Strichman, O., Szeider, S. (eds.) SAT 2010. LNCS, vol. 6175, pp. 44–57. Springer, Heidelberg (2010). https://doi.org/10.1007/978-3-642-14186-7_6

    Chapter  MATH  Google Scholar 

  10. Carlier, J., Pinson, E.: Adjustment of heads and tails for the job-shop problem. Eur. J. Oper. Res. 78(2), 146–161 (1994). Project Management and Scheduling

    Article  Google Scholar 

  11. Carlier, M., Dubois, C., Gotlieb, A.: FocalTest: a constraint programming approach for property-based testing. In: Cordeiro, J., Virvou, M., Shishkov, B. (eds.) ICSOFT 2010. CCIS, vol. 170, pp. 140–155. Springer, Heidelberg (2013). https://doi.org/10.1007/978-3-642-29578-2_9

    Chapter  Google Scholar 

  12. Carlier, M., Dubois, C., Gotlieb, A.: A certified constraint solver over finite domains. In: Giannakopoulou, D., Méry, D. (eds.) FM 2012. LNCS, vol. 7436, pp. 116–131. Springer, Heidelberg (2012). https://doi.org/10.1007/978-3-642-32759-9_12

    Chapter  Google Scholar 

  13. Claessen, K., Hughes, J.: Quickcheck: a lightweight tool for random testing of haskell programs. In: Odersky, M., Wadler, P. (eds.) Proceedings of the Fifth ACM SIGPLAN International Conference on Functional Programming (ICFP 2000), Montreal, Canada, 18–21 September 2000, pp. 268–279. ACM (2000)

    Google Scholar 

  14. Dahmen, M.: A debugger for constraints in prolog. Tech. Rep. ECRC-91-11. ECRC (1991)

    Google Scholar 

  15. Debruyune, R., Fekete, J.D., Jussien, N., Ghoniem, M.: Proposition de format concret pour des traces générées par des solveurs de contraintes réalisation rntl oadymppac 2.2. 2.1 (2001). http://pauillac.inria.fr/~contraintes/OADymPPaC/Public/d2.2.2.1.pdf

  16. Derval, G., Régin, J., Schaus, P.: Improved filtering for the bin-packing with cardinality constraint. Constraints 23(3), 251–271 (2018)

    Article  MathSciNet  Google Scholar 

  17. Dooms, G., Deville, Y., Dupont, P.: CP(Graph): introducing a graph computation domain in constraint programming. In: van Beek, P. (ed.) CP 2005. LNCS, vol. 3709, pp. 211–225. Springer, Heidelberg (2005). https://doi.org/10.1007/11564751_18

    Chapter  MATH  Google Scholar 

  18. Dubois, C.: Formally Verified Decomposition of Non-binary Constraints into Equivalent Binary Constraints. In: Magaud, N., Dargaye, Z. (eds.) Journées Francophones des Langages Applicatifs, January 2019. JFLA2019, Les Rousses, France (2019)

    Google Scholar 

  19. Dubois, C., Gotlieb, A.: Solveurs cp (fd) vérifiés formellement. In: Journées Francophones de Programmation par Contraintes (JFPC 2013), pp. 115–118. aix-en-provence, France (2013)

    Google Scholar 

  20. Ducassé, M.: Opium\({}^{\text{+}}\), a meta-debugger for prolog. In: ECAI, pp. 272–277 (1988)

    Google Scholar 

  21. Filliâtre, J.-C., Marché, C.: The Why/Krakatoa/Caduceus platform for deductive program verification. In: Damm, W., Hermanns, H. (eds.) CAV 2007. LNCS, vol. 4590, pp. 173–177. Springer, Heidelberg (2007). https://doi.org/10.1007/978-3-540-73368-3_21

    Chapter  Google Scholar 

  22. Filliâtre, J.-C., Paskevich, A.: Why3 — where programs meet provers. In: Felleisen, M., Gardner, P. (eds.) ESOP 2013. LNCS, vol. 7792, pp. 125–128. Springer, Heidelberg (2013). https://doi.org/10.1007/978-3-642-37036-6_8

    Chapter  Google Scholar 

  23. Frisch, A.M., Grum, M., Jefferson, C., Hernández, B.M., Miguel, I.: The design of ESSENCE: a constraint language for specifying combinatorial problems. In: Veloso, M.M. (ed.) IJCAI 2007, Proceedings of the 20th International Joint Conference on Artificial Intelligence, Hyderabad, India, 6–12 January 2007, pp. 80–87 (2007)

    Google Scholar 

  24. Hentenryck, P.V., Carillon, J.: Generality versus specificity: an experience with AI and OR techniques. In: Shrobe, H.E., Mitchell, T.M., Smith, R.G. (eds.) Proceedings of the 7th National Conference on Artificial Intelligence, St. Paul, MN, USA, 21–26 August 1988, pp. 660–664. AAAI Press/The MIT Press (1988)

    Google Scholar 

  25. Hoare, C.A.R.: The emperor’s old clothes. Commun. ACM 24(2), 75–83 (1981)

    Article  Google Scholar 

  26. Kirchner, F., Kosmatov, N., Prevosto, V., Signoles, J., Yakobowski, B.: Frama-C: a software analysis perspective. Formal Asp. Comput. 27(3), 573–609 (2015)

    Article  MathSciNet  Google Scholar 

  27. Kuchcinski, K., Szymanek, R.: JaCoP-java constraint programming solver. In: CP Solvers: Modeling, Applications, Integration, and Standardization, co-located with the 19th International Conference on Principles and Practice of Constraint Programming (2013)

    Google Scholar 

  28. Lazaar, N., Gotlieb, A., Lebbah, Y.: A CP framework for testing CP. Constraints 17(2), 123–147 (2012)

    Article  Google Scholar 

  29. Lecoutre, C., Prosser, P.: Maintaining singleton arc consistency. In: Proceedings of the 3rd International Workshop on Constraint Propagation and Implementation (CPAI 2006) held with CP 2006, pp. 47–61. Springer (2006)

    Google Scholar 

  30. Lecoutre, C., Tabary, S.: Abscon 112: towards more robustness. In: 3rd International Constraint Solver Competition (CSC 2008), Sydney, Australia, pp. 41–48 (2008). https://hal.archives-ouvertes.fr/hal-00870841

  31. Lesaint, D., Mehta, D., O’Sullivan, B., Quesada, L., Wilson, N.: Solving a telecommunications feature subscription configuration problem. In: Stuckey, P.J. (ed.) CP 2008. LNCS, vol. 5202, pp. 67–81. Springer, Heidelberg (2008). https://doi.org/10.1007/978-3-540-85958-1_5

    Chapter  Google Scholar 

  32. Martin, P., Shmoys, D.B.: A new approach to computing optimal schedules for the job-shop scheduling problem. In: Cunningham, W.H., McCormick, S.T., Queyranne, M. (eds.) IPCO 1996. LNCS, vol. 1084, pp. 389–403. Springer, Heidelberg (1996). https://doi.org/10.1007/3-540-61310-2_29

    Chapter  Google Scholar 

  33. Massart, A., Rombouts, V., Schaus, P.: Testing global constraints. CoRR abs/1807.03975 (2018). http://arxiv.org/abs/1807.03975

  34. Meier, M.: Debugging constraint programs. In: Montanari, U., Rossi, F. (eds.) CP 1995. LNCS, vol. 976, pp. 204–221. Springer, Heidelberg (1995). https://doi.org/10.1007/3-540-60299-2_13

    Chapter  Google Scholar 

  35. Michel, L., Schaus, P., Van Hentenryck, P.: MiniCP: a lightweight solver for constraint programming (2018). www.minicp.org

  36. Nethercote, N., Stuckey, P.J., Becket, R., Brand, S., Duck, G.J., Tack, G.: MiniZinc: towards a standard CP modelling language. In: Bessière, C. (ed.) CP 2007. LNCS, vol. 4741, pp. 529–543. Springer, Heidelberg (2007). https://doi.org/10.1007/978-3-540-74970-7_38

    Chapter  Google Scholar 

  37. Niemetz, A., Preiner, M., Biere, A.: Model-based API testing for SMT solvers. In: Proceedings of the 15th International Workshop on Satisfiability Modulo Theories, p. 10. SMT (2017)

    Google Scholar 

  38. van Omme, N., Perron, L., Furnon, V.: OR-Tools user’s manual. Google Inc. (2014). https://developers.google.com/optimization/

  39. OscaR Team: OscaR: Scala in OR (2012). https://bitbucket.org/oscarlib/oscar

  40. Pesant, G.: A regular language membership constraint for finite sequences of variables. In: Wallace, M. (ed.) CP 2004. LNCS, vol. 3258, pp. 482–495. Springer, Heidelberg (2004). https://doi.org/10.1007/978-3-540-30201-8_36

    Chapter  MATH  Google Scholar 

  41. Prud’homme, C., Fages, J.G., Lorca, X.: Choco Documentation. TASC - LS2N CNRS UMR 6241, COSLING S.A.S. (2017). http://www.choco-solver.org

  42. Régin, J.: A filtering algorithm for constraints of difference in CSPs. In: Hayes-Roth, B., Korf, R.E. (eds.) Proceedings of the 12th National Conference on Artificial Intelligence, Seattle, WA, USA, vol. 1, pp. 362–367, 31 July – 4 August 1994. AAAI Press/The MIT Press (1994)

    Google Scholar 

  43. Schulte, C., Tack, G.: Weakly monotonic propagators. In: Gent, I.P. (ed.) CP 2009. LNCS, vol. 5732, pp. 723–730. Springer, Heidelberg (2009). https://doi.org/10.1007/978-3-642-04244-7_56

    Chapter  Google Scholar 

  44. Simonis, H., Davern, P., Feldman, J., Mehta, D., Quesada, L., Carlsson, M.: A generic visualization platform for CP. In: Cohen, D. (ed.) CP 2010. LNCS, vol. 6308, pp. 460–474. Springer, Heidelberg (2010). https://doi.org/10.1007/978-3-642-15396-9_37

    Chapter  Google Scholar 

  45. Sutton, M.: Fuzzing: Brute Force Vulnerability Discovery. Addison-Wesley Professional, Reston (2007)

    Google Scholar 

  46. Team, G.: Gecode: generic constraint development environment (2008). https://www.gecode.org/

  47. Coq development team, T.: The coq proof assistant (2019). https://coq.inria.fr/

  48. Vilím, P.: Timetable edge finding filtering algorithm for discrete cumulative resources. In: Achterberg, T., Beck, J.C. (eds.) CPAIOR 2011. LNCS, vol. 6697, pp. 230–245. Springer, Heidelberg (2011). https://doi.org/10.1007/978-3-642-21311-3_22

    Chapter  MATH  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Université Catholique de Louvain, Ottignies-Louvain-la-Neuve, BE, Belgium

    Xavier Gillard, Pierre Schaus & Yves Deville

Authors
  1. Xavier Gillard
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Pierre Schaus
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Yves Deville
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding authors

Correspondence to Xavier Gillard , Pierre Schaus or Yves Deville .

Editor information

Editors and Affiliations

  1. INRA, Castanet Tolosan, France

    Thomas Schiex

  2. INRA, Castanet Tolosan, France

    Simon de Givry

Rights and permissions

Reprints and permissions

Copyright information

© 2019 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Gillard, X., Schaus, P., Deville, Y. (2019). SolverCheck: Declarative Testing of Constraints. In: Schiex, T., de Givry, S. (eds) Principles and Practice of Constraint Programming. CP 2019. Lecture Notes in Computer Science(), vol 11802. Springer, Cham. https://doi.org/10.1007/978-3-030-30048-7_33

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-30048-7_33

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-30047-0

  • Online ISBN: 978-3-030-30048-7

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation