Skip to main content
SpringerLink
Log in

Benchmarking a model checker for algorithmic improvements and tuning for performance

  • Published:
Formal Methods in System Design Aims and scope Submit manuscript

Abstract

This paper describes a portfolio-based approach for model checking, i.e., an approach in which several model checking engines are orchestrated to reach the best possible performance on a broad and real set of designs. Model checking algorithms are evaluated through experiments, and experimental data inspire package tuning, as well as new algorithmic features and methodologies. This approach, albeit similar to several industrial and academic experiences, and already applied in other domains, is somehow new to the model checking field. Its contributions lie in the description of how we: (1) characterize and classify benchmarks in a dynamic way, throughout experimental runs, (2) relate model checking problems to algorithms and engines, (3) introduce a dynamic tuning of sub-engines, exploiting an on-the-fly performance analysis, (4) record results of different approaches, and sort out heuristics to target different classes of problems. We provide a detailed description of the experiments performed in preparation of the Model Checking Competition 2010, where PdTRAV, our academic verification tool, won the UNSAT division, while ranking second in the OVERALL category.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Bjesse P, Leonard T, Mokkedem A (2001) Finding bugs in an alpha microprocessor using satisfiability solvers. In: Berry G, Comon H, Finkel A (eds) Proc computer aided verification, Paris, France, July 2001. LNCS, vol 2102. Springer, Berlin, pp 454–464

    Chapter  Google Scholar 

  2. Biere A, Klaessen KL (2010) The hardware model checking competition web page. http://fmv.jku/hwmcc10

  3. Rice JR (1976) The algorithm selection problem. Adv Comput 15:65–118

    Article  Google Scholar 

  4. Huberman B, Lukose R, Hogg T (1997) An economics approach to hard computational problems. Science 275(5296):51–54

    Article  Google Scholar 

  5. Xu L, Hutter F, Hoos HH, Leyton-Brown L (2008) Satzilla: Portfolio-based algorithm selection for sat. J Artif Intell Res 32(1):565–606

    MATH  Google Scholar 

  6. Pulina L, Tacchella A (2009) A self-adaptive multi-engine solver for quantified boolean formulas. Constraints 14(1):80–116

    Article  MathSciNet  MATH  Google Scholar 

  7. Gordon M (1989) Mechanizing programming logics in higher order logic. In: Current trends in hardware verification and automated theorem proving. Springer, Berlin

    Google Scholar 

  8. Srivas M, RueßH, Cyrluk D (1997) Hardware verification using PVS. In: Formal hardware verification: methods and systems in comparison. Springer, Berlin

    Google Scholar 

  9. Kaufmann M, Manolios P, Moore JS (2000) Computer-aided reasoning: an approach. Kluwer Academic, Dordrecht

    Book  Google Scholar 

  10. Kamhi G, Fix L, Binyamini Z (1998) Symbolic model checking visualization. In: Proceedings of the second international conference on formal methods in computer-aided design. FMCAD ’98, London, UK. Springer, Berlin, pp 290–303

    Google Scholar 

  11. Mony H, Baumgartner J, Paruthi V, Kanzelman R, Kuehlmann A (2004) Scalable automated verification via expert-system guided transformations. In: Proc formal methods in computer-aided design. Springer, Berlin, pp 159–173

    Chapter  Google Scholar 

  12. Cabodi G, Camurati P, Quer S (1994) Detecting hard faults with combined approximate forward/backward symbolic techniques. In: Proc IEEE ISCAS’94, London, UK, May 1994, pp 25–30

    Google Scholar 

  13. Biere A, Cimatti A, Clarke EM, Fujita M, Zhu Y (1999) Symbolic model checking using SAT procedures instead of BDDs. In: Proc 36th design automation conf, New Orleans, Louisiana, June 1999. IEEE Computer Society, Los Alamitos, pp 317–320

    Google Scholar 

  14. Sheeran M, Singh S, Stålmarck G (2000) Checking safety properties using induction and a SAT solver. In: Hunt WA, Johnson SD (eds) Proc formal methods in computer-aided design, Austin, TX, USA, November 2000. LNCS, vol 1954. Springer, Berlin, pp 108–125

    Google Scholar 

  15. Bjesse P, Claessen K (2000) SAT-based verification without state space traversal. In: Proc formal methods in computer-aided design, Austin, TX, USA, LNCS, vol 1954. Springer, Berlin

    Google Scholar 

  16. McMillan KL (2003) Interpolation and SAT-based model checking. In: Hunt WA Jr, Somenzi F (eds) Proc computer aided verification, Boulder, CO, USA, LNCS, vol 2725. Springer, Berlin, pp 1–13

    Chapter  Google Scholar 

  17. Cabodi G, Murciano M, Nocco S, Quer S (2006) Stepping forward with interpolants in unbounded model checking. In: Proc int’l conf on computer-aided design, San Jose, California, November 2006. ACM Press, New York, pp 772–778

    Google Scholar 

  18. Cabodi G, Camurati P, Murciano M (2008) Automated abstraction by incremental refinement in interpolant-based model checking. In: Proc int’l conf on computer-aided design, San Jose, California, November 2008. ACM Press, New York, pp 129–136

    Google Scholar 

  19. Cabodi G, Murciano M, Nocco S, Quer S (2008) Boosting interpolation with dynamic localized abstraction and redundancy removal. ACM Trans Des Autom Electron Syst 13(1):309–340

    Article  Google Scholar 

  20. Cabodi G, Camurati P, Garcia L, Murciano M, Nocco S, Quer S (2008) Trading-off SAT search and variable quantifications for effective unbounded model checking. In: Proc formal methods in computer-aided design, Portland, Oregon, November 2008, pp 205–212

    Google Scholar 

  21. Cabodi G, Garcia LA, Murciano M, Nocco S, Quer S (2010) Partitioning interpolant-based verification for effective unbounded model checking. IEEE Trans Comput-Aided Des Integr Circuits Syst 29(3):382–395

    Article  Google Scholar 

  22. Cabodi G, Nocco S, Quer S (2009) Strengthening model checking techniques with inductive invariants. IEEE Trans Comput-Aided Des Integr Circuits Syst 28(1):154–158

    Article  Google Scholar 

  23. Biere A, Jussila T (2007) The hardware model checking competition web page. http://fmv.jku.at/hwmcc07

  24. Berkeley Logic Interchange Format Technical report, September 1996

  25. The AIGER format. http://fmv.jku.at/aiger/

  26. Cabodi G, Camurati P, Quer S (2000) Symbolic forward/backward traversals of large finite state machines. Euromicro J 46(12):1137–1158

    Google Scholar 

  27. Cabodi G, Nocco S, Quer S (2002) Mixing forward and backward traversals in guided-prioritized BDD-based verification. In: Brinksma E, Larsen KG (eds) Proc computer aided verification, Copenhagen, Denmark, July 2002. LNCS, vol 2102. Springer, Berlin, pp 471–484

    Chapter  Google Scholar 

  28. Cabodi G, Camurati P, Quer S (1999) Improving the efficiency of BDD-based operators by means of partitioning. IEEE Trans Comput-Aided Des Integr Circuits Syst 18(5):545–556

    Article  Google Scholar 

  29. Cabodi G (2001) Meta-BDDs: a decomposed representation for layered symbolic manipulation of boolean functions. In: Berry G, Comon H, Finkel A (eds) Proc computer aided verification, Paris, France, July 2001. LNCS, vol 2102. Springer, Berlin, pp 118–130

    Chapter  Google Scholar 

  30. Cabodi G, Nocco S, Quer S (2007) Boosting the role of inductive invariants in model checking. In: Proc design automation & test in Europe conf, Nice, France, April 2007. IEEE Computer Society, Los Alamitos

    Google Scholar 

  31. Kuehlmann A, Ganai MK, Paruthi V (2001) Circuit-based Boolean reasoning. In: Proc design automation conference, Las Vegas, Nevada, June 2001. IEEE Computer Society, Los Alamitos

    Google Scholar 

  32. Cabodi G, Crivellari M, Nocco S, Quer S (2005) Circuit based quantification: back to state set manipulation within unbounded model checking. In: Proc design automation & test in Europe conf, Munich, Germany, March 2005. IEEE Computer Society, Alamitos

    Google Scholar 

  33. Brayton R, Mishchenko A (2010) Abc: an academic industrial-strength verification tool

  34. Baumgartner J, Mony H, Paruthi V, Kanzelman R, Janssen G (2006) Scalable sequential equivalence checking across arbitrary design transformations. In: Proc formal methods in computer-aided design

    Google Scholar 

  35. Een N, Mishchenko A, Amla N (2010) A single-instance incremental SAT formulation of proof and counterexample abstraction. In: Proc int’l workshop on logic synthesis, May 2010

    Google Scholar 

  36. Coudert O, Berthet C, Madre JC (1989) Verification of sequential machines based on symbolic execution. In: LNCS, vol 407. Springer, Berlin, pp 365–373

    Google Scholar 

  37. Burch JR, Clarke EM, McMillan KL, Dill DL, Hwang LJ (1990) Symbolic model checking: 1020 states and beyond. In: Proc symposium on logic in computer science, Washington, DC, June 1990. Springer, Berlin, pp 428–439

    Chapter  Google Scholar 

  38. Craig W (1957) Three uses of the Herbrand-Gentzen theorem in relating model theory and proof theory. J Symb Log 22(3):269–285

    Article  MathSciNet  MATH  Google Scholar 

  39. Pudlák P (1997) Lower bounds for resolution and cutting plane proofs and monotone computations. J Symb Log 62(3):981–998

    Article  MATH  Google Scholar 

  40. Somenzi F (2005) CUDD: CU decision diagram package—release 2.4.1. http://vlsi.colorado.edu/~fabio/CUDD/

  41. Brayton RK et al. (1996) VIS: a system for verification and synthesis. In: Alur R, Henzinger TA (eds) Proc computer aided verification. LNCS, vol 1102. Springer, Berlin, pp 428–432

    Google Scholar 

  42. Eén N, Sörensson N (2009) The Minisat SAT solver, April 2009. http://minisat.se

  43. Mishchenko A, Chatterjee S, Brayton RK (2005) FRAIGs: a unifying representation for logic synthesis and verification. Technical report, EECS Dept, UC Berkeley, March 2005

  44. Biere A, Jussila T (2008) The hardware model checking competition web page. http://fmv.jku.at/hwmcc08

  45. Cabodi G, Camurati P, Garcia L, Murciano M, Nocco S, Quer S (2009) Speeding up model checking by exploiting explicit and hidden verification constraints. In: Proc design automation & test in Europe conf, April 2009, pp 1686–1691

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Dip. di Automatica e Informatica, Politecnico di Torino, Turin, Italy

    Gianpiero Cabodi, Sergio Nocco & Stefano Quer

Authors
  1. Gianpiero Cabodi
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Sergio Nocco
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Stefano Quer
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Stefano Quer.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Cabodi, G., Nocco, S. & Quer, S. Benchmarking a model checker for algorithmic improvements and tuning for performance. Form Methods Syst Des 39, 205–227 (2011). https://doi.org/10.1007/s10703-011-0123-3

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10703-011-0123-3

Keywords

Search

Navigation