Skip to main content
Springer Nature Link
Log in

Sanity Checks in Formal Verification

  • Conference paper
CONCUR 2006 – Concurrency Theory (CONCUR 2006)

Part of the book series: Lecture Notes in Computer Science ((LNTCS,volume 4137))

Included in the following conference series:

Abstract

One of the advantages of temporal-logic model-checking tools is their ability to accompany a negative answer to the correctness query by a counterexample to the satisfaction of the specification in the system. On the other hand, when the answer to the correctness query is positive, most model-checking tools provide no additional information. In the last few years there has been growing awareness to the importance of suspecting the system or the specification of containing an error also in the case model checking succeeds. The main justification of such suspects are possible errors in the modeling of the system or of the specification. The goal of sanity checks is to detect such errors by further automatic reasoning. Two leading sanity checks are vacuity and coverage. In vacuity, the goal is to detect cases where the system satisfies the specification in some unintended trivial way. In coverage, the goal is to increase the exhaustiveness of the specification by detecting components of the system that do not play a role in verification process. For both checks, the challenge is to define vacuity and coverage formally, develop algorithms for detecting vacuous satisfaction and low coverage, and suggest methods for returning to the user helpful information. We survey existing work on vacuity and coverage and argue that, in many aspects, the two checks are essentially the same: both are based on repeating the verification process on some mutant input. In vacuity, mutations are in the specifications, whereas in coverage, mutations are in the system. This observation enables us to adopt work done in the context of vacuity to coverage, and vise versa.

The paper is based on joint work with Hana Chockler, Moshe Y. Vardi, and Robert Kurshan, appearing in [CKV01, CKKV01, CKV03, Cho03].

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Institutional subscriptions

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

Similar content being viewed by others

References

  1. Armoni, R., Fix, L., Flaisher, A., Gerth, R., Ginsburg, B., Kanza, T., Landver, A., Mador-Haim, S., Singerman, E., Tiemeyer, A., Vardi, M.Y., Zbar, Y.: The ForSpec temporal logic: A new temporal property-specification logic. In: Katoen, J.-P., Stevens, P. (eds.) TACAS 2002. LNCS, vol. 2280, pp. 296–296. Springer, Heidelberg (2002)

    Chapter  Google Scholar 

  2. Armoni, R., Fix, L., Flaisher, A., Grumberg, O., Piterman, N., Tiemeyer, A., Vardi, M., Y.: Enhanced vacuity detection in linear temporal logic. In: Hunt Jr., W.A., Somenzi, F. (eds.) CAV 2003. LNCS, vol. 2725, pp. 368–380. Springer, Heidelberg (2003)

    Chapter  Google Scholar 

  3. Beatty, D., Bryant, R.: Formally verifying a microprocessor using a simulation methodology. In: Proc. 31st Design Automation Conference, pp. 596–602. IEEE Computer Society Press, Los Alamitos (1994)

    Google Scholar 

  4. Beer, I., Ben-David, S., Eisner, C., Fisman, D., Gringauze, A., Rodeh, Y.: The temporal logic Sugar. In: Berry, G., Comon, H., Finkel, A. (eds.) CAV 2001. LNCS, vol. 2102, pp. 363–367. Springer, Heidelberg (2001)

    Chapter  Google Scholar 

  5. Beer, I., Ben-David, S., Eisner, C., Rodeh, Y.: Efficient detection of vacuity in ACTL formulas. Formal Methods in System Design 18(2), 141–162 (2001)

    Article  MATH  Google Scholar 

  6. Biere, A., Cimatti, A., Clarke, E.M., Fujita, M., Zhu, Y.: Symbolic model checking using SAT procedures instead of BDDs. In: Proc. 36th Design Automation Conference, pp. 317–320. IEEE Computer Society Press, Los Alamitos (1999)

    Google Scholar 

  7. Burch, J.R., Clarke, E.M., McMillan, K.L., Dill, D.L., Hwang, L.J.: Symbolic model checking: 1020 states and beyond. Information and Computation 98(2), 142–170 (1992)

    Article  MathSciNet  MATH  Google Scholar 

  8. Bening, L., Foster, H.: Principles of verifiable RTL design – a functional coding style supporting verification processes. Kluwer Academic Publishers, Dordrecht (2000)

    MATH  Google Scholar 

  9. Bustan, D., Flaisher, A., Grumberg, O., Kupferman, O., Vardi, M.Y.: Regular Vacuity. In: Borrione, D., Paul, W. (eds.) CHARME 2005. LNCS, vol. 3725, pp. 191–206. Springer, Heidelberg (2005)

    Chapter  Google Scholar 

  10. Bruns, G., Godefroid, P.: Temporal logic query checking. In: Proceedings of the 16th Annual IEEE Symposium on Logic in Computer Science (LICS 2001), June 16–19, 2001, pp. 409–420. IEEE Computer Society, Los Alamitos (2001)

    Chapter  Google Scholar 

  11. Godefroid, P., Bruns, G.: Model Checking with Multi-valued Logics. In: Díaz, J., Karhumäki, J., Lepistö, A., Sannella, D. (eds.) ICALP 2004. LNCS, vol. 3142, pp. 281–293. Springer, Heidelberg (2004)

    Chapter  Google Scholar 

  12. Chechik, M., Gurfinkel, A.: Extending extended vacuity. In: Hu, A.J., Martin, A.K. (eds.) FMCAD 2004. LNCS, vol. 3312, pp. 306–321. Springer, Heidelberg (2004)

    Chapter  Google Scholar 

  13. Chechik, M., Gurfinkel, A.: How Vacuous Is Vacuous? In: Jensen, K., Podelski, A. (eds.) TACAS 2004. LNCS, vol. 2988, pp. 451–466. Springer, Heidelberg (2004)

    Chapter  Google Scholar 

  14. Clarke, E.M., Grumberg, O., Long, D.: Verification tools for finite-state concurrent systems. In: de Bakker, J.W., de Roever, W.-P., Rozenberg, G. (eds.) REX 1993. LNCS, vol. 803, pp. 124–175. Springer, Heidelberg (1994)

    Google Scholar 

  15. Clarke, E.M., Grumberg, O., McMillan, K.L., Zhao, X.: Efficient generation of counterexamples and witnesses in symbolic model checking. In: Proc. 32nd Design Automation Conference, pp. 427–432. IEEE Computer Society Press, Los Alamitos (1995)

    Chapter  Google Scholar 

  16. Chan, W.: Temporal-logic queries. In: Emerson, E.A., Sistla, A.P. (eds.) CAV 2000. LNCS, vol. 1855, pp. 450–463. Springer, Heidelberg (2000)

    Chapter  Google Scholar 

  17. Chockler, H.: Coverage metrics for model checking. PhD thesis, Hebrew University, Jerusalem, Israel (2003)

    Google Scholar 

  18. Chockler, H., Kupferman, O.: Coverage of implementations by simulating specifications. In: Baeza-Yates, R.A., Montanari, U., Santoro, N. (eds.) Proceedings of 2nd IFIP International Conference on Theoretical Computer Science, Montreal, Canada, August 2002. IFIP Conference Proceedings, vol. 223, pp. 409–421. Kluwer Academic Publishers, Dordrecht (2002)

    Google Scholar 

  19. Chockler, H., Kupferman, O., Kurshan, R.P., Vardi, M.Y.: A practical approach to coverage in model checking. In: Berry, G., Comon, H., Finkel, A. (eds.) CAV 2001. LNCS, vol. 2102, pp. 66–78. Springer, Heidelberg (2001)

    Chapter  Google Scholar 

  20. Chockler, H., Kupferman, O., Vardi, M.Y.: Coverage metrics for temporal logic model checking. In: Margaria, T., Yi, W. (eds.) TACAS 2001. LNCS, vol. 2031, pp. 528–542. Springer, Heidelberg (2001)

    Chapter  Google Scholar 

  21. Vardi, M.Y., Kupferman, O., Chockler, H.: Coverage Metrics for Formal Verification. In: Geist, D., Tronci, E. (eds.) CHARME 2003. LNCS, vol. 2860, pp. 111–125. Springer, Heidelberg (2003)

    Chapter  Google Scholar 

  22. Dill, D.L.: What’s between simulation and formal verification? In: Proc. 35st Design Automation Conference, pp. 328–329. IEEE Computer Society Press, Los Alamitos (1998)

    Google Scholar 

  23. Hardin, R.H., Har’el, Z., Kurshan, R.P.: COSPAN. In: Alur, R., Henzinger, T.A. (eds.) CAV 1996. LNCS, vol. 1102, pp. 423–427. Springer, Heidelberg (1996)

    Google Scholar 

  24. Hoskote, Y., Kam, T., Ho, P.-H., Zhao, X.: Coverage estimation for symbolic model checking. In: Proc. 36th Design automation conference, pp. 300–305 (1999)

    Google Scholar 

  25. Katz, S., Grumberg, O., Geist, D.: Have I Written Enough Properties? - a method of comparison between specification and implementation. In: Pierre, L., Kropf, T. (eds.) CHARME 1999. LNCS, vol. 1703, pp. 280–297. Springer, Heidelberg (1999)

    Chapter  Google Scholar 

  26. Kurshan, R.P.: FormalCheck User’s Manual. Cadence Design, Inc. (1998)

    Google Scholar 

  27. Kupferman, O., Vardi, M.Y.: Vacuity detection in temporal model checking. Journal on Software Tools For Technology Transfer 4(2), 224–233 (2003)

    Article  Google Scholar 

  28. Namjoshi, K.S.: Certifying Model Checkers. In: Berry, G., Comon, H., Finkel, A. (eds.) CAV 2001. LNCS, vol. 2102, pp. 2–13. Springer, Heidelberg (2001)

    Chapter  Google Scholar 

  29. Namjoshi, K.S.: An efficiently checkable, proof-based formulation of vacuity in model checking. In: Alur, R., Peled, D.A. (eds.) CAV 2004. LNCS, vol. 3114, pp. 57–69. Springer, Heidelberg (2004)

    Chapter  Google Scholar 

  30. Peled, D.: Software Reliability Methods. Springer, Heidelberg (2001)

    MATH  Google Scholar 

  31. Purandare, M., Somenzi, F.: Vacuum cleaning CTL formulae. In: Brinksma, E., Larsen, K.G. (eds.) CAV 2002. LNCS, vol. 2404, pp. 485–499. Springer, Heidelberg (2002)

    Chapter  Google Scholar 

  32. Tasiran, S., Keutzer, K.: Coverage metrics for functional validation of hardware designs. IEEE Design and Test of Computers 18(4), 36–45 (2001)

    Article  Google Scholar 

  33. Verisity. Surecove’s code coverage technology (2003), http://www.verisity.com/products/surecov.html

  34. Vardi, M.Y., Wolper, P.: Reasoning about infinite computations. Information and Computation 115(1), 1–37 (1994)

    Article  MathSciNet  MATH  Google Scholar 

  35. Zhu, H., Hall, P.V., May, J.R.: Software unit test coverage and adequacy. ACM Computing Surveys 29(4), 366–427 (1997)

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. School of Engineering and Computer Science, Hebrew University, Jerusalem, 91904, Israel

    Orna Kupferman

Authors
  1. Orna Kupferman
    View author publications

    You can also search for this author in PubMed Google Scholar

Editor information

Editors and Affiliations

  1. Institute for Theoretical Computer Science, Technical University Dresden, Germany

    Christel Baier

  2. INRIA, VASY, Grenoble Rhône-Alpes, France

    Holger Hermanns

Rights and permissions

Reprints and permissions

Copyright information

© 2006 Springer-Verlag Berlin Heidelberg

About this paper

Cite this paper

Kupferman, O. (2006). Sanity Checks in Formal Verification. In: Baier, C., Hermanns, H. (eds) CONCUR 2006 – Concurrency Theory. CONCUR 2006. Lecture Notes in Computer Science, vol 4137. Springer, Berlin, Heidelberg. https://doi.org/10.1007/11817949_3

Download citation

  • DOI: https://doi.org/10.1007/11817949_3

  • Publisher Name: Springer, Berlin, Heidelberg

  • Print ISBN: 978-3-540-37376-6

  • Online ISBN: 978-3-540-37377-3

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics