Skip to main content
SpringerLink
Log in

Coverage metrics for temporal logic model checking*

  • Published:
Formal Methods in System Design Aims and scope Submit manuscript

Abstract

In formal verification, we verify that a system is correct with respect to a specification. Even when the system is proved to be correct, there is still a question of how complete the specification is, and whether it really covers all the behaviors of the system. In this paper we study coverage metrics for model checking. Coverage metrics are based on modifications we apply to the system in order to check which parts of it were actually relevant for the verification process to succeed. We introduce two principles that we believe should be part of any coverage metric for model checking: a distinction between state-based and logic-based coverage, and a distinction between the system and its environment. We suggest several coverage metrics that apply these principles, and we describe two algorithms for finding the non-covered parts of the system under these definitions. The first algorithm is a symbolic implementation of a naive algorithm that model checks many variants of the original system. The second algorithm improves the naive algorithm by exploiting overlaps in the variants. We also suggest a few helpful outputs to the user, once the non-covered parts are found.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2

Similar content being viewed by others

Notes

  1. The approach in [25] also has some technical and computational drawbacks: the specification considered is the (big) conjunction of all the properties the system should satisfy, the complexity of the algorithm is exponential in the specification (for ϕ in ACTL), and it is restricted to universal safety specifications whose tableaux have no fairness constraints.

  2. The restricted syntax of acceptable ACTL and the observability transformation force \(\tilde{K}_{{ w},q}\), for all states w, to satisfy ϕ in exactly the same way K does. For example, if a path in K satisfies αUβ by fulfilling β in the present, this path is expected to satisfy β in the present also in \(\tilde{K}_{{ w},q}\). This restriction is what makes the algorithm so efficient.

  3. The logic-based definition of coverage is closely related to the notion of observability don't care conditions as presented in [16]. In [16], there is a set \(C' \subseteq C\) of control signals and an assignment \(\alpha: C' \rightarrow \{ {\bf true,false} \}\) for the signals in C′, such that the behavior of all states of the system that agree with α about the assignment to the control signals in C′ is the same (in terms of the assignment to the output signals and the transition function). Thus, in these states the value of the control signals from \(C \setminus C'\) is “don't care”.

  4. The algorithm for computing the set of q-covered states in [21] runs in time \(O(|K| \cdot |\varphi|)\). As we discuss in Section 1, however, the algorithm calculates the set of q-covered states according to a different definition of coverage, which we found less intuitive, and it handles only a restricted subset of ACTL.

  5. The sizes |C n ∊| and |C n | are both discrete random variables. The value \({\bf E}(|C_n^\epsilon| / |C_n|)\) denotes the expected ratio between the shrunken and original circuit; that is, it the average ratio, where average takes into an account the probability of the probabilistic events.

  6. Note that this is different from the case where x can be fixed to both 0 and to 1. Consider, for example, a circuit \({\cal S}\) and a state s such that all the successors of s satisfy p and all the successors of \(\mathit{twin_x(s)}\) do not satisfy p. While we can fix x to 0 and to 1 without violating the satisfaction of the CTL specification \(AX(AXp \vee AX\neg{p})\) in a state s 0 that goes to both s and \(\mathit{twin_x(s)}\), we cannot replace x with don't care. Indeed, combining s with s′ in the domain of ρ results in a structure in which the successors of s 0 have successors that do not agree on the satisfaction of p.

  7. Note that this is different from the don't-care states in [21], which just removes from the set of q-covered states the states in which the label of q is don't care. Indeed, our definition takes into account the fact that q being don't care in one state may influence the q-coverage of other states.

References

  1. R. Armoni, L. Fix, A. Flaisher, O. Grumberg, N. Piterman, A. Tiemeyer, and M.Y. Vardi, “Enhanced vacuity detection in linear temporal logic,” in Proceedings of 15th International Conference in Computer Aided Verification (CAV), vol. 2725 of Lecture Notes in Computer Science, Springer, 2003. pp. 368–380.

  2. D. Beaty and R. Bryant, “Formally verifying a microprocessor using a simulation methodology,” in Proc. 31st Design Automation Conference, IEEE Computer Society, 1994, pp. 596–602.

  3. I. Beer, S. Ben-David, C. Eisner, and Y. Rodeh, “Efficient detection of vacuity in ACTL formulas,” Form. Meth. in Sys. Des., Vol. 18, No. 2, pp. 141–163, 2001.

  4. J.P. Bergmann and M.A. Horowitz, “Improving coverage analysis and test generation for large designs,” In IEEE International Conference for Computer-Aided Design, Nov. 1999, pp. 580–584.

  5. D. Bustan, A. Flaisher, O. Grumberg, O. Kupferman, and M.Y. Vardi, “Regular vacuity,” in 13th Advanced Research Working Conference on Correct Hardware Design and Verification Methods, volume 3725 of Lecture Notes in Computer Science, Springer-Verlag, 2005, pp. 191–206.

  6. K.-T. Cheng and A. Krishnakumar, “Automatic functional test generation using the extended finite state machine model,” in Proceedings of 30th Design Automation Conference, June 1993, pp. 86–91.

  7. H. Chockler, O. Kupferman, and M.Y. Vardi, “Coverage metrics for formal verification. charme 2003: 111–125,” in Proceedings of 12th Advanced Research Working Conference on Correct Hardware Design and Verification Methods (CHARME), number 2860 in Lecture Notes in Computer Science, Springer-Verlag, 2003, pp. 111–125.

  8. E.M. Clarke and E.A. Emerson, “Design and synthesis of synchronization skeletons using branching time temporal logic,” in Proc. Workshop on Logic of Programs, vol. 131 of Lecture Notes in Computer Science, Springer-Verlag, 1981, pp. 52–71.

  9. E.M. Clarke, O. Grumberg, and D.E. Long, “Model checking and abstraction,” in Proc. 19th ACM Symp. on Principles of Programming Languages, Albuquerque, New Mexico, Jan. 1992, pp. 343–354.

  10. E.M. Clarke, O. Grumberg, K.L. McMillan, and X. Zhao, “Efficient generation of counterexamples and witnesses in symbolic model checking,” in Proc. 32nd Design Automation Conference, IEEE Computer Society, 1995, pp. 427–432.

  11. E.M. Clarke, O. Grumberg, and D. Peled, Model Checking. MIT Press, 1999.

  12. S. Devadas, A. Ghosh, and K. Keutzer, “An observability-based code coverage metric for functional simulation,” in Proceedings of the International Conference on Computer-Aided Design, Nov. 1996, pp. 418–425.

  13. E.A. Emerson, “Temporal and modal logic,” Handbook of Theoretical Computer Science, 1990, pp. 997–1072.

  14. F. Fallah, P. Ashar, and S. Devadas, “Simulation vector generation from hdl descriptions for observability enhanced-statement coverage,” in Proceedings of the 36th Design Automation Conference, June 1999, pp. 666–671.

  15. F. Fallah, S. Devadas, and K. Keutzer, “OCCOM: Efficient computation of observability-based code coverage metrics for functional simulation,“ in Proceedings of the 35th Design Automation Conference, June 1998, pp. 152–157.

  16. G.D. Hachtel and F. Somenzi, Logic Synthesis and Verification Algorithms. Kluwer Academic Publishers, Norwell, Massachusetts, 1996.

  17. D. Harel and A. Pnueli, “On the development of reactive systems,” in K. Apt (Ed.) Logics and Models of Concurrent Systems, vol. F-13 of NATO Advanced Summer Institutes, Springer-Verlag, 1985, pp. 477–498.

  18. R.C. Ho and M.A. Horowitz, “Validation coverage analysis for complex digital designs,” in International Conference on Computer Aided Design, Nov. 1996, pp. 146–151.

  19. R. Ho, C. Yang, M. Horowitz, and D. Dill, “Architecture validation for processors,” in Proceedings of the 22nd Annual Symp. on Computer Architecture, June 1995, pp. 404–413.

  20. C.A.R. Hoare, Communicating Sequential Processes. Prentice-Hall, 1985.

  21. Y. Hoskote, T. Kam, P.-H Ho, and X. Zhao, “Coverage estimation for symbolic model checking,” in Proc. 36th Design automation conference, 1999, pp. 300–305.

  22. Y. Hoskote, D. Moundanos, and J. Abraham, “Automatic extraction of the control flow machine and application to evaluating coverage of verification vectors,” in Proceedings of ICDD, Oct. 1995, pp. 532–537.

  23. N. Jayakumar, M. Purandare, and F. Somenzi, “Dos and don'ts of ctl state coverage estimation,” in Proc. 40th Design Automation Conference (DAC), IEEE Computer Society, 2003.

  24. M. Kantrowitz and L. Noack, “I'm done simulating: Now what? verification coverage analysis and correctness checking of the dec chip 21164 alpha microprocessor,” in Proceedings of Design Automation Conference, June 1996, pp. 325–330.

  25. S. Katz, D. Geist, and O. Grumberg, “Have I written enough properties ?” A method of comparison between specification and implementation,” in 10th Advanced Research Working Conference on Correct Hardware Design and Verification Methods, Lecture Notes in Computer Science. Springer-Verlag, 1999.

  26. O. Kupferman and M.Y. Vardi, “Vacuity detection in temporal model checking,” International Journal on Software Tools for Technology Transfer (STTT), Vol. 4, No. 2, pp. 224–233, 2003.

  27. O. Kupferman, M.Y. Vardi, and P. Wolper, “An automata-theoretic approach to branching-time model checking,” J. ACM, Vol. 47, No. 2, pp. 312–360, March 2000.

  28. R.P. Kurshan, FormalCheck User's Manual. Cadence Design, Inc., 1998.

  29. O. Lichtenstein and A. Pnueli, “Checking that finite state concurrent programs satisfy their linear specification,” in Proc. 12th ACM Symp. on Principles of Programming Languages, New Orleans, Jan. 1985, pp. 97–107.

  30. Z. Manna and A. Pnueli, The Temporal Logic of Reactive and Concurrent Systems: Specification. Springer-Verlag, Berlin, Jan. 1992.

  31. D. Moumdanos, J.A. Abraham, and Y.V. Hoskote, “Abstraction techniques for validation coverage analysis and test generation,” IEEE Trans. Comp., Jan. 1998.

  32. R.G. Nigmatulin, The Complexity of Boolean Functions. Nauka, Main Editorial Board for Physical and Mathematical Literature, Moscow, 1990.

  33. M. Purandare and F. Somenzi, “Vacuum cleaning ctl formulae,” in Proceedings of 14th International Conference in Computer Aided Verification (CAV), Lecture Notes in Computer Science, Springer, 2002. pp. 485–499.

  34. J.P. Queille and J. Sifakis, “Specification and verification of concurrent systems in Cesar,” in Proc. 5th International Symp. on Programming, vol. 137 of Lecture Notes in Computer Science, Springer-Verlag, 1981. pp. 337–351.

  35. I. Wegener, The Complexity of Boolean Functions. John Wiley & Sons, Chichester, 1987.

Download references

Acknowledgment

We thank Orna Grumberg, Yatin Hoskote, Amir Pnueli, and Uri Zwick for helpful discussions. Supported in part by NSF grant CCR-9700061, NSF grant CCR-9988322, and by a grant from the Intel Corporation.

Author information

Authors and Affiliations

  1. IBM Haifa Research Laboratory, Mount Carmel, Haifa, 31905, Israel

    Hana Chockler

  2. Hebrew University, School of Engineering and Computer Science, Jerusalem, 91904, Israel

    Orna Kupferman

  3. Department of Computer Science, Rice University, Houston, TX, 77251-1892, U.S.A.

    Moshe Y. Vardi

Authors
  1. Hana Chockler
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Orna Kupferman
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Moshe Y. Vardi
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Hana Chockler.

Additional information

*A preliminary version appears in the Proceedings of the 7th International Conference on Tools and algorithms for the construction and analysis of systems, 2001.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Chockler, H., Kupferman, O. & Vardi, M.Y. Coverage metrics for temporal logic model checking* . Form Method Syst Des 28, 189–212 (2006). https://doi.org/10.1007/s10703-006-0001-6

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10703-006-0001-6

Keywords

Search

Navigation