Skip to main content
SpringerLink
Log in

The incorporation of testing into formal verification: Direct, modular, and hierarchical correctness degrees

  • Published:
Formal Methods in System Design Aims and scope Submit manuscript

Abstract

We describe a paradigm for the incorporation of software or hardware testing into formal verification and give some preliminary results. The “correctness degree” is essentially the fraction of computations of a program (or other computational description) that satisfy a given specification (this fraction can be weighted to account for a non-uniform distribution of inputs in the application environment.) We give some results about using test sets to establish correctness degrees, and we indicate how to find the correctness degree of a system composed of partially verified, partially tested modules.

In addition we discuss how to combine probabilistic correctness results at different levels of abstraction in the computer hierarchy.

We believe that this proposed methodology has the potential to help achieve rigorous estimates of correctness in cases that have proven elusive for pure verification.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. M. Abadi and L. Lamport, “Composing specifications.” ACM Transactions on Programming Languages and Systems, vol. 15, no. 1, pp. 73–132, January 1993.

    Google Scholar 

  2. E. W. Adams, Probability and the logic of conditionals. In Hintikka and Suppes, editors, Aspects of Inductive Logic, pp. 265–316. North-Holland, 1966.

  3. W. R. Adrion, M.A. Branstad, and J. C. Cherniavsky, “Validation, verification and testing of computer software.” Computing Surveys, ACM, vol. 14, no. 2, pp. 159–192, June 1982.

    Google Scholar 

  4. J. Alilovic-Curgus and S. T. Vuong, A metric based theory of test selection and coverage. In Proceedings of the IFIP WG 6.I Twelfth International Symposium on Protocol Specification, Testing, and Verification, 1993. North-Holland, 1993.

  5. F. B. Bastani, “On the uncertainty in the correctness of computer programs.” IEEE Transactions on Software Engineering, vol. SE-11, no. 9, pp. 857–864, September 1985.

    Google Scholar 

  6. G. Bernot, Testing against formal specifications: a theoretical view. In TAPSOFT 91, pages 99–119. Springer-Verlag, 1991. Lecture Notes in Computer Science, Volume 494.

  7. G. Bernot, M. Claude Gaudel, and B. Marre, “Software testing based on formal specifications: a theory and a tool.” Software Engineering Journal, pp. 387–405, November 1991.

  8. M. Blum and P. Raghavan, “Program correctness: Can one test for it?” In G. X. Ritter, editor, Information Processing 89, pp. 127–134. Elsevier Science Publishers, 1989.

  9. L. Bouge, “A contribution to the theory of program testing.” Theoretical Computer Science, vol. 37, pp. 151–181, 1985.

    Google Scholar 

  10. E. Brinksma, “A theory for the derivation of tests.” In S. Aggarwal and K. Sabnani, editors, Protocol Specification, Testing, and Verification VIII, pp. 63–74. Elsevier Science Publishers, 1988.

  11. S. Brocklehurst and B. Littlewood, “New ways to get accurate reliability measures.” IEEE Software, vol. 9, pp. 34–42, July 1992.

    Google Scholar 

  12. T.A. Budd and D. Angluin, “Two notions of correctness and their relation to testing.” Technical Report TR-80-19, Department of Computer Science, The University of Arizona, June 1980.

  13. R.C. Cheung, “A user-oriented software reliability model.” IEEE Transactions on Software Engineering, vol. SE-6, no. 2, pp. 118–125, March 1980.

    Google Scholar 

  14. J. V. Cook, I. V. Filippenko, B. H. Levy, L. G. Marcus, and T. K. Menas, “Formal Computer Verification in the State Delta Verification System (SDVS).” In Proceedings of the AIAA Computing in Aerospace Conference, pp. 77–87, Baltimore, Maryland, October 1991. American Institute of Aeronautics and Astronautics.

    Google Scholar 

  15. C. J. Dahl, E. W. Dijkstra, and C. A. R. Hoare, Structured Programming. Academic Press, London, 1972.

    Google Scholar 

  16. M. Geller, “Test data as an aid in proving program correctness.” Communications of the ACM, vol. 21, pp. 368–375, May 1978.

    Google Scholar 

  17. P., Gemmell and M., Harchol, “Tight bounds on expected time to add correctly and add mostly correctly.” Technical Report UCB/CSD 93/737, Computer Science Division, UC Berkeley, April 1993.

    Google Scholar 

  18. S. Gerhart, “Program validation.” In T. Anderson and B. Randell, editors, Computing Systems Reliability, pp. 66–108. Cambridge University Press, 1979.

  19. S.L. Gerhart, “A broad spectrum toolset for upstream testing, verification, and analysis.” In Workshop on Software Testing, Verification, and Analysis, pp. 4–12. IEEE, 1988.

  20. J.B. Goodenough and S. L. Gerhart, “Toward a theory of test data selection.” IEEE Transactions on Software Engineering, vol. SE-1, no. 2, pp. 156–173, June 1975.

    Google Scholar 

  21. J.S. Gourlay, “A mathematical framework for the investigation of testing. IEEE Transactions on Software Engineering, vol. SE-9, no. 6, pp. 686–709, November 1983.

    Google Scholar 

  22. G. Corporation, The Annotated Ada Reference Manual (ANSI/MIL-STD-1815A-1983), June 1989.

  23. J. Y. Halpern, “An analysis of first-order logics of probability.” Artificial Intelligence, vol. 46, pp. 311–350, 1990.

    Google Scholar 

  24. D. Hamlet, “Testing for probable correctness.” In Proceedings of the '86 Workshop on Software Testing, pp. 92–97. IEEE, 1986.

  25. D. Hamlet and R. Taylor. “Partition testing does not inspire confidence.” IEEE Transactions on Software Engineering, vol. SE-16, no. 12, pp. 1402–1411, December 1990.

    Google Scholar 

  26. R. Hamlet, “A patent problem for abstract programming languages: Machine-independent computations.” in Proceedings of the 4th Annual Symposium on Theory of Computing, pp. 193–197. ACM, 1972.

  27. R. Hamlet, “Testing programs with finite sets of data.” The Computer Journal, vol. 20, pp. 232–237, 1977.

    Google Scholar 

  28. R. Hamlet, “Reliability theory of program testing.” Acta Informatica, vol. 16, pp. 31–43, 1981.

    Google Scholar 

  29. R. Hamlet, “Guest editor's introduction to special section on software testing.” Communications of the ACM, vol. 31, pp. 662–667, June 1988

    Google Scholar 

  30. M. Hennessey and R. Milner, “Algebraic laws for nondeterminism and concurrency.” Journal of the ACM, vol. 332, no. 1, pp. 137–162, 1985.

    Google Scholar 

  31. W. E. Howden, “Reliability of the path analysis testing strategy.” IEEE Transactions on Software Engineering, vol. SE-2, no. 3, pp. 208–215, September 1976.

    Google Scholar 

  32. W. E. Howden, “Algebraic program testing.” Acta Informatica, vol. 10, pp. 53–66, 1978.

    Google Scholar 

  33. W. E. Howden, “Program testing versus proofs of correctness.” Journal of Software Testing, Verification, and Reliability, vol. 1, no. 1, pp. 5–15, 1993.

    Google Scholar 

  34. H. Hungar, “Combining model checking and theorem proving to verify parallel processes.” In Proceedings of the 1993 Workshop on Computer-Aided Verification, June 1993.

  35. IEEE, Standard VHDL Language Reference Manual, 1988. IEEE Std. 1076-1987.

  36. D. Ince, “The validation, verification, and testing of software.” In Oxford Surveys in Information Technology, vol. 2, pp. 1–40. Oxford University Press, 1985.

  37. A. M. Leone, “Selecting an appropriate model for software reliability.” In Proceedings 1988 Annual Reliability and Maintainability Symposium, pp. 208–213. IEEE, 1988.

  38. R. J. Lipton, “New directions in testing.” In Distributed Computing and Cryptography; Proceedings of a DIMACS Workshop, October 4–6, 1989, pp. 191–202. AMS, ACM, 1991.

  39. L. G. Marcus, “The incorporation of formal testing into verification: An introduction to a paradigm and summary of preliminary results.” Technical Report ATR-93(8354)-1, The Aerospace Corporation, September 1993.

  40. K. W. Miller, L. J. Morell, tE. Noonan, S. K. Park, D. M. Nicol, B. W. Murrill, and J. M. Voas, “Estimating the probability of failure when testing reveals no failures.” IEEE Transactions on Software Engineering, vol. SE-18, no. 1, pp. 33–43, January 1992.

    Google Scholar 

  41. H. D. Mills, “Zero defect software.” In Marshall C. Yovits, editor, Advances in Computers, vol. 36, pp. 1–41. Academic Press, Inc., 1993.

  42. T. J. Ostrand, R. Sigal, and E. J. Weyuker, “Design for a tool to manage specification-based testing.” In Proceedings of the 86 Workshop on Software Testing, pp. 41–50. IEEE, 1986.

  43. J. Pearl, “Reasoning under uncertainty.” In Joseph F. Traub, editor, Annual Review of Computer Science, vol. 4, pp. 37–72. Annual Reviews, Inc., 1989–1990.

  44. M. Phalippou, “The limited power of testing.” In Proceedings of the 5th Internation Workshop on Protocol Test Systems: Montreal, September 1992.

  45. D. H. Pitt and D. Freestone. “The derivation of conformance tests from LOTOS specifications.” IEEE Transactions on Software Engineering, vo. SE-16, no. 12, pp. 1337–1343, December 1990.

    Google Scholar 

  46. D. J. Richardson and L. A. Clarke, “Testing techniques based on symbolic evaluation.” In T. Anderson, editor, Software Requirements, Specification, and Testing: Proceedings of CSR Workshop, pp. 93–110. Blackwell Scientific Publications, 1984.

  47. D. J. Richardson and L. A. Clarke, “Partition analysis: A method combining testing and verification.” IEEE Transactions on Software Engineering, vol. SE-11, no. 12, pp. 1477–1490, December 1985.

    Google Scholar 

  48. K.A.R. Romanik, “Approximate testing theory.” Technical Report CS-TR-2988, UMIACS-TR-92–121, University of Maryland, College Park, November 1992.

    Google Scholar 

  49. J. M. Spivey, The Z Notation. Prentice Hall, Hertfordshire, U. K., 1989.

    Google Scholar 

  50. M. Veeraraghavan and K. Trivedi, “A combinatorial algorithm for performance and reliability analysis using multistate models.” To appear, IEEE-TC, 1993.

  51. S. N. Weiss, “Methods of comparing test data adequacy criteria.” In Proceedings of COMPSAC '90-The Fourteenth Annual International Computer Software and Applications Conference, Chicago, IL, October 1990.

  52. S. N. Weiss and E. J. Weyuker, “An extended domain-based model of software reliability.” IEEE Transactions on Software Engineering, vol. SE-14, no. 10, pp. 1512–1524, October 1988.

    Google Scholar 

  53. L. J. White, “Domain testing and several outstanding research problems in program testing.” INFOR, vol. 23, no. 1, pp. 55–68, February 1985.

    Google Scholar 

  54. L. J. White, “Software testing and verification.” In Advances in Computers, vol. 26, pp. 335–391. Academic Press, Inc., 1987.

  55. J. A. Whittaker, Markov Chain Techniques for Software Testing and Reliability Analysis. PhD thesis, University of Tennessee, Knoxville, 1992.

  56. D. M. Woit, “Specifying operational profiles for modules.” In Proceedings ISSTA (International Symposium on Software Testing and Analysis). ACM, June 1993.

  57. P. Zave and M. Jackson, “Conjunctions as composition.” ACM Transactions on Software Engineering and Methodology, vol. 2, no. 4, pp. 379–411, October 1993.

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Computer Systems Division M1-055, The Aerospace Corporation, P.O. Box 92957, 90009, Los Angeles, CA

    Leo Marcus

Authors
  1. Leo Marcus
    View author publications

    You can also search for this author in PubMed Google Scholar

Rights and permissions

Reprints and permissions

About this article

Cite this article

Marcus, L. The incorporation of testing into formal verification: Direct, modular, and hierarchical correctness degrees. Form Method Syst Des 9, 235–261 (1996). https://doi.org/10.1007/BF00122083

Download citation

  • Received:

  • Revised:

  • Issue Date:

  • DOI: https://doi.org/10.1007/BF00122083

Keywords

Search

Navigation