Skip to main content
SpringerLink
Log in

On the effect of test-suite reduction on automatically generated model-based tests

  • Published:
Automated Software Engineering Aims and scope Submit manuscript

Abstract

Model checking techniques can be successfully employed as a test-case generation technique to generate tests from formal models. The number of tests-cases produced, however, is typically large for complex coverage criteria such as MC/DC. Test-suite reduction can provide us with a smaller set of test-cases that preserve the original coverage—often a dramatically smaller set. Nevertheless, one potential drawback with test-suite reduction is that this might affect the quality of the test-suite in terms of fault finding. Previous empirical studies provide conflicting evidence on this issue. To further investigate the problem and determine its effect when testing implementations derived from formal models of software we performed an experiment using a large case example of a Flight Guidance System, generated reduced test-suites for a variety of structural coverage criteria while preserving coverage, and recorded their fault finding effectiveness. Our results indicate that the size of the specification based test-suites can be dramatically reduced and that the fault detection of the reduced test-suites is adversely affected. In this report we describe our experiment, analyze the results, and discuss the implications for testing based on formal specifications.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

Notes

  1. We thank Dr. Steve Miller, Dr. Alan Tribble, and Dr. Mike Whalen of Rockwell Collins Inc. for the information on flight control systems and for letting us use the models they developed.

  2. We use here the ASCII version of RSML-e since it is much more compact than the typeset version.

References

  • NuS: The NuSMV Toolset. Available at http://nusmv.irst.itc.it/ (2005)

  • Ammann, P.E., Black, P.E.: A specification-based coverage metric to evaluate test sets. In: Proceedings of the Fourth IEEE International Symposium on High-Assurance Systems Engineering. IEEE Computer Society (1999)

  • Ammann, P.E., Black, P.E., Majurski, W.: Using model checking to generate tests from specifications. In: Proceedings of the Second IEEE International Conference on Formal Engineering Methods (ICFEM’98), pp. 46–54. IEEE Computer Society (1998)

  • Archer, M., Heitmeyer, C., Simsm S.: TAME: A PVS interface to simplify proofs for automata models. In: User Interfaces for Theorem Provers (1998)

  • Bensalem, S., Caspi, P., Parent-Vigouroux, C., Dumas, C.: A methodology for proving control systems with Lustre and PVS. In: Proceedings of the Seventh Working Conference on Dependable Computing for Critical Applications (DCCA 7), pp. 89–107. IEEE Computer Society, San Jose, CA (1999)

  • Blackburn, M.R., Busser, R.D., Fontaine, J.S.: Automatic generation of test vectors for SCR-style specifications. In: Proceedings of the 12th Annual Conference on Computer Assurance, COMPASS’97 (1997)

  • Callahan, J., Schneider, F., Easterbrook, S.: Specification-based testing using model checking. In: Proceedings of the SPIN Workshop (1996)

  • Chan, W., Anderson, R., Beame, P., Burns, S., Modugno, F., Notkin, D., Reese, J.: Model checking large software specifications. IEEE Trans. Softw. Eng. 24(7), 498–520 (1998)

    Google Scholar 

  • Chilenski, J., Miller, S.: Applicability of modified condition/decision coverage to software testing. Softw. Eng. J. 9, 193–200 (1994a)

    Google Scholar 

  • Chilenski, J.J., Miller, S.P.: Applicability of modified condition/decision coverage to software testing. Softw. Eng. J. 193–200 (1994b)

  • Choi, Y., Heimdahl, M.: Model checking RSML-e requirements. In: Proceedings of the 7th IEEE/IEICE International Symposium on High Assurance Systems Engineering, pp. 109–118. Tokyo, Japan (2002)

  • Clarke, E.M., Grumberg, O., Peled, D.: Model Checking. MIT Press (1999)

  • Engels, A., Feijs, L.M.G., Mauw, S.: Test generation for intelligent networks using model checking. In: Proceedings of TACAS’97, LNCS 1217, pp. 384–398. Springer (1997)

  • Esterel Technologies: SCADE suite product description. http://www.esterel-technologies.com/v2/scadeSuiteForSafetyCriticalSoftwareDevelopment/index.html (2004)

  • FAA, C.A.S.T.: What is a “decision” in application of modified condition/decision coverage and decision coverage (DC)? Technical Report position paper (2002)

  • Frankl, P., Weiss, S.N.: An experimental comparison of the effectiveness of the all-uses and all-edges adequacy criteria. In: Proceedings of the symposium on Testing, analysis, and verification (1991)

  • Garey, M., Johnson, D.: Computers and Intractability. Freeman, New York (1979)

  • Gargantini, A., Heitmeyer, C.: Using model checking to generate tests from requirements specifications. Softw. Eng. Notes 24(6), 146–162 (1999)

    Google Scholar 

  • Grumberg, O., Long, D.E.: Model checking and modular verification. ACM Trans. Program. Lang. Syst. 16(3), 843–871 (1994)

    Google Scholar 

  • Harel, D.: Statecharts: A visual formalism for complex systems. Sci. Comput. Program. 8(3), 231–274 (1987)

    Google Scholar 

  • Harel, D., Lachover, H., Naamad, A., Pnueli, A., Politi, M., Sherman, R., Shtull-Trauring, A., Trakhtenbrot, M. STATEMATE: A working environment for the development of complex reactive systems. IEEE Trans. Softw. Eng. 16(4), 403–414 (1990)

    Google Scholar 

  • Hayhurst, K., Veerhusen, D., Rierson, L.: A practical tutorial on modified condition/decision coverage. Technical Report TM-2001-210876, NASA (2001)

  • Heimdahl, M.P., Devaraj, G.: Test-suite reduction for model based tests: Effects on test quality and implications for testing. In: Proceedings of the 19th IEEE International Conference on Automated Software Engineering (ASE). Linz, Austria (2004)

  • Heimdahl, M.P., Devaraj, G., Weber, R.J.: Specification test coverage adequacy criteria = specification test generation inadequacy criteria? In: Proceedings of the Eighth IEEE International Symposium on High Assurance Systems Engineering (HASE). Tampa, Florida (2004)

  • Heimdahl, M.P., Rayadurgam, S., Visser, W.: Specification centered testing. In: Second International Workshop on Analysis, Testing and Verification (2001)

  • Heimdahl, M.P., Rayadurgam, S., Visser, W., Devaraj, G., Gao, J.: Auto-generating test sequences using model checkers: A case study’. In: 3rd International Worshop on Formal Approaches to Testing of Software (FATES 2003) (2003)

  • Heimdahl, M.P.E., Leveson, N.G.: Completeness and consistency in hierarchical state-base requirements. IEEE Trans. Softw. Eng. 22(6), 363–377 (1996)

    Google Scholar 

  • Heitmeyer, C., Bull, A., Gasarch, C., Labaw, B.: SCR*: A toolset for specifying and analyzing requirements. In: Proceedings of the Tenth Annual Conference on Computer Assurance, COMPASS 95 (1995)

  • Heitmeyer, C., Jeffords, R., Labaw, B.: Automated consistency checking of requirements specifications. ACM Trans. Softw. Eng. Methodol. 5(3), 231–261 (1996)

    Google Scholar 

  • Heitmeyer, C., Jr, J.K., Labaw, B., Archer, M., Bharadwaj, R.: Using abstraction and model checking to detect safety violations in requirements specifications. IEEE Trans. Softw. Eng. 24(11), 927–948 (1998)

    Google Scholar 

  • Hong, H.S., Cha, S.D., Lee, I., Sokolsky, O., Ural, H.: Data flow testing as model checking. In: Proceedings of the International Conference on Software Engineering. Portland, Oregon (2003)

  • Hong, H.S., Lee, I., Sokolsky, O., Ural, H.: A temporal logic based theory of test coverage and generation. In: Proceedings of the International Conference on Tools and Algorithms for Construction and Analysis of Systems (TACAS’02). Grenoble, France (2002)

  • Jasper, R., Brennan, M., Williamson, K., Currier, B., Zimmerman, D.: Test data generation and feasible path analysis. In: Proceedings of International Symposium on Software Testing and Analysis (1994), pp. 95–107.

  • Jones, J.A., Harrold, M.J.: Test-suite reduction and prioritization for modified condition/decision coverage. IEEE Trans. Softw. Eng. 29(3), 195–209 (2003)

    Google Scholar 

  • Lee, E.A.: Overview of the Ptolemy Project. Technical Report Technical Memorandum UCB/ERL M03/25, University of California, Berkeley, CA, 94720, USA (2003)

  • Leveson, N., Heimdahl, M., Hildreth, H., Reese, J.: Requirements specification for process-control systems. IEEE Trans. Softw. Eng. 20(9), 684–706 (1994)

    Google Scholar 

  • Mathworks Inc.: Mathworks Inc. Simulink Product Web Site. via the world-wide-web: http://www.mathworks.com.

  • Offutt, A.J., Xiong, Y., Liu, S.: Criteria for generating specification-based tests. In: Proceedings of the Fifth IEEE International Conference on Engineering of Complex Computer Systems (ICECCS’99) (1999)

  • Owre, S., Shankar, N., Rushby, J.: The PVS specification language. Computer Science Laboratory; SRI International, Menlo Park, CA 94025, beta relese edition (1993)

  • Pnueli, A.: Applications of temporal logic to specification and verification of reactive systems: A survey of current trends. Lecture Notes in Computer Science Number 224, pp. 510–584 (1986)

  • Rayadurgam, S.: Automatic test-case generation from formal models of software. Ph.D. thesis, University of Minnesota (2003)

  • Rayadurgam, S., Heimdahl, M.P.: Coverage based test-case generation using model checkers. In: Proceedings of the 8th Annual IEEE International Conference and Workshop on the Engineering of Computer Based Systems (ECBS 2001), pp. 83–91. IEEE Computer Society (2001a)

  • Rayadurgam, S., Heimdahl, M.P.: Test-sequence generation from formal requirement models. In: Proceedings of the 6th IEEE International Symposium on the High Assurance Systems Engineering (HASE 2001). Boca Raton, Florida (2001b)

  • Rothermel, G., Harrold, M., Ostrin, J., Hong, C.: An empirical study of the effects of minimization on the fault detection capabilities of test suites. In: Proceedings of the International Conference on Software Maintenance, pp. 34–43 (1998)

  • RTCA: Software Considerations In Airborne Systems and Equipment Certification. RTCA (1992)

  • Thompson, J.M., Heimdahl, M.P.: An integrated development environment prototyping safety critical systems. In: Tenth IEEE International Workshop on Rapid System Prototyping (RSP) 99, (1999), pp. 172–177.

  • Thompson, J.M., Heimdahl, M.P., Miller, S.P.: Specification based prototyping for embedded systems. In: Seventh ACM SIGSOFT Symposium on the Foundations on Software Engineering, pp. 163–179 (1999)

  • Whalen, M.W., Heimdahl, M.P.: On the requirements of high-integrity code generation. In: 4th IEEE International Symposium on High Assurance Systems Engineering, Vol. LNCS yyy, (1999), pp. 217–226.

  • Wong, W., Horgan, J., Mathur, A., Pasquini, A.: Test set size minimization and fault detection effectiveness: a case study in a space application. In: Proceedings of the 21st Annual International Computer Software and Applications Conference, (1997), pp. 522–528.

  • Wong, W., Horgan, J., London, S., Mathur, A.: Effect of test set minimization on fault detection effectiveness. Softw. Pract. Exp. 28(4), 347–369 (1998)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Computer Science and Engineering, University of Minnesota, 200 Union Street S.E., 4-192, Minneapolis, MN, 55455, USA

    Mats P. E. Heimdahl & Devaraj George

Authors
  1. Mats P. E. Heimdahl
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Devaraj George
    View author publications

    You can also search for this author in PubMed Google Scholar

Additional information

This work has been partially supported by NASA grant NAG-1-224 and NASA contract NCC-01001. We also want to thank the McKnight Foundation for their generous support over the years.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Heimdahl, M.P.E., George, D. On the effect of test-suite reduction on automatically generated model-based tests. Autom Softw Eng 14, 37–57 (2007). https://doi.org/10.1007/s10515-006-0004-y

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10515-006-0004-y

Keywords

Search

Navigation