Skip to main content
SpringerLink
Log in

Bridging the gap between test cases and requirements by abstract testing

  • SI: QUATIC 2010
  • Published:
Innovations in Systems and Software Engineering Aims and scope Submit manuscript

Abstract

In this article we propose a technique, called abstract testing, which replaces traditional test cases by abstract test cases. By doing so, fewer test cases are needed, and they are linked more closely to the requirements. Abstract tests can be considered as verification scenarios on the source code level which are derived from the requirements. Checking verification scenarios against the source code is done automatically using a software model checker. We also suggest a migration path from traditional tests to abstract test cases, which provides a smooth transition towards this new technique. Finally, we demonstrate feasibility of abstract testing by a case study from the automotive systems domain.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3

Similar content being viewed by others

Notes

  1. We mainly have black-box unit testing of functional properties in mind when talking about a test case.

  2. Inlining is done by software bounded model checking tools automatically as part of the preprocessing of the code.

  3. MISRA-C is a software development standard developed by the The Motor Industry Software Reliability Association (MISRA) containing rules and guidelines for the development of safe, portable and reliably source code for embedded systems in the automotive industry [28]

  4. The havoc() procedure call in the example sets some global variables to undefined (non-deterministic) values.

  5. Due to a non-disclosure agreement we cannot publish the precise number of test cases created during software testing.

  6. Available at http://junit.org. Accessed 24/09/2014).

References

  1. Arnold D (2009) An open framework for the specification and execution of a testable requirements model. Ph.D. thesis, Ottawa-Carleton Institute for Computer Science, Carleton University

  2. Arnold D, Corriveau JP, Shi W (2010) Modeling and validating requirements using executable contracts and scenarios. In: Proceedings of the 8th international conference on software engineering research, management and applications (SERA)

  3. Biere A, Cimatti A, Clarke EM, Zhu Y (1999) Symbolic model checking without BDDs. In: Proceedings of the 5th international conference on tools and algorithms for construction and analysis of systems (TACAS). Springer, London, pp 193–207

  4. Bringmann E, Krämer A (2008) Model-based testing of automotive systems. In: ICST, pp 485–493

  5. Burnstein I (2003) Practical software testing: a process-oriented approach. Springer, Berlin

  6. Chaki S, Clarke E, Groce A, Jha S, Veith H (2003) Modular verification of software components in C. In: Proceedings of the 25th international conference on software engineering (ICSE). IEEE Computer Society, pp 385–395

  7. Chechik M, Gannon JD (2001) Automatic analysis of consistency between requirements and designs. IEEE Trans Softw Eng 27(7):651–672

    Article  Google Scholar 

  8. Clarke E, Kroening D, Lerda F (2004) A tool for checking ANSI-C programs. In: Jensen K, Podelski A (eds) Tools and algorithms for the construction and analysis of systems (TACAS). LNCS, vol 2988. Springer, Berlin, pp 168–176

  9. Cousot P (2007) Proving the absence of run-time errors in safety-critical avionics code. In: Proceedings of the 7th ACM and IEEE international conference on embedded software (EMSOFT). ACM, New York, pp 7–9

  10. Cousot P, Cousot R (1977) Abstract interpretation: a unified lattice model for static analysis of programs by construction or approximation of fixpoints. In: Proceedings of the fourth ACM symposium on principles of programming languages (POPL), Los Angeles, California, January 1977, pp 238–252

  11. Cousot P, Cousot R, Feret J, Mauborgne L, Miné A, Monniaux D, Rival X (2005) The ASTREÉ analyzer. In: Proceedings of the programming languages and systems, 14th European symposium on programming (ESOP), Edinburgh, UK, April 4–8, pp 21–30

  12. Crow J, Di Vito B (1998) Formalizing space shuttle software requirements: four case studies. ACM Trans Softw Eng Methodol 7(3):296–332

    Article  Google Scholar 

  13. Das Verisoft Projekt. http://www.verisoft.de. Accessed 22 Sept 2014

  14. Dutertre B, Stavridou V (1997) Formal requirements analysis of an avionics control system. IEEE Trans Softw Eng 23:267–278

    Article  Google Scholar 

  15. Fraser G, Wotawa F (2006) Property relevant software testing with model-checkers. SIGSOFT Softw Eng Notes 31(6):1–10

    Article  Google Scholar 

  16. Graham D (2002) Requirements and testing: seven missing-link myths. Softw IEEE 19(5):15–17

    Article  Google Scholar 

  17. Heitmeyer CL, Archer M, Leonard EI, McLean J (2008) Applying formal methods to a certifiably secure software system. IEEE Trans Softw Eng 34(1):82–98

    Article  Google Scholar 

  18. Heitmeyer CL, Jeffords RD, Labaw BG (1996) Automated consistency checking of requirements specifications. ACM Trans Softw Eng Methodol 5(3):231–261

    Article  Google Scholar 

  19. Henzinger T, Jhala R, Majumdar R, Sutre G (2003) Software verification with BLAST. In: Proceedings of the 10th international SPIN workshop. LNCS, vol 2648. Springer, Berlin, pp 235–239

  20. Henzinger TA, Jhala R, Majumdar R, Sutre G (2003) Software verification with blast. In: Proceedings of the 10th international workshop on model checking of software (SPIN). LNCS, vol 2648. Springer, Berlin, pp 235–239

  21. Hunt JJ, Jenn E, Leriche S, Schmitt P, Tonin I, Wonnemann C (2006) A case study of specification and verification using JML in an avionics application. In: Proceedings of the 4th workshop on java technologies for real-time and embedded systems (JTRES). ACM Press, New York, pp 107–116

  22. Jones C, Bonsignour O (2011) The economics of software quality. Prentice Hall, New Jersey (2011)

  23. Kim M, Kim Y, Kim H (2008) Unit testing of flash memory device driver through a SAT-based model checker. In: Proceedings of the international conference on automated software engineering (ASE). IEEE Computer Society Press, Washington, DC, pp 198–207

  24. Merz F, Falke S, Sinz C (2012) LLBMC: bounded model checking of C and C++ programs using a compiler IR. In: Proceedings of the 4th international conference on verified software: theories, tools, experiments (VSTTE’12). Springer, Berlin, pp 146–161

  25. Merz F, Sinz C, Post H, Gorges T, Kropf T (2010) Abstract testing: connecting source code verification with requirements. In: Proceedings of the 7th international conference on the quality of information and communications technology, pp 89–96

  26. Meyer B (2005) Eiffel as a framework for verification. In: First IFIP TC 2/WG 2.3 conference on verified software: theories, tools, experiments (VSSTE’05), pp 301–307

  27. Miller SP, Tribble AC, Whalen MW, Heimdahl MPE (2006) Proving the shalls: early validation of requirements through formal methods. J Softw Tools Technol Transf (STTT) 8(4–5):303–319

    Article  Google Scholar 

  28. MISRA C Working Group (2004) MISRA-C:2004—guidelines for the use of the C language in critical systems. The Motor Industry Software Reliability Association

  29. PolySpace Technologies (2008) Polyspace Client/Server for C/C++, Version 4.1.1.6. http://www.mathworks.de/products/polyspace. Accessed 24 Sept 2014

  30. Post H, Küchlin W (2007) Integration of static analysis for linux device driver verification. In: Davies J, Gibbons J (eds) Proceedings of the 6th international conference integrated formal methods (IFM). LNCS, vol 4591. Springer, Berlin, pp 518–537

  31. RTI-Health, Social, and Economics Research (2002) The economic impacts of inadequate infrastructure for software testing. Planning Report 02-3. National Institute of Standards and Technology

  32. SLAM-Microsoft Research. http://research.microsoft.com/slam. Accessed 22 Sept 2014

  33. Staats M, Heimdahl MPE (2008) Partial translation verification for untrusted code-generators. In: 10th international conference on formal engineering methodsformal (ICFEM’08), Kitakyushu-City, Japan, pp 226–237

  34. Tillmann N, Schulte W (2005) Parameterized unit tests. SIGSOFT Softw Eng Notes 30:253–262

    Article  Google Scholar 

  35. Uusitalo EJ, Komssi M, Kauppinen M, Davis AM (2008) Linking requirements and testing in practice. In: Proceedings of the 2008 16th IEEE international requirements engineering conference RE’08. IEEE Computer Society, Washington, DC, pp 265–270

  36. Visser W, Havelund K, Brat GP, Park S, Lerda F (2003) Model checking programs. Autom Softw Eng 10(2):203–232

    Article  Google Scholar 

  37. Wassyng A, Lawford M (2006) Software tools for safety-critical software development. STTT 8(4–5):337–354

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Karlsruhe Institute of Technology (KIT), Am Fasanengarten 5, 76131, Karlsruhe, Germany

    Florian Merz & Carsten Sinz

  2. Robert Bosch GmbH, Leonberg, Germany

    Hendrik Post, Thomas Gorges & Thomas Kropf

Authors
  1. Florian Merz
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Carsten Sinz
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Hendrik Post
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Thomas Gorges
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Thomas Kropf
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Florian Merz.

Additional information

This work was supported in part by the “Concept for the Future” of Karlsruhe Institute of Technology within the framework of the German Excellence Initiative. A preliminary version of this paper appeared in Proc. of the 7th Intl. Conf. on the Quality of Information and Communications Technology (QUATIC 2010), Porto, Portugal, 2010 [25].

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Merz, F., Sinz, C., Post, H. et al. Bridging the gap between test cases and requirements by abstract testing. Innovations Syst Softw Eng 11, 233–242 (2015). https://doi.org/10.1007/s11334-015-0245-7

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11334-015-0245-7

Keywords

Search

Navigation