Skip to main content
SpringerLink
Log in

Complete model-based equivalence class testing

  • ICTSS 2013
  • Published:
International Journal on Software Tools for Technology Transfer Aims and scope Submit manuscript

Abstract

In this article, we present a model-based black-box equivalence partition testing strategy, together with a formal proof of its completeness properties. The results apply to reactive systems with large, possibly infinite input data types and finite internal and output data ranges that may be enumerated with acceptable effort. The investigation is performed on a semantic level and applies to all concrete test models whose behavioural semantics can be encoded as a variant of state transition systems. Test suite construction is performed in relation to a given fault model \(\mathcal{F}\) for which a finite black-box test suite can be constructed which is complete with respect to \(\mathcal{F}\). It is shown how the test suite generation can be effectively implemented by model-based testing tools, using propositional representations of behavioural model semantics and constraint solvers. A SysML model of the ceiling speed monitoring function of the European Train Control System is presented as a case study, to explain theory application to a concrete modelling formalism.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Similar content being viewed by others

Notes

  1. Note that other authors, for example [13], reserve the term ‘exhaustiveness’ for test suites containing all possible tests and use validity instead, if (1) is fulfilled, and unbias in situation (2).

  2. In [20, Chapter 11], for example, it is shown how to construct \(\mathcal{R}_1\) for a class of SysML models that also contains the model used in the case study in Sect. 4.

  3. The full CSM specification distinguishes between service and emergency brakes, while the configuration presented here corresponds to the situation where the train is equipped with emergency brakes only. A test model comprising the full CSM functionality has been made available by the authors under http://www.mbt-benchmarks.org.

  4. This would be an exceptional behaviour situation, caused, for example, by temporary unavailability of odometry data, so that a “sudden jump” of \(v\) would be observed by the CSM.

References

  1. Anand, Saswat, Burke, Edmund K., Chen, Tsong Yueh, Clark, John A., Cohen, Myra B., Grieskamp, W., Harman, M., Harrold, M.J., McMinn, P.: An orchestrated survey of methodologies for automated software test case generation. J. Syst. Softw. 86(8), 1978–2001 (2013)

    Article  Google Scholar 

  2. Binder, R.V.: Testing object-oriented systems: models, patterns, and tools. Addison-Wesley, Reading (2000)

    Google Scholar 

  3. Brucker, A.D., Wolff, B.: On theorem prover-based testing. Formal Asp. Comput. 25(5), 683–721 (2013)

    Article  MathSciNet  MATH  Google Scholar 

  4. Chen, T.Y., Tse, T.H., Yu, Y.T.: Proportional sampling strategy: a compendium and some insights. J. Syst. Softw. 58(1), 65–81 (2001)

  5. Chow, T.S.: Testing software design modeled by finite-state machines. IEEE Trans. Softw. Eng. 4(3), 178–187 (1978)

  6. Clarke, E.M., Grumberg, O., Peled, D.A.: Model Checking. The MIT Press, Cambridge (1999)

    Google Scholar 

  7. Dick, Jeremy, Faivre, Alain: Automating the generation and sequencing of test cases from model-based specifications. In: Woodcock, J.C.P., Larsen, P.G. (eds.) FME ’93: Industrial-strength formal methods. Lecture notes in computer science, pp. 268–284. Springer, Berlin (1993)

    Chapter  Google Scholar 

  8. Doucet, F., Menarini, M., Krüger, I.H., Gupta, R.K., Talpin, J.-P.: A verification approach for gals integration of synchronous components. Electr. Notes Theor. Comput. Sci. 146(2), 105–131 (2006)

    Article  Google Scholar 

  9. European Committee for Electrotechnical Standardization: EN 50128—Railway applications—Communications, signalling and processing systems—Software for railway control and protection systems. CENELEC, Brussels (2001)

  10. European Railway Agency: ERTMS—System Requirements Specification—UNISIG SUBSET-026 February 2012. http://www.era.europa.eu/Document-Register/Pages/Set-2-System-Requirements-Specification.aspx

  11. Frantzen, L., Tretmans, J., Willemse, Tim A.C.: Test generation based on symbolic specifications. In: Grabowski, J., Nielsen, B. (eds.) Formal approaches to software testing. Lecture notes in computer science, pp. 1–15. Springer, Berlin (2005)

    Chapter  Google Scholar 

  12. Fujiwara, S., Bochmann, G., Khendek, F., Amalou, M., Ghedamsi, A.: Test selection based on finite state models. IEEE Trans. Softw. Eng. 17(6), 591–603 (1991)

    Article  Google Scholar 

  13. Gaudel, M.-C.: Testing can be formal, too. In: Mosses, P.D., Nielsen, M., Schwartzbach, M.I. (eds.) TAPSOFT. Lecture Notes in Computer Science, pp. 82–96. Springer, Berlin (1995)

    Google Scholar 

  14. Gnesi, S., Latella, D., Massink, M.: Formal test-case generation for uml statecharts. In: Ninth IEEE International Conference on Engineering Complex Computer Systems (ICECCS’04), iceccs, pp. 75–84 (2004)

  15. Gill, A.: Introduction to the Theory of Finite-State Machines. McGraw-Hill, New York (1962)

  16. Grieskamp, W., Gurevich, Y., Schulte, W., Veanes, M.: Generating finite state machines from abstract state machines. ACM SIGSOFT Softw. Eng. Notes 27(4), 112–122 (2002)

    Article  Google Scholar 

  17. Helke, S., Neustupny, T., Santen, T.: Automating test case generation from z specifications with isabelle. In: Bowen, J.P., Hinchey, M.G., Till, D. (eds.) ZUM ’97: The Z Formal Specification Notation, vol. 1212. Lecture Notes in Computer Science, pp. 52–71. Springer, Berlin (1997)

  18. Hong, H.S., Lee, I., Sokolsky, O., Ural, H.: A temporal logic based theory of test coverage and generation. In: Katoen, J.-P., Stevens, P. (eds.) TACAS. Lecture Notes in Computer Science, pp. 327–341. Springer, Berlin (2002)

    Google Scholar 

  19. Huang, Wen-ling, Peleska, Jan: Exhaustive model-based equivalence class testing. In: Yenigün, Hüsnü, Yilmaz, Cemal, Ulrich, Andreas (eds.) Testing software and systems. Lecture notes in computer science, pp. 49–64. Springer, Berlin (2013)

    Google Scholar 

  20. Huang, W., Peleska, J., Schulze, U.: Comprehensive modelling for advanced systems of systems—test automation support. Public Document D34.1, COMPASS, October 2013. http://www.compass-research.eu/deliverables.html

  21. ISO/DIS 26262-4 Road vehicles—Functional safety—Part 4: Product development: system level. Technical report, International Organization for Standardization (2009)

  22. ISO/IEC/IEEE 29119-1:2013(e): Software and systems engineering—software testing—part 1: Concepts and definitions, Sept (2013)

  23. ISO/IEC/IEEE 29119-2:2013(e): Software and systems engineering—software testing—part 2: Test processes. Sept (2013)

  24. Kalaji, A.S., Hierons, R.M., Swift, S.: Generating feasible transition paths for testing from an extended finite state machine (efsm). In: ICST, IEEE Computer Society, Silver Spring, pp. 230–239 (2009)

  25. Lapschies, F.: SONOLAR homepage. http://www.informatik.uni-bremen.de/agbs/florian/sonolar/. Accessed June 2014

  26. Object Management Group: OMG Systems Modeling Language (\(\text{ OMG } \text{ SysML }^{{TM}}\)). Technical report, Object Management Group, 2010. OMG Document Number: formal/2010-06-02 (2010)

  27. Object Management Group: OMG Unified Modeling Language (OMG UML), superstructure, version 2.4.1. Technical report, OMG (2011)

  28. Peleska, J., Siegel, M.: Test automation of safety–critical reactive systems. S. Afr. Comput. J. 19, 53–77 (1997)

    Google Scholar 

  29. Peleska, J.: Industrial-strength model-based testing—state of the art and current challenges. In: Petrenko, A.K., Schlingloff, H. (eds.) Proceedings Eighth Workshop on Model-Based Testing, Rome, Italy, 17th March 2013. Electronic Proceedings in Theoretical Computer Science, vol. 111, pp. 3–28. Open Publishing Association (2013)

  30. Peleska, J., Honisch, A., Lapschies, F., Löding, H., Schmid, H., Smuda, P., Vorobev, E., Zahlten, C.: A real-world benchmark model for testing concurrent real-time systems in the automotive domain. In: Wolff, B., Zaidi, F. (eds.) Testing Software and Systems. Proceedings of the 23rd IFIP WG 6.1 International Conference, ICTSS 2011. LNCS, vol. 7019, pp. 146–161, Nov 2011. IFIP WG 6.1, Springer, Berlin (2011)

  31. Peleska, J., Vorobev, E., Lapschies, F.: Automated test case generation with SMT-solving and abstract interpretation. In: Bobaru, M., Havelund, K., Holzmann, G.J., Joshi, R. (eds.) Nasa Formal Methods. Third International Symposium, NFM 2011. LNCS, vol. 6617, pp. 298–312. Springer, Berlin (2011)

  32. Petrenko, A., Yevtushenko, N., Bochmann, G.v: Fault models for testing in context. In: Gotzhein, Reinhard, Bredereke, Jan (eds.) Formal description techniques IX—theory, application and tools, pp. 163–177. Chapman & Hall, London (1996)

    Chapter  Google Scholar 

  33. Petrenko, A., Simao, A., Maldonado, J.C.: Model-based testing of software and systems: recent advances and challenges. Int. J. Softw. Tools Technol. Transf. 14(4), 383–386 (2012)

    Article  Google Scholar 

  34. Ranise, S., Tinelli, C.: Satisfiability modulo theories. IEEE Mag. Intell. Syst. Trends Controv. 21(6), 71–81 (2006)

    Google Scholar 

  35. RTCA, SC-167 Software Considerations in Airborne Systems and Equipment Certification, RTCA/DO-178B RTCA (1992)

  36. Spillner, A., Linz, T., Schaefer, H.: Software testing foundations. Dpunkt.verlag, Heidelberg (2006)

    Google Scholar 

  37. Springintveld, J.G., Vaandrager, F.W., D’Argenio, P.R.: Testing timed automata. Theor. Comput. Sci. 254(1–2), 225–257 (2001)

    Article  MathSciNet  MATH  Google Scholar 

  38. Tretmans, J.: Conformance testing with labelled transition systems: implementation relations and test generation. Comput. Netw. ISDN Syst. 29(1), 49–79 (1996)

    Article  Google Scholar 

  39. Tretmans, J.: Model based testing with labelled transition systems. In: Hierons, R.M., Bowen, J.P., Harman, M. (eds.) Formal Methods and Testing. Lecture Notes in Computer Science, vol. 4949, pp. 1–38. Springer, Berlin (2008)

  40. UNISIG: ERTMS/ETCS System Requirements Specification, chap. 3. Principles, vol. Subset-026-3, chapt. 3. Issue 3.3.0 (2012)

  41. Vasilevskii, M.P.: Failure diagnosis of automata. Kibernetika (Transl.) 4, 98–108 (1973)

    MathSciNet  Google Scholar 

Download references

Acknowledgments

The authors are grateful to Alexandre Petrenko and to the anonymous reviewers for their helpful comments and suggestions for improvements.

Author information

Authors and Affiliations

  1. Department of Mathematics and Computer Science, University of Bremen, Bremen, Germany

    Wen-ling Huang & Jan Peleska

Authors
  1. Wen-ling Huang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jan Peleska
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jan Peleska.

Additional information

The authors’ research is funded by the EU FP7 COMPASS project under Grant Agreement No. 287829.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Huang, Wl., Peleska, J. Complete model-based equivalence class testing. Int J Softw Tools Technol Transfer 18, 265–283 (2016). https://doi.org/10.1007/s10009-014-0356-8

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10009-014-0356-8

Keywords

Search

Navigation