Skip to main content
SpringerLink
Log in

A systematic review of state-based test tools

  • regular paper
  • Published:
International Journal on Software Tools for Technology Transfer Aims and scope Submit manuscript

Abstract

Model-based testing (MBT) is about testing a software system using a model of its behaviour. To benefit fully from MBT, automation support is required. The goal of this systematic review is determining the current state of the art of prominent MBT tool support where we focus on tools that rely on state-based models. We automatically searched different source of information including digital libraries and mailing lists dedicated to the topic. Precisely defined criteria are used to compare selected tools and comprise support for test adequacy and coverage criteria, level of automation for various testing activities and support for the construction of test scaffolding. Simple adequacy criteria are supported but not advanced ones; data(-flow) criteria are seldom supported; support for creating test scaffolding varies a great deal. The results of this review should be of interest to a wide range of stakeholders: software companies interested in selecting the most appropriate MBT tool for their needs; organizations willing to invest into creating MBT tool support; researchers interested in setting research directions.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

Notes

  1. For instance, Mathur defines MBT as “the generation of tests from a formal model of application behaviour” and identifies test generation from combinatorial designs as MBT [8].

  2. AGEDIS project: http://www.agedis.de/index.html (last accessed June 2012).

  3. D-MINT project: http://www.d-mint.org/ (last accessed June 2012).

  4. This tool is based on UniTESK technology.

  5. http://graphml.graphdrawing.org (last accessed June 2012).

  6. http://www.yworks.com (last accessed June 2012).

  7. http://testng.org (last accessed June 2012).

  8. http://www.gentleware.com (last accessed June 2012).

  9. http://www.artisansoftwaretools.com (last accessed June 2012).

  10. http://www.elvior.ee/messagemagic/general/ (last accessed June 2012).

  11. http://www.cs.aau.dk/~behrmann/cora/ (last accessed June 2012).

  12. http://graphml.graphdrawing.org (last accessed June 2012).

  13. http://www.omg.org/technology/documents/formal/xmi.htm (last accessed June 2012).

  14. http://strategoxt.org/Transform/GraphXML (last accessed June 2012).

  15. http://www.objecteering.com (last accessed June 2012).

  16. http://www.altova.com/xml-editor (last accessed June 2012).

  17. http://www.ibm.com/developerworks/rational/products/rsa/ (last accessed June 2012).

  18. http://www-01.ibm.com/software/awdtools/rhapsody/ (last accessed June 2012).

  19. http://www.sparxsystems.com.au/ (last accessed June 2012).

  20. http://www.topcased.org/ (last accessed June 2012).

  21. https://h10078.www1.hp.com/cda/hpms/display/main/hpms_content.jsp?zn=bto&cp=1-11-127-24\(\wedge \)1131_4000_100 (last accessed June 2012).

  22. https://h10078.www1.hp.com/cda/hpms/display/main/hpms_content.jsp?zn=bto&cp=1-11-127-24\(\wedge \)1352_4000_100 (last accessed June 2012).

  23. http://www-01.ibm.com/software/awdtools/modeler/swmodeler/ (last accessed June 2012).

  24. http://www-01.ibm.com/software/awdtools/architect/swarchitect/ (last accessed June 2012).

  25. http://www.borland.com/us/products/together/index.html (last accessed June 2012).

References

  1. Ammann, P., Offutt, J.: Introduction to Software Testing. Cambridge University Press, Cambridge (2008)

    Book  MATH  Google Scholar 

  2. Beizer, B.: Software Testing Techniques. Van Nostrand Reinhold, New York (1990)

    Google Scholar 

  3. Patton, R.: Software Testing. SAMS, Indianapolis (2005)

    Google Scholar 

  4. Boberg, J.: Early fault detection with model-based testing. In: ACM SIGPLAN Workshop on ERLANG, pp. 9–20. ACM, New york (2008)

  5. Pretschner, A.: Model-based testing in Practice. In: Formal Methods. Lecture Notes in Computer Science, vol. 3582, pp. 537–541. Springer, Heidelberg (2005)

  6. Broy, M., Jonsson, B., Katoen, J., Leucker, M., Pretschner, A.: Model-based testing of reactive systems, vol. 3472, Springer, Heidelberg (2005)

  7. Utting, M., Legeard, B.: Practical Model-Based Testing: A Tools Approach. Morgan-Kaufmann, San Francisco (2006)

    Google Scholar 

  8. Mathur, A.P.: Foundations of Software Testing. Pearson Education, Upper Saddle River (2009)

    Google Scholar 

  9. Gossens, S., Belli, F., Beydeda, S., Cin, M.D.: View graphs for analysis and testing of programs at different abstraction levels. In: IEEE International Symposium on High-Assurance Systems Engineering, pp. 121–130 (2005)

  10. Vegas, S., Juristo, N., Basili, V. R.: Maturing software engineering knowledge through classifications: a case study on unit testing techniques. IEEE Trans. Softw. Eng. 35(4), 551–565 (2009)

    Google Scholar 

  11. Pender, T.: UML Bible. Wiley, New York (2003)

    Google Scholar 

  12. Monin, J.: Understanding Formal Methods. Springer, Berlin (2003)

    Book  MATH  Google Scholar 

  13. van Eijk, P.H.J., Vissers, C.A., Díaz, M.: The Formal Description Technique LOTOS: Results of the ESPRIT/SEDOS Project. Elsevier, Amsterdam (1989)

    MATH  Google Scholar 

  14. Karris, S.T.: Introduction to Stateflow with Applications. Orchard Publications, Fremont (2007)

    Google Scholar 

  15. Murata, T.: Petri nets: properties, analysis and applications. Proc. IEEE. 77(4), 541–580 (1989)

    Article  Google Scholar 

  16. Halbwachs, N.: Synchronous Programming of Reactive Systems. Springer, Berlin (1993)

    Book  MATH  Google Scholar 

  17. Binder, R.V.: Model-Based Testing User Survey: Results and Analysis. http://www.robertvbinder.com/docs/arts/MBT-User-Survey.pdf (2011). Accessed June 2012

  18. Kitchenham, B., Charters, S.: Guidelines for Performing Systematic Literature Reviews in Software Engineering. Computer Science, Keele University, Staffordshire (2007)

    Google Scholar 

  19. Dyba, T., Kitchenham, B., Jorgensen, M.: Evidence-based software engineering for practitioners. IEEE Softw. 22(1), 58–65 (2005)

    Article  Google Scholar 

  20. Kitchenham, B., Dyba, T., Jorgensen, M.: Evidence-based software engineering. In: ACM/IEEE International Conference on Software Engineering, pp. 273–281 (2004)

  21. Brereton, P., Kitchenham, B., Budgen, D., Turner, M., Khalil, M.: Lessons from applying the systematic literature review process within the software engineering domain. J. Syst. Softw. 80(4), 571–583 (2007)

    Article  Google Scholar 

  22. Dias Neto, A.C., Subramanyan, R., Vieira, M., Travassos, G.H.: A survey on model-based testing approaches: a systematic review. In: International Workshop on Empirical Assessment of Software Engineering Languages and Technologies (in Conjunction with IEEE/ACM ASE), pp. 31–36 (2007)

  23. Hartman, A.: Model-Based Test Generation Tools. http://cm.techwell.com/sites/default/files/articles/XDD6047filelistfilename1_0.pdf (2002). Accessed June 2012 (2002)

  24. Bruegge, B., Dutoit, A.H.: Object-Oriented Software Engineering using UML, Patterns, and Java. Prentice Hall, Englewood Cliffs (2004)

    Google Scholar 

  25. Chanson, S.T., Zhu, J.: A unified approach to protocol test sequence generation. In: Proceedings. Twelfth Annual Joint Conference of the IEEE Computer and Communications Societies. Networking: Foundation for the Future, pp. 106–114 (1993)

  26. Jacky, J., Veanes, M., Campbell, C., Schulte, W.: Model-Based Software Testing and Analysis with C#. Cambridge University Press, Cambridge (2008)

    Google Scholar 

  27. Binder, R.: Testing Object-Oriented Systems: Models, Patterns, and Tools. Addison-Wesley, Reading (1999)

    Google Scholar 

  28. Jorgensen, P.: Software Testing-a Craftsman’s Approach. Auerbach Publications, Boca Raton (2008)

    MATH  Google Scholar 

  29. Lewis, W.: Software Testing and Continuous Quality Improvement. CRC Press, Boca Raton (2008)

    Book  Google Scholar 

  30. Pezze, M., Young, M.: Software Testing and Analysis. Wiley, New York (2007)

    MATH  Google Scholar 

  31. Poston, R.M.: Automating Specification-Based Software Testing. IEEE Computer Society, Los Alamitos (1997)

    Google Scholar 

  32. Dustin, E., Garrett, T., Gauf, B.: Implementing Automated Software Testing: How to Lower Costs while Raising Quality. Addison-Wesley, Upper Saddle River (2009)

    Google Scholar 

  33. Dustin, E., Rashka, J., Paul, J.: Automated Software Testing-introduction, Management, and Performance. Addison-Wesley, Upper Saddle River (1999)

    Google Scholar 

  34. Fewster, M.G.D.: Software Test Automation-Effective use of Test Execution Tools. Addison-Wesley, Upper Saddle River (1999)

    MATH  Google Scholar 

  35. Li, K., Wu, M.: Effective Software Test Automation: Developing an Automated Software Testing Tool. Sybex, Alamdea (2004)

    Google Scholar 

  36. Weyuker, E.J.: On testing non-testable programs. Comput. J. 25(4), 465–470 (1982)

    Article  Google Scholar 

  37. Briand, L., Labiche, Y., He, S.: Automating regression test selection based on UML Designs. Inf. Softw. Technol. 51(1), 16–30 (2009)

    Article  Google Scholar 

  38. Mansour, N., Takkoush, H., Nehme, A.: UML-based regression testing for OO software. J. Softw. Maint. Evolut. Res. Pract. 23, 51–68 (2011)

    Article  Google Scholar 

  39. Goga, N.: Comparing TorX, Autolink, TGV and UIO Test Algorithms, SDL 2001: Meeting UML, pp. 379–402. Springer, Berlin (2001)

    Book  Google Scholar 

  40. Offutt, A.J., Liu, S., Abdurazik, A., Ammann, P.: Generating test data from state-based specifications. Softw. Test. Verif. Reliab. 13, 25–53 (2003)

    Article  Google Scholar 

  41. Lee, D., Yannakakis, M.: Principles and methods of testing finite state machines: a survey. Proc. IEEE 84, 1090–1123 (1996)

    Article  Google Scholar 

  42. Mouchawrab, S., Briand, L.C., Labiche, Y., Penta, M.D.: Assessing, comparing, and combining state machine-based testing and structural testing: a series of experiments. IEEE Trans. Softw. Eng. 37, 161–187 (2011)

    Google Scholar 

  43. Khalil, M., Labiche, Y.: On the round trip path testing strategy. In: IEEE 21st International Symposium on Software Reliability Engineering (ISSRE), pp. 388–397 (2010)

  44. Briand, L.C., Labiche, Y., Wang, Y.: Using simulation to empirically investigate test coverage criteria. In: IEEE/ACM International Conference on Software Engineering, pp. 86–95 (2004)

  45. Ammann, P., Offutt, A., Hong, H.: Coverage criteria for logical expressions. In: International Symposium on Software Reliability Engineering, pp. 99–107 (2003)

  46. Vouk, M., Tai, K., Paradkar, A.: Empirical studies of predicate-based software testing. In: 5th International Symposium on Software Reliability Engineering, pp. 55–64 (1994)

  47. Ostrand, T.J., Balcer, M.J.: The category-partition method for specifying and generating functional test. Commun. ACM 31, 676–686 (1988)

    Article  Google Scholar 

  48. Reid, S.: An empirical analysis of equivalence partitioning, boundary value analysis and random testing. In: Software Metrics Symposium, pp. 64–73 (1997)

  49. Maity, S., Nayak, A.: Improved test generation algorithms for pair-wise testing. In: IEEE International Symposium on Software Reliability Engineering, pp. 235–244 (2005)

  50. Burr, K., Young, W.: Combinatorial test techniques: table-based automation, test generation, and code coverage. In: International Conference on Software Testing Analysis and Review, pp. 503–513 (1998)

  51. Antoniol, G., Briand, L.C., Penta, M.D., Labiche, Y.: A case study using the round-trip strategy for state-based class testing. In: 13th International Symposium on Software Reliability Engineering, pp. 269–279 (2002)

  52. Saifan, A., Dingel, J.: Model-Based Testing of Distributed Systems. School of Computing, Queen’s University, Canada (2008)

  53. Saifan, A., Dingel, J.: A survey of using model-based testing to improve quality attributes in distributed systems. In: Advanced Techniques in Computing Sciences and Software Engineering, pp. 283–288, Springer, Netherlands (2010)

  54. Hartman, A.: AGEDIS-model based test generation tool. In: ACM SIGSOFT International Symposium on Software Testing and Analysis, pp. 129–132 (2004)

  55. Utting, M., Pretschner, A., Legeard, B.: A Taxonomy of Model-Based Testing. Department of Computer Science, The University of Waikato, New Zealand (2006)

    Google Scholar 

  56. Belinfante, A., El-Ramly, M., Horstmann, M.: Tools for test case generation. In: Model-Based Testing of Reactive Systems. LNCS, pp. 391–438. Springer, Berlin (2005)

  57. Utting, M., Pretschner, A., Legeard, B.: A taxonomy of model-based testing approaches. J. Softw. Test. Verif. Reliab. 22(5), 297–312 (2011)

    Article  Google Scholar 

  58. Zander, J., Schieferdecker, I., Mostermna, P.J.: A taxonomy of model-based testing for embedded systems from multiple industry domains. In: Model-Based Testing for Embedded Systems, pp. 3–22. CRC Press, Boca Raton (2012)

  59. Schieferdecker, I.: Model-based testing. IEEE Softw. 29(1), 14–18 (2012)

    Article  Google Scholar 

  60. Sinha, A., Williams, C.E., Santhanam, P.: A measurement framework for evaluating model-based test generation tools. IBM Syst. J. 45(3), 501–514 (2006)

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Computer Systems Group, School of Computer Science, University of Waterloo, 200 University Ave West, Waterloo, ON, N2L3G1, Canada

    Muhammad Shafique

  2. Software Quality Engineering Lab, Systems and Computer Engineering, Carleton University, 1125 Colonel By Drive, Ottawa, ON, K1S5B6, Canada

    Yvan Labiche

Authors
  1. Muhammad Shafique
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yvan Labiche
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Muhammad Shafique.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Shafique, M., Labiche, Y. A systematic review of state-based test tools. Int J Softw Tools Technol Transfer 17, 59–76 (2015). https://doi.org/10.1007/s10009-013-0291-0

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10009-013-0291-0

Keywords

Search

Navigation