Skip to main content
SpringerLink
Log in

H-Switch Cover: a new test criterion to generate test case from finite state machines

  • Published:
Software Quality Journal Aims and scope Submit manuscript

Abstract

Test cases generation based on Finite State Machines (FSMs) has been addressed for quite some time. Model-based testing has drawn attention from researchers and practitioners as one of the approaches to support software verification and validation. Several test criteria have been proposed in the literature to generate test cases based on formal methods, such as FSM. However, there is still a lot to be done on this aspect in order to clearly direct a test designer to choose a test criterion most suitable to generate test cases for a certain application domain. This work presents a new test criterion for model-based test case generation based on FSM, H-Switch Cover. H-Switch Cover relies on the traditional Switch Cover test criterion, but H-Switch Cover uses new heuristics to improve its performance, for example, adoption of rules to optimize graph balancing and traverse the graph for test cases generation. We conducted an investigation of cost and efficiency of this new test criterion by comparing it with unique input/output and distinguishing sequence. We used two embedded software products (space application software products) and mutation analysis for assessing efficiency. In general, for the case studies proposed in this paper in terms of cost (amount of events) and efficiency (mutation score), H-Switch Cover test criterion presented an average and a standard deviation better than the other two test criteria.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14

Similar content being viewed by others

Notes

  1. In the academic community, a test criterion is also known as a method.

  2. We consider that a heuristic is a set of rules that lead to improvement or resolution of problems.

  3. In GTSC, the criterion implemented is the all-transitions of the statechart Coverage Criteria Family (SCCF) proposed by Souza (2000).

  4. In the context of this work, the term “time” refers to the instant that a defect was found in the code, that is, the amount of test cases necessary to identify the mutant.

References

  • Andrade, J., Ares, J., Martínez, M., Pazos, J., Rodríguez, S., Romera, J., et al. (2013). An architectural model for software testing lesson learned systems. Information and Software Technology, 55, 18–34).

  • Antoniol, G., Briand, L. C., Di Penta, M., & Labiche, Y. (2002). A case study using the round-trip path strategy for state-based class testing. In the 13th International symposium on software reliability engineering. Annapolis, MD, USA, pp. 269–279.

  • Andrews, A. A., Offutt, J., & Alexander, R. T. (2005). Testing web applications by modeling with FSMs. Software and System Modeling, 4(3), 326–345.

    Article  Google Scholar 

  • Arantes, A. O., Vijaykumar, N. L., Santiago, V., & Guimaraes, D. (2008). Test case generation for critical systems through a collaborative web-based tool. In International conference on inovation in software engineering (ISE2008). Viena, Aústria, pp. 163–168.

  • Arantes, A., Santiago Júnior, V. A., Vijaykumar, N. L., & Souza, E. F. (2014). Tool support for generating model-based test cases via web. International Journal of Web Engineering and Technology, 9(1), 62–96.

    Article  Google Scholar 

  • Arantes, A. (2008). WEB-PerformCharts: A web-based test case generator from statecharts modeling. Master (Master at post graduation course in applied computing)—National Institute for Space Research, São José dos Campos, Brazil.

  • Aho, A. V., Dahbura, A. T., Lee, D., & Uyar, M. U. (1988). An optimization technique for protocol conformance test generation based on UIO sequences and rural Chinese postman tours. In 8th Symposium on protocol specification, testing, and verification, IFIP, pp. 75–86.

  • Amaral, A. S. M. S. (2005). Geração de casos de testes para sistemas especificados em Statecharts. In Master (Master at post graduation course in applied computing)—National Institute for Space Research, São José dos Campos, Brazil.

  • Bang-Jensen, J., & Gutin, G. Z. (2009). Digraphs: Theory, algorithms and applications (2nd ed.). New York: Springer.

    Book  MATH  Google Scholar 

  • Binder, R. V. (2005). Testing object-oriented systems models, patterns, and tools (6th ed.). San Diego: Addison Wesley.

    Google Scholar 

  • Bondy, J. A., & Murty, U. S. R. (2008). Graduate texts in mathematics: Graph theory. Editorial Board, USA: Springer.

    MATH  Google Scholar 

  • Briand, L. C., Labiche, Y., & Wang, Y. (2004). Using simulation to empirically investigate test coverage criteria based on statechart. In 26th International conference on software engineering. Edinburgh/Scotland:IEEE, pp. 86–95.

  • Broy, M., Jonsson, B., Katoen, J. P., Leucker, M., & Pretschner, A. (2005). Model-based testing of reactive systems. New York: Springer.

    Book  MATH  Google Scholar 

  • Burguillo, J. C., Llamas, M., Fernández, M. J., & Robles, T. (2002). Heuristic-driven techniques for test case selection. Electronic Notes in Theoretical Computer Science, 66(2), 50–65.

    Article  Google Scholar 

  • Chow, T. S. (1978). Testing software design modeled by finite-state machines. IEEE Transactions on Software Engineering, Piscataway, USA, NJ, 4(3), 178–187.

    Article  MATH  Google Scholar 

  • De Bruijn, N. G. (1946). A combinatorial problem. Koninklijke Nederlandse Akademie v. Wetenschappen, 49, 758–764.

    MATH  Google Scholar 

  • Dorofeeva, R., El-Fakih, K., & Yevtushenko, N. (2005). An improved conformance testing method. In IFIP international conference on formal techniques for networked and distributed systems (FORTE), pp. 204–218.

  • Dorofeeva, R., El-Fakih, K., Maag, S., Cavalli, A. R., & Yevtushenko, N. (2010). FSM-based conformance testing methods: A survey annotated with experimental evaluation. Information and Software Technology, 52(12), 1286–1297.

    Article  Google Scholar 

  • El-Far, I. K., & Whittaker, J. A. (2001). Model-based software testing. In J. J. Marciniak (Ed.), Encyclopedia of software engineering. USA: Wiley.

    Google Scholar 

  • El-Fakih, A., Simao, A., Jadoon, N., & Maldonado, J. C. (2014). On studying the effectiveness of extended finite state machine based test selection criteria. In International conference on software testing, verification, and validation workshops, Cleveland, OH, pp. 222–229.

  • Endo, A. T., Linschulte, M., Simao, A., & Souza, S. R. S. (2010). Event- and coverage-based testing of web services. In Fourth international conference on secure software integration and reliability improvement companion (SSIRI-C), pp. 62–69.

  • Endo, A. T. & Simao, A. (2011). Model-based testing of service-oriented applications via state models. In IEEE international conference on services computing (SCC), pp. 432–439.

  • Endo, A. T. & Simao, A. (2012). Experimental comparison of test case generation methods for finite state machines. In Fifth international conference on software testing, verification and validation. Montreal, QC, pp. 549–558.

  • Endo, A. T., & Simao, A. (2013). Evaluating test suite characteristics, cost, and effectiveness of FSM-based testing methods. Information and Software Technology, 55(6), 1045–1062.

    Article  Google Scholar 

  • European Space Agency (ESA). (2015). Ariane-5: Learning from ight 501 and preparing for 502. Paris, France, 1997. ESA bulletin Nr. 89. http://www.esa.int/esapub/bulletin/bullet89/dalma89.htm. Accessed March 2015.

  • Fujiwara, S., Bochmann, G. V., Khendek, F., Amalou, M., & Ghedamsi, A. (1991). Test selection based on finite state models. IEEE Transactions on Software Engineering, 17(6), 591–603.

    Article  Google Scholar 

  • Fraser, M. D., Kumar, K., & Vaishnavi, V. K. (1991). Informal and formal requirements specification languages: Bridging the gap. IEEE Transactions on Software Engineering, 17(5), 454–466.

    Article  Google Scholar 

  • Fraser, G., & Gargantini, A. (2009). Experiments on the test case length in specification based test case generation. In ICSE Workshop on Automation of Software Test (AST), IEEE, (pp. 18–26).

  • Fraser, G., & Wotawa, A. (2007). Redundancy based test-suite reduction. Fundamental Approaches to Software Engineering Lecture Notes in Computer Science., 4422, 291–305.

    Article  Google Scholar 

  • Gill, A. (1962). Introduction to the theory of finite-state machines. New York: McGraw-Hill.

    MATH  Google Scholar 

  • Gonnenc, G. (1970). A method for the design of fault detection experiments. IEEE Transactions on Computers, 19(6), 551–558.

    Article  Google Scholar 

  • Grieskamp, W., Kicillof, N., Stobie, K., & Braberman, V. (2011). Model-based quality assurance of protocol documentation: Tools and Methodology. Software Testing, Verification and Reliability, 5(1), 55–71.

    Article  Google Scholar 

  • Harel, D. (1987). Statecharts: S visual formalism for complex systems. Science of Computer Programming North-Holland, 8, 231–274.

    Article  MathSciNet  MATH  Google Scholar 

  • Hierons, R. M. (1997). Testing from a Z specification. The Journal of Software Testing, Verification and Reliability, 7(1), 19–33.

    Article  Google Scholar 

  • Hierons, R. M., Bogdanov, K., Bowen, J. P., Cleaveland, R., Derrick, J., Dick, J., et al. (2009). Using formal specifications to support testing. ACM Computing Surveys (CSUR) North-Holland, 41(2), 1–76.

    Article  Google Scholar 

  • IEEE Std 829. (1998). IEEE standard for software test documentation. New York, NY, USA.

  • Jia, Yue, & Harman, M. (2011). An analysis and survey of the development of mutation testing. IEEE Transactions on Software Engineering, 37(5), 649–678.

    Article  Google Scholar 

  • Lee, D., & Yannakakis, M. (2001). Principles and methods of testing finite state machines—A survey. Proceedings of the IEEE, 84(8), 1090–1123.

    Article  Google Scholar 

  • Leveson, N. G., & Turner, C. S. (1993). An investigation of the Therac-25 accidents. Computer, 26, 18–41.

    Article  Google Scholar 

  • Lipschutz, S., & Lipson, M. (1997). Matematica discreta (2nd ed.). Porto Alegre, RS: Bookman-Coleção Schaum.

    Google Scholar 

  • Luo, G., Petrenko, A. V., & Bochmann, G. (1995) Selecting test sequences for partially-specified nondeterministic finite state machines. In International Workshop on Protocol Test Systems, Chapman & Hall, Ltd., (pp. 95–110), London, UK, UK: ACM.

  • Martins, E., Sabião, S. B., & Ambrosio, A. M. (1999). Condata: A tool for automating specification-based test case generation for communication systems. In International conference on system sciences (pp. 303–319). Hawaii, USA.

  • Mathur, A. P. (2008). Foundations of software testing. Delhi, India: Dorling Kindersley (India), Pearson Education in South Asia.

  • NASA jet propulsion laboratory. (2014). Technical report, Cape Canaveral Air Force Station, Florida. http://www.jpl.nasa.gov/missions/mars-climate-orbiter/. Accessed March 2015.

  • Naito, S., & Tsunoyama, M. (1981). Fault detection for sequential machines by transition tours. In Proceedings of the 11th IEEE fault tolerant computing conference (FTCS 1981), pp. 238–243.

  • Neumann, F. (2004). Expected runtimes of evolutionary algorithms for the Eulerian cycle problem. In Congress on evolutionary computation, CEC2004, pp. 904–910.

  • Nguyen, C. D., Marchetto, A., & Tonella, P. (2012). Combining model-based and combinatorial testing for effective test case generation. In (International symposium on software testing and analysis (ISSTA)) (pp. 15–20). Minneapolis, MN, USA.

  • Offutt, A. J. (1993). Experimental results from an automatic test case generator. ACM Transactions on Software Engineering Methodology, 2, 109–127.

    Article  Google Scholar 

  • Oishi, M., Tomlin C, & Degani, A. (2003). Discrete abstractions of hybrid systems: Verification of safety and application to user-interface design. NASA/TM-2003-212803. http://goo.gl/AJCl1K. Accessed March 2015.

  • Peterson, J. L. (1977). Petri nets. ACM Computing Surveys, 9(3), 223–252.

    Article  MathSciNet  MATH  Google Scholar 

  • Petrenko, A., & Yevtushenko, N. (2005). Testing from partial deterministic FSM specifications (Vol. 54, No. 4, pp. 1154–1165). Washington, DC, USA.

  • Pretschner, A., Prenninger, W., Wagner, S., Kuhnel, C., Baumgartner, M., Sostawa, B., et al. (2005). One evaluation of model based testing and its automation. In 27th International conference on software engineering (ICSE), St Louis, USA, (pp. 392–401).

  • Pimont, S., & Rault, J. (1976). A software reliability assessment based on a structural and behavioral analysis of programs. In International conference on software engineering (ICSE) (pp. 486–491). CA: IEEE.

  • Pinheiro, A. C., Simão, A., & Ambrosio, A. M. (2014). FSM-based test case generation methods applied to test the communication software on board the ITASAT University Satellite: A Case Study. Journal of Aerospace Technology and Management, 6(4), 447–461.

    Article  Google Scholar 

  • Pontes, R. P., Véras, P. C., Ambrosio, A. M., & Villani, E. (2014). Contributions of model checking and CoFI methodology to the development of space embedded software. Empirical Software Engineering, 19, 39–68.

    Article  Google Scholar 

  • Robinson-Mallett, C., & Liggesmeyer P. (2006). State Identification and verification using a model checker, Software Engineering, pp. 131–142.

  • Sabnani, K. K., & Dahbura, A. (1988). A protocol test generation procedure. Computer NetWorks and ISDN Systems, 15(4), 285–297.

    Article  Google Scholar 

  • Santiago, V., Amaral, A. S. M., Vijaykumar, N. L., Matiello-Francisco, M. F., Martins, E., & Lopes, O. C. (2006). A practical approach for automated test case generation using Statecharts. In Annual international computer software & applications conference (COMPSAC)—International Workshop on Testing and Quality Assurance for Component - Based Systems (TQACBS), 30. Chicago, IL, USA, pp. 183–188.

  • Santiago, V., Mattiello, F., Costa, R., Silva, W. P., & Ambrósio, A. M. (2007). QSEE project: An experience in outsourcing software development for space. In International conference on software engineering and knowledge engineering (SEKE’07) (pp. 183–188). Boston, USA.

  • Santiago, V., Vijaykumar, N. L., Guimarães, D., Amaral, A. S., & Souza, E. F. (2008). An environment for automated test case generation from statechart-based and finite state machine-based behavioral models. In 4th A-MOST (2008), first IEEE international conferente on software testing verification and validation (ICST2008) (pp. 63–72). Lillehammer: IEEE.

  • Santiago Júnior, V. A. (2011). SOLIMVA: A methodology for generating model-based test cases from natural language requirements and detecting incompleteness in software specifications, 264 p. Thesis (Doctorate at Post Graduation Course in Applied Computing)—National Institute for Space Research, São José dos Campos, SP, Brazil.

  • Santiago Júnior, V. A., Cristiá, M., & Vijaykumar, N. L. (2010). Model-based test case generation using Statecharts and Z: A comparison and a combined approach, 72 p. (INPE-16677-RPQ/850).

  • Santiago Júnior, V. A., & Vijaykumar, N. L. (2012). Generating model-based test cases from natural language requirements for space application software. Software Quality Journal, 20, 77–143.

    Article  Google Scholar 

  • Santiago Júnior, V. A., Vijaykumar, N. L., Souza, E. F., Guimarães, D., & Costa, R. C. (2012). GTSC: Automated model-based test case generation from statecharts and finite state machines. In Tools session of the III the Brazilian conference on software: Theory and oractice, Natal-RN, pp. 25–30.

  • Sidhu, D. P., & Leung, T. (1989). Formal methods for protocol testing: A detailed study. Transactions on software engineering. IEEE Computer Society Department of Computer Science (Vol. 13, No. 4, pp. 413–426). Baltimore, MD: Maryland University.

  • Simão, A., & Petrenko, A. (2010). Fault coverage-driven incremental test generation. Computer Journal, 53, 1508–1522.

    Article  Google Scholar 

  • Simão, A., Petrenko, A., & Maldonado, J. C. (2009). Comparing finite state machine test coverage criteria. IET Software, 3(2), 91–105.

    Article  Google Scholar 

  • Shen, X., & Li, G. (1992). A new protocol conformance test generation method and experimental results. ACM/SIGAPP symposium on applied computing (pp. 75–84). New York, NY, USA: ACM.

  • Shen, X., Scoggins, S., & Tang, A. (1991). An improved rcp-method for protocol test generation using backward UIO sequences. In Symposium on Applied Computing (pp. 284–293). New York: ACM.

  • Shen, Y. N., Lombardi, F., & Dahbura, A. T. (1984). Protocol conformance testing using multiple UIO sequences. In International symposium on protocol specification, testing and verification IX (pp. 131–143). Amsterdam, The Netherlands: North-Holland Publishing Co.

  • Shirole, M., Suthar, A., & Kumar, R. (2011). Generation of improved test cases from UML state diagram using genetic algorithm. In 4th India software engineering conference (ISEC), (pp. 23–27). Thiruvananthapuram, Kerala, India.

  • Schur, M., Roth, A., & Zeller, A. (2014). ProCrawl: Mining test models from multi-user web applications. In International symposium on software testing and analysis (ISSTA) (pp. 413–416). San Jose, CA.

  • Soucha, M. (2014). Finite state machine state identification sequences.Open informatics—computer and information science. Faculty of Electrical Engineering. Department of Cybernetics.

  • Souza, S. R. S. (2000). Validação de especificações de sistemas reativos: Definição e análise de critérios de teste. Thesis (PhD in computational physics)—Institute of Physics of São Carlos (IFSC/USP).

  • Souza, E. F., Santiago, V., Guimaraes, D., & Vijaykumar, N. L. (2008). Evaluation of test criteria for space application software modeling in statecharts. In International conference on inovation in software engineering (ISE. (2008). IEEE computer society (pp. 157–162). Viena: Aústria.

  • Souza, E. F. (2010). Geração de casos de teste para sistemas da área espacial usando critérios de teste para máquinas de estados finitos. (INPE-16682-TDI/1627). Dissertation (Master in Applied Computing)—National Institute for Space Research (INPE), São José dos Campos, SP, Brazil. http://urlib.net/8JMKD3MGP7W/36T9LLB. Accessed March 2015.

  • Sun, H., Gaol M., & Lian, A. (2001). Study on UIO sequence generation for sequential machine’s functional test. In International conference (pp. 628–632). Shanghai.

  • Utting, M., & Legeard, B. (2006). Practical model-based testing: A tools approach. Waltham, MA, USA: Morgan Kaufmann Publishers Inc.

    Google Scholar 

  • Wu, C., & Huang, C. (2013). The web services composition testing based on extended finite state machine and UML model. In 5th International conference on service science and innovation (ICSSI) (pp. 29–31). Kaohsiung.

  • Yang, B., & Ural, H. (1990). Protocol conformance test generation using multiple UIO sequences with overlapping. SIGCOMM Computer Communication Review, 20(4), 118–125.

    Article  Google Scholar 

  • Zander, J., Schieferdecker, I., & Mosterman, P. (2011). Model-based testing for embedded systems (computational analysis, synthesis, and design of dynamic systems). New York: Taylor & Francis Group.

    Google Scholar 

  • Zhang, Xinchang, Yang, Meihong, Geng, Guanggang, & Luo, Wanming. (2011). A DFSM-based protocol conformance testing and diagnosing method. Informatica, 22(3), 447–469.

    MathSciNet  MATH  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Departamento de Computação, Universidade Tecnológica Federal do Paraná (UTFPR), Av. Alberto Carazzai, 1640, Cornélio Procópio, PR, 86300-000, Brazil

    Érica Ferreira de Souza

  2. Laboratório Associado de Computação e Matemática Aplicada (LAC), Instituto Nacional de Pesquisas Espaciais (INPE), Av. dos Astronautas, 1758, São José dos Campos, SP, 12227-010, Brazil

    Valdivino Alexandre de Santiago Júnior & Nandamudi Lankalapalli Vijaykumar

Authors
  1. Érica Ferreira de Souza
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Valdivino Alexandre de Santiago Júnior
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Nandamudi Lankalapalli Vijaykumar
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Érica Ferreira de Souza.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Souza, É.F.d., Santiago Júnior, V.A.d. & Vijaykumar, N.L. H-Switch Cover: a new test criterion to generate test case from finite state machines. Software Qual J 25, 373–405 (2017). https://doi.org/10.1007/s11219-015-9300-8

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11219-015-9300-8

Keywords

Search

Navigation