research-article

Time for Networks: Mutation Testing for Timed Automata Networks

Authors:

Gilles Perrouin,

Pierre Yves SchobbensAuthors Info & Claims

FormaliSE '24: Proceedings of the 2024 IEEE/ACM 12th International Conference on Formal Methods in Software Engineering (FormaliSE)

Pages 44 - 54

https://doi.org/10.1145/3644033.3644378

Published: 06 June 2024 Publication History

Abstract

Mutation Testing (MT) is a technique employed to assess the efficacy of tests by introducing artificial faults, known as mutations, into the system. The goal is to evaluate how well the tests can detect these mutations. These artificial faults are generated using mutation operators, which produce a set of mutations derived from the original system. Mutation operators and frameworks exist for a variety of programming languages, and model-based mutation testing is gaining traction, particularly for timed safety-critical systems. This paper focuses on extending MT to Networks of Timed Automata (NTAs), an area that has not been extensively explored. We introduce mutation operators designed for NTAs specified in UPPAAL, aiming to create temporal interaction faults. We assess the effectiveness of these operators on five UPPAAL NTAs sourced from the literature, specifically examining the generation of equivalent and duplicate mutants. Our results demonstrate a varied prevalence of equivalent mutants (from 12% to 71%) while the number of duplicates is less. In all cases, timed bisimulation was able to process each mutant pair in less than one second.

References

[1]

M Saeed AbouTrab, Steve Counsell, and Robert M Hierons. 2012. Specification mutation analysis for validating timed testing approaches based on timed automata. In 2012 IEEE 36th Annual Computer Software and Applications Conference. IEEE, 660--669.

Digital Library

[2]

Bernhard K. Aichernig and He Jifeng. 2008. Mutation testing in UTP. Formal Aspects of Computing 21, 1--2 (Feb. 2008), 33--64.

Digital Library

[3]

Bernhard K Aichernig, Florian Lorber, and Dejan Ničković. 2013. Time for mutants---model-based mutation testing with timed automata. In Tests and Proofs: 7th International Conference, TAP 2013, Budapest, Hungary, June 16--20, 2013. Proceedings 7. Springer, 20--38.

[4]

Alex Donizeti Betez Alberto. [n. d.]. Formal mutation testing in Circus process algebra. Doutorado em Ciěncias de Computação e Matemática Computacional.

[5]

Rajeev Alur and David L Dill. 1994. A theory of timed automata. Theoretical computer science 126, 2 (1994), 183--235.

[6]

Rajeev Alur, Thomas A. Henzinger, and Moshe Y. Vardi. 1993. Parametric realtime reasoning. In STOC. ACM, 592--601.

[7]

Davide Basile, Maurice H ter Beek, Sami Lazreg, Maxime Cordy, and Axel Legay. 2022. Static detection of equivalent mutants in real-time model-based mutation testing: An Empirical Evaluation. Empirical Software Engineering 27, 7 (2022), 160.

Digital Library

[8]

Davide Basile, Maurice H. ter Beek, Maxime Cordy, and Axel Legay. 2020. Tackling the equivalent mutant problem in real-time systems: the 12 commandments of model-based mutation testing. In SPLC'20: 24th ACM International Systems and Software Product Line Conference, Volume A, Roberto Erick Lopez-Herrejon (Ed.). ACM, 30:1--30:11.

Digital Library

[9]

Gerd Behrmann, Alexandre David, and Kim G Larsen. 2006. A tutorial on Uppaal 4.0. Department of computer science, Aalborg university (2006).

[10]

B. Berthomieu and M. Diaz. 1991. Modeling and verification of time dependent systems using time Petri nets. IEEE Transactions on Software Engineering 17, 3 (1991), 259--273.

Digital Library

[11]

Marius Bozga, Conrado Daws, Oded Maler, Alfredo Olivero, Stavros Tripakis, and Sergio Yovine. 1998. Kronos: A model-checking tool for real-time systems. In Computer Aided Verification: 10th International Conference, CAV'98 Vancouver, BC, Canada, June 28-July 2, 1998 Proceedings 10. Springer, 546--550.

[12]

Kārlis Cerāns. 1993. Decidability of Bisimulation Equivalences for Parallel Timer Processes. In Proceedings of the 4th International Workshop on Computer Aided Verification (CAV'92) (Lecture Notes in Computer Science, Vol. 663), Gregor von Bochmann and David K. Probst (Eds.). Springer-Verlag, 302--315.

[13]

David Cortes. 2023. Mechanical Ventilator Case Study. https://github.com/ventynet/ventynet.

[14]

Jaime Cuartas. 2022. Model-Based Mutation Testing Prototype for Timed Automata. Bachelor's Thesis.

[15]

Jaime Cuartas, Jesús Aranda, Maxime Cordy, James Ortiz, Gilles Perrouin, and Pierre-Yves Schobbens. 2023. MUPPAAL: Reducing and Removing Equivalent and Duplicate Mutants in UPPAAL. In 2023 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW). IEEE, 52--61.

[16]

Jaime Cuartas, David Cortés, Joan S. Betancourt, Jesús Aranda, José I. García, Andrés M. Valencia, and James Ortiz. 2023. Formal Verification of a Mechanical Ventilator Using UPPAAL. In Proceedings of the 9th ACM SIGPLAN International Workshop on Formal Techniques for Safety-Critical Systems (Cascais, Portugal) (FTSCS 2023). Association for Computing Machinery, New York, NY, USA, 2--13.

Digital Library

[17]

Alexandre David, Kim G Larsen, Axel Legay, Ulrik Nyman, and Andrzej Wasowski. 2010. Timed I/O automata: a complete specification theory for real-time systems. In Proceedings of the 13th ACM international conference on Hybrid systems: computation and control. 91--100.

Digital Library

[18]

Rebeka Farkas. 2023. Case Studies. https://github.com/farkasrebus/XtaBenchmarkSuite.

[19]

Phyllis G Frankl, Stewart N Weiss, and Cang Hu. 1997. All-uses vs mutation testing: an experimental comparison of effectiveness. Journal of Systems and Software 38, 3 (1997), 235--253.

Digital Library

[20]

Pierre Ganty, Nicolas Manini, and Francesco Ranzato. 2023. Computing Reachable Simulations. arXiv:2204.11804 [cs.LO]

[21]

M. R. Hansen. 1998. Duration Calculus: A Logical Approach to Real-Time Systems. DFKI Saarbrücken. http://www2.compute.dtu.dk/pubdb/pubs/1902-full.html

[22]

Thomas A Henzinger, Pei-Hsin Ho, and Howard Wong-Toi. 1997. HyTech: A model checker for hybrid systems. In Computer Aided Verification: 9th International Conference, CAV'97 Haifa, Israel, June 22--25, 1997 Proceedings 9. Springer, 460--463.

[23]

John E. Hopcroft, Rajeev Motwani, and Jeffrey D. Ullman. 2006. Introduction to Automata Theory, Languages, and Computation (3rd Edition). Addison-Wesley Longman Publishing Co., Inc., USA.

Digital Library

[24]

Henrik Jensen, Kim Larsen, and Arne Skou. 2002. Modelling and Analysis of a Collision Avoidance Protocol using SPIN and UPPAAL. BRICS Report Series 3 (01 2002).

[25]

Yue Jia and Mark Harman. 2010. An analysis and survey of the development of mutation testing. IEEE transactions on software engineering 37, 5 (2010), 649--678.

[26]

Dilsun Kaynar, Nancy Lynch, Roberto Segala, and Frits Vaandrager. 2010. The theory of timed I/O automata. Morgan & Claypool Publishers.

[27]

Moez Krichen and Stavros Tripakis. 2009. Conformance testing for real-time systems. Formal Methods Syst. Des. 34, 3 (2009), 238--304.

Digital Library

[28]

Antonin Kučera and Richard Mayr. 2002. Why is Simulation Harder than Bisimulation?. In Proceedings of the 13th International Conference on Concurrency Theory (CONCUR'02) (Lecture Notes in Computer Science, Vol. 2421), Luboś Brim, Petr Jančar, Mojmír Křetínský, and Antonín Kučera (Eds.). Springer-Verlag, 594--609.

[29]

Kim G Larsen, Florian Lorber, Brian Nielsen, and Ulrik M Nyman. 2017. Mutation-based test-case generation with ecdar. In 2017 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW). IEEE, 319--328.

[30]

Magnus Lindahl, Paul Pettersson, and Wang Yi. 2001. Formal design and analysis of a gear controller. International Journal on Software Tools for Technology Transfer 3, 3 (01 Aug 2001), 353--368.

[31]

Florian Lorber. 2015. Model-Based Mutation Testing of Synchronous and Asynchronous Real-Time Systems. In 8th IEEE International Conference on Software Testing, Verification and Validation, ICST 2015,. IEEE Computer Society, 1--2.

[32]

Florian Lorber, Kim G Larsen, and Brian Nielsen. 2018. Model-based mutation testing of real-time systems via model checking. In 2018 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW). IEEE, 59--68.

[33]

Lech Madeyski, Wojciech Orzeszyna, Richard Torkar, and Mariusz Jozala. 2013. Overcoming the equivalent mutant problem: A systematic literature review and a comparative experiment of second order mutation. IEEE Transactions on Software Engineering 40, 1 (2013), 23--42.

Digital Library

[34]

Xavier Nicollin and Joseph Sifakis. 1994. The Algebra of Timed Processes, ATP: Theory and Application. Inf. Comput. 114, 1 (1994), 131--178.

Digital Library

[35]

Robert Nilsson, Jeff Offutt, and Sten F Andler. 2004. Mutation-based testing criteria for timeliness. In Proceedings of the 28th Annual International Computer Software and Applications Conference, 2004. COMPSAC 2004. IEEE, 306--311.

[36]

James Jerson Ortiz, Moussa Amrani, and Pierre-Yves Schobbens. 2017. Multi-timed Bisimulation for Distributed Timed Automata. In NASA Formal Methods - 9th International Symposium, NFM 2017, Moffett Field, CA, USA, May 16--18, 2017, Proceedings. 52--67.

[37]

Mike Papadakis, Marinos Kintis, Jie Zhang, Yue Jia, Yves Le Traon, and Mark Harman. 2019. Mutation testing advances: an analysis and survey. In Advances in Computers. Vol. 112. Elsevier, 275--378.

[38]

Alessandro Viola Pizzoleto, Fabiano Cutigi Ferrari, Jeff Offutt, Leo Fernandes, and Márcio Ribeiro. 2019. A systematic literature review of techniques and metrics to reduce the cost of mutation testing. Journal of Systems and Software 157 (2019), 110388.

[39]

Faezeh Siavashi, Dragos Truscan, and Jüri Vain. 2018. Vulnerability Assessment of Web Services with Model-Based Mutation Testing. In 2018 IEEE International Conference on Software Quality, Reliability and Security (QRS).

[40]

Thitima Srivatanakul, John A. Clark, Susan Stepney, and Fiona Polack. 2003. Challenging Formal Specifications by Mutation: a CSP Security Example. In Proceedings of the 10th Asia-Pacific Software Engineering Conference (APSEC'03). Chiang Mai, Thailand, 340--350.

[41]

UppalTeam. [n. d.]. Uppaal Case Studies. https://uppaal.org/casestudies/.

[42]

James Jerson Ortiz Vega, Gilles Perrouin, Moussa Amrani, and Pierre-Yves Schobbens. 2018. Model-based mutation operators for timed systems: a taxonomy and research agenda. In 2018 IEEE International Conference on Software Quality, Reliability and Security (QRS). IEEE, 325--332.

[43]

Wang Yi. 1991. CCS + time = an interleaving model for real time systems. In Automata, Languages and Programming, Javier Leach Albert, Burkhard Monien, and Mario Rodriguez Artalejo (Eds.). Springer Berlin Heidelberg, Berlin, Heidelberg, 217--228.

Cited By

Cuartas JCortés DBetancourt JAranda JCordy MOrtiz JPerrouin GSchobbens P(2024)MUPPAAL: Efficient Elimination and Reduction of Useless Mutants in Real‐Time Model‐Based SystemsSoftware Testing, Verification and Reliability10.1002/stvr.190735:1Online publication date: 12-Nov-2024
https://doi.org/10.1002/stvr.1907

Index Terms

Time for Networks: Mutation Testing for Timed Automata Networks
1. Software and its engineering
  1. Software creation and management
    1. Software verification and validation
      1. Software defect analysis
        Software testing and debugging
2. Theory of computation
  1. Models of computation
    1. Timed and hybrid models

Recommendations

Learning Deterministic One-Clock Timed Automata via Mutation Testing
Automated Technology for Verification and Analysis
Abstract
In active learning, an equivalence oracle is supposed to answer whether a hypothesis model is equivalent to the system under learning. Its implementation in real applications is considered a major bottleneck for active automata learning. The ...
Model-based mutation testing-Approach and case studies

This paper rigorously introduces the concept of model-based mutation testing (MBMT) and positions it in the landscape of mutation testing. Two elementary mutation operators, insertion and omission, are exemplarily applied to a hierarchy of graph-based ...
Bounded determinization of timed automata with silent transitions

Deterministic timed automata are strictly less expressive than their non-deterministic counterparts, which are again less expressive than those with silent transitions. As a consequence, timed automata are in general non-determinizable. This is ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

FormaliSE '24: Proceedings of the 2024 IEEE/ACM 12th International Conference on Formal Methods in Software Engineering (FormaliSE)

April 2024

154 pages

ISBN:9798400705892

DOI:10.1145/3644033

Chair:
Nico Plat,
General Chair:
Stefania Gnesi
Istituto di Scienza e Tecnologie dell'Informazione "Alessandro Faedo", Italy
,
Program Chairs:
Carlo A. Furia,
Antónia Lopes
University of Lisbon, Portugal

Copyright © 2024 Copyright is held by the owner/author(s). Publication rights licensed to ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 06 June 2024

Check for updates

Author Tags

Qualifiers

Research-article

Conference

FormaliSE '24

Sponsor:

SIGSOFT

FormaliSE '24: 2024 IEEE/ACM 12th International Conference on Formal Methods in Software Engineering (FormaliSE)

April 14 - 15, 2024

Lisbon, Portugal

Upcoming Conference

ICSE 2025

2025 IEEE/ACM 46th International Conference on Software Engineering

April 26 - May 3, 2025

Ottawa , ON , Canada

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

1
Total Citations
View Citations
50
Total Downloads

Downloads (Last 12 months)50
Downloads (Last 6 weeks)7

Reflects downloads up to 03 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Cuartas JCortés DBetancourt JAranda JCordy MOrtiz JPerrouin GSchobbens P(2024)MUPPAAL: Efficient Elimination and Reduction of Useless Mutants in Real‐Time Model‐Based SystemsSoftware Testing, Verification and Reliability10.1002/stvr.190735:1Online publication date: 12-Nov-2024
https://doi.org/10.1002/stvr.1907

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten