Hoda Khalil
ABSTRACT
Research demonstrated that faults seeded mutation using operators can be representative of faults in real systems. In this paper, we study the relationship between the different operators used to insert mutants in the fault domain of the system under test and the effectiveness of different state machine test suites at killing those mutants. We are particularly interested in the effectiveness of two interrelated state machine testing strategies at finding different types of faults. Those are the round-trip paths strategy and the transition tree strategy. Using empirical evaluation, we compare the effectiveness of more than two thousand unique test suites at killing mutants seeded using eight different mutation operators. We perform experiments on four experimental objects and provide qualitative analysis of the results. We conclude that neither of the two studied strategies is more effective than the other at killing a certain type of mutants. However, the structure of the finite state machine and the nature of the system under test affect the type of faults detected by the different testing strategies.
- Paul Amman and Jeff Offutt. 2016. Introduction to Software Testing (2nd. ed.). Cambridge University Press, New York, NY. DOI: https://10.1017/9781316771273Google Scholar
- James H. Andrews, Lionel C. Briand, and Yvan Labiche. 2005. Is Mutation an Appropriate Tool for Testing Experiments ? In Proceedings of the 27th International Conference of Software Engineering (ICSE), May 15-21, 2005, St. Louis, MO, 402-411. DOI: https://doi.org/10.1145/1062455.1062530 Google ScholarDigital Library
- James H. Andrews, Lionel C. Briand, Yvan Labiche, and Akbar Siami Namin. 2006. Using Mutation Analysis for Assessing and Comparing Testing Coverage Criteria. IEEE Trans. Softw. Eng. 32, 8 (Sept. 2006 ), 608-624. DOI: https://doi.org/10.1109/TSE. 2006.83 Google ScholarDigital Library
- Boris Beizer. 1990. Software Testing Techniques (2nd. ed.). Van Nostrand Reinhold Co., New York, NY.Google Scholar
- Robert V. Binder. 1999. Testing Object-Oriented Systems : Models, Patterns, and Tools (1st. ed.). Addison-Wesley Professional, Boston, MA.Google Scholar
- Tsun S. Chow. 1978. Testing Software Design Modeled by FiniteState Machines. IEEE Trans. Softw. Eng. SE-4, 3 (May 1978 ), 178-187. DOI: https://doi.org/10.1109/TSE. 1978.231496 Google ScholarDigital Library
- Andre Takeshi Endo and Adenilso Simao. 2013. Evaluating Test Suite Characteristics, Cost, and Effectiveness of FSM-Based Testing Methods. Inf. Softw. Technol. 55, 6 ( June 2013 ), 1045-1062. DOI: https://doi.org/10.1016/j.infsof. 2013. 01.001 Google ScholarDigital Library
- Nina Elisabeth Holt, Lionel C. Briand, and Richard Torkar. 2014. Empirical Evaluations on the Cost-Effectiveness of State-Based testing: an Industrial Case Study. Inf. Softw. Technol. 56, 8 (Aug. 2014 ), 890-910. DOI: https://doi.org/10.1016/j.infsof. 2014. 02.011 Google ScholarCross Ref
- Nina Elisabeth Holt, Richard Torkar, Lionel Briand, and Kai Hansen. 2012. State-Based Testing: Industrial Evaluation of the CostEffectiveness of Round-Trip Path and Sneak-Path Strategies. In Proceedings of the IEEE 23rd International Symposium of Software Reliability Engineering (ISSRE), November 27-30, 2012, Dallas, TX, 321-330. DOI: https://doi.org/10.1109/ISSRE. 2012.17 Google ScholarDigital Library
- Rene Just, Franz Schweiggert, and Gregory M. Kapfhammer. 2011. MAJOR: An Efficient and Extensible Tool for Mutation Analysis in a Java Compiler. In Proceedings of the 26th IEEE/ACM International Conference on Automated Software Engineering (ASE), November 6-10, 2011, Lawrence, KS, 612-615. DOI: https://doi.org/10.1109/ASE. 2011.6100138 Google ScholarDigital Library
- Rene. Just, Michael D. Ernst, and Gordon Fraser. 2014. Efficient Mutation Analysis by Propagating and Partitioning Infected Execution States. In Proceedings of the International Symposium on Software Testing and Analysis (ISSTA), July 21-25, 2014, San Jose, CA, 315-326. DOI: https://doi.org/10.1145/2610384.2610388 Google ScholarDigital Library
- Hoda A. Khalil. 2018. FSM Testing Based on Transition Trees and Complete Round Trip Paths Testing Criteria. Ph.D. Dissertation. Carleton University, Ottawa, ON, Canada.Google Scholar
- Hoda Khalil and Yvan Labiche. 2017. Finding All Breadth First Full Spanning Trees in a Directed Graph. In proceedings of the 41st International Computer Software and Applications Conference (COMPSAC), July 4-8, 2017, Turin, Italy, 372-377. DOI: https://doi.org/10.1109/COMPSAC. 2017.128 Google ScholarCross Ref
- Hoda Khalil and Yvan Labiche. 2017. State-Based Tests Suites Automatic Generation Tool (STAGE-1). In Proceedings of the 41st International Computer Software and Applications Conference (COMPSAC), July 4-8, 2017, Turin, Italy, 357-363. DOI: https://doi.org/10.1109/COMPSAC. 2017.221 Google ScholarCross Ref
- Hoda Khalil and Yvan Labiche. 2017. On FSM-Based Testing: An Empirical Study: Complete Round-Trip Versus Transition Trees. In Proceedings of the IEEE 28th International Symposium on Software Reliability Engineering (ISSRE), October 23-26, 2017, Toulouse, France, 305-315. DOI: https://doi.org/10.1109/ISSRE. 2017.34 Google ScholarCross Ref
- Hoda Khalil and Yvan Labiche. 2017. FSM-Testing-TranstiontionTrees-and-RTP. Retrieved from https://github.com/hakhalil/FSMTesting-Transtiontion-Trees-and-RTP.Google Scholar
- David Lee and Mihalis Yannakakis. 1996. Principles and Methods of Testing Finite State Machines-A Survey. Proc. IEEE. 84, 8 ( August 1996 ), 1090-1123. DOI: https://doi.org/10.1109/5.533956 Google ScholarCross Ref
- Yu Seung Ma, Jeff Offutt, and Yong Rae Kwon. 2005. MuJava: An Automated Class Mutation System. Softw. Test. Verif. Reliab. 15, 2 ( June 2005 ), 97-133. DOI: https://doi.org/10.5555/1077303.1077304 Google ScholarDigital Library
- Samar Mouchawrab, Lionel C. Briand, Yvan Labiche, and Massimiliano Di Penta. 2011. Assessing, Comparing, and Combining State Machine-Based Testing and Structural Testing: A Series of Experiments. IEEE Trans. Softw. Eng. 37, 2 (March-April 2011 ), 161-187. DOI: https://doi.org/10.1109/TSE. 2010.32 Google ScholarDigital Library
- Adenilso Simão and Alexandre Petrenko. 2010. Fault CoverageDriven Incremental Test Generation. Comput. J. 53, 9 (November 2010 ), 1508-1522. DOI: https://doi.org/10.1093/comjnl/bxp073 Google ScholarCross Ref
Index Terms
- Comparing transition trees test suites effectiveness for different mutation operators
Recommendations
Achieving scalable mutation-based generation of whole test suites
Without complete formal specification, automatically generated software tests need to be manually checked in order to detect faults. This makes it desirable to produce the strongest possible test set while keeping the number of tests as small as ...
Mutation Operators in BPMN Model
ICIDE 2017: Proceedings of the 2017 International Conference on Industrial Design EngineeringMutation Testing is a technique to evaluate quality of test cases in Software Testing. A tester injects some faults into a source code then tests the source code against the original test cases for evaluating the test case quality. Business Process ...
Growing a Reduced Set of Mutation Operators
ARES '14: Proceedings of the 2014 Ninth International Conference on Availability, Reliability and SecurityAlthough widely considered to be quite powerful, mutation testing is also known for its expense. Three fundamental (and related) sources for much of the expense are (1) the number of mutants, (2) the number of equivalent mutants, and (3) the number of ...
Comments