Beatriz Souza
ABSTRACT
Comparing the mutation scores achieved for test suites, one is able to judge which test suite is more effective. However, it is not known if the mutation score is a fair metric to do such comparison. In this paper, we present an empirical study, which compares developer-written and automatically generated test suites in terms of mutation score and in relation to the detection ratios of 7 mutation types. Our results indicate fairness on the mutation score.
- 2019. EvoSuite: Automatic Test Suite Generation for Java (2019). http://www.evosuite.org/evosuite/ Accessed: 2019-09-04.Google Scholar
- 2019. PITest Mutation Testing Tool for Java (2019). http://pitest.org/ Accessed: 2019-09-04.Google Scholar
- 2019. Randoop: Automatic unit test generation for Java (2019). https://randoop.github.io/randoop/ Accessed: 2019-09-04.Google Scholar
- Michael Andersson. 2017. An Experimental Evaluation of PIT's Mutation Operators. Bachelor thesis. Umeå University.Google Scholar
- J. H. Andrews, L. C. Briand, and Y. Labiche. 2005. Is mutation an appropriate tool for testing experiments? [software testing]. In Proceedings. 27th International Conference on Software Engineering, 2005. ICSE 2005. IEEE, Saint Louis, MO, USA, USA, 402-411. Google ScholarDigital Library
- Henry Coles, Thomas Laurent, Christopher Henard, Mike Papadakis, and Anthony Ventresque. 2016. PIT: A Practical Mutation Testing Tool for Java (Demo). In Proceedings of the 25th International Symposium on Software Testing and Analysis (ISSTA 2016). ACM, New York, NY, USA, 449-452. Google ScholarDigital Library
- Leonardo Fernandes, Márcio Ribeiro, Luiz Carvalho, Rohit Gheyi, Melina Mongiovi, André Santos, Ana Cavalcanti, Fabiano Ferrari, and José Carlos Maldonado. 2017. Avoiding Useless Mutants. In Proceedings of the 16th ACM SIGPLAN International Conference on Generative Programming: Concepts and Experiences (GPCE 2017). ACM, New York, NY, USA, 187-198. Google ScholarDigital Library
- Gordon Fraser and Andrea Arcuri. 2011. EvoSuite: Automatic Test Suite Generation for Object-oriented Software. In Proceedings of the 19th ACM SIGSOFT Symposium and the 13th European Conference on Foundations of Software Engineering (ESEC/FSE '11). ACM, New York, NY, USA, 416-419. Google ScholarDigital Library
- James H. Andrews, Lionel Briand, Yvan Labiche, and Akbar Siami Namin. 2006. Using Mutation Analysis for Assessing and Comparing Testing Coverage Criteria. Software Engineering, IEEE Transactions on 32 (09 2006), 608-624. Google ScholarDigital Library
- Y. Jia and M. Harman. 2011. An Analysis and Survey of the Development of Mutation Testing. IEEE Transactions on Software Engineering 37, 5 (Sep. 2011), 649-678. Google ScholarDigital Library
- René Just, Darioush Jalali, Laura Inozemtseva, Michael D. Ernst, Reid Holmes, and Gordon Fraser. 2014. Are Mutants a Valid Substitute for Real Faults in Software Testing?. In Proceedings of the 22Nd ACM SIGSOFT International Symposium on Foundations of Software Engineering (FSE 2014). ACM, New York, NY, USA, 654-665. Google ScholarDigital Library
- M. Kintis, M. Papadakis, Y. Jia, N. Malevris, Y. Le Traon, and M. Harman. 2018. Detecting Trivial Mutant Equivalences via Compiler Optimisations. IEEE Transactions on Software Engineering 44, 4 (April 2018), 308-333.Google ScholarCross Ref
- J. S. Kracht, J. Z. Petrovic, and K. R. Walcott-Justice. 2014. Empirically Evaluating the Quality of Automatically Generated and Manually Written Test Suites. In 2014 14th International Conference on Quality Software. IEEE, Dallas, TX, USA, 256-265. Google ScholarDigital Library
- Carlos Pacheco and Michael D. Ernst. 2007. Randoop: Feedback-directed Random Testing for Java. In Companion to the 22Nd ACM SIGPLAN Conference on Object-oriented Programming Systems and Applications Companion (OOPSLA '07). ACM, New York, NY, USA, 815-816. Google ScholarDigital Library
Index Terms
- Is mutation score a fair metric?
Recommendations
Are mutation scores correlated with real fault detection?: a large scale empirical study on the relationship between mutants and real faults
ICSE '18: Proceedings of the 40th International Conference on Software EngineeringEmpirical validation of software testing studies is increasingly relying on mutants. This practice is motivated by the strong correlation between mutant scores and real fault detection that is reported in the literature. In contrast, our study shows that ...
Investigating the Correlation between Mutation Score and Coverage Score
UKSIM '13: Proceedings of the 2013 UKSim 15th International Conference on Computer Modelling and SimulationStrong test suites ensure the correctness and quality of software. Coverage and Mutation score are widespread metrics for evaluating the quality of a test suite. Mutation analysis process improves a test suite to obtain higher coverage scores. We ...
On the Improvement of the Mutation Score Using Distinguishing Test Cases
ICST '11: Proceedings of the 2011 Fourth IEEE International Conference on Software Testing, Verification and ValidationSoftware testing, i.e., discovering software failures through test case execution, plays a crucial role in the software development process. A high quality software must have a strong test suite. Therefore it is of high importance for a software to ...
Comments