Alessandro Pizzoleto
Fabiano Ferrari
ABSTRACT
Context: Mutation testing has been investigated since the late 70’s. Ever since, researchers have devised dozens of mutation approaches, including ways of generating, executing, and analyzing mutants, as well as ways of reducing the costs for its application. However, research on this field falls short when it comes to producing a representative and manageable set of artifacts to enable experiments with the plethora of existing mutation approaches. Objective: In this paper, we describe the process and current results of creating a knowledge base of mutation-related artifacts to support experiments with mutation testing. Method: We setup the Evosuite tool for generating test cases, and the PIT tool for generating and running mutants. We also created scripts to import the results to a relational database. The database includes some procedures to generate killing matrices for the tested Java classes. Results: Beyond establishing the tooling infrastructure, we populated our database with classes extracted from five Java projects, from which four are open source projects hosted in GitHub and the fifth one is composed by simple Java programs. Currently, the database includes around 2,000 classes, 50,000 test cases, and 195,000 mutants. Conclusion: The database structure eases the addition of other Java programs and the related mutation artifacts. Furthermore, it allows for tasks such as minimizing test sets and mutant sets (e.g. by removing redundant tests and trivial mutants), thus providing researchers with a well-established and extensible basis to support varied experiments.
- J. H. Andrews, L. C. Briand, and Y. Labiche. 2005. Is Mutation an Appropriate Tool for Testing Experiments?. In ICSE’05.Google Scholar
- E. F. Barbosa, J. C. Maldonado, and A. M. R. Vincenzi. 2001. Toward the Determination of Sufficient Mutant Operators for C. STVR 11, 2 (2001).Google Scholar
- L. D. Dallilo, A. V. Pizzoleto, and F. C. Ferrari. 2019. An Evaluation of Internal Program Metrics as Predictors of Mutation Operator Score. In SAST’19.Google Scholar
- R. A. DeMillo, R. J. Lipton, and F. G. Sayward. 1978. Hints on Test Data Selection: Help for the Practicing Programmer. IEEE Computer 11, 4 (1978).Google Scholar
- H. Do, S. Elbaum, and G. Rothermel. 2005. Supporting Controlled Experimentation with Testing Techniques: An Infrastructure and Its Potential Impact. Empirical Software Engineering 10, 4 (2005).Google Scholar
- M. Hutchins et al.1994. Experiments on the Effectiveness of Dataflow- and control-flow-based test adequacy criteria. In ICSE’94.Google Scholar
- Y. Jia and M. Harman. 2011. An Analysis and Survey of the Development of Mutation Testing. IEEE Trans. on Soft. Eng. 37, 5 (2011).Google ScholarDigital Library
- R. Just, D. Jalali, and M. D. Ernst. 2014. Defects4J: A Database of existing faults to enable controlled testing studies for Java programs. In ISSTA’14 - Tools Track.Google Scholar
- R. Just et al.2014. Are Mutants a Valid Substitute for Real Faults in Software Testing?. In FSE’14.Google Scholar
- M. Kintis and N. Malevris. 2015. MEDIC: A Static Analysis Framework for Equivalent Mutant Identification. Information and Software Technology 68 (2015).Google Scholar
- M. Kintis, M. Papadakis, Y. Jia, N. Malevris, Y. Le Traon, and M. Harman. 2018. Detecting Trivial Mutant Equivalences via Compiler Optimisations. IEEE Transactions on Software Engineering 44, 4 (2018).Google ScholarCross Ref
- L. F. M. Oliveira. 2010. Tackling The Useless Mutants Problem. Tese. Centro de Informática, Universidade Federal de Pernambuco.Google Scholar
- M. Papadakis, Y. Jia, M. Harman, and Y. Le Traon. 2015. Trivial Compiler Equivalence: A Large Scale Empirical Study of a Simple, Fast and Effective Equivalent Mutant Detection Technique. In ICSE’15.Google Scholar
- M. Papadakis, M. Kintis, J. Zhang, Y. Jia, Yves Le Traon, and M. Harman. 2019. Mutation Testing Advances: An Analysis and Survey. In Advances in Computers.Google Scholar
- D. K. Phan, Y. Kim, and M. Kim. 2018. MUSIC: Mutation Analysis Tool with High Configurability and Extensibility. In Mutation’18.Google Scholar
- A. V. Pizzoleto, F. C. Ferrari, L. D. Dallilo, and J. Offutt. 2020. SiMut: Exploring Program Similarity to Support the Cost Reduction of Mutation Testing. In Mutation’20.Google Scholar
- A. V. Pizzoleto, F. C. Ferrari, A. J. Offutt, L. Fernandes, and M. Ribeiro. 2019. A Systematic Literature Review of Techniques and Metrics to Reduce the Cost of Mutation Testing. Journal of Systems and Software 157 (2019).Google Scholar
- J. M. Rojas, J. Campos, M. Vivanti, G. Fraser, and A. Arcuri. 2015. Combining Multiple Coverage Criteria in Search-Based Unit Test Generation. In Search-Based Software Engineering. Springer, Cham, 93–108.Google Scholar
- Definition of a Knowledge Base Towards a Benchmark for Experiments with Mutation Testing
Recommendations
Mitigating the effects of flaky tests on mutation testing
ISSTA 2019: Proceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and AnalysisMutation testing is widely used in research as a metric for evaluating the quality of test suites. Mutation testing runs the test suite on generated mutants (variants of the code under test), where a test suite kills a mutant if any of the tests fail ...
Faster mutation testing inspired by test prioritization and reduction
ISSTA 2013: Proceedings of the 2013 International Symposium on Software Testing and AnalysisMutation testing is a well-known but costly approach for determining test adequacy. The central idea behind the approach is to generate mutants, which are small syntactic transformations of the program under test, and then to measure for a given test ...
Predictive mutation testing
ISSTA 2016: Proceedings of the 25th International Symposium on Software Testing and AnalysisMutation testing is a powerful methodology for evaluating test suite quality. In mutation testing, a large number of mutants are generated and executed against the test suite to check the ratio of killed mutants. Therefore, mutation testing is widely ...
Comments