Rahul Gopinath
Mohammad Amin Alipour
Iftekhar Ahmed
Alex Groce
ABSTRACT
Although mutation analysis is considered the best way to evaluate the effectiveness of a test suite, hefty computational cost often limits its use. To address this problem, various mutation reduction strategies have been proposed, all seeking to reduce the number of mutants while maintaining the representativeness of an exhaustive mutation analysis. While research has focused on the reduction achieved, the effectiveness of these strategies in selecting representative mutants, and the limits in doing so have not been investigated, either theoretically or empirically.
We investigate the practical limits to the effectiveness of mutation reduction strategies, and provide a simple theoretical framework for thinking about the absolute limits. Our results show that the limit in improvement of effectiveness over random sampling for real-world open source programs is a mean of only 13.078%. Interestingly, there is no limit to the improvement that can be made by addition of new mutation operators.
Given that this is the maximum that can be achieved with perfect advance knowledge of mutation kills, what can be practically achieved may be much worse. We conclude that more effort should be focused on enhancing mutations than removing operators in the name of selective mutation for questionable benefit.
- A. T. Acree, Jr. On Mutation. PhD thesis, Georgia Institute of Technology, Atlanta, GA, USA, 1980. Google Scholar
Digital Library
- P. Ammann. Transforming mutation testing from the technology of the future into the technology of the present. In International Conference on Software Testing, Verification and Validation Workshops. IEEE, 2015.Google Scholar
- P. Ammann, M. E. Delamaro, and J. Offutt. Establishing theoretical minimal sets of mutants. In International Conference on Software Testing, Verification and Validation, pages 21--30, Washington, DC, USA, 2014. IEEE Computer Society. Google ScholarDigital Library
- J. H. Andrews, L. C. Briand, and Y. Labiche. Is mutation an appropriate tool for testing experiments? In International Conference on Software Engineering, pages 402--411. IEEE, 2005. Google ScholarDigital Library
- J. H. Andrews, L. C. Briand, Y. Labiche, and A. S. Namin. Using mutation analysis for assessing and comparing testing coverage criteria. IEEE Transactions on Software Engineering, 32(8), 2006. Google ScholarDigital Library
- Apache Software Foundation. Apache commons. http://commons.apache.org/.Google Scholar
- Apache Software Foundation. Apache maven project. http://maven.apache.org.Google Scholar
- E. F. Barbosa, J. C. Maldonado, and A. M. R. Vincenzi. Toward the determination of sufficient mutant operators for c. Software Testing, Verification and Reliability, 11(2):113--136, 2001.Google ScholarCross Ref
- T. A. Budd. Mutation Analysis of Program Test Data. PhD thesis, Yale University, New Haven, CT, USA, 1980. Google ScholarDigital Library
- T. A. Budd, R. A. DeMillo, R. J. Lipton, and F. G. Sayward. Theoretical and empirical studies on using program mutation to test the functional correctness of programs. In ACM SIGPLAN-SIGACT symposium on Principles of programming languages, pages 220--233. ACM, 1980. Google ScholarDigital Library
- T. A. Budd, R. J. Lipton, R. A. DeMillo, and F. G. Sayward. Mutation analysis. Yale University, Department of Computer Science, 1979.Google ScholarCross Ref
- V. Chvatal. A greedy heuristic for the set-covering problem. Mathematics of operations research, 4(3):233--235, 1979. Google ScholarDigital Library
- H. Coles. Pit mutation testing. http://pitest.org/.Google Scholar
- H. Coles. Pit mutation testing: Mutators. http://pitest.org/quickstart/mutators.Google Scholar
- M. Daran and P. Thévenod-Fosse. Software error analysis: A real case study involving real faults and mutations. In ACM SIGSOFT International Symposium on Software Testing and Analysis, pages 158--171. ACM, 1996. Google ScholarDigital Library
- R. A. DeMillo, D. S. Guindi, W. McCracken, A. Offutt, and K. King. An extended overview of the mothra software testing environment. In International Conference on Software Testing, Verification and Validation Workshops, pages 142--151. IEEE, 1988.Google ScholarCross Ref
- R. A. DeMillo, R. J. Lipton, and F. G. Sayward. Hints on test data selection: Help for the practicing programmer. Computer, 11(4):34--41, 1978. Google ScholarDigital Library
- L. Deng, J. Offutt, and N. Li. Empirical evaluation of the statement deletion mutation operator. In IEEE 6th ICST, Luxembourg, 2013. Google ScholarDigital Library
- A. Derezińska. A quality estimation of mutation clustering in c# programs. In New Results in Dependability and Computer Systems, pages 119--129. Springer, 2013.Google ScholarCross Ref
- A. Derezińska. Toward generalization of mutant clustering results in mutation testing. In Soft Computing in Computer and Information Science, pages 395--407. Springer, 2015.Google ScholarCross Ref
- U. Feige. A threshold of ln n for approximating set cover. pages 634--652, 1998. Google ScholarDigital Library
- GitHub Inc. Software repository. http://www.github.com.Google Scholar
- R. Gopinath. Replication: Limits of mutation reduction strategies. http://eecs.osuosl.org/rahul/icse2016/.Google Scholar
- R. Gopinath, A. Alipour, I. Ahmed, C. Jensen, and A. Groce. Do mutation reduction strategies matter? Technical report, Oregon State University, Aug 2015. http://hdl.handle.net/1957/56917.Google Scholar
- R. Gopinath, A. Alipour, I. Ahmed, C. Jensen, and A. Groce. How hard does mutation analysis have to be, anyway? In International Symposium on Software Reliability Engineering. IEEE, 2015. Google ScholarDigital Library
- A. (http://cstheory.stackexchange.com/users/37275/anonymous). What is the reverse of greedy algorithm for setcover? Theoretical Computer Science Stack Exchange. url:http://cstheory.stackexchange.com/q/33685 (version: 2016-01-29).Google Scholar
- S. Hussain. Mutation clustering. Master's thesis, King's College London, Strand, London, UK, 2008.Google Scholar
- C. Ji, Z. Chen, B. Xu, and Z. Zhao. A novel method of mutation clustering based on domain analysis. In SEKE, pages 422--425, 2009.Google Scholar
- Y. Jia and M. Harman. Constructing subtle faults using higher order mutation testing. In IEEE International Working Conference on Source Code Analysis and Manipulation, pages 249--258. IEEE, 2008.Google ScholarCross Ref
- Y. Jia and M. Harman. Higher Order Mutation Testing. Information and Software Technology, 51(10):1379--1393, Oct. 2009. Google ScholarDigital Library
- Y. Jia and M. Harman. An analysis and survey of the development of mutation testing. IEEE Transactions on Software Engineering, 37(5):649--678, 2011. Google ScholarDigital Library
- R. Just, D. Jalali, L. Inozemtseva, M. D. Ernst, R. Holmes, and G. Fraser. Are mutants a valid substitute for real faults in software testing? In ACM SIGSOFT Symposium on The Foundations of Software Engineering, pages 654--665, Hong Kong, China, 2014. ACM. Google ScholarDigital Library
- R. M. Karp. Reducibility among combinatorial problems. Springer, 1972.Google ScholarCross Ref
- B. Kurtz, P. Ammann, and J. Offutt. Static analysis of mutant subsumption. In Software Testing, Verification and Validation Workshops (ICSTW), 2015 IEEE Eighth International Conference on, pages 1--10, April 2015.Google Scholar
- R. Kurtz, P. Ammann, M. Delamaro, J. Offutt, and L. Deng. Mutant subsumption graphs. In Workshop on Mutation Analysis, 2014.Google ScholarDigital Library
- R. J. Lipton. Fault diagnosis of computer programs. Technical report, Carnegie Mellon Univ., 1971.Google Scholar
- Y.-S. Ma and S.-W. Kim. Mutation testing cost reduction by clustering overlapped mutants. Journal of Systems and Software, pages --, 2016. Google ScholarDigital Library
- A. Mathur. Performance, effectiveness, and reliability issues in software testing. In Annual International Computer Software and Applications Conference, COMPSAC, pages 604--605, 1991.Google Scholar
- A. P. Mathur. Foundations of Software Testing. Addison-Wesley, 2012. Google ScholarDigital Library
- A. P. Mathur and W. E. Wong. An empirical comparison of data flow and mutation-based test adequacy criteria. Software Testing, Verification and Reliability, 4(1):9--31, 1994.Google ScholarCross Ref
- E. S. Mresa and L. Bottaci. Efficiency of mutation operators and selective mutation strategies: An empirical study. Software Testing, Verification and Reliability, 9(4):205--232, 1999.Google ScholarCross Ref
- A. S. Namin and J. H. Andrews. Finding sufficient mutation operators via variable reduction. In Workshop on Mutation Analysis, page 5, 2006. Google ScholarDigital Library
- A. J. Offutt, G. Rothermel, and C. Zapf. An experimental evaluation of selective mutation. In International Conference on Software Engineering, pages 100--107. IEEE Computer Society Press, 1993. Google ScholarDigital Library
- A. J. Offutt and R. H. Untch. Mutation 2000: Uniting the orthogonal. In Mutation testing for the new century, pages 34--44. Springer, 2001. Google ScholarDigital Library
- A. J. Offutt and J. M. Voas. Subsumption of condition coverage techniques by mutation testing. Technical report, Technical Report ISSE-TR-96-01, Information and Software Systems Engineering, George Mason University, 1996.Google Scholar
- D. Schuler and A. Zeller. Javalanche: Efficient mutation testing for java. In ACM SIGSOFT Symposium on The Foundations of Software Engineering, pages 297--298, Aug. 2009. Google ScholarDigital Library
- A. Siami Namin, J. H. Andrews, and D. J. Murdoch. Sufficient mutation operators for measuring test effectiveness. In International Conference on Software Engineering, pages 351--360. ACM, 2008. Google ScholarDigital Library
- J. Strug and B. Strug. Machine learning approach in mutation testing. In Testing Software and Systems, pages 200--214. Springer, 2012.Google ScholarCross Ref
- R. H. Untch. On reduced neighborhood mutation analysis using a single mutagenic operator. In Annual Southeast Regional Conference, ACM-SE 47, pages 71:1--71:4, New York, NY, USA, 2009. ACM. Google ScholarDigital Library
- W. Wong and A. P. Mathur. Reducing the cost of mutation testing: An empirical study. Journal of Systems and Software, 31(3):185--196, 1995. Google ScholarDigital Library
- W. E. Wong. On Mutation and Data Flow. PhD thesis, Purdue University, West Lafayette, IN, USA, 1993. UMI Order No. GAX94-20921. Google ScholarDigital Library
- J. Zhang, M. Zhu, D. Hao, and L. Zhang. An empirical study on the scalability of selective mutation testing. In International Symposium on Software Reliability Engineering. ACM, 2014. Google ScholarDigital Library
- L. Zhang, M. Gligoric, D. Marinov, and S. Khurshid. Operator-based and random mutant selection: Better together. In IEEE/ACM Automated Software Engineering. ACM, 2013.Google Scholar
- L. Zhang, S.-S. Hou, J.-J. Hu, T. Xie, and H. Mei. Is operator-based mutant selection superior to random mutant selection? In International Conference on Software Engineering, NY, USA, 2010. ACM. Google ScholarDigital Library
Index Terms
- On the limits of mutation reduction strategies
Recommendations
On reduced neighborhood mutation analysis using a single mutagenic operator
ACM-SE 47: Proceedings of the 47th Annual Southeast Regional ConferenceMutation analysis evaluates the quality of a test suite based on its ability to reveal simple faults, termed mutations, injected in the program under test. Unfortunately, the need to execute many slightly different versions, or mutants, of a program ...
Mutation testing cost reduction by clustering overlapped mutants
We defined the term conditionally-overlapped (c-overlapped) mutants.C-overlapped mutants are expected to produce the same results against a test case.Clustering c-overlapped mutants effectively reduces the cost of mutation testing.Clustering c-...
Evaluating N-selective mutation for C programs: unit and integration testing
Mutation testing for the new centuryMutation Testing has been found to be an effective fault revealing criterion. However, its high cost of application, mainly due to the high number of mutants created and the effort to determine the equivalent mutants, has motivated the position of ...
Comments