Skip to main content
SpringerLink
Log in

Automatically performing weak mutation with the aid of symbolic execution, concolic testing and search-based testing

  • Published:
Software Quality Journal Aims and scope Submit manuscript

Abstract

Automating software testing activities can increase the quality and drastically decrease the cost of software development. Toward this direction, various automated test data generation tools have been developed. The majority of existing tools aim at structural testing, while a quite limited number aim at a higher level of testing thoroughness such as mutation. In this paper, an attempt toward automating the generation of mutation-based test cases by utilizing existing automated tools is proposed. This is achieved by reducing the killing mutants’ problem into a covering branches one. To this extent, this paper is motivated by the use of state of the art techniques and tools suitable for covering program branches when performing mutation. Tools and techniques such as symbolic execution, concolic execution, and evolutionary testing can be easily adopted toward automating the test input generation activity for the weak mutation testing criterion by simply utilizing a special form of the mutant schemata technique. The propositions made in this paper integrate three automated tools in order to illustrate and examine the method’s feasibility and effectiveness. The obtained results, based on a set of Java program units, indicate the applicability and effectiveness of the suggested technique. The results advocate that the proposed approach is able to guide existing automating tools in producing test cases according to the weak mutation testing criterion. Additionally, experimental results with the proposed mutation testing regime show that weak mutation is able to speedup the mutant execution time by at least 4.79 times when compared with strong mutation.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

Similar content being viewed by others

Notes

  1. This conclusion was based on the Java programming language.

  2. Short circuit operators (logical) execute or evaluate the second argument of a Boolean expression if the first argument does not suffice to determine the value of the expression.

  3. Schematic functions take both expression operands as arguments and thus, they execute or evaluate the second argument of the expression at the schematic call point.

  4. For the Siemens programs the selected tests were chosen among those that kill all the killable mutants by the available test suite. Thus, they might not kill all the killable mutants.

  5. Siemens programs were rewritten in Java for the purposes of the experiments.

  6. In the experiments, a timeout of 3 s was employed in order to overcome the problems caused by the endless loops.

References

  • Ammann, P., & Offutt, J. (2008). Introduction to software testing. Cambridge: Cambridge University Press.

  • Andrews, J. H., Briand, L. C., & Labiche, Y. (2005). Is mutation an appropriate tool for testing experiments? In Proceedings of the 27th international conference on software engineering, St. Louis, MO, USA (pp. 402–411). 1062530: ACM. doi:10.1145/1062455.1062530.

  • Andrews, J. H., Briand, L. C., Labiche, Y., & Namin, A. S. (2006). Using mutation analysis for assessing and comparing testing coverage criteria. IEEE Transaction on Software Engineering, 32(8), 608–624. doi:10.1109/TSE.2006.83.

    Google Scholar 

  • Ayari, K., Bouktif, S., & Antoniol, G. (2007). Automatic mutation test input data generation via ant colony. In Proceedings of the 9th annual conference on Genetic and evolutionary computation, London, England (pp. 1074–1081). 1277172: ACM. doi:10.1145/1276958.1277172.

  • Baluda, M., Braione, P., Denaro, G., & Pezzè, M. (2010). Structural coverage of feasible code. In Proceedings of the 5th workshop on automation of software test, Cape Town, South Africa (pp. 59–66). 1808275: ACM. doi:10.1145/1808266.1808275.

  • Baresi, L., Lanzi, P. L., & Miraz, M. (2010). Testful: An evolutionary test approach for java. In Software testing, verification and validation (ICST), 2010 third international conference on, 6–10 April 2010 (pp. 185–194). doi:10.1109/ICST.2010.54.

  • Bottaci, L. (2001). A genetic algorithm fitness function for mutation testing. In SEMINAL: Software engineering using metaheuristic inovative algorithms, workshop (pp. 3–7).

  • Burnim, J., & Sen, K. (2008). Heuristics for scalable dynamic test generation. In Proceedings of the 2008 23rd IEEE/ACM international conference on automated software engineering (pp. 443–446). 1642995: IEEE Computer Society. doi:10.1109/ASE.2008.69.

  • DeMillo, R. A., Lipton, R. J., & Sayward, F. G. (1978). Hints on test data selection: Help for the practicing programmer. Computer, 11(4), 34–41. doi:10.1109/C-M.1978.218136.

    Google Scholar 

  • DeMillo, R. A., & Offutt, A. J. (1991). Constraint-based automatic test data generation. IEEE Transaction on Software Engineering, 17(9), 900–910. doi:10.1109/32.92910.

    Google Scholar 

  • Fraser, G., & Zeller, A. (2010). Mutation-driven generation of unit tests and oracles. In Proceedings of the 19th international symposium on software testing and analysis, Trento, Italy (pp. 147–158). 1831728: ACM. doi:10.1145/1831708.1831728.

  • Hamlet, R. G. (1977). Testing programs with the aid of a compiler. IEEE Transaction on Software Engineering, 3(4), 279–290. doi:10.1109/TSE.1977.231145.

  • Harman, M., & McMinn, P. (2007). A theoretical & empirical analysis of evolutionary testing and hill climbing for structural test data generation. In Proceedings of the 2007 international symposium on Software testing and analysis, London, United Kingdom (pp. 73–83). 1273475: ACM. doi:10.1145/1273463.1273475.

  • Harrold, J., & Rothermel, G. (2010). Siemens programs, hr variants. http://www.cc.gatech.edu/aristotle/Tools/subjects/, May 2010.

  • Howden, W. E. (1982). Weak mutation testing and completeness of test sets. IEEE Transaction on Software Engineering, 8(4), 371–379. doi:10.1109/TSE.1982.235571.

  • Jia, Y., & Harman, M. (2010). An analysis and survey of the development of mutation testing. IEEE Transactions on Software Engineering, 99(PrePrints), doi:10.1109/TSE.2010.62.

  • King, J. C. (1976). Symbolic execution and program testing. Communication ACM, 19(7), 385–394. doi:10.1145/360248.360252.

  • King, K. N., & Offutt, A. J. (1991). A fortran language system for mutation-based software testing. Software Practical Experiment, 21(7), 685–718. doi:10.1002/spe.4380210704.

  • Lakhotia, K., Harman, M., & Gross, H. (2010). Austin: A tool for search based software testing for the c language and its evaluation on deployed automotive systems. In 2nd International symposium on search based software engineering. doi:10.1109/SSBSE.2010.21.

  • Li, N., Praphamontripong, U., & Offutt, A. J. (2009). An experimental comparison of four unit test criteria: Mutation, edge-pair, all-uses and prime path coverage. In Proceedings of the 4th international workshop on mutation analysis (MUTATION’09), Denver, Colorado, 1–4 April 2009 (pp. 220–229). doi:10.1109/ICSTW.2009.30.

  • Ma, Y.-S., Offutt, J., & Kwon, Y. R. (2005). Mujava: An automated class mutation system: Research articles. Software Test Verification Reliability, 15(2), 97–133. doi:10.1002/stvr.v15:2.

    Google Scholar 

  • Maldonado, J. C., Delamaro, M. E., Fabbri, S. C. P. F., Simão, A. D. S., Sugeta, T., Vincenzi, A. M. R., et al. (2001). Proteum: A family of tools to support specification and program testing based on mutation. In W. E. Wong (Ed.), Mutation testing for the new century (pp. 113–116). Kluwer.

  • Mastorantonakis, M., & Malevris, N. (2003). An effective metrod for mutating java programs. In Proceedings of the IASTED international conference on software engineering and applications (Vol. 7, pp. 252–257).

  • Offutt, A. J. (1992). Investigations of the software testing coupling effect. ACM Transaction on Software Engineering Methodology, 1(1), 5–20. doi:10.1145/125489.125473.

  • Offutt, A. J., Jin, Z., & Pan, J. (1999). The dynamic domain reduction procedure for test data generation. Software Practical Experiment, 29(2), 167–193. doi:10.1002/(SICI)1097-024X(199902)29:2<167::AID-SPE225>3.3.CO;2-M.

  • Offutt, A. J., & Lee, S. D. (1994). An empirical evaluation of weak mutation. IEEE Transaction Software Engineering, 20(5), 337–344. doi:10.1109/32.286422.

    Google Scholar 

  • Offutt, A. J., Lee, A., Rothermel, G., Untch, R. H., & Zapf, C. (1996). An experimental determination of sufficient mutant operators. ACM Transaction Software Engineering Methodology, 5(2), 99–118. doi:10.1145/227607.227610.

    Google Scholar 

  • Offutt, A. J., & Pan, J. (1997). Automatically detecting equivalent mutants and infeasible paths. Software Testing, Verification and Reliability, 7, 165–192, doi:10.1002/(SICI)1099-1689(199709)7:3<165::AID-STVR143>3.0.CO;2-U.

  • Offutt, A. J., & Untch, R. H. (2001). Mutation 2000: Uniting the orthogonal. In Mutation testing for the new century (pp. 34–44). Kluwer.

  • Offutt, J., & Voas, J. (1996). Subsumption of condition coverage techniques by mutation testing. Dept. of Information and Software Systems Engineering, George Mason Univ., Fairfax, VA.

  • Papadakis, M., & Malevris, N. (2009). An effective path selection strategy for mutation testing. In Proceedings of the 16th Asia-Pacific software engineering conference, 2009 (pp. 422–429). IEEE Computer Society. doi:10.1109/APSEC.2009.68.

  • Papadakis, M., & Malevris, N. (2010a). Automatic mutation test case generation via dynamic symbolic execution. In Software reliability engineering (ISSRE), 2010 IEEE 21st international symposium on, 1–4 Nov. 2010 (pp. 121–130). doi:10.1109/ISSRE.2010.38.

  • Papadakis, M., & Malevris, N. (2010b). An empirical evaluation of the first and second order mutation testing strategies. In Software testing, verification, and validation workshops (ICSTW), 2010 third international conference on, 6–10 April 2010 (pp. 90–99). doi:10.1109/ICSTW.2010.50.

  • Papadakis, M., Malevris, N., & Kallia, M. (2010). Towards automating the generation of mutation tests. In Proceedings of the 5th workshop on automation of software test, Cape Town, South Africa, 2010 (pp. 111–118). 1808283: ACM. doi:10.1145/1808266.1808283.

  • Păsăreanu, C. S., Mehlitz, P. C., Bushnell, D. H., Gundy-Burlet, K., Lowry, M., Person, S., et al. (2008). Combining unit-level symbolic execution and system-level concrete execution for testing nasa software. In Proceedings of the 2008 international symposium on software testing and analysis, Seattle, WA, USA (pp. 15–26). 1390635: ACM. doi:10.1145/1390630.1390635.

  • Polo, M., Piattini, M., & García-Rodríguez, I. (2009). Decreasing the cost of mutation testing with second-order mutants. Software Testing, Verification and Reliability, 19(2), 111–131. doi:10.1002/stvr.392.

    Google Scholar 

  • Schuler, D., & Zeller, A. (2010). (un-)covering equivalent mutants. In Software testing, verification and validation (ICST), 2010 third international conference on, 6–10 April 2010 (pp. 45–54). doi:10.1109/ICST.2010.30.

  • Sen, K., Marinov, D., & Agha, G. (2005). Cute: A concolic unit testing engine for c. In Proceedings of the 10th European software engineering conference held jointly with 13th ACM SIGSOFT international symposium on Foundations of software engineering, Lisbon, Portugal (pp. 263–272). 1081750: ACM. doi:10.1145/1081706.1081750.

  • Smith, B. H., & Williams, L. (2009). On guiding the augmentation of an automated test suite via mutation analysis. Empirical Software Engineering, 14(3), 341–369. doi:10.1007/s10664-008-9083-7.

    Google Scholar 

  • Tonella, P. (2004). Evolutionary testing of classes. SIGSOFT Software Engineering Notes, 29(4), 119–128. doi:10.1145/1013886.1007528.

  • Untch, R. H., Offutt, A. J., & Harrold, M. J. (1993). Mutation analysis using mutant schemata. In Proceedings of the 1993 ACM SIGSOFT international symposium on software testing and analysis, Cambridge, Massachusetts, United States (pp. 139–148). 154265: ACM. doi:10.1145/154183.154265.

  • Visser, W., Păsăreanu, C. S., & Khurshid, S. (2004). Test input generation with java pathfinder. In Proceedings of the 2004 ACM SIGSOFT international symposium on software testing and analysis, Boston, Massachusetts, USA (pp. 97–107). 1007526: ACM. doi:10.1145/1007512.1007526.

  • Yates, D., & Malevris, N. (1989). Reducing the effects of infeasible paths in branch testing. SIGSOFT Software Engineering Notes 14(8), 48–54. doi:10.1145/75309.75315.

Download references

Author information

Authors and Affiliations

  1. Department of Informatics, Athens University of Economics and Business, Athens, Greece

    Mike Papadakis & Nicos Malevris

Authors
  1. Mike Papadakis
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Nicos Malevris
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Nicos Malevris.

Additional information

This paper is an extended version of the (Papadakis et al. 2010) paper.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Papadakis, M., Malevris, N. Automatically performing weak mutation with the aid of symbolic execution, concolic testing and search-based testing. Software Qual J 19, 691–723 (2011). https://doi.org/10.1007/s11219-011-9142-y

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11219-011-9142-y

Keywords

Search

Navigation