Skip to main content
SpringerLink
Log in
Book cover

Asian Conference on Intelligent Information and Database Systems

ACIIDS 2016: Intelligent Information and Database Systems pp 235–244Cite as

Higher Order Mutation Testing to Drive Development of New Test Cases: An Empirical Comparison of Three Strategies

  • Conference paper

  • 2341 Accesses

Part of the book series: Lecture Notes in Computer Science ((LNAI,volume 9621))

Abstract

Mutation testing, which includes first order mutation (FOM) testing and higher order mutation (HOM) testing, appeared as a powerful and effective technique to evaluate the quality of test suites. The live mutants, which cannot be killed by the given test suite, make up a significant part of generated mutants and may drive the development of new test cases. Generating live higher order mutants (HOMs) able to drive development of new test cases is considered in this paper. We apply multi-objective optimization algorithms based on our proposed objectives and fitness functions to generate higher order mutants using three strategies: HOMT1 (HOMs generated from all first order mutants), HOMT2 (HOMs generated from killed first order mutants) and HOMT3 (HOMs generated from not-easy-to-kill first order mutants). We then use mutation score indicator to evaluate, which of the three approaches is better suited to drive development of new test cases and, as a result, to improve the software quality.

This is a preview of subscription content, log in via an institution.

Chapter
USD   29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD   39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD   54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Learn about institutional subscriptions

References

  1. DeMillo, R.A., Lipton, R.J., Sayward, F.G.: Hints on test data selection: help for the practicing programmer. IEEE Comput. 11(4), 34–41 (1978)

    Article  Google Scholar 

  2. Hamlet, R.G.: Testing programs with the aid of a compiler. IEEE Trans. Softw. Eng. SE-3(4), 279–290 (1977)

    Article  MathSciNet  MATH  Google Scholar 

  3. Jia, Y., Harman, M.: Higher order mutation testing. Inf. Softw. Technol. 51, 1379–1393 (2009)

    Article  Google Scholar 

  4. Harman, M., Jia, Y., Langdon, W.B.: A manifesto for higher order mutation testing. In: Third International Conference on Software Testing, Verification, and Validation Workshops, (2010)

    Google Scholar 

  5. Langdon, W.B., Harman, M., Jia, Y.: Efficient multi-objective higher order mutation testing with genetic programming. J. Syst. Softw. 83, 2416–2430 (2010)

    Article  Google Scholar 

  6. Jia, Y., Harman, M.: Constructing subtle faults using higher order mutation testing. In: Proceedings of the Eighth International Working Conference Source Code Analysis and Manipulation (2008)

    Google Scholar 

  7. Madeyski, L., Orzeszyna, W., Torkar, R., Józala, M.: Overcoming the equivalent mutant problem: a systematic literature review and a comparative experiment of second order mutation. IEEE Trans. Softw. Eng. 40(1), 23–42 (2014). doi:10.1109/TSE.2013.44

    Article  Google Scholar 

  8. Mresa, E.S., Bottaci, L.: Efficiency of mutation operators and selective mutation strategies: an empirical study. Softw. Test. Verification Reliab. 9(4), 205–232 (1999)

    Article  Google Scholar 

  9. Papadakis, M., Malevris, N.: An empirical evaluation of the first and second order mutation testing strategies. In: Proceedings of the 2010 Third International Conference on Software Testing, Verification, and Validation Workshops, ser. ICSTW 2010, IEEE Computer Society, pp. 90–99 (2010)

    Google Scholar 

  10. Vincenzi, A.M.R., Nakagawa, E.Y., Maldonado, J.C., Delamaro, M.E., Romero, R.A.F.: Bayesian-learning based guidelines to determine equivalent mutants. Int. J. Softw. Eng. Knowl. Eng. 12(6), 675–690 (2002)

    Article  Google Scholar 

  11. Polo, M., Piattini, M., Garcia-Rodriguez, I.: Decreasing the cost of mutation testing with second-order mutants. Softw. Test. Verification Reliab. 19(2), 111–131 (2008)

    Article  Google Scholar 

  12. Nguyen, Q.V., Madeyski, L.: Problems of mutation testing and higher order mutation testing. In: van Do, T., Thi, H.A.L., Nguyen, N.T. (eds.) Advanced Computational Methods for Knowledge Engineering. AISC, vol. 282, pp. 157–172. Springer, Heidelberg (2014). doi:10.1007/978-3-319-06569-4_12

    Chapter  Google Scholar 

  13. Zhu, H., Hall, P.A.V., May, J.H.R.: Software Unit Test Coverage and Adequacy. ACM Comput. Surv. 29(4), 366–427 (1997)

    Article  Google Scholar 

  14. Madeyski, L.: On the effects of pair programming on thoroughness and fault-finding effectiveness of unit tests. In: Münch, J., Abrahamsson, P. (eds.) PROFES 2007. LNCS, vol. 4589, pp. 207–221. Springer, Heidelberg (2007). doi:10.1007/978-3-540-73460-4_20

    Chapter  Google Scholar 

  15. Madeyski, L.: The impact of pair programming on thoroughness and fault detection effectiveness of unit tests suites. Softw. Process: Improv. Pract. 13(3), 281–295 (2008). doi:10.1002/spip.382

    Article  Google Scholar 

  16. Madeyski, L.: The impact of test-first programming on branch coverage and mutation score indicator of unit tests: An experiment. Inf. Softw. Technol. 52(2), 169–184 (2010). doi:10.1016/j.infsof.2009.08.007

    Article  Google Scholar 

  17. Madeyski, L., Radyk, N.: Judy - a mutation testing tool for Java. IET Softw. 4(1), 32–42 (2010). doi:10.1049/iet-sen.2008.0038

    Article  Google Scholar 

  18. Deb, K., Pratap, A., Agarwal, S., Meyarivan, T.: A fast and elitist multi objective genetic algorithm: NSGA-II. IEEE Trans. Evol. Comput. 6(2), 182–197 (2002)

    Article  Google Scholar 

  19. Deb, K., Jain, H.: An evolutionary many-objective optimization algorithm using reference-point-based nondominated sorting approach, Part I: solving problems with box constraints. IEEE Trans. Evol. Comput. 18(4), 577–601 (2014)

    Article  Google Scholar 

  20. Kollat, J.B., Reed, P.M., The value of online adaptive search: a performance comparison of NSGAII, ε-NSGAII and ε-MOEA. In: Coello Coello, C.A., Aguirre, A.H., Zitzler, E. (Eds.), Evolutionary Multi-Criterion Optimization, Third International Conference, EMO 2005 Guanajuato, Mexico, March 9–11 (2005)

    Google Scholar 

  21. Deb, K., Mohan, M., Mishra, S.: A fast multi-objective evolutionary algorithm for finding well-spread pareto-optimal solutions. KenGAL, Report No. 2003002. Indian Institute of Technology, Kanpur, India (2003)

    Google Scholar 

  22. Nguyen, Q.V., Madeyski, L.: Searching for strongly subsuming higher order mutants by applying multi-objective optimization algorithm. In: Le Thi, H.A., Nguyen, N.T., Do, T.V. (eds.) Advanced Computational Methods for Knowledge Engineering. AISC, vol. 358, pp. 391–402. Springer, Heidelberg (2015). doi:10.1007/978-3-319-17996-4_35

    Google Scholar 

  23. Purushothaman, R., Perry, D.E.: Toward Understanding the Rhetoric of small source code changes. IEEE Trans. Softw. Eng. 31(6), 511–526 (2005)

    Article  Google Scholar 

  24. Madeyski, L., Jureczko, M.: Which process metrics can significantly improve defect prediction models? An empirical study. Softw. Qual. J. 23(3), 393–422 (2015). doi:10.1007/s11219-014-9241-7

    Article  Google Scholar 

  25. Jureczko, M., Madeyski, L.: A review of process metrics in defect prediction studies. Metody Informatyki Stosowanej 30(5), 133–145 (2011). http://madeyski.e-informatyka.pl/download/Madeyski11.pdf

    Google Scholar 

  26. Jureczko, M., Madeyski, L.: Towards identifying software project clusters with regard to defect prediction. In: Proceedings of the 6th International Conference on Predictive Models in Software Engineering (PROMISE 2010). ACM, New York, NY, USA, Article 9, 9:1–9:10. doi: 10.1145/1868328.1868342

  27. Madeyski, L., Majchrzak, M.: Software measurement and defect prediction with DePress extensible framework. Found. Comput. Decis. Sci. 39(4), 249–270 (2014). doi:10.2478/fcds-2014-0014

    Google Scholar 

  28. Hryszko, J., Madeyski, L.: Bottlenecks in software defect prediction implementation in industrial projects. Found. Comput. Decis. Sci. 40(1), 17–33 (2015). doi:10.1515/fcds-2015-0002

    Google Scholar 

  29. Nguyen, Q.V., Madeyski, L.: Empirical evaluation of multi-objective optimization algorithms searching for higher order mutants. Cybern. Syst. Int. J. (Accepted 2016). DOI: 10.1080/01969722.2016.1128763 URL: http://madeyski.e-informatyka.pl/download/NguyenMadeyski16CS.pdf

Download references

Author information

Authors and Affiliations

  1. Faculty of Computer Science and Management, Wroclaw University of Technology, Wybrzeze Wyspianskiego 27, 50-370, Wroclaw, Poland

    Quang Vu Nguyen & Lech Madeyski

Authors
  1. Quang Vu Nguyen
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Lech Madeyski
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Quang Vu Nguyen .

Editor information

Editors and Affiliations

  1. Wrocław University of Technology, Wrocław, Poland

    Ngoc Thanh Nguyen

  2. Wrocław University of Technology, Wrocław, Poland

    Bogdan Trawiński

  3. Iwate Prefectural University, Takizawa, Japan

    Hamido Fujita

  4. National University of Kaohsiung, Kaohsiung, Taiwan

    Tzung-Pei Hong

Rights and permissions

Reprints and permissions

Copyright information

© 2016 Springer-Verlag Berlin Heidelberg

About this paper

Cite this paper

Nguyen, Q.V., Madeyski, L. (2016). Higher Order Mutation Testing to Drive Development of New Test Cases: An Empirical Comparison of Three Strategies. In: Nguyen, N.T., Trawiński, B., Fujita, H., Hong, TP. (eds) Intelligent Information and Database Systems. ACIIDS 2016. Lecture Notes in Computer Science(), vol 9621. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-662-49381-6_23

Download citation

  • DOI: https://doi.org/10.1007/978-3-662-49381-6_23

  • Publisher Name: Springer, Berlin, Heidelberg

  • Print ISBN: 978-3-662-49380-9

  • Online ISBN: 978-3-662-49381-6

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation