Skip to main content

Advertisement

Log in

Evolution or revolution: the critical need in genetic algorithm based testing

  • Published:
Artificial Intelligence Review Aims and scope Submit manuscript

Abstract

Software testing is one of the most inevitable processes in software development. The field of software testing has seen an extensive use of search based techniques in the last decade. Among the search based techniques, it is the metaheuristic techniques such as genetic algorithm that has garnered the major share of attention from researchers. Looking at the large body of work that has happened and is happening in this field, we feel that it is high time someone studied how well genetic algorithm based techniques fare in practical testing process. Method: In this work, we present a roadmap to the future of genetic algorithm based software testing, based on a review of literature. We have mainly reviewed the works which use genetic algorithm for software test data generation. This independent review is designed to direct the attention of future researchers to the deficiencies of genetic algorithm based testing, their possible solutions and the extent to which they are correctable. The observations from the selected primary studies highlight the issues faced when genetic algorithm is applied in software testing. The observations form the review reveal that the type of genetic algorithm used, fitness function design, population initialization and parameter settings does impact the quality of solution obtained in software testing using genetic algorithm. From the review we conclude that, more generalized approaches can make genetic algorithm based software testing one of the strongest methods in practical software testing. We hope that, this review will be a major breakthrough in genetic algorithm based software testing field.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12

Similar content being viewed by others

References

  • Ahmed MA, Hermadi I (2008) GA-based multiple paths test data generator. Comp Opera Res 35:3107–3127

    Article  Google Scholar 

  • Ali S, Briand LC, Hemmati H, Panesar-Walawege RK (2009) A systematic review of the application and empirical investigation of search-based test case generation. In: IEEE Transactions on Software Engineering, vol. 99

  • Ali S, Iqbal MZ, Arcuri A, Briand L (2011) A search-based OCL constraint solver for model-based test data generation. In: Proceedings of the 11th international conference on quality software, pp. 41–50

  • Alshahwan N, Harman M (2014) Coverage and fault detection of the output-uniqueness test selection criteria. In: Proc. ISSTA, pp. 1–12

  • Arcuri A, Fraser G (2011) On parameter tuning in search based software engineering. In: Proc. SSBSE, pp. 33–47

  • Arcuri A, Fraser G (2013) Parameter tuning or default values? An empirical investigation in search-based software engineering. Empir Softw Eng 18(3):594–623

    Article  Google Scholar 

  • Beizer B (1990) Software testing technique. Van Nostrand Rheinhold, New York

    MATH  Google Scholar 

  • Berndt DJ, Watkins A (2004) Investigating the performance of genetic algorithm-based software test case generation. In: Proc. HASE

  • Binkley D, Harman M, Lakhotia K (2009) FlagRemover: a testability transformation for transforming loop assigned Flag. ACM Trans Softw Eng Methodol 2(3):110–146

    Google Scholar 

  • Bueno PM, Jino S (2002) Automatic test data generation for program paths using genetic algorithms. Int J Softw Eng Knowl Eng 12(6):691–709

    Article  Google Scholar 

  • Campos J, Arcuri A, Fraser G, Abreu R (2014) Continuous test generation: enhancing continuous integration with automated test generation. In: Proc. Automated Software Engineering (ASE)

  • Cao Y, Hu C, Li L (2009) An approach to generate software test data for a specific path automatically with genetic algorithm. In: Proc. ICRMS, Chengdu, pp. 888–892

  • Chen C, Xu X, Chen Y, Li X, Guo D (2009) A new method of test data generation for branch coverage in software testing based on EPDG and genetic algorithm. In: Proc. ASID, pp. 307–310

  • Doungsa-ard C, Dahal K, Hossain A, Suwannasart T (2007) Test data generation from UML state machine diagrams using gas. In: Proc. ICSEA

  • Feldt R, Poulding S (2015) Broadening the search in search-based software testing: it need not be evolutionary. In: Proc. Eighth international workshop on search-based software testing, pp. 1–7

  • Fischer M (2012) Generating test data for black-box testing using Genetic algorithms. In: Proc. 17th ETFA, pp. 1–6

  • Fraser G, Arcuri A, McMinn P (2015) A memetic algorithm for whole test suite generation. J Syst Softw 103:311–327. doi:10.1016/j.jss.2014.05.032

    Article  Google Scholar 

  • Fraser G, Arcuri A (2011a) EvoSuite: automatic test suite generation for object-oriented software. In: Proc. ESEC/FSE, Szeged, Hungary, pp. 416–419

  • Fraser G, Arcuri A (2011b) It is not the length that matters, it is how you control it. In: Proc. IEEE international conference on software testing, verification and validation, pp. 150–159

  • Fraser G, Arcuri A (2012) The seed is strong: seeding strategies in search-based software testing. In: Proc. ICST, pp. 121–130

  • Fraser G, Arcuri A (2013) Whole test suite generation. IEEE Trans Softw Eng 39(2):276–291

    Article  Google Scholar 

  • Fraser G, Arcuri A, McMinn P (2013) Test suite generation with memetic algorithms. In: Proc. GECCO

  • Galeotti JP, Fraser G, Arcuri A (2014) Extending a search-based test generator with adaptive dynamic symbolic execution. In: Proc. ISSTA

  • Girgis MR (2005) Automatic test data generation for data flow testing using a genetic algorithm. J Univ Comp Sci 11(5):898–915

    Google Scholar 

  • Goldberg D (1989) Genetic algorithms in search, optimization and machine learning. Addison-Wesley, Boston

    MATH  Google Scholar 

  • Gonga D, Zhanga W, Yaob X (2011) Evolutionary generation of test data for many paths coverage based on grouping. J Syst Softw 84:2222–2233

    Article  Google Scholar 

  • Graham-Rowe D (2002) Radio emerges from the electronic soup. New Sci 175(2358):19

    Google Scholar 

  • Harman M (2007) The current state and future of search based software engineering. In: Proc. FOSE, pp. 342–357

  • Harman M, Mansouri SA, Zhang Y (2012) Search-based software engineering: trends, techniques and applications. ACM Comp Surv 45(1):1–66

    Article  Google Scholar 

  • Harman M, Jia Y, Zhang Y (2015) Achievements, open problems and challenges for search based software testing. In: Proc. 8th IEEE international conference on software testing, verification and validation

  • Harman M, McMinn P (2010) A theoretical and empirical study of search based testing: local, global and hybrid search. IEEE Trans Softw Eng 36(2):226–247

    Article  Google Scholar 

  • Harman M, McMinn P, Wegener J (2007) The impact of input domain reduction on search based test data generation. In: Proc. ESEC/FSE, Croatia, pp. 155–164

  • Hermadi I, Lokan C, Sarker R (2014) Dynamic stopping criteria for search-based test data generation for path testing. Inf Softw Technol J 56(4):395–407

    Article  Google Scholar 

  • Holland JH (1975) Adaptation in natural and artificial systems. University of Michigan Press, Ann Arbor

    Google Scholar 

  • Jones B, Sthamer H, Eyres D (1996) Automatic structural testing using genetic algorithms. Softw Eng J 11(5):299–306

    Article  Google Scholar 

  • Jorgensen PC (2008) Software testing: a Craftsman’s approach. Auerbach Publications, Boca Raton

  • Khor S, Grogono P (2004) Using a genetic algorithm and formal concept analysis to generate branch coverage test data automatically. In: Proc. ASE

  • Kitchenham BA (2007) Guidelines for performing systematic literature reviews in software engineering. Technical report EBSE- 2007-01

  • Korel B (1990) Automated software test data generation. IEEE Trans Softw Eng 16(8):870–879

    Article  Google Scholar 

  • Latiu GI, Cret OA, Vacariu L (2012) Automatic test data generation for software path testing using evolutionary algorithms. In: Proc. Third international conference on emerging intelligent data and web technologies, pp.1–8

  • Li J, Baob W, Zhaoa Y, Maa Z, Donga H (2009) Evolutionary generation of unique input/output sequences for class behavioral testing. Comput Math Appl 57:1800–1807

    Article  Google Scholar 

  • Lin P, Bao XL, Shu ZY, Wang XJ, Liu JM (2012) Test case generation based on adaptive genetic algorithm. In: International conference on quality, reliability, risk, maintenance, and safety engineering, Chengdu, China, pp. 863–866

  • Liu D, Wang X, Wang J (2013) Automatic test case generation based on genetic algorithm. J Theor Appl Inf Technol 48(1):411–416

    Google Scholar 

  • Louzada J, Camilo-Junior CG, Vincenzi A, Rodrigues C (2012) An elitist evolutionary algorithm for automatically generating test data. In: Proc. IEEE WCCI, Brisbane, Australia

  • Malburg J, Fraser G (2011) Combining search based and constraint-based testing. In: Proc. IEEE ASE, Lawrence, KS, pp. 436–439

  • Mantere T, Alander JT (2005) Evolutionary software engineering, A review. J Appl Soft Comput 5:315–333

    Article  Google Scholar 

  • Mao C, Yu X (2013) Test data generation for software testing based on quantum-inspired genetic algorithm. Int J Comput Intell Appl 12(1) pp. 1350004 (21 pages)

  • McMinn P (2004) Search-based software test data generation: a survey. Softw Test Verif Reliab 14(2):105–156

  • McMinn P (2005) Evolutionary search for test data in the presence of state behavior. Ph. D. Thesis, University of Sheffield, Sheffield

  • McMinn P (2011) Search-based software testing: past, present and future. In: Proc. 4th international conference on software testing, verification and validation workshops, pp. 153–163

  • McMinn P (2013) An identification of program factors that impact crossover performance in evolutionary test input generation for the branch coverage of C programs. Inf Softw Technol 55:153–172

    Article  Google Scholar 

  • Michael CC, McGraw GE, Schatz MA (2001) Generating software test data by evolution. IEEE Trans Softw Eng 27(12):1085–1110

    Article  Google Scholar 

  • Miller J, Reformat M, Zhang H (2006) Automatic test data generation using genetic algorithm and program dependence graphs. Inf Softw Technol 48:586–605

    Article  Google Scholar 

  • Myers G (1979) The art of software testing. Wiley, New York

    Google Scholar 

  • Oxman AD (1994) Systematic reviews: checklists for review articles. BMJ 309(6955):648–651

    Article  Google Scholar 

  • Pachauri A, Gursaran (2011) Software test data generation using path prefix strategy and genetic algorithm. In: Proc. international conference on science and engineering, pp. 131–140

  • Pargas RP, Harrold MJ, Peck RR (1999) Test data generation using genetic algorithms. J Softw Test Verif Reliab 9:263–282

    Article  Google Scholar 

  • Pei M, Goodman ED, Gao Z, Zhong K (1994) Automated software test data generation using a genetic algorithm. Technical report, Michigan State University

  • Pocatilu P, Ivan I (2013) A genetic algorithm based system for automatic control of test data generation. Stud Inf Control 22(2):219–226

    Google Scholar 

  • Rauf A, Anwar S, Jaffer MA, Shahid A (2010) Automated GUI test coverage analysis using GA. In: Proc. 7th international conference on information technology, pp. 1057–1062

  • Roper M, Maclean I, Brooks A, Miller J, Wood M (1995) Genetic Algorithms and the automatic generation of test data. Technical report RR/95/195[EFoCS-19-95], Department of Computer Science, University of Strathclyde

  • Shamshiri S, Rojas JM, Fraser G, McMinn P (2015) Random or genetic algorithm search for object-oriented test suite generation?. In: Proc. Genetic and Evolutionary Computation Conference (GECCO 2015), pp. 1367–1374

  • Sharma N, Pasala A, Kommineni R (2012) Generation of character test input data using GA for functional testing. In: Proc. 19th APSEC-SATA workshop, pp. 87–94

  • Sofokleous AA, Andreou AS (2008) Automatic, evolutionary test data generation for dynamic software testing. J Sys Softw 81:1883–1898

    Article  Google Scholar 

  • Srinivas M (1994) Genetic algorithms: a survey. J Comput 27:17–26

    Article  Google Scholar 

  • Sthamer HH (1996) The automatic generation of software test data using genetic algorithms. Ph. D thesis, University of Glamorgan, Pontypridd, Wales

  • Suresh Y, Rath SK (2013a) A genetic algorithm based approach for test data generation in basis path testing. In: International Journal of Soft Computing and Software Engineering [JSCSE], Special Issue: The proceeding of international conference on soft computing and software engineering, 3(3)

  • Suresh Y, Rath S (2013b) Application of meta-heuristic algorithms for automated software test data generation. Int J Comput Intell Stud 4(2):113–133

  • Tracey N (2000) A search-based automated test data generation framework for safety critical software. Ph. D. thesis, University of York

  • Watkins A (1995) The automatic generation of test data using genetic algorithms. In: Proc. 4th software quality conference, pp. 300–309

  • Wegener J, Baresel A, Sthamer H (2001) Evolutionary test environment for automatic structural testing. J Inf Softw Technol 43:841–854

    Article  Google Scholar 

  • Xanthakis S, Ellis C, Skourlas C, Le Gall A, Katsikas S, Karapoulios K (1992) Application of genetic algorithms to software testing. In: Proc. 5th ICSE, pp. 625–636

  • Xiao J, Afzal W (2010) Search-based resource scheduling for bug fixing tasks. In: Proc. 2nd SSBSE, pp. 133–14

  • Xue-ying MA, Sheng BK, Zhen-feng HE, Cheng-qing YE (2005) A genetic algorithm for test-suite reduction. In: Proc. ICSMC, pp. 133–139

  • Zhang N, Wu B, Bao X (2015) Automatic generation of test cases based on multi-population genetic algorithm. Int J Multimed Ubiquitous Eng 10(6):113–122

    Article  Google Scholar 

  • Zhang W, Gong D, Yao X, Zhang Y (2007) Evolutionary generation of test data for many paths coverage. In: Proc. Chinese Control and Decision Conference, pp. 230–235

Download references

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Anupama Surendran.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Surendran, A., Samuel, P. Evolution or revolution: the critical need in genetic algorithm based testing. Artif Intell Rev 48, 349–395 (2017). https://doi.org/10.1007/s10462-016-9504-8

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10462-016-9504-8

Keywords

Navigation