Abstract
Software testing is both a time and resource-consuming activity in software development. The most difficult parts of software testing are the generation and prioritization of test data. Principally these two parts are performed manually. Hence introducing an automation approach will significantly reduce the total cost incurred in the software development lifecycle. A number of automatic test case generation (ATCG) and prioritization approaches have been explored. In this paper, we propose two approaches: (1) a pathspecific approach for ATCG using the following metaheuristic techniques: the genetic algorithm (GA), particle swarm optimization (PSO) and artificial bee colony optimization (ABC); and (2) a test case prioritization (TCP) approach using PSO. Based on our experimental findings, we conclude that ABC outperforms the GA and PSO-based approaches for ATC.G Moreover, the results for PSO on TCP arguments demonstrate biased applicability for both small and large test suites against random, reverse and unordered prioritization schemes. Therefore, we focus on conducting a comprehensive and exhaustive study of the application of metaheuristic algorithms in solving ATCG and TCP problems in software engineering.
Similar content being viewed by others
Notes
Detailed MATLAB code can be provided by the authors upon prior request for academic use only.
A test requirement identifies what needs to be tested, including conditions, business logic, and functional and non-functional benchmarks.
Detailed MATLAB code for PSO for TCP can be provided by the authors upon prior request for academic use only.
References
Tassey G (2002) The economic impacts of inadequate infrastructure for software testing. National Institute of Standards and Technology, RTI Project 7007
Beizerm B (1990) Software Testing Techniques, Van Nostrand Reinhold. New York
Rothermel G, Untch RH, Chu C, Harrold MJ (1999) Test case prioritization: An empirical study. In: Proceedings IEEE international conference on software maintenance, (ICSM’99). IEEE, pp 179–188
Hou Y, Zhao C, Liao Y (2006) A new method of test generation for sequential circuits. In: International conference on communications, circuits and systems proceedings. IEEE, pp 2181–2185
Liang Y, Liu L, Wang D, Wu R (2010) Optimizing particle swarm optimization to solve knapsack problem Information computing and applications. Springer, pp 437–443
Liu H, Sun S, Abraham A (2006) Particle swarm approach to scheduling work-flow applications in distributed data-intensive computing environments. In: Sixth international conference on intelligent systems design and applications, ISDA’06, pp 661– 666
Lope HS, Coelho LS (2005) Particle swarn optimization with fast local search for the blind traveling salesman problem. In: Fifth international conference on hybrid intelligent systems, HIS’05. IEEE, pp 245–250
Zhao F, Zhang Q, Yang Y (2006) An improved particle swarm optimization-based approach for production scheduling problems. In: IEEE international conference on mechatronics and automation. IEEE, 2279–2283
Do H, Rothermel G (2006) On the use of mutation faults in empirical assessments of test case prioritization techniques. IEEE Trans Softw Eng 32:733–752
Elbaum S, Malishevsky AG, Rothermel G (2002) Test case prioritization: A family of empirical studies. IEEE Trans Softw Eng 28:159–182
Arcuri A, Briand L (2012) Formal analysis of the probability of interaction fault detection using random testing. IEEE Trans Softw Eng 38:1088–1099
Mills HD, Dyer M, Linger RC (1987) Cleanroom software engineering. IEEE Softw 4:19
Voas J, Morell L, Miller K (1991) Predicting where faults can hide from testing. IEEE Softw 8:41–48
Bird DL, Munoz CU (1983) Automatic generation of random self-checking test cases. IBM Syst J 22:229–245
Ferguson R, Korel B (1996) The chaining approach for software test data generation. ACM Trans Softw Eng Methodol (TOSEM) 5:63–86
Chang K-H, Cross IIJH, Carlisle WH, Brown DB (1992) A framework for intelligent test data generation. J Intell Robot Syst 5:147–165
Korel B (1996) Automated test data generation for programs with procedures ACM SIGSOFT software engineering notes. ACM, pp 209–215
Korel B (1990) A dynamic approach of test data generation. In: Proceedings, conference on software maintenance. IEEE, pp 311–317
McMinn P (2004) Search-based software test data generation: a survey. Software Testing Verification and Reliability 14:105–156
Clarke LA (1976) A system to generate test data and symbolically execute programs. IEEE Trans Softw Eng :215–222
Howden WE (1977) Symbolic testing and the DISSECT symbolic evaluation system. IEEE Trans Softw Eng :266–278
Ramamoorthy CV, Ho S-B, Chen WT (1976) On the automated generation of program test data. IEEE Trans Softw Eng :293–300
Shan J-H, Wang J, Qi Z-C (2004) Survey on path-wise automatic generation of test data. Acta Electron Sin 32:109–113
Malhotra R, Garg M (2011) An adequacy based test data generation technique using genetic algorithms. J Inf Process Syst 7:363–384
Michael CC, McGraw G, Schatz MA (2001) Generating software test data by evolution. IEEE Trans Softw Eng 27:1085–1110
Fraser G, Arcuri A (2014) A large-scale evaluation of automated unit test generation using EvoSuite. ACM Trans Softw Eng Methodol (TOSEM) 24:8
Tracey N, Clark J, Mander K (1998) Automated program flaw finding using simulated annealing ACM SIGSOFT software engineering notes. ACM, pp 73–81
Malhotra R, Khari M (2013) Heuristic search-based approach for automated test data generation: a survey. International Journal of Bio-Inspired Computation 5:1–18
Do H, Rothermel G, Kinneer A (2004) Empirical studies of test case prioritization in a JUnit testing environment. In: 5th International symposium on software reliability engineering ISSRE. IEEE, pp 113–124
Kim J-M, Porter A (2002) A history-based test prioritization technique for regression testing in resource constrained environments Proceedings of the 24rd international conference on software engineering, ICSE. IEEE, pp 119–129
Rothermel G, Untch RH, Chu C, Harrold MJ (2001) Prioritizing test cases for regression testing. IEEE Trans Softw Eng 27:929–948
Leon D, Podgurski A (2003) A comparison of coverage-based and distribution-based techniques for filtering and prioritizing test cases. In: 14th international symposium on software reliability engineering, ISSRE. IEEE, pp 442–453
Mirarab S, Tahvildari L (2008) An empirical study on bayesian network-based approach for test case prioritization. In: 1st International conference on software testing, verification, and validation. IEEE, pp 278–287
Yoo S, Harman M (2012) Regression testing minimization, selection and prioritization: a survey. Software testing, verification and reliability 22:67–120
Li Z, Harman M, Hierons RM (2007) Search algorithms for regression test case prioritization. IEEE Trans Softw Eng 33:225–237
Kaur DA, Goyal S (2011) A bee colony optimization algorithm for code coverage test suite prioritization. Int J Eng Sci Technol 1:2786–2795
Catal C, Mishra D (2013) Test case prioritization: a systematic mapping study. Softw Qual J 21:445–478
Jacob TP, Ravi T (2013) Optimization of test cases by prioritization. J Comput Sci 9:972
Singh Y, Kaur A, Suri B (2010) Test case prioritization using ant colony optimization. ACM SIGSOFT Software Engineering Notes 35:1–7
Ahmed AA, Shaheen M, Kosba E (2012) Software testing suite prioritization using multi-criteria fitness function. In: 222nd International conference on computer theory and applications (ICCTA). IEEE, pp 160–166
Chen Y, Zhong Y (2008) Automatic path-oriented test data generation using a multi-population genetic algorithm. In: Fourth international conference on natural computation, ICNC’08. IEEE, pp 566–570
Mohapatra D (2011) GA Based Test Case Generation Approach for Formation of Efficient Set of Dynamic Slices. International Journal on Computer Science and Engineering (IJCSE) 3:
Pargas RP, Harrold MJ, Peck RR (1999) Test-data generation using genetic algorithms. Software Testing Verification and Reliability 9:263–282
Berndt D, Fisher J, Johnson L et al (2003) Breeding software test cases with genetic algorithms. In: Proceedings of the 36th annual hawaii international conference on system sciences
Ahmed M A, Hermadi I (2008) GA-Based multiple paths test data generator. Comput Oper Res 35:3107–3124
Mukesh MOPS (2015) Generating and prioritizing optimal paths using ant colony optimization. Computational Ecology and Software 5:1
Watkins AL (1995) The automatic generation of test data using genetic algorithms. In: Proceedings of the 4th software quality conference, pp 300–309
Flipcart Shop Flipcart, availabte: www.flipcart.com, accessed june 2016
Arcuri A, Briand L (2014) A hitchhiker’s guide to statistical tests for assessing randomized algorithms in software engineering. Software Testing, Verification and Reliability 24:219– 250
Rothermel G, Harrold MJ (1996) Analyzing regression test selection techniques. IEEE Trans Softw Eng 22:529–551
Di Caro G, Dorigo M (1998) Antnet: Distributed stigmergetic control for communications networks. J Artif Intell Res :317– 365
Dorigo M, Maniezzo V, Colorni A (1996) Ant system: optimization by a colony of cooperating agents. IEEE Transactions on Systems, Man, and Cybernetics 26:29–41
Li H, Lam CP (2004) Software Test Data Generation using Ant Colony Optimization. In: International conference on computational intelligence, pp 1–4
Ayari K, Bouktif S, Antoniol G (2007) Automatic mutation test input data generation via ant colony. In: Proceedings of the 9th annual conference on Genetic and evolutionary computation. ACM, pp 1074–1081
Parpinelli RS, Lopes HS, Freitas AA (2002) Data mining with an ant colony optimization algorithm. IEEE Trans Evol Comput 6:321–332
Zhao P, Zhao P, Zhang X (2006) A new ant colony optimization for the knapsack problem. In: CAIDCD’06 7th international conference on computer-aided industrial design and conceptual design. IEEE, pp 1–3
Atif M (2016) Software Benchmark repository for TeraPaint3. http://www.cs.umd.edu/~atif/Benchmarks/common/TerpPaint3-fault_matrix.txt. Accessed 1 June 2016
Atif M (2016) Software Benchmark repository for TeraPresent3. http://www.cs.umd.edu/~atif/Benchmarks/common/TerpPresent3-fault_matrix.txt. Accessed 20 June 2016
Atif M (2016) Software Benchmark repository for TerpSpreadSheet3, available. http://www.cs.umd.edu/~atif/Benchmarks/common/TerpSpreadSheet3-fault_matrix.txt
Krishnamoorthi R, Mary SASA (2009) Regression test suite prioritization using genetic algorithms. International Journal of Hybrid Information Technology 2:35–52
Srivastava PR (2008) Test case prioritization. Journal of Theoretical and Applied Information Technology 4:178–181
Kaur A, Bhatt D (2011) Hybrid particle swarm optimization for regression testing. Int J Comput Sci Eng 3:1815–1824
Kaur A, Bhatt D (2011) Particle swarm optimization with cross-over operator for prioritization in regression testing. Int J Comput Appl 27:27–34
Author information
Authors and Affiliations
Corresponding author
Appendices
Appendix 1: Small application test suite- fault matrix
Only the main modules are provided in this paper. Detailed source code is available from the authors upon prior request.
Large real application test suite: data can be accessed at [57].
Appendix 2: Objective function formulation for TeraPaint3
The following code is implemented in MATLAB and forms the objective function. The objective function is given as an input to the PSO algorithm. Note that the lower and upper bounds in the PSO algorithm depend on the system under test. Because TeraPaint3 has 424 test cases, the lower and upper bounds are set to [1] and [424], respectively.
The APFD scores for the other two applications, TeraPresent3 and TeraSpreadSheet3, are calculated in a similar manner.
Rights and permissions
About this article
Cite this article
Mann, M., Tomar, P. & Sangwan, O.P. Bio-inspired metaheuristics: evolving and prioritizing software test data. Appl Intell 48, 687–702 (2018). https://doi.org/10.1007/s10489-017-1003-3
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10489-017-1003-3