Huayao Wu
Changhai Nie
ABSTRACT
Combinatorial testing (CT) is a branch of software testing, which aims to detect the interaction triggered failures as much as possible. Search based combinatorial testing is to use the search techniques to solve the problem in combinatorial testing. It has been shown to be effective and promising. In this paper, we aim to provide an overview of search based combinatorial testing, especially focusing on test suite generation without constraint, and discuss the potential future directions in this field.
- B. S. Ahmed and K. Z. Zamli. Pstg a t-way strategy adopting particle swarm optimization. In Proceedings of the Asia International Conference on Mathematical Analytical Modelling and Computer Simulation, pages 1–5, 2010. Google Scholar
Digital Library
- B. S. Ahmed and K. Z. Zamli. A variable strength interaction test suites generation strategy using particle swarm optimization. Journal of Systems and Software, 84(12):2171–2185, 2011. Google ScholarDigital Library
- B. S. Ahmed, K. Z. Zamli, and C. P. Lim. Application of particle swarm optimization to uniform and variable strength covering array construction. Applied Soft Computing, 12(4):1330–1347, 2012. Google ScholarDigital Library
- S. Ali, L. C. Briand, H. Hemmati, and R. K. Panesar-Walawege. A systematic review of the application and empirical investigation of search-based test case generation. IEEE Transactions on Software Engineering, 36(6):742–762, 2010. Google ScholarDigital Library
- S. Anand, E. K. Burke, T. Y. Chen, J. Clark, M. B. Cohen, W. Grieskamp, M. Harman, M. J. Harrold, and P. McMinn. An orchestrated survey of methodologies for automated software test case generation. Journal of Systems and Software, 86(8):1978 – 2001, 2013. Google ScholarDigital Library
- A. Arcuri and L. Briand. Formal analysis of the probability of interaction fault detection using random testing. IEEE Transactions on Software Engineering, 38(5):1088–1099, 2012. Google ScholarDigital Library
- R. C. Bryce and C. J. Colbourn. One-test-at-a-time heuristic search for interaction test suites. In Proceedings of the annual conference on Genetic and evolutionary computation, pages 1082–1089, 2007. Google ScholarDigital Library
- R. C. Bryce, S. Sampath, J. B. Pedersen, and S. Manchester. Test suite prioritization by cost-based combinatorial interaction coverage. International Journal of System Assurance Engineering and Management, 2(2):126–134, 2011.Google ScholarCross Ref
- X. Chen, Q. Gu, A. Li, and D. Chen. Variable strength interaction testing with an ant colony system approach. In Proceedings of the Asia-Pacific Software Engineering Conference, pages 160–167, 2009. Google ScholarDigital Library
- X. Chen, Q. Gu, J. Qi, and D. Chen. Applying particle swarm optimization to pairwise testing. In Proceedings of the Annual Computer Software and Applications Conference, pages 107–116, 2010. Google ScholarDigital Library
- X. Chen, Q. Gu, X. Zhang, and D. Chen. Building prioritized pairwise interaction test suites with ant colony optimization. In Proceedings of the International Conference on Quality Software, pages 347–352, 2009. Google ScholarDigital Library
- M. Cohen, C. Colbourn, and A. Ling. Augmenting simulated annealing to build interaction test suites. In Proceedings of the International Symposium on Software Reliability Engineering, pages 394–405, 2003. Google ScholarDigital Library
- M. Cohen, P. Gibbons, W. Mugridge, and C. Colbourn. Constructing test suites for interaction testing. In Proceedings of the International Conference on Software Engineering, pages 38–48, 2003. Google ScholarDigital Library
- M. B. Cohen, C. J. Colbourn, and A. C. Ling. Constructing strength three covering arrays with augmented annealing. Discrete Mathematics, 308(13):2709–2722, 2008.Google ScholarDigital Library
- C. J. Colbourn. Combinatorial aspects of covering arrays. Le Matematiche (Catania), 58(121-167):0–10, 2004.Google Scholar
- J. Ferrer, P. M. Kruse, F. Chicano, and E. Alba. Evolutionary algorithm for prioritized pairwise test data generation. In Proceedings of the International Conference on Genetic and Evolutionary Computation Conference, pages 1213–1220, 2012. Google ScholarDigital Library
- B. Garvin, M. Cohen, and M. Dwyer. Evaluating improvements to a meta-heuristic search for constrained interaction testing. Empirical Software Engineering, 16(1):61–102, 2011. Google ScholarDigital Library
- B. J. Garvin, M. B. Cohen, and M. B. Dwyer. Evaluating improvements to a meta-heuristic search for constrained interaction testing. Empirical Software Engineering, 16(1):61–102, 2010. Google ScholarDigital Library
- S. Ghazi and M. Ahmed. Pair-wise test coverage using genetic algorithms. In Proceedings of the Congress on Evolutionary Computation, volume 2, pages 1420–1424, 2003.Google ScholarCross Ref
- M. Harman, S. A. Mansouri, and Y. Zhang. Search-based software engineering: Trends, techniques and application. ACM Computing Survey, 45(1):11:1–11:61, 2012. Google ScholarDigital Library
- R. A. W. II and C. J. Colbourn. Tabu search for covering arrays using permutation vectors. Journal of Statistical Planning and Inference, 139(1):69–80, 2009.Google ScholarCross Ref
- J. McCaffrey. An empirical study of pairwise test set generation using a genetic algorithm. In Proceedings of the International Conference on Information Technology: New Generations, pages 992–997, 2010. Google ScholarDigital Library
- P. McMinn. Search-based software testing: Past, present and future. In Proceedings of the International Conference on Software Testing, Verification and Validation Workshops, pages 153–163, 2011. Google ScholarDigital Library
- C. Nie and H. Leung. A survey of combinatorial testing. ACM Computing Surveys, 43(2):11:1–11:29, 2011. Google ScholarDigital Library
- C. Nie, H. Leung, and K.-Y. Cai. Adaptive Combinatorial Testing. In Proceedings of the International Conference on Quality Software, pages 284–287, 2013. Google ScholarDigital Library
- C. Nie, H. Wu, Y. Liang, H. Leung, F.-C. Kuo, and Z. Li. Search based combinatorial testing. In Proceedings of the Asia-Pacific Software Engineering Conference, pages 778–783, 2012. Google ScholarDigital Library
- C. Nie, B. Xu, and H. Leung. Using computational search to generate 2-way covering array. In Proceedings of the International Symposium on Search Based Software Engineering, Fast Abstract, 2009.Google Scholar
- K. J. Nurmela. Upper bounds for covering arrays by tabu search. Discrete Applied Mathematics, 138(1-2):143–152, 2004. Google ScholarDigital Library
- J. Petke, S. Yoo, M. B. Cohen, and M. Harman. Efficiency and early fault detection with lower and higher strength combinatorial interaction testing. In Proceedings of the Joint Meeting on Foundations of Software Engineering, pages 26–36. ACM, 2013. Google ScholarDigital Library
- E. Rodriguez-Tello and J. Torres-Jimenez. Memetic algorithms for constructing binary covering arrays of strength three. In Artifical Evolution, volume 5975, pages 86–97. 2010. Google ScholarDigital Library
- T. Shiba, T. Tsuchiya, and T. Kikuno. Using artificial life techniques to generate test cases for combinatorial testing. In Proceedings of the Annual International Computer Software and Applications Conference, volume 1, pages 72–77, 2004. Google ScholarDigital Library
- Z. H. Soh, S. A. Abdullah, K. Z. Zamli, and M. I. Younis. Distributed t-way test suite data generation using exhaustive search method with map and reduce framework. In Proceedings of the Symposium on Industrial Electronics & Applications, pages 340––344, 2010.Google ScholarCross Ref
- J. Torres-Jimenez and E. Rodriguez-Tello. Simulated annealing for constructing binary covering arrays of variable strength. In Proceedings of the Congress on Evolutionary Computation, pages 1–8, 2010.Google ScholarCross Ref
- J. Torres-Jimenez and E. Rodriguez-Tello. New bounds for binary covering arrays using simulated annealing. Information Sciences, 185(1):137–152, 2012. Google ScholarDigital Library
- L. Yalan, C. Nie, J. M. Kauffman, G. M. Kapfhammer, and H. Leung. Empirically identifying the best genetic algorithm for covering array generation. In Proceedings of the International Symposium on Search Based Software Engineering, Fast abstract, 2011. Google ScholarDigital Library
- C. Yilmaz, S. Fouche, M. Cohen, A. A. Porter, G. Demiroz, and U. Koc. Moving forward with combinatorial interaction testing. Computer, 99(PrePrints), 2013. Google ScholarDigital Library
- L. Zekaoui. Mixed covering arrays on graphs and tabu search algorithms. PhD thesis, University of Ottawa, 2006.Google Scholar
- Y. Zhang. Sbse repository. http://crestweb.cs.ucl.ac.uk/resources/sbse repository/.Google Scholar
Index Terms
- An overview of search based combinatorial testing
Recommendations
Prioritizing random combinatorial test suites
SAC '17: Proceedings of the Symposium on Applied ComputingThe behaviour of a system under test can be influenced by several factors, such as system configurations, user inputs, and so on. It has also been observed that many failures are caused by only a small number of factors. Combinatorial testing aims at ...
A combinatorial testing strategy for concurrent programs
One approach to testing concurrent programs is called reachability testing, which derives test sequences automatically and on-the-fly, without constructing a static model. Existing reachability testing algorithms are exhaustive in that they are intended ...
A backtracking search tool for constructing combinatorial test suites
Combinatorial testing is an important testing method. It requires the test cases to cover various combinations of parameters of the system under test. The test generation problem for combinatorial testing can be modeled as constructing a matrix which ...
Comments