ABSTRACT
Test minimization techniques aim at identifying and eliminating redundant test cases from test suites in order to reduce the total number of test cases to execute, thereby improving the efficiency of testing. In the context of software product line, we can save effort and cost in the selection and minimization of test cases for testing a specific product by modeling the product line. However, minimizing the test suite for a product requires addressing two potential issues: 1) the minimized test suite may not cover all test requirements compared with the original suite; 2) the minimized test suite may have less fault revealing capability than the original suite. In this paper, we apply weight-based Genetic Algorithms (GAs) to minimize the test suite for testing a product, while preserving fault detection capability and testing coverage of the original test suite. The challenge behind is to define an appropriate fitness function, which is able to preserve the coverage of complex testing criteria (e.g., Combinatorial Interaction Testing criterion). Based on the defined fitness function, we have empirically evaluated three different weight-based GAs on an industrial case study provided by Cisco Systems, Inc. Norway. We also presented our results of applying the three weight-based GAs on five existing case studies from the literature. Based on these case studies, we conclude that among the three weight-based GAs, Random-Weighted GA (RWGA) achieved significantly better performance than the other ones.
- J. McGregor. Testing a Software Product Line. Technical Report Carnegie Mellon University/SEI-2001-TR-022. Software Engineering Institute, Carnegie Mellon University, Pittsburgh, Pennsylvania. 2001. http://www.sei.cmu.edu/library/abstracts/reports/01tr022.cfmGoogle ScholarCross Ref
- D. Benavides, S. Segura, and A. R. Cortés. Automated analysis of feature models 20 years later: A literature review. Information Systems. (35), 615--636. 2010. Google ScholarDigital Library
- T. Chen, and M. Lau. Dividing strategies for the optimization of a test suite. Information Processing Letters. 60(3), pp. 135-- 141. 1996. Google ScholarDigital Library
- S. Wang, A. Gotlieb, M. Liaaen, and L. C. Briand. Automatic selection of test execution plans from a Video Conference System Product Line. In Proceedings of the ACM MODELS Workshop VARiability for You (VARY' 12), pp. 30--35. 2012. Google ScholarDigital Library
- S. Wang, A. Gotlieb, S. Ali, M. Liaaen. Automated Selection of Test Cases using Feature Model: An Industrial Case Study. Technical Report (2012--20), Simula Research Laboratory. 2012.Google Scholar
- A. Konak, D. W. Coit, and A. E. Smith. Multi-objective optimization using genetic algorithms: A tutorial. Reliability Engineering & System Safety. 91(9), pp. 992--1007. 2007.Google ScholarCross Ref
- K. Czarnecki, C. Kim, and K. Kalleberg. Feature models are views on ontologies. In Proceedings of International Software Product Line Conference, pp. 41--51. 2006. Google ScholarDigital Library
- Pure systems GmbH. Variant management with pure::variants. Technical white paper. Available from http://web.pure- systems.com, 2006.Google Scholar
- M. Harman, S. A. Mansouri, and Y. Zhang. Search based software engineering: A comprehensive analysis and review of trends techniques and applications. Technical Report TR-09-032009. King's College, London. 2009.Google Scholar
- K. C. Tai, and Y. Lei. A Test-Generation Strategy for Pairwise Testing. IEEE Trans. of Software Engineering. 28(1), pp. 109 - 111. 2002. Google ScholarDigital Library
- R. Kuhn, Y. Lei, and R. Kacker. Practical combinatorial testing: Beyond pairwise. IT Professional. 10(3), pp. 19--23. 2008. Google ScholarDigital Library
- T. Murata, T. Ishibuchi, and H. Tanaka. Multi-objective genetic algorithm and its applications to flowshop scheduling. Comput Ind Eng. 30(4), pp. 957--968. 1996. Google ScholarDigital Library
- Cisco Systems. Cisco telepresence codec c90. Data sheet. 2010. Available from http://www.cisco.com.Google Scholar
- D. Benavides. On the Automated Analysis of Software Product Lines Using Feature Models. Doctoral Thesis. Universidad de Sevilla. 2007.Google Scholar
- A. Arcuri, and L. C. Briand. A Practical Guide for Using Statistical Tests to Assess Randomized Algorithms in Software Engineering. In Proceedings of the International Conference on Software Engineering. pp. 21--28. 2011. Google ScholarDigital Library
- S. Ali, L. C. Briand, H. Hemmati, and R. K. Panesar-Walawege. A Systematic Review of the Application and Empirical Investigation of Search-Based Test Case Generation. IEEE Trans on Software Engineering, 36(6), pp. 742--762. 2010. Google ScholarDigital Library
- M. B. Cohen, M. B. Dwyer, and J. Shi. Coverage and Adequacy in Software Product Line Testing. In Proceedings ACM ISSTA Workshop on Role of Software Architecture for Testing and Analysis. pp. 53--63. 2006. Google ScholarDigital Library
- H. Muccini, and A. Van Der Hoek. Towards Testing Product Line Architectures. Electronic Notes in Theoretical Computer Science. 82(6), pp. 99--109. 2003.Google ScholarCross Ref
- S. Yoo, and M. Harman. Regression test minimization, selection and prioritization: a survey. Software Testing, Verification, and Reliability; 22(2), pp. 67--120. 2012. Google ScholarDigital Library
- T. Y. Chen, and M. F. Lau. Dividing strategies for the optimization of a test suite. Information Processing Letters. 60(3), pp. 135--141. 1996. Google ScholarDigital Library
- S. Tallam, and N. Gupta. A concept analysis inspired greedy algorithm for test suite minimization. SIGSOFT Software Engineering Notes. 31(1), 35--42. 2006. Google ScholarDigital Library
- D. Jeffrey, and N. Gupta. Test suite reduction with selective redundancy. In Proceedings of the International Conference on Software Maintenance. pp. 549--558. 2005. Google ScholarDigital Library
- S. Yoo, and M. Harman. Pareto Efficient Multi-Objective Test Case Selection. In Proceedings of the international symposium on Software testing and analysis (ISSTA), pp. 140--150.2007. Google ScholarDigital Library
Index Terms
- Minimizing test suites in software product lines using weight-based genetic algorithms
Recommendations
Fine-grained test minimization
ICSE '18: Proceedings of the 40th International Conference on Software EngineeringAs a software system evolves, its test suite can accumulate redundancies over time. Test minimization aims at removing redundant test cases. However, current techniques remove whole test cases from the test suite using test adequacy criteria, such as ...
Probability Model-Based Test Suite Reduction
The paper presents a test suite reduction approach based on probability models in regression testing. First, we model the process of software testing by using the joint probability of both test cases and program faults, and propose two probability ...
Comparing non-adequate test suites using coverage criteria
ISSTA 2013: Proceedings of the 2013 International Symposium on Software Testing and AnalysisA fundamental question in software testing research is how to compare test suites, often as a means for comparing test-generation techniques. Researchers frequently compare test suites by measuring their coverage. A coverage criterion C provides a set ...
Comments