Preeti Satish
Abstract
The efficiency of prioritization algorithms depends on how early the faults are detected. In this paper, we present a novel prioritization algorithm for combinatorial testing. Our approach takes ACTS tool generated test cases with "don't care" values as the starting point and refines them for increased effectiveness without increasing the number of test cases. Our algorithm maximizes the number of higher order combinations tested, by filling the "don't care" values in the test suite effectively. It also orders the test cases using a cost function that includes higher order coverage, thereby achieving early fault detection. The effectiveness of our algorithms is demonstrated by performing a comparative evaluation using the metric t way Rate of Fault Detection, on 2 real life case studies and numerous synthetic covering arrays of different sizes. The results show that our algorithms perform better in terms of covering higher order pairs and also faster.
- Bryce RC, Colbourn CJ. Test prioritization for pairwise interaction coverage. In ACM SIGSOFT Software Engineering Notes 2005;30(4): 1-7. Google Scholar
Digital Library
- Bryce RC, Sampath S, Memon AM. Developing a single model and test prioritization strategies for event-driven software. Software Engineering, IEEE Transactions 2011; 37(1):48-64. Google ScholarDigital Library
- Bryce RC, Sampath S, Pedersen JB, Manchester S. Test suite prioritization by cost-based combinatorial interaction coverage. International Journal of System Assurance Engineering and Management 2011; 2(2):126-34.Google Scholar
- Bryce, Renée C., and Atif M. Memon. Test suite prioritization by interaction coverage. In Workshop on Domain specific approaches to software test automation in conjunction with the 6th ESEC/FSE joint meeting 2007; 1-7. ACM. Google ScholarDigital Library
- Bryce, Renée C., and Charles J. Colbourn. Prioritized interaction testing for pair-wise coverage with seeding and constraints. Information and Software Technology 2006; 48(10): 960-970.Google Scholar
- Chen X, Gu Q, Zhang X, Chen D. Building prioritized pairwise interaction test suites with ant colony optimization. In Quality Software,QSIC'09. 9th International Conference on 2009; (pp. 347-352). IEEE. Google ScholarDigital Library
- Choi EH, Kitamura T, Artho C, Yamada A, Oiwa Y. Priority Integration for Weighted Combinatorial Testing. In Computer Software and Applications Conference (COMPSAC), IEEE 39th Annual 2015 (Vol. 2, pp. 242-247). IEEE. Google ScholarDigital Library
- Colbourn CJ. Combinatorial aspects of covering arrays. Le Matematiche (Catania) 2004; 58(121-167):0-10.Google Scholar
- Czerwonka J. Pairwise testing in the real world: Practical extensions to test-case scenarios. In Proceedings of 24th Pacific Northwest Software Quality Conference, Citeseer 2006; 419-430.Google Scholar
- Ferrer J, Kruse PM, Chicano F, Alba E. Evolutionary algorithm for prioritized pairwise test data generation. In Proceedings of the 14th annual conference on Genetic and evolutionary computation 2012; (pp. 1213-1220). ACM. Google ScholarDigital Library
- Gao SW, Lv JH, Du BL, Colbourn CJ, Ma SL. Balancing Frequencies and Fault Detection in the In-Parameter-Order Algorithm. Journal of Computer Science and Technology 2015; 30(5):957-68Google Scholar
- Huang R, Chen J, Zhang T, Wang R, Lu Y. Prioritizing variable-strength covering array. InComputer Software and Applications Conference (COMPSAC) IEEE 37th Annual 2013; (pp. 502-511). IEEE. Google ScholarDigital Library
- Huang R, Xie X, Towey D, Chen TY, Lu Y, Chen J. Prioritization of combinatorial test cases by incremental interaction coverage. International Journal of Software Engineering and Knowledge Engineering 2013; 23(10):1427-57.Google Scholar
- Kuhn, D. Richard, Raghu N. Kacker, and Yu Lei. Introduction to combinatorial testing. CRC Press,2013 Google ScholarDigital Library
- Kuhn, D. Richard, Raghu N. Kacker, and Yu Lei. Introduction to combinatorial testing. CRC Press,2013 Google ScholarDigital Library
- Lei Y, Kacker R, Kuhn DR, Okun V, Lawrence J. IPOG/IPOG?D: efficient test generation for multi?way combinatorial testing. Software Testing, Verification and Reliability 2008;18(3):125-48. Google ScholarDigital Library
- National Institute of Standards and Technology (NIST), Automated Combinatorial Testing for Software (ACTS), http://csrc.nist.gov/groups/SNS/acts/documents/comparison-report.html . {13 April 2016}.Google Scholar
- Nie C, Leung H. A survey of combinatorial testing. ACM Computing Surveys (CSUR). 2011;43(2):11. Google ScholarDigital Library
- Petke J, Yoo S, Cohen MB, Harman M. Efficiency and early fault detection with lower and higher strength combinatorial interaction testing. In Proceedings of the 2013 9th Joint Meeting on Foundations of Software Engineering 2013; (pp. 26-36). ACM. Google ScholarDigital Library
- Qu X, Cohen MB, Rothermel G. Configuration-aware regression testing: an empirical study of sampling and prioritization. In Proceedings of the 2008 international symposium on Software testing and analysis 2008; (pp. 75-86). ACM. Google ScholarDigital Library
- Qu X, Cohen MB, Woolf KM. Combinatorial interaction regression testing: A study of test case generation and prioritization. In Software Maintenance, 2007. ICSM 2007. IEEE International Conference on 2007;(pp. 255-264). IEEE.Google Scholar
- Qu X, Cohen MB. A study in prioritization for higher strength combinatorial testing. In Software Testing, Verification and Validation Workshops (ICSTW), IEEE Sixth International Conference on 2013 (pp. 285-294). IEEE. Google ScholarDigital Library
- Rothermel G, Untch RH, Chu C, Harrold MJ. Prioritizing test cases for regression testing. Software Engineering, IEEE Transactions 2001; 27(10): 929-48. Google ScholarDigital Library
- Sampath S, Bryce R, Memon AM. A uniform representation of hybrid criteria for regression testing. Software Engineering, IEEE Transactions 2013; 39(10):1326-44. Google ScholarDigital Library
- Sampath S, Bryce RC, Jain S, Manchester S. A tool for combination-based prioritization and reduction of user-session-based test suites. InSoftware Maintenance (ICSM), 27th IEEE International Conference on 2011; (pp. 574-577). IEEE. Google ScholarDigital Library
- Satisfice, Inc.,ALLPAIRS Test case generation tool (Version 1.2.1) http://www.satisfice.com/tools.shtml, {13 April 2016}.Google Scholar
- Software-artifact Infrastructure Repository, http://sir.unl.edu/portal/index.php {13 April 2016}.Google Scholar
- Srikanth H, Cohen MB, Qu X. Reducing field failures in system configurable software: Cost-based prioritization. In 20th International Symposium on Software Reliability Engineering 2009; (pp. 61-70). IEEE. Google ScholarDigital Library
- Wu H, Nie C, Kuo FC. Test suite prioritization by switching cost. InSoftware Testing, Verification and Validation Workshops (ICSTW), 2014 IEEE Seventh International Conference on 2014; (pp. 133-142). IEEE Google ScholarDigital Library
Index Terms
- A Test Prioritization Algorithm That Cares for "Don't Care" Values and Higher Order Combinatorial Coverage
Recommendations
Faster mutation testing inspired by test prioritization and reduction
ISSTA 2013: Proceedings of the 2013 International Symposium on Software Testing and AnalysisMutation testing is a well-known but costly approach for determining test adequacy. The central idea behind the approach is to generate mutants, which are small syntactic transformations of the program under test, and then to measure for a given test ...
Prioritizing random combinatorial test suites
SAC '17: Proceedings of the Symposium on Applied ComputingThe behaviour of a system under test can be influenced by several factors, such as system configurations, user inputs, and so on. It has also been observed that many failures are caused by only a small number of factors. Combinatorial testing aims at ...
Experimental Comparison of Code-Based and Model-Based Test Prioritization
ICSTW '09: Proceedings of the IEEE International Conference on Software Testing, Verification, and Validation WorkshopsDuring regression testing, a modified system needs to beretested using the existing test suite. Since test suites may be very large, developers are interested in detecting faults in the system as early as possible. Test prioritization orders test cases ...
Comments