Eduard Baranov
Kuldeep S. Meel
ABSTRACT
Owing to the pervasiveness of software in our modern lives, software systems have evolved to be highly configurable. Combinatorial testing has emerged as a dominant paradigm for testing highly configurable systems. Often constraints are employed to define the environments where a given system under test (SUT) is expected to work. Therefore, there has been a sustained interest in designing constraint-based test suite generation techniques. A significant goal of test suite generation techniques is to achieve t-wise coverage for higher values of t. Therefore, designing scalable techniques that can estimate t-wise coverage for a given set of tests and/or the estimation of maximum achievable t-wise coverage under a given set of constraints is of crucial importance. The existing estimation techniques face significant scalability hurdles.
The primary scientific contribution of this work is the design of scalable algorithms with mathematical guarantees to estimate (i) t-wise coverage for a given set of tests, and (ii) maximum t-wise coverage for a given set of constraints. In particular, we design a scalable framework ApproxCov that takes in a test set u, a coverage parameter t, a tolerance parameter ε, and a confidence parameter δ, and returns an estimate of the t-wise coverage of u that is guaranteed to be within (1 ± ε)-factor of the ground truth with probability at least 1 - δ. We design a scalable framework ApproxMaxCov that, for a given formula F, a coverage parameter t, a tolerance parameter ε, and a confidence parameter δ, outputs an approximation which is guaranteed to be within (1 ± ε) factor of the maximum achievable t-wise coverage under F, with probability ≥ 1 - δ. Our comprehensive evaluation demonstrates that ApproxCov and ApproxMaxCov can handle benchmarks that are beyond the reach of current state-of-the-art approaches. We believe that the availability of ApproxCov and ApproxMaxCov will enable test suite designers to evaluate the effectiveness of their generators and thereby significantly impact the development of combinatorial testing techniques.
- Iago Abal, Claus Brabrand, and Andrzej Wasowski. 2014. 42 variability bugs in the linux kernel: a qualitative analysis. In Proceedings of the 29th ACM/IEEE international conference on Automated software engineering. 421--432.Google Scholar
Digital Library
- Mustafa Al-Hajjaji, Sebastian Krieter, Thomas Thüm, Malte Lochau, and Gunter Saake. 2016. IncLing: efficient product-line testing using incremental pairwise sampling. ACM SIGPLAN Notices 52, 3 (2016), 144--155.Google ScholarDigital Library
- Eduard Baranov, Axel Legay, and Kuldeep S. Meel. 2020. Baital: an adaptive weighted sampling approach for improved t-wise coverage. In ESEC/FSE '20: 28th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, Virtual Event, USA, November 8--13, 2020. 1114--1126. Google ScholarDigital Library
- Thorsten Berger, Ralf Rublack, Divya Nair, Joanne M Atlee, Martin Becker, Krzysztof Czarnecki, and Andrzej Wasowski. 2013. A survey of variability modeling in industrial practice. In Proceedings of the Seventh International Workshop on Variability Modelling of Software-intensive Systems. 1--8.Google ScholarDigital Library
- Thorsten Berger, Steven She, Rafael Lotufo, Andrzej Wasowski, and Krzysztof Czarnecki. 2013. A study of variability models and languages in the systems software domain. IEEE Transactions on Software Engineering 39, 12 (2013), 1611--1640.Google ScholarDigital Library
- Renée C Bryce and Charles J Colbourn. 2009. A density-based greedy algorithm for higher strength covering arrays. Software Testing, Verification and Reliability 19, 1 (2009), 37--53.Google ScholarDigital Library
- Andrea Calvagna and Angelo Gargantini. 2010. A formal logic approach to constrained combinatorial testing. Journal of Automated Reasoning 45, 4 (2010), 331--358.Google ScholarDigital Library
- Supratik Chakraborty, Kuldeep S. Meel, and Moshe Y. Vardi. 2013. A Scalable and Nearly Uniform Generator of SAT Witnesses. In Proc. of CAV. 608--623.Google Scholar
- Supratik Chakraborty, Kuldeep S. Meel, and Moshe Y. Vardi. 2013. A Scalable Approximate Model Counter. In Proc. of CP. 200--216.Google Scholar
- Supratik Chakraborty, Kuldeep S. Meel, and Moshe Y. Vardi. 2014. Balancing Scalability and Uniformity in SAT Witness Generator. In Proc. of DAC. 1--6.Google Scholar
- Supratik Chakraborty, Kuldeep S. Meel, and Moshe Y. Vardi. 2016. Algorithmic Improvements in Approximate Counting for Probabilistic Inference: From Linear to Logarithmic SAT Calls. In Proc. of IJCAI.Google ScholarDigital Library
- Tsong Yueh Chen, Hing Leung, and IK Mak. 2004. Adaptive random testing. In Annual Asian Computing Science Conference. Springer, 320--329.Google ScholarDigital Library
- Anastasia Cmyrev and Ralf Reissing. 2014. Efficient and effective testing of automotive software product lines. King Mongkuts University of Technology North Bangkok International Journal of Applied Science and Technology 7, 2 (2014), 53--57.Google Scholar
- David M. Cohen, Siddhartha R. Dalal, Michael L. Fredman, and Gardner C. Patton. 1997. The AETG system: An approach to testing based on combinatorial design. IEEE Transactions on Software Engineering 23, 7 (1997), 437--444.Google ScholarDigital Library
- Myra B. Cohen, Peter B. Gibbons, Warwick B. Mugridge, and Charles J. Colbourn. [n.d.]. Constructing Test Suites for Interaction Testing. In Proceedings of the 25th International Conference on Software Engineering, May 3--10, 2003, Portland, Oregon, USA. 38--48.Google Scholar
- R. Dechter, K. Kask, E. Bin, and R. Emek. 2002. Generating Random Solutions for Constraint Satisfaction Problems. In Proc. of AAAI/IAAI. 15--21.Google Scholar
- Joe W Duran and Simeon C Ntafos. 1984. An evaluation of random testing. IEEE transactions on Software Engineering 4 (1984), 438--444.Google ScholarDigital Library
- Arnaud Durand, Miki Hermann, and Phokion G Kolaitis. 2005. Subtractive reductions and complete problems for counting complexity classes. Theoretical Computer Science 340, 3 (2005), 496--513.Google ScholarDigital Library
- Rafael Dutra, Kevin Laeufer, Jonathan Bachrach, and Koushik Sen. 2018. Efficient sampling of SAT solutions for testing. In 2018 IEEE/ACM 40th International Conference on Software Engineering (ICSE). IEEE, 549--559.Google ScholarDigital Library
- Rafael Dutra, Kevin Laeufer, Jonathan Bachrach, and Koushik Sen. 2018. Efficient sampling of SAT solutions for testing. In Proc. of ICSE. 549--559.Google ScholarDigital Library
- Brady J Garvin, Myra B Cohen, and Matthew B Dwyer. 2009. An improved meta-heuristic search for constrained interaction testing. In 2009 1st International Symposium on Search Based Software Engineering. IEEE, 13--22.Google ScholarDigital Library
- V. Gogate and R. Dechter. 2006. A New Algorithm for Sampling CSP Solutions Uniformly at Random. In Proc. of CP. 711--715.Google Scholar
- Carla P. Gomes, Ashish Sabharwal, and Bart Selman. 2006. Model counting: A new strategy for obtaining good bounds. In Proc. of AAAI. 54--61.Google Scholar
- Carla P. Gomes, Ashish Sabharwal, and Bart Selman. 2007. Near-Uniform sampling of combinatorial spaces using XOR constraints. In Proc. of NIPS. 670--676.Google Scholar
- Loreto Gonzalez-Hernandez, Nelson Rangel-Valdez, and Jose Torres-Jimenez. 2010. Construction of mixed covering arrays of variable strength using a tabu search approach. In International Conference on Combinatorial Optimization and Applications. Springer, 51--64.Google ScholarCross Ref
- Rahul Gupta, Shubham Sharma, Subhajit Roy, and Kuldeep S. Meel. 2019. WAPS: Weighted and Projected Sampling. In Proceedings of Tools and Algorithms for the Construction and Analysis of Systems (TACAS).Google Scholar
- Yue Jia, Myra B. Cohen, Mark Harman, and Justyna Petke. 2015. Learning Combinatorial Interaction Test Generation Strategies Using Hyperheuristic Search. In 37th IEEE/ACM International Conference on Software Engineering, ICSE 2015, Florence, Italy, May 16--24, 2015, Volume 1. IEEE Computer Society, 540--550.Google Scholar
- Martin Fagereng Johansen, Øystein Haugen, and Franck Fleurey. 2012. An algorithm for generating t-wise covering arrays from large feature models. In Proceedings of the 16th International Software Product Line Conference-Volume 1. 46--55.Google ScholarDigital Library
- R.M. Karp and M. Luby. 1983. Monte-Carlo algorithms for enumeration and reliability problems. Proc. of FOCS (1983).Google Scholar
- Richard M Karp, Michael Luby, and Neal Madras. 1989. Monte-Carlo Approximation Algorithms for Enumeration Problems. Journal of Algorithms 10, 3 (1989), 429--448.Google ScholarCross Ref
- Nathan Kitchen. 2010. Markov Chain Monte Carlo stimulus generation for constrained random simulation. Ph.D. Dissertation. UC Berkeley.Google Scholar
- Alexander Knüppel, Thomas Thüm, Stephan Mennicke, Jens Meinicke, and Ina Schaefer. 2017. Is there a mismatch between real-world feature models and product-line research?. In Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering. ACM, 291--302.Google ScholarDigital Library
- D Richard Kuhn, Raghu N Kacker, and Yu Lei. 2010. Practical combinatorial testing. NIST special Publication 800, 142 (2010), 142.Google ScholarDigital Library
- D Richard Kuhn, Raghu N Kacker, and Yu Lei. 2013. Introduction to combinatorial testing. CRC press.Google Scholar
- James H Kukula and Thomas R Shiple. 2000. Building circuits from relations. In Proc. of CAV.Google ScholarCross Ref
- Yu Lei, Raghu Kacker, D Richard Kuhn, Vadim Okun, and James Lawrence. 2007. IPOG: A general strategy for t-way software testing. In 14th Annual IEEE International Conference and Workshops on the Engineering of Computer-Based Systems (ECBS'07). IEEE, 549--556.Google ScholarDigital Library
- Yu Lei, Raghu Kacker, D Richard Kuhn, Vadim Okun, and James Lawrence. 2008. IPOG/IPOG-D: efficient test generation for multi-way combinatorial testing. Software Testing, Verification and Reliability 18, 3 (2008), 125--148.Google ScholarDigital Library
- Jia Hui Liang, Vijay Ganesh, Krzysztof Czarnecki, and Venkatesh Raman. 2015. Sat-based analysis of large real-world feature models is easy. In Proceedings of the 19th International Conference on Software Product Line. ACM, 91--100.Google ScholarDigital Library
- Chuan Luo, Binqi Sun, Bo Qiao, Junjie Chen, Hongyu Zhang, Jinkun Lin, Qingwei Lin, and Dongmei Zhang. 2021. LS-Sampling: An Effective Local Search Based Sampling Approach for Achieving High t-Wise Coverage (ESEC/FSE 2021). 1081--1092.Google Scholar
- Robert Mandl. 1985. Orthogonal Latin squares: an application of experiment design to compiler testing. Commun. ACM 28, 10 (1985), 1054--1058.Google ScholarDigital Library
- Dusica Marijan, Arnaud Gotlieb, Sagar Sen, and Aymeric Hervieu. 2013. Practical pairwise testing for software product lines. In Proceedings of the 17th international software product line conference. 227--235.Google ScholarDigital Library
- James D McCaffrey. 2009. Generation of pairwise test sets using a genetic algorithm. In 2009 33rd annual IEEE international computer software and applications conference, Vol. 1. IEEE, 626--631.Google Scholar
- Flávio Medeiros, Christian Kästner, Márcio Ribeiro, Rohit Gheyi, and Sven Apel. 2016. A comparison of 10 sampling algorithms for configurable systems. In 2016 IEEE/ACM 38th International Conference on Software Engineering (ICSE). IEEE, 643--654.Google ScholarDigital Library
- Kuldeep S. Meel. 2020. Model counting and uniform sampling instances. Google ScholarCross Ref
- Kuldeep S Meel and S Akshay. 2020. Sparse hashing for scalable approximate model counting: Theory and practice. In Proc. of LICS.Google ScholarDigital Library
- Matthew W Moskewicz, Conor F Madigan, Ying Zhao, Lintao Zhang, and Sharad Malik. 2001. Chaff: Engineering an efficient SAT solver. In Proc. of DAC.Google ScholarDigital Library
- Changhai Nie and Hareton Leung. 2011. A survey of combinatorial testing. ACM Computing Surveys (CSUR) 43, 2 (2011), 1--29.Google ScholarDigital Library
- Kari J Nurmela. 2004. Upper bounds for covering arrays by tabu search. Discrete applied mathematics 138, 1--2 (2004), 143--152.Google Scholar
- Jeho Oh, Paul Gazzillo, and Don S. Batory. 2019. t-wise coverage by uniform sampling. In Proceedings of the 23rd International Systems and Software Product Line Conference, SPLC 2019, Volume A, Paris, France, September 9--13, 2019. 15:1--15:4.Google Scholar
- David Lorge Parnas. 1976. On the design and development of program families. IEEE Transactions on software engineering 1 (1976), 1--9.Google ScholarDigital Library
- Gilles Perrouin, Sagar Sen, Jacques Klein, Benoit Baudry, and Yves Le Traon. 2010. Automated and scalable t-wise test case generation strategies for software product lines. In 2010 Third international conference on software testing, verification and validation. IEEE, 459--468.Google ScholarDigital Library
- Tobias Pett, Thomas Thüm, Tobias Runge, Sebastian Krieter, Malte Lochau, and Ina Schaefer. [n.d.]. Product Sampling for Product Lines: The Scalability Challenge. ([n.d.]).Google Scholar
- Quentin Plazar, Mathieu Acher, Gilles Perrouin, Xavier Devroey, and Maxime Cordy. 2019. Uniform sampling of sat solutions for configurable systems: Are we there yet?. In 2019 12th IEEE Conference on Software Testing, Validation and Verification (ICST). IEEE, 240--251.Google ScholarCross Ref
- Iran Rodrigues, Márcio Ribeiro, Flávio Medeiros, Paulo Borba, Baldoino Fonseca, and Rohit Gheyi. 2016. Assessing fine-grained feature dependencies. Inf. Softw. Technol. 78 (2016), 27--52. Google ScholarDigital Library
- Gadiel Seroussi and Nader H Bshouty. 1988. Vector sets for exhaustive testing of logic circuits. IEEE Transactions on Information Theory 34, 3 (1988), 513--522.Google ScholarDigital Library
- Shubham Sharma, Rahul Gupta, Subhajit Roy, and Kuldeep S. Meel. 2018. Knowledge Compilation meets Uniform Sampling. In Proc. of LPAR-22. 620--636.Google Scholar
- Neil JA Sloane. 1993. Covering arrays and intersecting codes. Journal of combinatorial designs 1, 1 (1993), 51--63. Google ScholarCross Ref
- Mate Soos, Stephan Gocht, and Kuldeep S Meel. 2020. Tinted, Detached, and Lazy CNF-XOR solving and its Applications to Counting and Sampling. In International Conference on Computer Aided Verification (CAV).Google ScholarDigital Library
- Mate Soos and Kuldeep S Meel. 2019. BIRD: Engineering an Efficient CNF-XOR SAT Solver and its Applications to Approximate Model Counting. In Proc. of AAAI.Google ScholarDigital Library
- Larry Stockmeyer. 1983. The complexity of approximate counting. In Proc. of STOC. 118--126.Google ScholarDigital Library
- Reinhard Tartler, Daniel Lohmann, Christian Dietrich, Christoph Egger, and Julio Sincero. 2011. Configuration coverage in the analysis of large-scale system software. In Proceedings of the 6th Workshop on Programming Languages and Operating Systems. 1--5.Google ScholarDigital Library
- Keizo Tatsumi. 1987. Test case design support system. In Proc. International Conference on Quality Control (ICQC'87). 615--620.Google Scholar
- Thomas Thüm, Sven Apel, Christian Kästner, Ina Schaefer, and Gunter Saake. 2014. A classification and survey of analysis strategies for software product lines. ACM Computing Surveys (CSUR) 47, 1 (2014), 1--45.Google ScholarDigital Library
- Leslie G. Valiant. 1979. The complexity of enumeration and reliability problems. SIAM J. Comput. 8, 3 (1979), 410--421.Google ScholarDigital Library
- Ziyuan Wang, Baowen Xu, and Changhai Nie. 2008. Greedy heuristic algorithms to generate variable strength combinatorial test suite. In 2008 The Eighth International Conference on Quality Software. IEEE, 155--160.Google ScholarDigital Library
- Wei Wei and Bart Selman. 2005. A new approach to model counting. In Proc. of SAT.Google ScholarDigital Library
- Akihisa Yamada, Armin Biere, Cyrille Artho, Takashi Kitamura, and Eun-Hye Choi. 2016. Greedy combinatorial test case generation using unsatisfiable cores. In Proceedings of the 31st IEEE/ACM International Conference on Automated Software Engineering. 614--624.Google ScholarDigital Library
- Cemal Yilmaz, Sandro Fouché, Myra B. Cohen, Adam A. Porter, Gülsen Demiröz, and Ugur Koc. 2014. Moving Forward with Combinatorial Interaction Testing. Computer 47, 2 (2014), 37--45.Google ScholarDigital Library
Index Terms
- A scalable t-wise coverage estimator
Index terms have been assigned to the content through auto-classification.
Recommendations
Continuous T-Wise Coverage
SPLC '23: Proceedings of the 27th ACM International Systems and Software Product Line Conference - Volume AQuality assurance for highly configurable systems uses t-wise feature interaction coverage as a metric to measure the quality of selected samples for testing. Achieving t-wise feature interaction coverage requires testing many configurations, often ...
Baital: an adaptive weighted sampling approach for improved t-wise coverage
ESEC/FSE 2020: Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software EngineeringThe rise of highly configurable complex software and its widespread usage requires design of efficient testing methodology. t-wise coverage is a leading metric to measure the quality of the testing suite and the underlying test generation engine. While ...
Checked Coverage and Object Branch Coverage: New Alternatives for Assessing Student-Written Tests
SIGCSE '15: Proceedings of the 46th ACM Technical Symposium on Computer Science EducationMany educators currently use code coverage metrics to assess student-written software tests. While test adequacy criteria such as statement or branch coverage can also be used to measure the thoroughness of a test suite, they have limitations. Coverage ...
Comments