Abstract
Covering arrays are combinatorial objects used in testing large-scale systems to increase confidence in their correctness. To do so, each interaction of at most a specified number t of factors is represented in at least one test; that is, the covering array has strength t and index 1. For certain systems, the outcome of running a test may be altered by variability of the interaction effect or by measurement error of the test result. To improve the efficacy of testing, one can ensure that each interaction of t or fewer factors is represented in at least \(\lambda \) tests. When \(\lambda > 1\), this leads to covering arrays of higher index. We explore two algorithmic methods for constructing covering arrays of higher index. One is based on the in-parameter-order algorithm, and the other employs a conditional expectation paradigm. We compare these two by performing experiments on real-world benchmarks and on uniform parameter sets.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Akhtar Y, Colbourn CJ, Syrotiuk VR (2021) Mixed covering, locating, and detecting arrays via cyclotomy Springer Nature PROMS, accepted Jul 22
Aldaco AN, Colbourn CJ, Syrotiuk VR (2015) Locating arrays: a new experimental design for screening complex engineered systems. SIGOPS Oper Syst Rev 49(1):31–40
Alon N, Spencer JH (2004) The probabilistic method. Wiley, Hoboken
Alon N, Gutner S (2007) Balanced families of perfect hash functions and their applications. In: Automata, languages and programming. Lecture Lecture Notes in Computer Science, vol 4596, pp 435–446. Springer, Berlin
Alzahrani F, Salem A (2018) Sharp bounds for the Lambert \(W\) function. Integr Transform Spec Funct 29(12):971–978
Bryce RC, Colbourn CJ (2007) The density algorithm for pairwise interaction testing. Softw Test Verif Reliab 17:159–182
Bryce RC, Colbourn CJ (2009) A density-based greedy algorithm for higher strength covering arrays. Softw Test Verif Reliab 19(1):37–53
Cohen DM, Dalal SR, Fredman ML, Patton GC (1997) The AETG system: an approach to testing based on combinatorial design. IEEE Trans Softw Eng 23(7):437–444
Colbourn CJ (2011) Covering arrays and hash families. In: Information security, coding theory, and related combinatorics. NATO Science for Peace and Security Series. IOS Press, Amsterdam, pp 99–135
Colbourn CJ, Lanus E, Sarkar K (2017) Asymptotic and constructive methods for covering perfect hash families and covering arrays. Designs Codes Cryptogr 86(4):1–31
Colbourn CJ (2004) Combinatorial aspects of covering arrays. Le Matematiche (Catania) 58:121–167
Colbourn CJ (2010) Covering arrays from cyclotomy. Des Codes Cryptogr 55:201–219
Colbourn CJ (2014) Conditional expectation algorithms for covering arrays. J Comb Math Comb Comput 90:97–115
Colbourn CJ, McClary DW (2008) Locating and detecting arrays for interaction faults. J Comb Optim 15:17–48
Colbourn CJ, Syrotiuk VR (2018) On a combinatorial framework for fault characterization. Math Comput Sci 12(4):429–451
Deng D, Stinson DR, Wei R (2004) The Lovász local lemma and its applications to some combinatorial arrays. Design Codes Cryptogr 32(1–3):121–134
Dougherty RE (2019) Hash families and applications to \(t\)-restrictions. PhD thesis, Arizona State University
Dougherty RE, Colbourn CJ (2020) Perfect hash families: the generalization to higher indices. In: Raigorodskii AM, Rassias MT (eds) Discrete mathematics and applications. Springer, Cham
Forbes M, Lawrence J, Lei Y, Kacker RN, Kuhn DR (2008) Refining the in-parameter-order strategy for constructing covering arrays. J Res Nat Inst Stand Technol 113(5):287
Kleine K, Simos DE (2018) An efficient design and implementation of the in-parameter-order algorithm. Math Comput Sci 12(1):51–67
Kuhn DR, Kacker RN, Lei Y (2013) Introduction to combinatorial testing. Chapman & Hall/CRC Innovations in software engineering and software development series. Taylor & Francis, Boca Raton
Lei Y, Kacker R, Kuhn DR, Okun V, Lawrence J (2007) IPOG: a general strategy for t-way software testing. In: 14th annual IEEE international conference and workshops on the engineering of computer-based systems, 2007. ECBS’07, IEEE. pp 549–556
Lei Y, Tai K-C (1998) In-parameter-order: a test generation strategy for pairwise testing. In: High-assurance systems engineering symposium, 1998. Proceedings. Third IEEE International, pp. 254–261. IEEE
Leithner M, Kleine K, Simos DE (2018) CAMETRICS: a tool for advanced combinatorial analysis and measurement of test sets. In: 2018 IEEE international conference on software testing, verification and validation workshops (ICSTW), pp 318–327
Martínez C, Moura L, Panario D, Stevens B (2009/10) Locating errors using ELAs, covering arrays, and adaptive testing algorithms. SIAM J Discrete Math 23:1776–1799
Moura L, Raaphorst S, Stevens B (2019) Upper bounds on the sizes of variable strength covering arrays using the Lovász local lemma. Theoret Comput Sci 800:146–154
Nie C, Leung H (2011) A survey of combinatorial testing. ACM Comput Surv 43:1–29
Ray-Chaudhuri DK, Singhi NM (1988) On existence and number of orthogonal arrays. J Comb Theory Ser A 47(1):28–36
Sherwood GB, Martirosyan SS, Colbourn CJ (2006) Covering arrays of higher strength from permutation vectors. J Comb Des 14(3):202–213
Wagner M, Colbourn CJ, Simos DE (2021) In-parameter-order strategies for covering perfect hash families. Appl Math Comput. https://doi.org/10.1016/j.amc.2022.126952
Wagner M, Kleine K, Simos DE, Kuhn R, Kacker R (2020) CAGEN: a fast combinatorial test generation tool with support for constraints and higher-index arrays. In: Proceedings of 2020 IEEE international conference on software testing, verification and validation workshops (ICSTW)
Acknowledgements
The views expressed in this article are those of the author(s) and do not reflect the official policy or position of the Department of the Army, Department of Defense, or the U.S. Government. This research of KK, DES, and MW was carried out partly in the context of the Austrian COMET K1 program and publicly funded by the Austrian Research Promotion Agency (FFG) and the Vienna Business Agency (WAW). Research of CJC is funded by the U.S. National Science Foundation Grant #1813729.
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Dougherty, R.E., Kleine, K., Wagner, M. et al. Algorithmic methods for covering arrays of higher index. J Comb Optim 45, 28 (2023). https://doi.org/10.1007/s10878-022-00947-x
Accepted:
Published:
DOI: https://doi.org/10.1007/s10878-022-00947-x