Abstract
The In-Parameter-Order-General algorithm is one of the most widely used algorithms for covering array generation. It has found wide adoption in the field of combinatorial testing as it produces acceptably small covering arrays while being faster than alternative algorithms. In this paper we describe in detail how to efficiently design an efficient implementation, referred to as Fast In-Parameter-Order-General algorithm, which improves test generation times significantly while maintaining a low memory footprint compared to existing implementations. Besides the general algorithmic engineering and optimization considerations, we analyse and evaluate their benefits in an experimental study.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Bozic, J., Garn, B., Simos, D.E., Wotawa, F.: Evaluation of the IPO-family algorithms for test case generation in web security testing. In: 2015 IEEE Eighth International Conference on Software Testing, Verification and Validation Workshops (ICSTW), pp. 1–10. IEEE (2015)
Bryce, R.C., Colbourn, C.J.: The density algorithm for pairwise interaction testing. Softw. Test. Verif. Reliab. 17(3), 159–182 (2007)
Bryce, R.C., Colbourn, C.J.: A density-based greedy algorithm for higher strength covering arrays. Softw. Test. Verif. Reliab. 19(1), 37–53 (2009)
Cohen, D.M., Dalal, S.R., Fredman, M.L., Patton, G.C.: The AETG system: an approach to testing based on combinatorial design. IEEE Trans. Softw. Eng. 23(7), 437–444 (1997)
Colbourn, C.J., Dinitz, J.H.: Handbook of Combinatorial Designs. CRC Press, Boca Raton (2006)
Duan, F., Lei, Y., Yu, L., Kacker, R.N., Kuhn, D.R.: Improving IPOG’s vertical growth based on a graph coloring scheme. In: 2015 IEEE Eighth International Conference on Software Testing, Verification and Validation Workshops (ICSTW), pp. 1–8. IEEE (2015)
Forbes, M., Lawrence, J., Lei, Y., Kacker, R.N., Kuhn, D.R.: Refining the in-parameter-order strategy for constructing covering arrays. J. Res. Natl. Inst. Stand. Technol. 113(5), 287 (2008)
Gao, S.-W., Lv, J.-H., Bing-Lei, D., Colbourn, C.J., Ma, S.-L.: Balancing frequencies and fault detection in the in-parameter-order algorithm. J. Comput. Sci. Technol. 30(5), 957–968 (2015)
Gao, S., Lv, J., Du, B., Jiang, Y., Ma, S.: General optimization strategies for refining the in-parameter-order algorithm. In: 2014 14th International Conference on Quality Software (QSIC), pp. 21–26. IEEE (2014)
Hartman, A., Raskin, L.: Problems and algorithms for covering arrays. Discret. Math. 284(1), 149–156 (2004)
Kitsos, P., Simos, D.E., Torres-Jimenez, J., Voyiatzis, A.G.: Exciting FPGA cryptographic trojans using combinatorial testing. In: 2015 IEEE 26th International Symposium on Software Reliability Engineering (ISSRE), pp. 69–76. IEEE (2015)
Kleine, K., Simos, D.E.: Coveringcerts: combinatorial methods for x. 509 certificate testing. In: 2017 IEEE International Conference on Software Testing, Verification and Validation (ICST), pp. 69–79. IEEE (2017)
Kreher, D.L., Stinson, D.R.: Combinatorial Algorithms: Generation, Enumeration, and Search, vol. 7. CRC Press, Boca Raton (1998)
Kuhn, D.R., Bryce, R., Duan, F., Ghandehari, L.S., Lei, Y., Kacker, R.N.: Combinatorial testing: theory and practice. Adv. Comput. 99, 1–66 (2015)
Kuhn, D.R., Wallace, D.R., Gallo, A.M.: Software fault interactions and implications for software testing. IEEE Trans. Softw. Eng. 30(6), 418–421 (2004)
Lei, Y., Kacker, R., Kuhn, D.R., Okun, V., Lawrence, J.: IPOG: a general strategy for t-way software testing. In: 14th Annual IEEE International Conference and Workshops on the Engineering of Computer-Based Systems, 2007. ECBS’07, pp. 549–556. IEEE (2007)
Lei, Y., Kacker, R., Kuhn, D.R., Okun, V., Lawrence, J.: IPOG/IPOG-D: efficient test generation for multi-way combinatorial testing. Softw. Test. Verif. Reliab. 18(3), 125–148 (2008)
Lei, Y., Tai, K.-C.: In-parameter-order: a test generation strategy for pairwise testing. In: High-Assurance Systems Engineering Symposium, 1998. Proceedings. Third IEEE International, pp. 254–261. IEEE (1998)
Seroussi, G., Bshouty, N.H.: Vector sets for exhaustive testing of logic circuits. IEEE Trans. Inf. Theory 34(3), 513–522 (1988)
Torres-Jimenez, J., Izquierdo-Marquez, I.: Survey of covering arrays. In: 2013 15th International Symposium on Symbolic and Numeric Algorithms for Scientific Computing (SYNASC), pp. 20–27. IEEE (2013)
Younis, M.I., Zamli, K.Z.: MIPOG-an efficient t-way minimization strategy for combinatorial testing. Int. J. Comput. Theory Eng. 3(3), 388 (2011)
Yu, L., Lei, Y., Kacker, R.N., Kuhn, D.R.: Acts: a combinatorial test generation tool. In: 2013 IEEE Sixth International Conference on Software Testing, Verification and Validation (ICST), pp. 370–375. IEEE (2013)
Acknowledgements
The research presented in this paper has been funded in part by the Austrian Research Promotion Agency (FFG) under grant 851205 (Security ProtocoL Interaction Testing in Practice - SPLIT) and the Austrian COMET Program (FFG).
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Kleine, K., Simos, D.E. An Efficient Design and Implementation of the In-Parameter-Order Algorithm. Math.Comput.Sci. 12, 51–67 (2018). https://doi.org/10.1007/s11786-017-0326-0
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11786-017-0326-0