Andrea Bombarda
ABSTRACT
Combinatorial interaction testing (CIT) is a testing technique that has proven to be effective in finding faults due to the interaction among inputs and in reducing the number of test cases, without losing effectiveness. Several tools have been proposed in the literature; however, generating tests remains a challenging task. In this paper, we present a technique for generating combinatorial test suites that uses a multi-thread architecture and exploits Satisfiability Modulo Theory (SMT) solvers to represent model parameters, constraints, and tuples, and it builds from SMT solver contexts the desired test suite. This technique is implemented by the tool KALI. The main advantage of using SMT solvers is that combinatorial models can contain all kinds of parameters and constraints. To evaluate our approach, we tested the impact of several optimizations and compared the performance of KALI with those of some existing tools for test generation. Our experiments confirm that the use of multi-threading is a promising technique but still requires some optimization for being more effective than the already available ones.
- H. Avila George. 2012. Constructing Covering Arrays using Parallel Computing and Grid Computing. Ph. D. Dissertation. Universitat Politècnica de València.Google Scholar
- A. Bombarda, E. Crippa, and A. Gargantini. 2021. An environment for benchmarking combinatorial test suite generators. In 2021 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW). IEEE.Google Scholar
- A. Bombarda and A. Gargantini. 2022. Parallel Test Generation for Combinatorial Models Based on Multivalued Decision Diagrams. In 2022 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW). IEEE.Google Scholar
- A. Bombarda, A. Gargantini, and A. Calvagna. 2022. Replication package for the paper "Multi-thread Combinatorial Test Generation with SMT solvers". https://github.com/fmselab/ct-tools/tree/main/KALIGoogle Scholar
- A. Calvagna, A. Gargantini, and E. Tramontana. 2009. Building T-wise Combinatorial Interaction Test Suites by Means of Grid Computing. In 2009 18th IEEE International Workshops on Enabling Technologies: Infrastructures for Collaborative Enterprises. 213--218.Google Scholar
- A. Calvagna, G. Pappalardo, and E. Tramontana. 2012. A Novel Approach to Effective Parallel Computing of t-Wise Covering Arrays. In 2012 IEEE 21st International Workshop on Enabling Technologies: Infrastructure for Collaborative Enterprises. 149--153.Google Scholar
- J. Christ, J. Hoenicke, and A. Nutz. 2012. SMTInterpol: An Interpolating SMT Solver. In Model Checking Software, Alastair Donaldson and David Parker (Eds.). Springer Berlin Heidelberg, Berlin, Heidelberg, 248--254.Google Scholar
- A. Cimatti, A. Griggio, B. J. Schaafsma, and R. Sebastiani. 2013. The MathSAT5 SMT Solver. In Tools and Algorithms for the Construction and Analysis of Systems, Nir Piterman and Scott A. Smolka (Eds.). Springer Berlin Heidelberg, Berlin, Heidelberg, 93--107.Google Scholar
- L. De Moura and N. Bjørner. 2008. Z3: An Efficient SMT Solver. In Tools and Algorithms for the Construction and Analysis of Systems, C. R. Ramakrishnan and Jakob Rehof (Eds.). Springer Berlin Heidelberg, Berlin, Heidelberg, 337--340.Google Scholar
- B. Dutertre. 2014. Yices 2.2. In Computer-Aided Verification (CAV'2014) (Lecture Notes in Computer Science, Vol. 8559), Armin Biere and Roderick Bloem (Eds.). Springer, 737--744.Google ScholarDigital Library
- A. Gargantini and M. Radavelli. 2018. Migrating Combinatorial Interaction Test Modeling and Generation to the Web. In 2018 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW). 308--317.Google Scholar
- W. Grieskamp et al. 2009. Interaction coverage meets path coverage by SMT constraint solving. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) 5826 LNCS (2009), 97--112.Google Scholar
- C. Henard, M. Papadakis, and Y. Le Traon. 2015. Flattening or not of the combinatorial interaction testing models?. In 2015 IEEE Eighth International Conference on Software Testing, Verification and Validation Workshops (ICSTW). IEEE, 1--4.Google Scholar
- Y. Lei, R. Kacker, D. R. Kuhn, V. Okun, and J. Lawrence. 2007. IPOG: A General Strategy for T-Way Software Testing. In 14th Annual IEEE International Conference and Workshops on the Engineering of Computer-Based Systems (ECBS'07). IEEE.Google Scholar
- H. Mercan, C. Yilmaz, and K. Kaya. 2019. CHiP: A Configurable Hybrid Parallel Covering Array Constructor. IEEE Transactions on Software Engineering 45, 12 (2019), 1270--1291.Google ScholarCross Ref
- J. Peleska, E. Vorobev, and F. Lapschies. 2011. Automated Test Case Generation with SMT-Solving and Abstract Interpretation. In Proceedings of the Third International Conference on NASA Formal Methods (Pasadena, CA) (NFM'11). Springer-Verlag, Berlin, Heidelberg, 298--312.Google Scholar
- P. Rümmer. 2008. A Constraint Sequent Calculus for First-Order Logic with Linear Integer Arithmetic. In Proc., 15th Int. Conf. on Logic for Programming, Artificial Intelligence and Reasoning (LNCS, Vol. 5330). Springer, 274--289.Google ScholarDigital Library
- K. C. Tai and Y. Lie. 2002. A Test Generation Strategy for Pairwise Testing. IEEE Trans. Softw. Eng. 28, 1 (jan 2002), 109--111.Google Scholar
- M. Veanes, C. Campbell, W. Schulte, and N. Tillmann. 2005. Online Testing with Model Programs. SIGSOFT Softw. Eng. Notes 30, 5 (sep 2005), 273--282. Google ScholarDigital Library
- Y. Wang et al. 2022. An Adaptive Penalty based Parallel Tabu Search for Constrained Covering Array Generation. Information and Software Technology 143 (2022).Google Scholar
- C. Wohlin, P. Runeson, M. Höst, M. C. Ohlsson, B. Regnell, and A. Wesslén. 2012. Experimentation in Software Engineering. Springer Berlin Heidelberg.Google Scholar
- Mohammed I. Younis and Kamal Z. Zamli. 2010. MC-MIPOG: A Parallel t-Way Test Generation Strategy for Multicore Systems. ETRI Journal 32, 1 (feb 2010), 73--83.Google ScholarCross Ref
- L. Yu, F. Duan, Y. Lei, R. N. Kacker, and D. R. Kuhn. 2015. Constraint handling in combinatorial test generation using forbidden tuples. In 2015 IEEE Eighth International Conference on Software Testing, Verification and Validation Workshops (ICSTW). 1--9.Google Scholar
- L. Yu, Y. Lei, R. N. Kacker, and D. R. Kuhn. 2013. ACTS: A Combinatorial Test Generation Tool. In 2013 IEEE Sixth International Conference on Software Testing, Verification and Validation. 370--375.Google Scholar
Index Terms
- Multi-thread Combinatorial Test Generation with SMT solvers
Recommendations
Prioritizing random combinatorial test suites
SAC '17: Proceedings of the Symposium on Applied ComputingThe behaviour of a system under test can be influenced by several factors, such as system configurations, user inputs, and so on. It has also been observed that many failures are caused by only a small number of factors. Combinatorial testing aims at ...
Evolutionary algorithm for prioritized pairwise test data generation
GECCO '12: Proceedings of the 14th annual conference on Genetic and evolutionary computationCombinatorial Interaction Testing (CIT) is a technique used to discover faults caused by parameter interactions in highly configurable systems. These systems tend to be large and exhaustive testing is generally impractical. Indeed, when the resources ...
Generating combinatorial test suite for interaction relationship
SOQUA '07: Fourth international workshop on Software quality assurance: in conjunction with the 6th ESEC/FSE joint meetingCombinatorial testing could detect the faults triggered by the interactions among factors in software. But in many cases, the pair-wise, N-way and even the variable strength combinatorial testing may lead test suite redundancy and fault detect ability ...
Comments