Design and implementation of a harmony-search-based variable-strength t-way testing strategy with constraints support
Highlights
► HSS is the first variable strength t-way strategy that adopts HS algorithm as its core implementation. ► HSS also addresses the support for high interaction strength (i.e. t > 6) and implements the support for constraints. ► HSS performance in term of test size is statistically significant as compared to competing strategies.
Introduction
Software systems are typically composed of many subcomponents. Each subcomponent is responsible for providing one or more functionalities to the overall system. Often, having a different implementation for each subcomponent, with each implementation tailored and configured for a specific set of applications or environments, is desirable. In this way, interoperability and portability can potentially be maximized.
Because each subcomponent has a family of potentially thousands or millions of program configurations and instantiations, quality assurances activities are heavily burdened (i.e., particularly in terms of increasing testing costs and time-to-market pressure) [1], [2], [3]. Testing all of the configurations and instantiations is not feasible (i.e., due to combinatorial explosion of configurable test data); thus, a sound sampling mechanism for covering test parameter interaction is also desirable.
A recent work on t-way interaction testing has demonstrated that a combinatorial approach based on variable-strength t-way combinations of inputs (i.e., where t denotes the strength of interaction) can be effective. As a result, many new variable-strength t-way strategies have been developed and are now part of the literature.
Existing variable-strength t-way strategies adopt many different approaches, including pure computational-based (e.g. TVG [4], [5], Density [6], ParaOrder [6], WHITCH [7], and PICT [8]) and artificial intelligence (AI)-based approaches (e.g. ACS [9], [10], SA [6], [10], [11], VS-PSTG [12]). Although both approaches have merits, the AI-based approach appears to give better prospects as far as optimality (i.e., covering all the interactions with the most minimal combinations) is concerned. For this reason, we have adopted the AI-based approach as the basis of our implementation.
Currently, a number of existing strategies adopt the AI-based approach. Although useful, strategies based on SA [11], [13] are prone to local minima problem [14], [15], which can potentially lead to non-convergence of solution. Strategies based on GA [9], [10], [16], [17], [18], [19], [20] and ACA [6], [9], [10], [16], [20] are known to be computationally expensive due to the frequent needs to interact with environments during computation. Although GA and ACA give optimal results for small set of configurations, the range of interaction strength covered is limited to (i.e., t ⩽ 3) [9]. At a glance, PSTG, a particle swarm optimization (PSO)-based strategy [21], [22], [23], usefully addresses the limitations of GA and ACA, specifically in terms of improving the interaction strength support up to t = 6 (i.e., attributed to lesser computational loads). Nonetheless, a counter argument suggests that supporting up to t = 6 may be inadequate for the testing of highly interacting software systems.
The need of support for high interaction (i.e., t > 6) can be attributed to the fact that software systems have grown tremendously in terms of size and functionality, resulting in more interactions with one or more subsets of high interaction strengths. There are always opportunities for new intertwined dependencies between the involved parameters; this can potentially be problematic and may cause failure [24].
Apart from the lack of support for high interaction strength, most AI-based strategies (e.g., GA, ACA, SA, and PSTG) generally do not address the support for constraints. Here, some parameter interactions (or combinations) are deemed impossible; hence, they are considered as constraints and must not be part of the final suite.
Complementing and addressing the aforementioned limitations of existing AI-based strategies, the current paper proposes to implement a new variable-strength t-way test generation strategy called Harmony Search Strategy (HSS). HSS is the first variable-strength t-way strategy to adopt harmony search (HS) algorithm as its core implementation. Unlike other competing AI-based variable-strength t-way strategies, HSS addresses the support for high interaction strength (i.e., t > 6), and implements seamless support for constraints. Experimental results demonstrate that HSS gives competitive results against most existing AI-based, as well as computational-based, variable-strength t-way strategies.
This paper is organized as follows. Section 2 discusses the mathematical notations for variable-strength interaction testing. Section 3 describes the problem definition model. Section 4 outlines the related work whilst Section 5 introduces the design and implementation of HSS. Section 6 illustrates the tuning of HSS parameters. Section 7 elaborates the benchmarking of HSS against other existing strategies. Section 8 illustrates the statistical analysis of the results. Finally, Section 9 provides the conclusion.
Section snippets
Covering array notations
Interaction test suites can be mapped to corresponding combinatorial objects, called covering arrays (CA). In order to aid discussion, this section elaborates useful notations for expressing CA.
Normally, the CA has four parameters; N, t, P, and v (i.e., CA (N, t, vP)). Here, the symbols P, v, and t are used to refer to number of parameters (or factors), values (or levels), and interaction strength for the CA, respectively. For example, CA (9, 2, 34) represents a test suite consisting of 9 × 4
Problem definition model
A simple configurable software system is used here as a model to illustrate the variable strength interaction testing and constraints. Table 1 represents the simplified pizza ordering system. The system takes different parameters and values (i.e., also termed factors and levels respectively) depending on the desired orders. In this example, the system consists of five parameters with three 2-value parameters (i.e., Pizza Type, Crust, and Delivery) and two 3-value parameters (i.e., Toppings and
Related work
Lei et al. in [27] classified two main approaches for constructing interaction test suites: algebraic and computational. Algebraic approach is based on the extension of mathematical functions. Examples of strategies that adopt this approach include orthogonal Latin squares (OLS) [28], CA [29] and MCA [30]. Mandl adopted an orthogonal test design to test a compiler [28]. Williams et al. developed TConfig Tool, which employs a recursive construction process based on orthogonal arrays to construct
Overview of HSS
HSS is a composition of two main algorithms as shown in Fig. 1: (1) HSS interaction generation algorithm for generating the interaction tuples and (2) HSS test suite generation for test suite construction.
Tuning HSS parameters
In order to ensure optimality, as far as the generated test size is concerned, the relevant HSS parameters need to be tuned. These parameters include maximum improvisation (i.e., iterations), HMS, HMCR, and PAR. For tuning, we select one case study involving a covering array of CA (N; 2, 57). The rationale for adopting this case study stems from the fact that it is a well-known case study in tuning many AI-based strategies, as described by Stardom [11] and Ahmed et al. [21]. To undertake the
Benchmarking results
We divide our benchmarking results into two parts. In the first part, we evaluate the performance of HSS against existing variable-strength strategies based on the published results in the works by Wang et al. [6], Chen et al. [9], and Ahmed and Zamli [12]. In the second part, we compare our HSS strategy supporting constraints with the published results in the works by Cohen et al. [34].
For each part, we have reported the best test suite size and the average size from repeating the HSS run 30
Statistical analysis
In order to assess the statistical significance in terms of the size performance, we perform the Wilcoxon signed-rank test between HSS and each individual strategy from all of our experimental table entries (i.e. from Table 8 till 14). The rationale for adopting Wilcoxon signed-rank test stemmed from the fact that the differences between the mean size results appear not normally distributed (i.e., rendering the need for a non-parametric test). Here, all the table entries with NA, and NS are
Conclusion
In the current paper, we have discussed and evaluated a novel harmony-search-based variable-strength t-way strategy with constraints support, called HSS. Our benchmark evaluation has demonstrated that HSS produces competitive results against most existing strategies. Unlike its AI counterparts, HSS also has the potential to address constrained covering arrays for high interaction strength. As part of our future work, we are augmenting HSS to support sequence-based covering array generation, as
Acknowledgements
This research is partially funded by the generous MOHE fundamental Grant “Investigating T-Way Test Data Reduction Strategy Using PSO Technique,” the USM RU Grant “Development of Variable-strength Interaction Testing Strategy for T-Way Test Data Generation” and the USM Short-term Grant “Development of Pairwise Interaction Testing Tool With Seeding and Constraints Support.”
References (54)
- et al.
An evaluation of methods for prioritizing software requirements
Information and Software Technology
(1998) - et al.
Improving test quality using robust unique input/output circuit sequences (UIOCs)
Information and Software Technology
(2006) - et al.
Search-based software engineering
Information and Software Technology
(2001) - et al.
A variable-strength interaction test suites generation strategy using particle swarm optimization
Journal of Systems and Software
(2011) - et al.
Constructing strength three covering arrays with augmented annealing
Discrete Mathematics
(2008) - et al.
Experimental evaluation of simulated annealing algorithms for the time-cost trade-off problem
Applied Mathematics and Computation
(2010) - et al.
A systematic review of search-based testing for non-functional system properties
Information and Software Technology
(2009) - et al.
Design and implementation of a t-way test data generation strategy with automated execution tool support
Information Sciences
(2011) - et al.
Problems and algorithms for covering arrays
Discrete Mathematics
(2004) - et al.
Particle swarm optimizer, ant colony strategy and harmony search scheme hybridized for optimization of truss structures
Computers & Structures
(2009)
Global-best harmony search
Applied Mathematics and Computation
A new structural optimization method based on the harmony search algorithm
Computers & Structures
An improved harmony search algorithm for solving optimization problems
Applied Mathematics and Computation
Metastrategy simulated annealing and tabu search algorithms for the vehicle routing problem
Annals of Operations Research
Constructing a t-way interaction test suite using the particle swarm optimization approach
International Journal of Innovative Computing, Information and Control
Cited by (135)
SCIPOG: Seeding and constraint support in IPOG strategy for combinatorial t-way testing to generate optimum test cases
2023, Journal of King Saud University - Computer and Information SciencesA three-phase approach to improve the functionality of t-way strategy
2024, Soft ComputingQ-learning whale optimization algorithm for test suite generation with constraints support
2023, Neural Computing and ApplicationsA novel combinatorial testing approach with fuzzing strategy
2023, Journal of Software: Evolution and ProcessAn Effective Approach to High Strength Covering Array Generation in Combinatorial Testing
2023, IEEE Transactions on Software Engineering