Design and implementation of a harmony-search-based variable-strength t-way testing strategy with constraints support

Abstract

Context

Although useful, AI-based variable strength t-way strategies are lacking in terms of the support for high interaction strength. Additionally, most AI-based strategies generally do not address the support for constraints. Addressing the aforementioned issues, this paper elaborates the design, implementation, and evaluation of a novel variable-strength-based on harmony search algorithm, called Harmony Search Strategy (HSS).

Objective

The objective of this work is to investigate the adoption of harmony search algorithm for constructing variable-strength t-way strategy.

Method

Implemented in Java, HSS integrates the harmony search algorithm as parts of its search engine.

Result

Benchmarking results demonstrate that HSS gives competitive results against most existing AI-based (and pure computational) counterparts. However, unlike other AI-based counterparts, HSS addresses the support for high interaction strength and permits the support for constraints.

Conclusion

AI-based t-way strategies tend to outperform the pure computational-based strategies in terms of test size.

Introduction

Software systems are typically composed of many subcomponents. Each subcomponent is responsible for providing one or more functionalities to the overall system. Often, having a different implementation for each subcomponent, with each implementation tailored and configured for a specific set of applications or environments, is desirable. In this way, interoperability and portability can potentially be maximized.

Because each subcomponent has a family of potentially thousands or millions of program configurations and instantiations, quality assurances activities are heavily burdened (i.e., particularly in terms of increasing testing costs and time-to-market pressure) [1], [2], [3]. Testing all of the configurations and instantiations is not feasible (i.e., due to combinatorial explosion of configurable test data); thus, a sound sampling mechanism for covering test parameter interaction is also desirable.

A recent work on t-way interaction testing has demonstrated that a combinatorial approach based on variable-strength t-way combinations of inputs (i.e., where t denotes the strength of interaction) can be effective. As a result, many new variable-strength t-way strategies have been developed and are now part of the literature.

Existing variable-strength t-way strategies adopt many different approaches, including pure computational-based (e.g. TVG [4], [5], Density [6], ParaOrder [6], WHITCH [7], and PICT [8]) and artificial intelligence (AI)-based approaches (e.g. ACS [9], [10], SA [6], [10], [11], VS-PSTG [12]). Although both approaches have merits, the AI-based approach appears to give better prospects as far as optimality (i.e., covering all the interactions with the most minimal combinations) is concerned. For this reason, we have adopted the AI-based approach as the basis of our implementation.

Currently, a number of existing strategies adopt the AI-based approach. Although useful, strategies based on SA [11], [13] are prone to local minima problem [14], [15], which can potentially lead to non-convergence of solution. Strategies based on GA [9], [10], [16], [17], [18], [19], [20] and ACA [6], [9], [10], [16], [20] are known to be computationally expensive due to the frequent needs to interact with environments during computation. Although GA and ACA give optimal results for small set of configurations, the range of interaction strength covered is limited to (i.e., t ⩽ 3) [9]. At a glance, PSTG, a particle swarm optimization (PSO)-based strategy [21], [22], [23], usefully addresses the limitations of GA and ACA, specifically in terms of improving the interaction strength support up to t = 6 (i.e., attributed to lesser computational loads). Nonetheless, a counter argument suggests that supporting up to t = 6 may be inadequate for the testing of highly interacting software systems.

The need of support for high interaction (i.e., t > 6) can be attributed to the fact that software systems have grown tremendously in terms of size and functionality, resulting in more interactions with one or more subsets of high interaction strengths. There are always opportunities for new intertwined dependencies between the involved parameters; this can potentially be problematic and may cause failure [24].

Apart from the lack of support for high interaction strength, most AI-based strategies (e.g., GA, ACA, SA, and PSTG) generally do not address the support for constraints. Here, some parameter interactions (or combinations) are deemed impossible; hence, they are considered as constraints and must not be part of the final suite.

Complementing and addressing the aforementioned limitations of existing AI-based strategies, the current paper proposes to implement a new variable-strength t-way test generation strategy called Harmony Search Strategy (HSS). HSS is the first variable-strength t-way strategy to adopt harmony search (HS) algorithm as its core implementation. Unlike other competing AI-based variable-strength t-way strategies, HSS addresses the support for high interaction strength (i.e., t > 6), and implements seamless support for constraints. Experimental results demonstrate that HSS gives competitive results against most existing AI-based, as well as computational-based, variable-strength t-way strategies.

This paper is organized as follows. Section 2 discusses the mathematical notations for variable-strength interaction testing. Section 3 describes the problem definition model. Section 4 outlines the related work whilst Section 5 introduces the design and implementation of HSS. Section 6 illustrates the tuning of HSS parameters. Section 7 elaborates the benchmarking of HSS against other existing strategies. Section 8 illustrates the statistical analysis of the results. Finally, Section 9 provides the conclusion.