Application of Particle Swarm Optimization to uniform and variable strength covering array construction

Abstract

Recently, researchers have started to explore the use of Artificial Intelligence (AI)-based algorithms as t-way (where t indicates the interaction strength) testing strategies. Many AI-based strategies have been developed, such as Ant Colony, Simulated Annealing, Genetic Algorithm, and Tabu Search. Although useful, most existing AI-based t-way testing strategies adopt complex search processes and require heavy computations. For this reason, existing AI-based t-way testing strategies have been confined to small interaction strengths (i.e., t ≤ 3) and small test configurations. Recent studies demonstrate the need to go up to t = 6 in order to capture most faults. In this paper, we demonstrate the effectiveness of our proposed Particle Swarm-based t-way Test Generator (PSTG) for generating uniform and variable strength covering arrays. Unlike other existing AI-based t-way testing strategies, the lightweight computation of the particle swarm search process enables PSTG to support high interaction strengths of up to t = 6. The performance of our proposed PSTG is evaluated using several sets of benchmark experiments. Comparatively, PSTG consistently outperforms its AI counterparts and other existing testing strategies as far as the size of the array is concerned. Furthermore, our case study demonstrates the usefulness of PSTG for facilitating fault detection owing to interactions of the input components.

Introduction

Software performance testing is an activity that aims to evaluate the capability of a program as well as to determine whether it meets its required results or not [1]. Owing to its usefulness in the software development life cycle, software performance testing comprehends a variety of activities, including stress, isolation, and configuration testing [2], [3]. In each activity, test cases are used in an established test plan to run experiments occupying the software system components. In large software systems, this process is limited by cost because the addition of each test case leads to additional expenditures. This, in turn, leads to the inability of exhaustive testing in performing such a testing process.

Design of Experiment (DOE) has been used to aid the software performance testing [4]. Here, each component of the system is called a “factor,” and each test case is called an “experimental run.” An experimental run represents a test case to comprehend the system components, where each component is represented by its valid numeric value or configuration [5]. When the system is tested exhaustively, the “full factorial” design of experiment is used [5]. However, when the system is large and the full factorial design is not desirable, the “fractional factorial” design is used to reduce the experimental run to a subset of the full factorial design. The fractional factorial design is used with systems of numeric factors; conversely, systems with categorical factors cannot use this method for experiments [6].

The D-Optimality design, on the other hand, has been used with systems, including categorical factors, to reduce the experimental run by selecting a subset of runs from the full factorial [7], [8]. Instead of a purely random selection of subsets of experimental runs from the full factorial design, the use of the D-Optimality design method in experiments leads to the production of experimental runs that are closer to full factorial design [4].

Recently, an alternative design based on Covering Array (CA) has been used for the approximation of full factorial design [4]. Compared with D-Optimality, empirical evidence demonstrates that CA produces better results than full factorial approximation experiments [4], [8]. In such a design, each t-set of factors (or system components) is covered by a set of experimental runs (at least once) to form a CA.

The use of CAs has proven to be adequate and effective in several applications, including drug screening, regulation of gene expression, data compression, code coverage and GUI testing [9], [10], [11], [12], [13], [14]. Motivated by the effectiveness of CAs, a number of recent studies have focused on the construction of CAs for combinatorial interaction testing using t-way strategies, where t signifies the interaction strength of the component. These strategies aim to optimally reduce the number of test cases (i.e., the number of rows in the CA) by ensuring that each test case greedily covers the required t-interactions (or t-set of factors) at least once for a typically large space of possible test values. This mechanism uniformly covers t-interactions of the system components to generate test cases. However, often, the interactions between parameter components are typically non-uniform [15], [16]. As an example, a system with an overall component values of two-way (pairwise) strength might have a subset of higher strength than the component values for the test [17]. Therefore, the strength might vary and be non-uniform during the testing process of the system component values. Taking both cases (i.e., uniform and variable interaction strength) as an NP hard computational optimization problem [16], [18], [19], many strategies based on Artificial Intelligence (AI) have been developed. Recent researches demonstrate that strategies based on Genetic Algorithm (GA), Ant Colony Algorithm (ACA), Simulated Annealing (SA), and Tabu Search (TS) can effectively generate small-sized CAs.

Although useful, most existing AI-based t-way testing strategies require complex computations (i.e., in terms of the need to deal with mutations, crossovers, and the local minima problem [17], [20], [21], [22]). For these reasons, existing AI-based t-way testing strategies have been confined to small interaction strengths (i.e., t ≤ 3) and small test configurations [15], [23], [24], [25]. To be effective, recent studies and empirical evidence demonstrate the need to go up to t = 6 in order to capture most faults in a software module [10], [26], [27], [28].

Particle Swarm Optimization (PSO) is known for the simplicity of its algorithm structure over other optimization methods [29], [30], [31], [32]. The usefulness of PSO for small uniform strength configurations has been studied in our earlier research work [33], [34]. In this paper, we demonstrate the competitiveness of our proposed Particle Swarm-based t-way Test Generator (PSTG) for uniform and variable strength CA generation. Unlike other existing AI-based t-way testing strategies, the use of PSO leads to lightweight computation in PSTG, thus, enabling it to support high interaction strengths of up to t = 6. Benchmark experiments demonstrate that PSTG is useful for generating uniform and variable strength CAs, as compared with other existing strategies. In addition, our case study demonstrates the usefulness of PSTG for generating uniform and variable strength CAs in order to tackle faults owing to interactions of the input components.

The rest of this paper is organized as follows. Section 2 discusses the theoretical framework and gives the necessary background on CAs. Section 3 reviews the state-of-the-art of t-way test case generation strategies for both uniform and variable strength CAs. Section 4 gives an overview of PSO. Section 5 outlines an extensive description of the design and implementation of PSTG, including its corresponding algorithms. Section 6 highlights the system parameter settings and demonstrates how the design parameters of PSO are tuned to achieve the best possible results. Section 7 gives an extensive evaluation of PSTG. Section 8 presents a case study to demonstrate the applicability and usefulness of the CAs generated by PSTG. Finally, Section 9 summarizes the paper and presents our conclusions.