Preference based multi-objective algorithms applied to the variability testing of software product lines

Highlights

• Preference-Based Evolutionary Multi-objective Algorithms applied to SPL testing.

• PEMOAs outperform NSGA-II generating more solutions in the Region of Interest (ROI).

• R-NSGA-II presents the best performance considering the R-HV indicator.

• r-NSGA-II should be considered to visualize few solutions inside the ROI.

• PEMOAs can reduce the DM’s burden by selecting better set of products for SPL testing.

Abstract

Evolutionary Multi-Objective Algorithms (EMOAs) have been applied to derive products for the variability testing of Software Product Lines (SPLs), which is a complex task impacted by many factors, such as the number of products to be tested, coverage criteria, and efficacy to reveal faults. But such algorithms generally produce a lot of solutions that are uninteresting to the tester. This happens because traditional search algorithms do not take into consideration the user preferences. To ease the selection of the best solutions and avoid effort generating uninteresting solutions, this work introduces an approach that applies Preference-Based Evolutionary Multi-objective Algorithms (PEMOAs) to solve the problem. The approach is multi-objective, working with the number of products to be tested, pairwise coverage and mutation score. It incorporates the preferences before the evolution process and uses the Reference Point (RP) method. Two PEMOAs are evaluated: R-NSGA-II and r-NSGA-II, using two different formulations of objectives, and three kinds of RPs. PEMOAs outperform the traditional NSGA-II by generating a greater number of solutions in the Region of Interest (ROI) associated to the RPs. The use of PEMOAs can reduce the tester’s burden in the task of selecting a better and reduced set of products for SPL testing.

Introduction

The variability testing of Software Product Lines (SPLs) is devoted to test a set of products derived from the SPL variability model. The most popular and used variability model is the Feature Model (FM) (Kang et al., 2002), considered as the facto standard in the SPL engineering. Ideally, all the products should be tested, but the number of possible products grows exponentially with the number of features and the exhaustive testing is, in general, infeasible (Cohen et al., 2006). Then, the use of a test criterion can help in the selection of the best products to be tested. A test criterion is a predicate to be satisfied by the test data set and is generally used to consider that the testing activity has ended (Rapps and Weyuker, 1985).

In the FM context, the test cases are SPL products and these ones need to satisfy some criteria. The most popular criteria (Cohen, Dwyer, Shi, 2006, Lamancha, Usaola, 2010, Perrouin, Sen, Klein, Baudry, le Traon, 2010, Henard, Papadakis, Perrouin, Klein, Heymans, Le Traon, 2014, McGregor, 2001) are the ones derived from combinatorial testing (Cohen et al., 1996). Among them, the pairwise testing is the most used (Cohen, Dwyer, Shi, 2006, Lamancha, Usaola, 2010, McGregor, 2001, Oster, Zink, Lochau, Grechanik, 2011, Uzuncaova, Khurshid, Batory, 2010), which requires the selection of products to include all valid pairs of features present in the FM. The number of covered pairs is used to evaluate the set of products generated.

More recent works are based on mutation testing (Ferreira, Vergilio, Quináia, 2013, Henard, Papadakis, Perrouin, Klein, Le Traon, 2013, Arcaini, Gargantini, Vavassori, 2015, Lackner, Schmidt, 2014, Reuling, Bürdek, Rotärmel, Lochau, Kelter, 2015). Mutant FMs are generated to describe typical faults that can be present in the FM. The idea is to find, for each mutant, a product that is validated by the mutant and is not validated by the original FM (Ferreira et al., 2013), or otherwise, it is valid for the FM but it is invalid for the mutant. When a difference exists the mutant is considered killed. At the end, the mutation score is produced and used to evaluate a set of products according to the number of killed mutants.

The selection of products for SPL testing can be impacted by different factors such as: criterion coverage; cost (given by the number of products); efficacy (related to the number of faults found); and so on. Different solutions can be considered as good alternatives, according to the testing goals and the testers’ needs. Due to this, such a problem has been efficiently solved by Evolutionary Multi-Objective Algorithms (EMOAs), in the field known as Search-Based Software Engineering (SBSE) (Harman et al., 2012). Surveys on search-based SPL engineering (Harman, Jia, Krinke, Langdon, Petke, Zhang, 2014, Lopez-Herrejon, Linsbauer, Egyed, 2015) shows that there are many works devoted to this research subject (Lopez-Herrejon, Chicano, Ferrer, Egyed, Alba, 2013, Matnei Filho, Vergilio, 2015, Matnei Filho, Vergilio, 2016). The most used algorithms are Non-dominated Sorting Genetic Algorithm-II (NSGA-II) (Deb et al., 2002), Strength Pareto Evolutionary Algorithm (SPEA2) (Zitzler et al., 2001), and Indicator-Based Evolutionary Algorithm (IBEA) (Zitzler and Künzli, 2004). Such algorithms provide a set called non-dominated solutions, distributed over the Pareto-front. At the end, all these solutions are considered Pareto-equivalent and good solutions. The Decision Maker (DM) is then responsible for choosing the solution to be used.

However, this choice is not always easy for the DM, because the cardinality of the non-dominated solution set is, most times, very large, especially if the number of objectives becomes high. In our problem, that is, product selection for SPL testing, it is not true that all the solutions are equivalent from the DM’s point of view. For instance, the DM, (the tester in our case) is not usually interested in solutions with low coverage. Therefore, it is clear that the objectives are not equally important for the DM.

Cases like that are very common in SBSE, and to cope with these limitations, a new field has emerged, named Preference and Search-Based Software Engineering (PSBSE) (Ferreira et al., 2017c). PSBSE is the SBSE sub-research field devoted to the application of preference and search-based algorithms to solve software engineering problems. A preference-based algorithm is the one that incorporates human preferences, intuition, emotion or psychological aspects in the optimization process (Takagi, 2001). There are different ways to incorporate the user preferences (Branke, Deb, Miettinen, Słowinski, 2008, Ferreira, Vergilio, de Souza, 2017): before (a priori), during (interactively), or after (a posteriori) the solution evolution process.

The best way depends on several conditions related to the DM’s personality, context, problem’s characteristics, and so on. For example, consider our SPL testing problem, in many cases there is a coverage level for the system under test or a cost budget, which are known a priori. In this way, it is suitable an a priori approach instead of an interactively one that requires a lot of effort and can increase the DM’s burden. It is important to notice that a recent mapping on PSBSE (Ferreira et al., 2017c) does not mention this kind of approach applied in the SPL context.

Considering the above-mentioned facts, in this paper we introduce a Preference-Based Evolutionary Multi-objective Algorithm (PEMOA) approach to select products for the variability test of SPLs. The goal is to find the best solutions according to the tester’s needs, and considering trade-offs related to some factors such as cost, efficacy and criteria coverage. The proposed approach has two main characteristics:

• It is a mutiobjective approach. This allows the use of multiple factors that impact our problem. Ferreira et al. (2017c) point out that most solutions in PSBSE are mono-objective. Besides, the authors consider the use of multi-objective approach is a research gap in PSBSE;

• It applies two PEMOAs based on the well-known and used algorithm NSGA-II. They are: Reference Point-based NSGA-II (R-NSGA-II) (Deb et al., 2006) and Reference Solution-based NSGA-II (r-NSGA-II) (Said et al., 2010). Both algorithms are based on the Reference Point (RP) concept. For this problem, the tester is interested only in a subset of solutions of the Pareto front, and the preferred part of the Pareto optimal region from the DM’s point of view is called Region of Interest (ROI). Then, the DM’s preference guides the PEMOA in the search, which results in a greater number of solutions in the ROI, close to the RP. This eases the task of choosing the final solution to be used.

These characteristics present some advantages: i) the approach contributes to the PSBSE field and can be used to solve other similar Software Engineering problems; ii) the preferences can be provided a priori, requiring reduced effort from the DM; iii) the existence of a ROI facilitates the visualization of the solutions (in the case of problems with two or three objectives) and recognizing if they satisfy the DM’s preferences; and iv) the approach can provide a greater number of solutions in the ROI space, reducing the number of uninteresting solutions and, consequently, easing the decision making task.

The paper also presents results that compare both PEMOAs, considering formulations to the problem with two and three objectives. The results show PEMOAs can generate better solutions than NSGA-II by avoiding uninteresting solutions (outside the ROI space). In general, R-NSGA-II presents the best performance and generates a greater number of solutions inside the ROI.

The paper is organized as follows. Section 2 reviews main PSBSE concepts and the PEMOAs used in this work. Section 3 contains related work. Section 4 introduces the proposed approach: problem formulation, search operators and objective functions. Section 5 describes how the empirical evaluation was conducted: research questions, quality indicators, used FMs, and algorithms configuration. Section Section 6 presents and analyses the results. Section 7 discusses the main threats to the validity of our results. Finally, Section 8 concludes the work and presents future research directions.