A survey of many-objective optimisation in search-based software engineering

Highlights

• Software engineering problems often present many decision factors to be optimised.

• Many-objective optimisation brings advanced search methods well-suited for SBSE.

• The literature survey identifies many-objective SBSE as an emerging research topic.

• We discuss current findings and outline open challenges and future work in the field.

Abstract

Search-based software engineering (SBSE) is changing the way traditional software engineering (SE) activities are carried out by reformulating them as optimisation problems. The natural evolution of SBSE is bringing new challenges, such as the need of a large number of objectives to formally represent the many decision criteria involved in the resolution of SE tasks. This suggests that SBSE is moving towards many-objective optimisation, an emerging area that provides advanced techniques to cope with high-dimensional optimisation problems. To analyse this phenomenon, this paper surveys relevant SBSE literature focused on the resolution of many-objective problems. From the gathered knowledge, current limitations regarding problem formulation, algorithm selection, experimental design and industrial applicability are discussed. Through the analysis of observed trends, this survey provides a historical perspective and future lines of research concerning the adoption of many-objective optimisation within SBSE.

Introduction

Software engineers daily make decisions for a variety of tasks, such as planning the best schedule to guarantee project completion or selecting the subset of requirements for the next release that satisfy stakeholders the most. In these situations, manually evaluating all possible alternatives is impractical, specially if many – often conflicting – decision criteria are involved. To support engineers, search-based software engineering proposes reformulating software engineering activities as optimisation problems to be automatically solved by search algorithms (Harman et al., 2012).

Particularly, a key element in SBSE is the definition of the fitness function, which allows numerically assessing the quality of solutions. It is usually based on software metrics, and serves to guide the search algorithm. For instance, code coverage can be used to find the minimum test suite, while the minimisation of the number of code smells can be used to guide a refactoring process. It is hard to imagine that complex SE activities could be simplified to the extend that only one metric is required. It is most likely that SE problems present several conflicting goals (Harman et al., 2012). However, it does not necessarily imply that formulating multiple objective functions is straightforward.

Nevertheless, even two or three metrics could be insufficient to express all the decision criteria appearing in standard SE tasks, leading to the formulation of SBSE problems with a larger number of objectives. This implies that SE multi-objective problems are turning into many-objective problems, a term coined to refer to those problems with four or more objectives (Chand and Wagner, 2015). This new perspective represents an important challenge, since popular multi-objective algorithms present serious difficulties when a large number of objectives is considered. Opportunely, many-objective optimisation has become an active focus of research (Chand and Wagner, 2015), with a proliferation of specialised algorithms in the last years (Li et al., 2015a). The parallelism in the development of both fields, i.e. SBSE and many-objective optimisation, suggests a potential synergy that some SBSE authors are beginning to exploit (Hierons, Li, Liu, Segura, Zheng, 2016, Fleck, Troya, Kessentini, Wimmer, 2017, Panichella, Kifetew, Tonella, 2018).

At this stage, the aim of this paper is to bring this emerging phenomenon to light as a way to encourage the SBSE community to further explore the benefits of many-objective optimisation. Following a methodological literature search and analysis (Kitchenham and Charters, 2007), this survey first digs into the origins of many-objective optimisation problems (MaOPs) within SBSE. After completing the search procedure, 72 papers were selected within the scope.

Although this is a significant number of publications that could suggest that substantial work has been done since the origin of SBSE, the reality is that SBSE is just beginning to discover the real potential of many-objective optimisation. Fig. 1 shows this upward trend by depicting the cumulative number of publications in SBSE addressing many-objective problems per year.¹ As can be seen, half of the contributions have been published after 2014.

This survey analyses for the first time how SBSE is progressively incorporating emerging ideas from many-objective optimisation. Our findings suggest that a variety of many-objective problems have already been defined in SBSE, but only few approaches actually apply many-objective optimisation techniques. Nevertheless, this observation seems to be changing, and recently published studies are exploring new algorithms and approaches to solve more complete problem formulations. Hence, there are still some gaps and challenges that could open interesting lines of future research.

The rest of the paper is organised as follows. Section 2 provides some background on SBSE and many-objective optimisation. Section 3 details the methodology followed for conducting the literature survey. Sections 4–7 cover the different phases of the software life cycle, proposed as optimization problems solved from the many-objective perspective. This phase categorisation follows the literature standards in SBSE (Harman et al., 2012). The analysis of the most relevant findings is explained in Section 8, while Section 9 discusses gaps and challenges. Finally, Section 10 concludes the paper.