Performance analysis of dynamic optimization algorithms using relative error distance

Abstract

Quantification of the performance of algorithms that solve dynamic optimization problems (DOPs) is challenging, since the fitness landscape changes over time. Popular performance measures for DOPs do not adequately account for ongoing fitness landscape scale changes, and often yield a confounded view of performance. Similarly, most popular measures do not allow for fair performance comparisons across multiple instances of the same problem type nor across different types of problems, since performance values are not normalized. Many measures also assume normally distributed input data values, while in reality the necessary conditions for data normality are often not satisfied. The majority of measures also fail to capture the notion of performance variance over time. This paper proposes a new performance measure for DOPs, namely the relative error distance. The measure shows how close to optimal an algorithm performs by considering the multi-dimensional distance between the vector comprising the normalized performance scores for specific algorithm iterations of interest, and the theoretical point of best possible performance. The new measure does not assume normally distributed performance data across fitness landscape changes, is resilient against fitness landscape scale changes, better incorporates performance variance across fitness landscape changes into a single scalar value, and allows easier algorithm comparisons using established nonparametric statistical methods.

Introduction

Dynamic optimization problems (DOPs) are challenging since, as time passes, optimal solutions may become sub-optimal while previously inferior regions of the search space may suddenly yield the best solutions. Most optimization algorithms that are specialized to solve problems in static environments tend to be ineffective in dynamic environments due to the dynamic nature of DOPs [1]. As a result, many swarm intelligence (SI) and evolutionary computation (EC) meta-heuristics have been developed that focus specifically on solving DOPs [2], [3], [4], [5].

Quantification of the performance of algorithms is not as straightforward for dynamic environments as it is for static environments. Cruz et al. [2] outline how, for static environments, it often suffices to report just the quality of the best-found solution, and (optionally) the memory/time cost to achieve the result. For DOPs, these simple assessments are not sufficient since practitioners are interested in other aspects of algorithm performance during the optimization process. Such details include understanding an algorithm’s ability to detect problem changes, continually explore and exploit different areas of the problem search space, track existing optima as they move through the problem space, find new optima as they appear over time, track the stability of solutions over time, and determine how constraints are handled.

Researchers have defined many specialized performance measures to assess algorithms that solve DOPs. A number of surveys discuss the most popular performance measures that are used to quantify the ability of algorithms in solving DOPs [2], [4], [5]. The majority of reviewed measures have shortcomings that make fair comparisons of algorithm performance challenging. These shortcomings are discussed at length in this paper. The most often-used measures in literature, namely the offline error and offline performance [6], [7], and average best error before change [8] have crippling limitations that can cause observers to misreport findings by up to 60% (as shown in Section 5). Such shortcomings make it hard for existing measures to faithfully represent the underlying reality of an algorithm’s performance while solving a DOP. This paper proposes a new performance measure called the relative error distance (RED), that addresses the limitations of popular performance measures. The paper therefore makes a contribution towards more fair and sound measures to analyze and compare the performance of algorithms focused on solving DOPs.

The paper outline is as follows. Section 2 provides background information on DOPs, performance measures for algorithms aimed at solving DOPs, and a discussion of the attributes and shortcomings of popular performance measures. Section 3 introduces the RED measure. Section 4 outlines the experimental approach to validate the noteworthy characteristics of the RED measure, and section 5 presents the results of the empirical investigation. Section 6 concludes the paper.