On the steady state analysis of covariance matrix self-adaptation evolution strategies on the noisy ellipsoid model

Abstract

This paper addresses the analysis of covariance matrix self-adaptive Evolution Strategies (CMSA-ES) on a subclass of quadratic functions subject to additive Gaussian noise: the noisy ellipsoid model. To this end, it is demonstrated that the dynamical systems approach from the context of isotropic mutations can be transferred to ES that also control the covariance matrix. Theoretical findings such as the component-wise quadratic progress rate or the self-adaptation response function can thus be reused for the CMSA-ES analysis. By deriving the steady state quantities approached on the noisy ellipsoid model for constant population size, a detailed description of the asymptotic CMSA-ES behavior is obtained. By providing self-adaptive ES with a population control mechanism, despite noise disturbances, the algorithm is able to realize continuing progress towards the optimum. Regarding the population control CMSA-ES (pcCMSA-ES), the analytical findings allow to specify its asymptotic long-term behavior and to identify influencing parameters. The finally obtained convergence rate matches the theoretical lower bound of all comparison-based direct search algorithms.

Introduction

In many real-world applications the influences of various uncertainties increase the complexity of a corresponding optimization problem. Such uncertainties are summarized by the term noise. Noise may originate from sensory disturbances, randomized simulations or modeling insufficiencies. It impairs the objective function evaluations or even the search space parameters of the optimization problem. In both cases, successive evaluations of the same parameter vector result in different objective function values. The task of finding an optimal solution for this kind of problems is a great challenge for optimization strategies.

In contrast to classical deterministic methods, direct optimization strategies such as Evolutionary Algorithms (EA) benefit from their intrinsically reduced exposure to noise disturbances. Their performance only relies on the observed objective function values of a candidate solution and, for example, the use of corrupted gradient information is evaded. In recent years, EA proved their ability to successfully deal with noisy optimization problems. This is endorsed by both empirical and theoretical investigations [1], [2].

The present paper focuses on the EA subclass of Evolution Strategies (ES) with mutative σ self-adaptation (σSA). Considering ES, a number of theoretical investigations on different test functions exists [3], [4], [5]. The theoretical analysis on such test functions is essential to understand the working principles of ES in the presence of noise. It allows to identify beneficial strategy parameter settings, supports the design of new successful algorithm variants, or discovers fundamental performance laws.

The work of [6] addressed the complete analysis of the σSA-ES on the noise-free ellipsoid model. While the respective analysis only takes into account isotropic mutations within the procreation process of new candidate solutions, that analysis of the ellipsoid model provides generality with respect to arbitrary rotations of the search space. The respective analysis was transferred to the noisy ellipsoid model in [7]. Recent theoretical analyses on the ellipsoid model also confined themselves on ES with isotropic mutations [8], [9]. That is, those investigations omit the covariance matrix adaptation which is an integral part of state-of-the-art ES, like the Covariance Matrix Adaptation ES (CMA-ES) [10] or the Covariance Matrix Self-Adaptation ES (CMSA-ES) [11].

Despite the fact that ES exhibit a certain robustness in the presence of moderate noise disturbances, strong noise degrades the ES performance. At worst, noise may cause the ES to prematurely converge or even to diverge from the optimum. Two common ways to address this potential performance degradation are either resampling of objective function values or increasing the population size of the ES. The first approach aims at a reduction of the noise influence by averaging over a number of κ objective function values. The second one uses growing population sizes in order to mitigate the noise impact via the internal intermediate recombination operator of the ES. Considering the $(\mu /{\mu }_I,\lambda )$-ES on quadratic functions increasing the population size turned out to be advantageous to performing resampling [12], [13]. On the downside, both methods result in escalating effort in terms of function evaluations.

In order to avoid excess of function evaluations adequate situations in the search process at which to apply resampling or to increase the population size have to be discovered. For detection of the right point to apply the above mentioned countermeasures, a number of strategies have been introduced over the years. These strategies range from simply reevaluating the objective function value of a single candidate solution several times to more sophisticated strategies like the uncertainty handling covariance matrix adaptation Evolution Strategy (UH-CMA-ES) [14] that considers rank changes within the offspring individuals after resampling their fitness ($\kappa =2$). A small number of rank changes indicate low noise intensity and vice versa. While the latter approach realizes a tangible notion of the noise intensity, it still turns out to be too pessimistic [5]. Even a lot of rank changes can be tolerated by the ES still being able to realize progress towards the optimum due to the genetic repair effect induced by the intermediate recombination [15]. That is, the population size matters and must be controlled in order to approximate the optimum as efficiently as possible. Already previously, a residual error-based population size control rule was introduced in [13]. The respective strategy increased the population size if the fitness dynamics on average do not exhibit further progress. The respective work revealed that standard self-adaptive ES exhibit an advantageous behavior in the presence of additive fitness noise. Unlike Cumulative Step-size Adaptation (CSA) which shows a continuous mutation strength decrease, σ-Self-Adaptive ES (σSA-ES) approach a steady state mutation strength. In fact, this bias towards larger step-sizes [16] is beneficial in noisy fitness environments as it helps to prevent premature convergence. For instance, the work of [17] revealed that a population control CMSA-ES variant, the pcCMSA-ES, does not behave like a “simple ES” [5] on the noisy ellipsoid model subject to additive Gaussian noise. That is, in the presence of strong noise the mutation strength of the pcCMSA-ES is not scaling with the distance to the optimum. Instead, the σ dynamics approach a steady state ${\sigma }_{ss}$ as the strategy approaches the optimum. A first theoretical analysis of the pcCMSA-ES was sketched in [18].

As it turns out, on the general ellipsoid model the dynamical systems approach appears to be transferable to self-adaptive ES that control their covariance matrix. To this end, the covariance matrix adaptation can be interpreted as a transformation of the fitness environment rather than a change of the mutation distribution. Essential parts of the analyses [6], [7]can thus be applied in the context of covariance-adaptive ES, and particularly to the CMSA-ES. Section 3 of the present paper is devoted to the clarification of this proposition.

Finding generalizations and providing additional insight into the working principles of the CMSA-ES in the presence of additive fitness noise on the ellipsoid model, the present paper is an extension of [18]. Apart from stating the motivations and the theoretical basis of our derivations more precisely, the present article elaborates on:

• The transfer of earlier theoretical predictions from the context of σSA-ES to CMSA-ES in Sec. 3.1.

• The derivations of the CMSA-ES stationary states in Sec. 4 and Sec. 5. In particular, the paper calculates the general steady-state mutation strength including noisy ellipsoid models that were disregarded in [18].

• The calculation of the pcCMSA-ES long-term dynamics and those of the corresponding convergence rate CR in Sec. 6. To this end, the estimates of the isolation time G are substantiated and the convergence rate CR in the marginal case $\varkappa =1$ is refined.

• The experimental verification of the theoretical predictions by taking into account ellipsoid test functions with different conditioning numbers.

After an introduction of the noisy optimization problem in Sec. 2, Section 3.1 considers the transfer of the analysis approach from ES using isotropic mutations towards covariance-adaptive ES. The concept of the dynamical systems approach is motivated and the fundamental equations from the context of isotropic ES [7] are presented. Parts of the theoretical analysis are addressing CMSA-ES with appropriate population control mechanism. The pcCMSA-ES [17] can be regarded as a standard example for these cases. Therefore, the working principles of the pcCMSA-ES are recapped in Sec. 3.3.

The first analysis step presented in Sec. 4 is the derivation of the steady state mutation strength of CMSA-ES on the noisy ellipsoid model. While [18] excluded some ellipsoid models with respect to the tractability of the analysis, the steady state derivation is extended to the general ellipsoid model. Providing a more convenient steady state representation for the ensuing analysis, the result that was already sketched in [18] is presented in more detail.

Having determined the mutation strength steady state, Section 5 is concerned with the description of the parameter vector dynamics of the CMSA-ES in the presence of noise. Assuming a fixed population size, a steady state expression of the parameter vector components is derived. This allows to rediscover previously obtained steady state terms of the noisy fitness values [12] as well as the residual distance from the optimum that is approached by the CMSA-ES algorithm without population adaptation. Up to this point, the analysis can (with respect to certain assumptions) be considered as an asymptotical examination of the CMSA-ES dynamics under additive Gaussian fitness noise.

Section 6 then takes up the concept of population control again. Given a CMSA-ES that appropriately elevates the population size when the evolutionary process is subject to strong noise disturbances, it is able to continuously reduce its residual distance from the optimum. A prototype of such strategies is the pcCMSA-ES. In this regard, Sec. 6 investigates the long-term behavior of the pcCMSA-ES and provides a theoretical derivation of its convergence rate.

The paper closes with a discussion of the results obtained and points out potential research directions.