Accelerate Evolution Strategy by Proximal Policy Optimization

A pivotal challenge in meta-heuristic optimization is the lack of knowledge inheritance in heuristic rules. For example, the Covariance Matrix Adaptation Evolution Strategy (CMA-ES) requires generating historical data from scratch for its adaptation mechanisms for each new instance, demanding an extensive number of fitness evaluations. This severely limits the practicality of meta-heuristics, especially under restricted evaluation budgets, hindering efficient navigation in high-dimensional spaces. To overcome this, we integrate Proximal Policy Optimization (PPO) with a vanilla Evolution Strategy (ES), forming the novel PPO-ES approach. Distinct from other adaptive ES variants like CMA-ES with cumulative path calculation and covariance matrix adaptation, it leverages PPO's capability for dynamic step-size adjustment. Our method streamlines the optimization process and incorporates a meticulously designed reward system to adeptly navigate the scalability challenge, significantly enhancing adaptability and efficiency of ES. PPO-ES, trained on part of the bbob benchmarks, was tested on these and the rest unseen problems, and further validated on bbob-largescale benchmarks with much higher dimensions. Results show that PPO-ES achieves faster or comparable convergence to CMA-ES. Our code can be found online: https://github.com/burningxt/PPO-ES_GECCO24.