skip to main content
10.1145/3071178.3071205acmconferencesArticle/Chapter ViewAbstractPublication PagesgeccoConference Proceedingsconference-collections
research-article

Algorithm configuration data mining for CMA evolution strategies

Published: 01 July 2017 Publication History

Abstract

In the past years, quite a number of algorithmic extensions of the Covariance Matrix Adaptation Evolution Strategy (CMA-ES) have been proposed. These extensions define a large algorithm design space, but relatively little is known about the performance of most of these variations and the interaction between them.
In this paper we investigate how various algorithmic extensions interact and what their impact is on objective functions from the Black Box Optimization Benchmark (BBOB). Based on the existing Estimated Running Time (ERT) and Fixed Cost Error (FCE) measures, a novel algorithm quality measure is proposed to quantify an impact-score of the variants studied.
Using performance data from running 4,608 available algorithmic variations in the configurable CMA-ES framework published previously, decision trees and other data mining methods are used to analyze performance data. Analysis identifies algorithmic variations required for obtaining best performance and identifies strong differences between objective functions, thereby helping to understand the interaction of algorithmic components for an objective function and, ultimately, for an objective function class. The results also quantitatively confirm that popular variants such as increasing population size and elitism generally have a positive impact on algorithm performance.

References

[1]
Anne Auger, Dimo Brockhoff, and Nikolaus Hansen. 2011. Mirrored sampling in evolution strategies with weighted recombination. In Proceedings of the 13th Annual Conference Companion on Genetic and Evolutionary Computation (GECCO'11). ACM, 861--868.
[2]
Anne Auger and Nikolaus Hansen. 2005. A restart CMA evolution strategy with increasing population size. In Evolutionary Computation, 2005. The 2005 IEEE Congress on, Vol. 2. IEEE, 1769--1776.
[3]
Anne Auger, Mohammed Jebalia, and Olivier Teytaud. 2006. Algorithms (x, sigma, eta): quasi-random mutations for evolution strategies. In Artificial Evolution: 7th International Conference, Revised Selected Papers. Springer, 296--307.
[4]
Bernd Bischl, Olaf Mersmann, Heike Trautmann, and Mike Preuß. 2012. Algorithm Selection Based on Exploratory Landscape Analysis and Cost-sensitive Learning. In Proceedings of the 14th Annual Conference on Genetic and Evolutionary Computation (GECCO '12). ACM, New York, NY, USA, 313--320.
[5]
Dimo Brockhoff, Anne Auger, Nikolaus Hansen, Dirk V. Arnold, and Tim Hohm. 2010. Mirrored Sampling and Sequential Selection for Evolution Strategies. In Parallel Problem Solving from Nature, PPSN XI: 11th International Conference, Kraków, Poland, September 11--15, 2010, Proceedings, Part I, Robert Schaefer, Carlos Cotta, Joanna Ko lodziej, and Günter Rudolph (Eds.). Springer Berlin Heidelberg, Berlin, Heidelberg, 11--21.
[6]
Pilar Caamaño, Jose A. Becerra, Francisco Bellas, and Richard J. Duro. 2011. Are Evolutionary Algorithm Competitions Characterizing Landscapes Appropriately. In Proceedings of the 13th Annual Conference Companion on Genetic and Evolutionary Computation (GECCO '11). ACM, New York, NY, USA, 695--702.
[7]
Nikolaus Hansen. 2008. CMA-ES with Two-Point Step-Size Adaptation. CoRR abs/0805.0231 (2008). http://arxiv.org/abs/0805.0231
[8]
Nikolaus Hansen. 2009. Benchmarking a BI-population CMA-ES on the BBOB-2009 function testbed. In Proceedings of the 11th Annual Conference Companion on Genetic and Evolutionary Computation Conference: Late Breaking Papers (GECCO '09). ACM, 2389--2396.
[9]
Nikolaus Hansen, Steffen Finck, Raymond Ros, and Anne Auger. 2009. Real-Parameter Black-Box Optimization Benchmarking 2009: Noiseless Functions Definitions. Research Report RR-6829. INRIA. https://hal.inria.fr/inria-00362633
[10]
Nikolaus Hansen and Andreas Ostermeier. 1996. Adapting arbitrary normal mutation distributions in evolution strategies: The covariance matrix adaptation. In Evolutionary Computation (CEC), 1996 IEEE Congress on. IEEE, 312--317.
[11]
Frank Hutter, Holger H. Hoos, and Kevin Leyton-Brown. 2011. Sequential Model-Based Optimization for General Algorithm Configuration. Springer Berlin Heidelberg, Berlin, Heidelberg, 507--523.
[12]
Grahame Jastrebski, Dirk V Arnold, and others. 2006. Improving evolution strategies through active covariance matrix adaptation. In Evolutionary Computation (CEC), 2006 IEEE Congress on. IEEE, 2814--2821.
[13]
Olaf Mersmann, Bernd Bischl, Heike Trautmann, Mike Preuss, Claus Weihs, and Günter Rudolph. 2011. Exploratory Landscape Analysis. In Proceedings of the 13th Annual Conference on Genetic and Evolutionary Computation (GECCO '11). ACM, New York, NY, USA, 829--836.
[14]
A. Piad-Morffis, S. Estevez-Velarde, A. Bolufe-Rohler, J. Montgomery, and S. Chen. 2015. Evolution strategies with thresheld convergence. In Evolutionary Computation (CEC), 2015 IEEE Congress on. 2097--2104.
[15]
John R. Rice. 1976. The Algorithm Selection Problem*. Advances in Computers, Vol. 15. Elsevier, 65 -- 118.
[16]
Sander van Rijn, Hao Wang, Matthijs van Leeuwen, and Thomas Bäck. 2016. Evolving the Structure of Evolution Strategies. Computer 49, 5 (May 2016), 54--63.
[17]
Hao Wang, Michael Emmerich, and Thomas Bäck. 2014. Mirrored Orthogonal Sampling with Pairwise Selection in Evolution Strategies. In Proceedings of the 29th Annual ACM Symposium on Applied Computing. ACM, 154--156.

Cited By

View all
  • (2025)Location, Size, and CapacityInto a Deeper Understanding of Evolutionary Computing: Exploration, Exploitation, and Parameter Control10.1007/978-3-031-75577-4_1(1-152)Online publication date: 18-Jan-2025
  • (2024)Quantifying Individual and Joint Module Impact in Modular Optimization Frameworks2024 IEEE Congress on Evolutionary Computation (CEC)10.1109/CEC60901.2024.10611779(1-8)Online publication date: 30-Jun-2024
  • (2023)Transfer of Multi-objectively Tuned CMA-ES Parameters to a Vehicle Dynamics ProblemEvolutionary Multi-Criterion Optimization10.1007/978-3-031-27250-9_39(546-560)Online publication date: 9-Mar-2023
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
GECCO '17: Proceedings of the Genetic and Evolutionary Computation Conference
July 2017
1427 pages
ISBN:9781450349208
DOI:10.1145/3071178
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 01 July 2017

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. empirical study
  2. evolution strategies
  3. metaheuristics
  4. parameter tuning
  5. performance measures

Qualifiers

  • Research-article

Conference

GECCO '17
Sponsor:

Acceptance Rates

GECCO '17 Paper Acceptance Rate 178 of 462 submissions, 39%;
Overall Acceptance Rate 1,669 of 4,410 submissions, 38%

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)11
  • Downloads (Last 6 weeks)3
Reflects downloads up to 19 Feb 2025

Other Metrics

Citations

Cited By

View all
  • (2025)Location, Size, and CapacityInto a Deeper Understanding of Evolutionary Computing: Exploration, Exploitation, and Parameter Control10.1007/978-3-031-75577-4_1(1-152)Online publication date: 18-Jan-2025
  • (2024)Quantifying Individual and Joint Module Impact in Modular Optimization Frameworks2024 IEEE Congress on Evolutionary Computation (CEC)10.1109/CEC60901.2024.10611779(1-8)Online publication date: 30-Jun-2024
  • (2023)Transfer of Multi-objectively Tuned CMA-ES Parameters to a Vehicle Dynamics ProblemEvolutionary Multi-Criterion Optimization10.1007/978-3-031-27250-9_39(546-560)Online publication date: 9-Mar-2023
  • (2022)Using structural bias to analyse the behaviour of modular CMA-ESProceedings of the Genetic and Evolutionary Computation Conference Companion10.1145/3520304.3534035(1674-1682)Online publication date: 9-Jul-2022
  • (2022)The importance of landscape features for performance prediction of modular CMA-ES variantsProceedings of the Genetic and Evolutionary Computation Conference10.1145/3512290.3528832(648-656)Online publication date: 8-Jul-2022
  • (2022)Benchmarking Feature-Based Algorithm Selection Systems for Black-Box Numerical OptimizationIEEE Transactions on Evolutionary Computation10.1109/TEVC.2022.316977026:6(1321-1335)Online publication date: Dec-2022
  • (2021)Tuning as a means of assessing the benefits of new ideas in interplay with existing algorithmic modulesProceedings of the Genetic and Evolutionary Computation Conference Companion10.1145/3449726.3463167(1375-1384)Online publication date: 7-Jul-2021
  • (2021)Explorative data analysis of time series based algorithm features of CMA-ES variantsProceedings of the Genetic and Evolutionary Computation Conference10.1145/3449639.3459399(510-518)Online publication date: 26-Jun-2021
  • (2020)Landscape-aware fixed-budget performance regression and algorithm selection for modular CMA-ES variantsProceedings of the 2020 Genetic and Evolutionary Computation Conference10.1145/3377930.3390183(841-849)Online publication date: 25-Jun-2020
  • (2020)Integrated vs. sequential approaches for selecting and tuning CMA-ES variantsProceedings of the 2020 Genetic and Evolutionary Computation Conference10.1145/3377930.3389831(903-912)Online publication date: 25-Jun-2020
  • Show More Cited By

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media