Raphael Patrick Prager
ABSTRACT
Artificial benchmark functions are commonly used in optimization research because of their ability to rapidly evaluate potential solutions, making them a preferred substitute for real-world problems. However, these benchmark functions have faced criticism for their limited resemblance to real-world problems. In response, recent research has focused on automatically generating new benchmark functions for areas where established test suites are inadequate. These approaches have limitations, such as the difficulty of generating new benchmark functions that exhibit exploratory landscape analysis (ELA) features beyond those of existing benchmarks.
The objective of this work is to develop a method for generating benchmark functions for single-objective continuous optimization with user-specified structural properties. Specifically, we aim to demonstrate a proof of concept for a method that uses an ELA feature vector to specify these properties in advance. To achieve this, we begin by generating a random sample of decision space variables and objective values. We then adjust the objective values using CMA-ES until the corresponding features of our new problem match the predefined ELA features within a specified threshold. By iteratively transforming the landscape in this way, we ensure that the resulting function exhibits the desired properties. To create the final function, we use the resulting point cloud as training data for a simple neural network that produces a function exhibiting the target ELA features. We demonstrate the effectiveness of this approach by replicating the existing functions of the well-known BBOB suite and creating new functions with ELA feature values that are not present in BBOB.
- R. L. Anderson. 1953. Recent Advances in Finding Best Operating Conditions. J. Amer. Statist. Assoc. 48, 264 (1953), 789--798. https://doi.org/10.1080/01621459.1953.10501200Google Scholar
Cross Ref
- Janez Demšar. 2006. Statistical comparisons of classifiers over multiple data sets. The Journal of Machine learning research 7 (2006), 1--30.Google ScholarDigital Library
- Konstantin Dietrich and Olaf Mersmann. 2022. Increasing the Diversity of Benchmark Function Sets Through Affine Recombination. In Parallel Problem Solving from Nature -- PPSN XVII: 17th International Conference, PPSN 2022, Dortmund, Germany, September 10-14, 2022, Proceedings, Part I (Dortmund, Germany). Springer-Verlag, Berlin, Heidelberg, 590--602. https://doi.org/10.1007/978-3-031-14714-2_41Google ScholarDigital Library
- M. Gallagher and Bo Yuan. 2006. A general-purpose tunable landscape generator. IEEE Transactions on Evolutionary Computation 10 (2006), 590--603. Issue 5. https://doi.org/10.1109/TEVC.2005.863Google ScholarCross Ref
- L. Grinsztajn, E. Oyallon, and G. Varoquaux. 2022. Why Do Tree-Based Models Still Outperform Deep Learning on Typical Tabular Data?. In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track.Google Scholar
- Nikolaus Hansen, Steffen Finck, Raymond Ros, and Anne Auger. 2009. Real-Parameter Black-Box Optimization Benchmarking 2009: Noiseless Functions Definitions. Research Report RR-6829. INRIA. https://hal.inria.fr/inria-00362633Google Scholar
- Nikolaus Hansen, Sibylle D. Müller, and Petros Koumoutsakos. 2003. Reducing the Time Complexity of the Derandomized Evolution Strategy with Covariance Matrix Adaptation (CMA-ES). Evol. Comput. 11, 1 (mar 2003), 1--18. https://doi.org/10.1162/106365603321828970Google ScholarDigital Library
- N. Hansen and A. Ostermeier. 2001. Completely Derandomized Self-Adaptation in Evolution Strategies. Evolutionary Computation 9 (6 2001), 159--195. Issue 2. https://doi.org/10.1162/106365601750190398Google ScholarDigital Library
- Steffen Herbold. 2020. Autorank: A Python package for automated ranking of classifiers. Journal of Open Source Software 5, 48 (2020), 2173. https://doi.org/10.21105/joss.02173Google ScholarCross Ref
- Kurt Hornik, Maxwell Stinchcombe, and Halbert White. 1989. Multilayer Feedforward Networks Are Universal Approximators. Neural Networks 2, 5 (1989), 359--366.Google ScholarCross Ref
- Anja Janković and Carola Doerr. 2019. Adaptive Landscape Analysis. In Proceedings of the Genetic and Evolutionary Computation Conference Companion (Prague, Czech Republic) (GECCO '19). Association for Computing Machinery, New York, NY, USA, 2032--2035. https://doi.org/10.1145/3319619.3326905Google ScholarDigital Library
- Terry Jones and Stephanie Forrest. 1995. Fitness Distance Correlation as a Measure of Problem Difficulty for Genetic Algorithms. In Proceedings of the 6th International Conference on Genetic Algorithms. Morgan Kaufmann Publishers Inc., San Francisco, CA, USA, 184--192.Google Scholar
- J. Kennedy and R. Eberhart. 1995. Particle swarm optimization. 4 (1995), 1942--1948 vol.4. https://doi.org/10.1109/ICNN.1995.488968Google Scholar
- Pascal Kerschke, Holger H. Hoos, Frank Neumann, and Heike Trautmann. 2019. Automated Algorithm Selection: Survey and Perspectives. Evolutionary Computation 27, 1 (2019), 3--45. https://doi.org/10.1162/evco_a_00242Google ScholarDigital Library
- Pascal Kerschke, Mike Preuss, Simon Wessing, and Heike Trautmann. 2015. Detecting Funnel Structures by Means of Exploratory Landscape Analysis. In Proceedings of the 2015 Annual Conference on Genetic and Evolutionary Computation (Madrid, Spain) (GECCO '15). Association for Computing Machinery, New York, NY, USA, 265--272. https://doi.org/10.1145/2739480.2754642Google ScholarDigital Library
- Pascal Kerschke, Mike Preuss, Simon Wessing, and Heike Trautmann. 2016. Low-Budget Exploratory Landscape Analysis on Multiple Peaks Models. In Proceedings of the Genetic and Evolutionary Computation Conference 2016 (Denver, Colorado, USA) (GECCO '16). Association for Computing Machinery, New York, NY, USA, 229--236. https://doi.org/10.1145/2908812.2908845Google ScholarDigital Library
- Pascal Kerschke and Heike Trautmann. 2019. Automated Algorithm Selection on Continuous Black-Box Problems By Combining Exploratory Landscape Analysis and Machine Learning. Evolutionary Computation 27, 1 (2019), 99--127. https://doi.org/10.1162/evco_a_00236Google ScholarDigital Library
- Ana Kostovska, Diederick Vermetten, Sašo Džeroski, Carola Doerr, Peter Korosec, and Tome Eftimov. 2022. The Importance of Landscape Features for Performance Prediction of Modular CMA-ES Variants. In Proceedings of the Genetic and Evolutionary Computation Conference (GECCO) (Boston, MA, USA), Jonathan E. Fieldsend and Markus Wagner (Eds.). ACM, 648--656. https://doi.org/10.1145/3512290.3528832Google ScholarDigital Library
- Ryan Dieter Lang and Andries Petrus Engelbrecht. 2021. An Exploratory Landscape Analysis-Based Benchmark Suite. Algorithms 14, 3 (2021), 78.Google ScholarCross Ref
- Pedro Larrañaga and Jose A Lazano. 2002. Estimation of Distribution Algorithms. Vol. 2. Springer Science & Buisness Media. https://doi.org/10.1007/978-1-4615-1539-5Google Scholar
- Fu Xing Long, Bas van Stein, Moritz Frenzel, Peter Krause, Markus Gitterle, and Thomas Bäck. 2022. Learning the Characteristics of Engineering Optimization Problems with Applications in Automotive Crash (GECCO '22). Association for Computing Machinery, New York, NY, USA. https://doi.org/10.1145/3512290.3528712Google ScholarDigital Library
- Ilya Loshchilov. 2014. A Computationally Efficient Limited Memory CMA-ES for Large Scale Optimization. In Proceedings of the 2014 Annual Conference on Genetic and Evolutionary Computation (Vancouver, BC, Canada) (GECCO '14). Association for Computing Machinery, New York, NY, USA, 397--404. https://doi.org/10.1145/2576768.2598294Google ScholarDigital Library
- Katherine Mary Malan and Andries Petrus Engelbrecht. 2009. Quantifying Ruggedness of Continuous Landscapes Using Entropy. In Proceedings of the IEEE Congress on Evolutionary Computation (CEC). IEEE, 1440--1447. https://doi.org/10.1109/CEC.2009.4983112Google ScholarCross Ref
- Olaf Mersmann, Bernd Bischl, Heike Trautmann, Mike Preuss, Claus Weihs, and Günter Rudolph. 2011. Exploratory Landscape Analysis. In Proceedings of the 13th Annual Conference on Genetic and Evolutionary Computation (Dublin, Ireland) (GECCO '11). Association for Computing Machinery, New York, NY, USA, 829--836. https://doi.org/10.1145/2001576.2001690Google ScholarDigital Library
- Mario Andrés Muñoz and Kate Smith-Miles. 2020. Generating New Space-Filling Test Instances for Continuous Black-Box Optimization. Evolutionary Computation 28 (2020), 379--404. Issue 3. https://doi.org/10.1162/EVCO_A_00262Google ScholarDigital Library
- J. A. Nelder and R. Mead. 1965. A Simplex Method for Function Minimization. Comput. J. 7 (1 1965), 308--313. Issue 4. https://doi.org/10.1093/COMJNL/7.4.308Google Scholar
- Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. 2019. PyTorch: An Imperative Style, High-Performance Deep Learning Library. In Advances in Neural Information Processing Systems 32, H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché Buc, E. Fox, and R. Garnett (Eds.). Curran Associates, Inc., 8024--8035. http://papers.neurips.cc/paper/9015-pytorch-an-imperative-style-high-performance-deep-learning-library.pdfGoogle ScholarDigital Library
- M. J. D. Powell. 1994. A Direct Search Optimization Method That Models the Objective and Constraint Functions by Linear Interpolation. Springer Netherlands, Dordrecht, 51--67. https://doi.org/10.1007/978-94-015-8330-5_4Google Scholar
- Raphael Patrick Prager, Moritz Vinzent Seiler, Heike Trautmann, and Pascal Kerschke. 2022. Automated Algorithm Selection in Single-Objective Continuous Optimization: A Comparative Study of Deep Learning and Landscape Analysis Methods. In Proceedings of the 17th International Conference on Parallel Problem Solving from Nature (PPSN XVII) (Dortmund, Germany), Günter Rudolph, Anna V. Kononova, Hernán E. Aguirre, Pascal Kerschke, Gabriela Ochoa, and Tea Tusar (Eds.). Springer, 3--17. https://doi.org/10.1007/978-3-031-14714-2_1Google ScholarDigital Library
- Raphael Patrick Prager and Heike Trautmann. 2023. Nullifying the Inherent Bias of Non-Invariant Exploratory Landscape Analysis Features. In Applications of Evolutionary Computation, Joao Correia, Stephen Smith, and Raneem Qaddoura (Eds.). Springer International Publishing, Cham.Google Scholar
- J. Rapin and O. Teytaud. 2018. Nevergrad - A Gradient-Free Optimization Platform. https://GitHub.com/FacebookResearch/Nevergrad.Google Scholar
- Quentin Renau, Carola Doerr, Johann Dreo, and Benjamin Doerr. 2020. Exploratory Landscape Analysis is Strongly Sensitive to the Sampling Strategy. Proceedings of the 16th International Conference on Parallel Problem Solving from Nature (PPSN 2020) 12270 LNCS (2020), 139--153. https://doi.org/10.1007/978-3-030-58115-2_10Google ScholarDigital Library
- Quentin Renau, Johann Dreo, Carola Doerr, and Benjamin Doerr. 2021. Towards Explainable Exploratory Landscape Analysis: Extreme Feature Selection for Classifying BBOB Functions. Proceedings of the 24th International Conference, EvoApplications2021 12694 LNCS (2021), 17--33. https://doi.org/10.48550/arxiv.2102.00736Google ScholarCross Ref
- Lennart Schneider, Lennart Schäpermeier, Raphael Patrick Prager, Bernd Bischl, Heike Trautmann, and Pascal Kerschke. 2022. HPO × ELA: Investigating Hyperparameter Optimization Landscapes by Means of Exploratory Landscape Analysis. In Parallel Problem Solving from Nature -- PPSN XVII, Günter Rudolph, Anna V. Kononova, Hernán Aguirre, Pascal Kerschke, Gabriela Ochoa, and Tea Tušar (Eds.). Springer International Publishing, Cham, 575--589.Google Scholar
- Moritz Vinzent Seiler, Raphael Patrick Prager, Pascal Kerschke, and Heike Trautmann. 2022. A Collection of Deep Learning-Based Feature-Free Approaches for Characterizing Single-Objective Continuous Fitness Landscapes. In Proceedings of the Genetic and Evolutionary Computation Conference (Boston, Massachusetts) (GECCO '22). Association for Computing Machinery, New York, NY, USA, 657--665. https://doi.org/10.1145/3512290.3528834Google ScholarDigital Library
- Rainer Storn and Kenneth Price. 1997. Differential Evolution --A Simple and Efficient Heuristic for global Optimization over Continuous Spaces. Journal of Global Optimization 11, 4 (1997), 341--359. https://doi.org/10.1023/A:1008202821328Google ScholarDigital Library
- Ke Tang, Xin Yáo, Ponnuthurai Nagaratnam Suganthan, Cara MacNish, Ying-Ping Chen, Chih-Ming Chen, and Zhenyu Yang. 2007. Benchmark Functions for the CEC'2008 Special Session and Competition on Large Scale Global Optimization. Nature inspired computation and applications laboratory USTC, China 24 (2007), 1--18.Google Scholar
- Konstantinos Varelas. 2019. Benchmarking Large Scale Variants of CMA-ES and L-BFGS-B on the Bbob-Large scale Testbed. In Proceedings of the Genetic andEvolutionary Computation Conference Companion (Prague, Czech Republic) (GECCO '19). Association for Computing Machinery, New York, NY, USA, 1937--1945. https://doi.org/10.1145/3319619.3326893Google ScholarDigital Library
Index Terms
- Neural Networks as Black-Box Benchmark Functions Optimized for Exploratory Landscape Features
Recommendations
Exploratory landscape analysis
GECCO '11: Proceedings of the 13th annual conference on Genetic and evolutionary computationExploratory Landscape Analysis subsumes a number of techniques employed to obtain knowledge about the properties of an unknown optimization problem, especially insofar as these properties are important for the performance of optimization algorithms. ...
Increasing the Diversity of Benchmark Function Sets Through Affine Recombination
Parallel Problem Solving from Nature – PPSN XVIIBlack-box optimization benchmarking of NEWUOA compared to BIPOP-CMA-ES: on the BBOB noiseless testbed
GECCO '10: Proceedings of the 12th annual conference companion on Genetic and evolutionary computationIn this paper, the performances of the NEW Unconstrained Optimization Algorithm (NEWUOA) on some noiseless functions are compared to those of the BI-POPulation Covariance Matrix Adaptation-Evolution Strategy (BIPOP-CMA-ES). The two algorithms were ...
Comments