skip to main content
10.1145/3594805.3607136acmconferencesArticle/Chapter ViewAbstractPublication PagesfogaConference Proceedingsconference-collections
research-article

Neural Networks as Black-Box Benchmark Functions Optimized for Exploratory Landscape Features

Published: 30 August 2023 Publication History

Abstract

Artificial benchmark functions are commonly used in optimization research because of their ability to rapidly evaluate potential solutions, making them a preferred substitute for real-world problems. However, these benchmark functions have faced criticism for their limited resemblance to real-world problems. In response, recent research has focused on automatically generating new benchmark functions for areas where established test suites are inadequate. These approaches have limitations, such as the difficulty of generating new benchmark functions that exhibit exploratory landscape analysis (ELA) features beyond those of existing benchmarks.
The objective of this work is to develop a method for generating benchmark functions for single-objective continuous optimization with user-specified structural properties. Specifically, we aim to demonstrate a proof of concept for a method that uses an ELA feature vector to specify these properties in advance. To achieve this, we begin by generating a random sample of decision space variables and objective values. We then adjust the objective values using CMA-ES until the corresponding features of our new problem match the predefined ELA features within a specified threshold. By iteratively transforming the landscape in this way, we ensure that the resulting function exhibits the desired properties. To create the final function, we use the resulting point cloud as training data for a simple neural network that produces a function exhibiting the target ELA features. We demonstrate the effectiveness of this approach by replicating the existing functions of the well-known BBOB suite and creating new functions with ELA feature values that are not present in BBOB.

References

[1]
R. L. Anderson. 1953. Recent Advances in Finding Best Operating Conditions. J. Amer. Statist. Assoc. 48, 264 (1953), 789--798. https://doi.org/10.1080/01621459.1953.10501200
[2]
Janez Demšar. 2006. Statistical comparisons of classifiers over multiple data sets. The Journal of Machine learning research 7 (2006), 1--30.
[3]
Konstantin Dietrich and Olaf Mersmann. 2022. Increasing the Diversity of Benchmark Function Sets Through Affine Recombination. In Parallel Problem Solving from Nature -- PPSN XVII: 17th International Conference, PPSN 2022, Dortmund, Germany, September 10-14, 2022, Proceedings, Part I (Dortmund, Germany). Springer-Verlag, Berlin, Heidelberg, 590--602. https://doi.org/10.1007/978-3-031-14714-2_41
[4]
M. Gallagher and Bo Yuan. 2006. A general-purpose tunable landscape generator. IEEE Transactions on Evolutionary Computation 10 (2006), 590--603. Issue 5. https://doi.org/10.1109/TEVC.2005.863
[5]
L. Grinsztajn, E. Oyallon, and G. Varoquaux. 2022. Why Do Tree-Based Models Still Outperform Deep Learning on Typical Tabular Data?. In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track.
[6]
Nikolaus Hansen, Steffen Finck, Raymond Ros, and Anne Auger. 2009. Real-Parameter Black-Box Optimization Benchmarking 2009: Noiseless Functions Definitions. Research Report RR-6829. INRIA. https://hal.inria.fr/inria-00362633
[7]
Nikolaus Hansen, Sibylle D. Müller, and Petros Koumoutsakos. 2003. Reducing the Time Complexity of the Derandomized Evolution Strategy with Covariance Matrix Adaptation (CMA-ES). Evol. Comput. 11, 1 (mar 2003), 1--18. https://doi.org/10.1162/106365603321828970
[8]
N. Hansen and A. Ostermeier. 2001. Completely Derandomized Self-Adaptation in Evolution Strategies. Evolutionary Computation 9 (6 2001), 159--195. Issue 2. https://doi.org/10.1162/106365601750190398
[9]
Steffen Herbold. 2020. Autorank: A Python package for automated ranking of classifiers. Journal of Open Source Software 5, 48 (2020), 2173. https://doi.org/10.21105/joss.02173
[10]
Kurt Hornik, Maxwell Stinchcombe, and Halbert White. 1989. Multilayer Feedforward Networks Are Universal Approximators. Neural Networks 2, 5 (1989), 359--366.
[11]
Anja Janković and Carola Doerr. 2019. Adaptive Landscape Analysis. In Proceedings of the Genetic and Evolutionary Computation Conference Companion (Prague, Czech Republic) (GECCO '19). Association for Computing Machinery, New York, NY, USA, 2032--2035. https://doi.org/10.1145/3319619.3326905
[12]
Terry Jones and Stephanie Forrest. 1995. Fitness Distance Correlation as a Measure of Problem Difficulty for Genetic Algorithms. In Proceedings of the 6th International Conference on Genetic Algorithms. Morgan Kaufmann Publishers Inc., San Francisco, CA, USA, 184--192.
[13]
J. Kennedy and R. Eberhart. 1995. Particle swarm optimization. 4 (1995), 1942--1948 vol.4. https://doi.org/10.1109/ICNN.1995.488968
[14]
Pascal Kerschke, Holger H. Hoos, Frank Neumann, and Heike Trautmann. 2019. Automated Algorithm Selection: Survey and Perspectives. Evolutionary Computation 27, 1 (2019), 3--45. https://doi.org/10.1162/evco_a_00242
[15]
Pascal Kerschke, Mike Preuss, Simon Wessing, and Heike Trautmann. 2015. Detecting Funnel Structures by Means of Exploratory Landscape Analysis. In Proceedings of the 2015 Annual Conference on Genetic and Evolutionary Computation (Madrid, Spain) (GECCO '15). Association for Computing Machinery, New York, NY, USA, 265--272. https://doi.org/10.1145/2739480.2754642
[16]
Pascal Kerschke, Mike Preuss, Simon Wessing, and Heike Trautmann. 2016. Low-Budget Exploratory Landscape Analysis on Multiple Peaks Models. In Proceedings of the Genetic and Evolutionary Computation Conference 2016 (Denver, Colorado, USA) (GECCO '16). Association for Computing Machinery, New York, NY, USA, 229--236. https://doi.org/10.1145/2908812.2908845
[17]
Pascal Kerschke and Heike Trautmann. 2019. Automated Algorithm Selection on Continuous Black-Box Problems By Combining Exploratory Landscape Analysis and Machine Learning. Evolutionary Computation 27, 1 (2019), 99--127. https://doi.org/10.1162/evco_a_00236
[18]
Ana Kostovska, Diederick Vermetten, Sašo Džeroski, Carola Doerr, Peter Korosec, and Tome Eftimov. 2022. The Importance of Landscape Features for Performance Prediction of Modular CMA-ES Variants. In Proceedings of the Genetic and Evolutionary Computation Conference (GECCO) (Boston, MA, USA), Jonathan E. Fieldsend and Markus Wagner (Eds.). ACM, 648--656. https://doi.org/10.1145/3512290.3528832
[19]
Ryan Dieter Lang and Andries Petrus Engelbrecht. 2021. An Exploratory Landscape Analysis-Based Benchmark Suite. Algorithms 14, 3 (2021), 78.
[20]
Pedro Larrañaga and Jose A Lazano. 2002. Estimation of Distribution Algorithms. Vol. 2. Springer Science & Buisness Media. https://doi.org/10.1007/978-1-4615-1539-5
[21]
Fu Xing Long, Bas van Stein, Moritz Frenzel, Peter Krause, Markus Gitterle, and Thomas Bäck. 2022. Learning the Characteristics of Engineering Optimization Problems with Applications in Automotive Crash (GECCO '22). Association for Computing Machinery, New York, NY, USA. https://doi.org/10.1145/3512290.3528712
[22]
Ilya Loshchilov. 2014. A Computationally Efficient Limited Memory CMA-ES for Large Scale Optimization. In Proceedings of the 2014 Annual Conference on Genetic and Evolutionary Computation (Vancouver, BC, Canada) (GECCO '14). Association for Computing Machinery, New York, NY, USA, 397--404. https://doi.org/10.1145/2576768.2598294
[23]
Katherine Mary Malan and Andries Petrus Engelbrecht. 2009. Quantifying Ruggedness of Continuous Landscapes Using Entropy. In Proceedings of the IEEE Congress on Evolutionary Computation (CEC). IEEE, 1440--1447. https://doi.org/10.1109/CEC.2009.4983112
[24]
Olaf Mersmann, Bernd Bischl, Heike Trautmann, Mike Preuss, Claus Weihs, and Günter Rudolph. 2011. Exploratory Landscape Analysis. In Proceedings of the 13th Annual Conference on Genetic and Evolutionary Computation (Dublin, Ireland) (GECCO '11). Association for Computing Machinery, New York, NY, USA, 829--836. https://doi.org/10.1145/2001576.2001690
[25]
Mario Andrés Muñoz and Kate Smith-Miles. 2020. Generating New Space-Filling Test Instances for Continuous Black-Box Optimization. Evolutionary Computation 28 (2020), 379--404. Issue 3. https://doi.org/10.1162/EVCO_A_00262
[26]
J. A. Nelder and R. Mead. 1965. A Simplex Method for Function Minimization. Comput. J. 7 (1 1965), 308--313. Issue 4. https://doi.org/10.1093/COMJNL/7.4.308
[27]
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. 2019. PyTorch: An Imperative Style, High-Performance Deep Learning Library. In Advances in Neural Information Processing Systems 32, H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché Buc, E. Fox, and R. Garnett (Eds.). Curran Associates, Inc., 8024--8035. http://papers.neurips.cc/paper/9015-pytorch-an-imperative-style-high-performance-deep-learning-library.pdf
[28]
M. J. D. Powell. 1994. A Direct Search Optimization Method That Models the Objective and Constraint Functions by Linear Interpolation. Springer Netherlands, Dordrecht, 51--67. https://doi.org/10.1007/978-94-015-8330-5_4
[29]
Raphael Patrick Prager, Moritz Vinzent Seiler, Heike Trautmann, and Pascal Kerschke. 2022. Automated Algorithm Selection in Single-Objective Continuous Optimization: A Comparative Study of Deep Learning and Landscape Analysis Methods. In Proceedings of the 17th International Conference on Parallel Problem Solving from Nature (PPSN XVII) (Dortmund, Germany), Günter Rudolph, Anna V. Kononova, Hernán E. Aguirre, Pascal Kerschke, Gabriela Ochoa, and Tea Tusar (Eds.). Springer, 3--17. https://doi.org/10.1007/978-3-031-14714-2_1
[30]
Raphael Patrick Prager and Heike Trautmann. 2023. Nullifying the Inherent Bias of Non-Invariant Exploratory Landscape Analysis Features. In Applications of Evolutionary Computation, Joao Correia, Stephen Smith, and Raneem Qaddoura (Eds.). Springer International Publishing, Cham.
[31]
J. Rapin and O. Teytaud. 2018. Nevergrad - A Gradient-Free Optimization Platform. https://GitHub.com/FacebookResearch/Nevergrad.
[32]
Quentin Renau, Carola Doerr, Johann Dreo, and Benjamin Doerr. 2020. Exploratory Landscape Analysis is Strongly Sensitive to the Sampling Strategy. Proceedings of the 16th International Conference on Parallel Problem Solving from Nature (PPSN 2020) 12270 LNCS (2020), 139--153. https://doi.org/10.1007/978-3-030-58115-2_10
[33]
Quentin Renau, Johann Dreo, Carola Doerr, and Benjamin Doerr. 2021. Towards Explainable Exploratory Landscape Analysis: Extreme Feature Selection for Classifying BBOB Functions. Proceedings of the 24th International Conference, EvoApplications2021 12694 LNCS (2021), 17--33. https://doi.org/10.48550/arxiv.2102.00736
[34]
Lennart Schneider, Lennart Schäpermeier, Raphael Patrick Prager, Bernd Bischl, Heike Trautmann, and Pascal Kerschke. 2022. HPO × ELA: Investigating Hyperparameter Optimization Landscapes by Means of Exploratory Landscape Analysis. In Parallel Problem Solving from Nature -- PPSN XVII, Günter Rudolph, Anna V. Kononova, Hernán Aguirre, Pascal Kerschke, Gabriela Ochoa, and Tea Tušar (Eds.). Springer International Publishing, Cham, 575--589.
[35]
Moritz Vinzent Seiler, Raphael Patrick Prager, Pascal Kerschke, and Heike Trautmann. 2022. A Collection of Deep Learning-Based Feature-Free Approaches for Characterizing Single-Objective Continuous Fitness Landscapes. In Proceedings of the Genetic and Evolutionary Computation Conference (Boston, Massachusetts) (GECCO '22). Association for Computing Machinery, New York, NY, USA, 657--665. https://doi.org/10.1145/3512290.3528834
[36]
Rainer Storn and Kenneth Price. 1997. Differential Evolution --A Simple and Efficient Heuristic for global Optimization over Continuous Spaces. Journal of Global Optimization 11, 4 (1997), 341--359. https://doi.org/10.1023/A:1008202821328
[37]
Ke Tang, Xin Yáo, Ponnuthurai Nagaratnam Suganthan, Cara MacNish, Ying-Ping Chen, Chih-Ming Chen, and Zhenyu Yang. 2007. Benchmark Functions for the CEC'2008 Special Session and Competition on Large Scale Global Optimization. Nature inspired computation and applications laboratory USTC, China 24 (2007), 1--18.
[38]
Konstantinos Varelas. 2019. Benchmarking Large Scale Variants of CMA-ES and L-BFGS-B on the Bbob-Large scale Testbed. In Proceedings of the Genetic andEvolutionary Computation Conference Companion (Prague, Czech Republic) (GECCO '19). Association for Computing Machinery, New York, NY, USA, 1937--1945. https://doi.org/10.1145/3319619.3326893

Index Terms

  1. Neural Networks as Black-Box Benchmark Functions Optimized for Exploratory Landscape Features

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image ACM Conferences
    FOGA '23: Proceedings of the 17th ACM/SIGEVO Conference on Foundations of Genetic Algorithms
    August 2023
    169 pages
    ISBN:9798400702020
    DOI:10.1145/3594805
    Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

    Sponsors

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 30 August 2023

    Permissions

    Request permissions for this article.

    Check for updates

    Author Tags

    1. Benchmarking
    2. Black-Box Continuous Optimization
    3. Exploratory Landscape Analysis
    4. Instance Generator
    5. Neural Networks

    Qualifiers

    • Research-article
    • Research
    • Refereed limited

    Conference

    FOGA '23
    Sponsor:
    FOGA '23: Foundations of Genetic Algorithms XVII
    August 30 - September 1, 2023
    Potsdam, Germany

    Acceptance Rates

    Overall Acceptance Rate 72 of 131 submissions, 55%

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • 0
      Total Citations
    • 172
      Total Downloads
    • Downloads (Last 12 months)86
    • Downloads (Last 6 weeks)8
    Reflects downloads up to 27 Jan 2025

    Other Metrics

    Citations

    View Options

    Login options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Figures

    Tables

    Media

    Share

    Share

    Share this Publication link

    Share on social media