skip to main content
10.1145/3319619.3326830acmconferencesArticle/Chapter ViewAbstractPublication PagesgeccoConference Proceedingsconference-collections
research-article

Exploring the MLDA benchmark on the nevergrad platform

Published: 13 July 2019 Publication History

Abstract

This work presents the integration of the recently released benchmark suite MLDA into Nevergrad, a likewise recently released platform for derivative-free optimization. Benchmarking evolutionary and other optimization methods on this collection enables us to learn how algorithms deal with problems that are often treated by means of standard methods like clustering or gradient descent. As available computation power nowadays allows for running much 'slower' methods without noticing a performance difference it is an open question which of these standard methods may be replaced by derivative-free and (in terms of quality) better performing optimization algorithms. Additionally, most MLDA problems are suitable for landscape analysis and other means of understanding problem difficulty or algorithm behavior, due to their tangible nature.
We present the open-source reimplementation of MLDA inside the Nevergrad platform and further discuss some first findings, which result from exploratory experiments with this platform. These include superior performance of advanced quasi-random sequences in some highly multimodal cases (even in non-parallel optimization), great performance of CMA for the perceptron and the Sammon tasks, success of DE on clustering problems, and straight forward implementations of highly competitive algorithm selection models by means of competence maps.

References

[1]
Ossama Abdel-Hamid, Abdel-rahman Mohamed, Hui Jiang, Li Deng, Gerald Penn, and Dong Yu. 2014. Convolutional Neural Networks for Speech Recognition. IEEE/ACM Transactions on Audio, Speech & Language Processing 22, 10 (2014), 1533--1545. https://ieeexplore.ieee.org/document/6857341
[2]
Daniel Aloise, Amit Deshpande, Pierre Hansen, and Preyas Popat. 2009. NP-hardness of Euclidean sum-of-squares clustering. Machine learning 75, 2 (2009), 245--248.
[3]
SME Artelys. 2015. (2015). https://www.artelys.com/news/159/16/KNITRO-wins-the-GECCO-2015-Black-Box-Optimization-Competition
[4]
Adil M Bagirov. 2008. Modified global k-means algorithm for minimum sum-of-squares clustering problems. Pattern Recognition 41, 10 (2008), 3192--3199.
[5]
Nicolas Baskiotis and Michèle Sebag. 2004. C4.5 competence map: a phase transition-inspired approach. In ICML (ACM International Conference Proceeding Series), Vol. 69. ACM.
[6]
Nacim Belkhir, Johann Dréo, Pierre Savéant, and Marc Schoenauer. 2017. Per Instance Algorithm Configuration of CMA-ES with Limited Budget. In Proceedings of the Genetic and Evolutionary Computation Conference (GECCO '17). ACM, New York, NY, USA, 681--688.
[7]
Hans-Georg Beyer. 2001. The Theory of Evolution Strategies. Springer, Heideberg.
[8]
Bernd Bischl, Pascal Kerschke, Lars Kotthoff, Thomas Marius Lindauer, Yuri Malitsky, Alexandre Fréchette, Holger H. Hoos, Frank Hutter, Kevin Leyton-Brown, Kevin Tierney, and Joaquin Vanschoren. 2016. ASlib: A Benchmark Library for Algorithm Selection. Artificial Intelligence (AIJ) 237 (2016), 41--58.
[9]
C. M. Bishop. 1995. Neural Networks for Pattern Recognition. Oxford University Press, Oxford.
[10]
Olivier Bousquet, Sylvain Gelly, Kurach Karol, Olivier Teytaud, and Damien Vincent. 2017. Critical Hyper-Parameters: No Random, No Cry. (2017). Preprint https://arxiv.org/pdf/1706.03200.pdf.
[11]
Chenyi Chen, Ari Seff, Alain Kornhauser, and Jianxiong Xiao. 2015. DeepDriving: Learning Affordance for Direct Perception in Autonomous Driving. In Proceedings of the International Conference on Computer Vision. IEEE, 2722--2730. https://ieeexplore.ieee.org/document/7410669
[12]
Carola Doerr, Johann Dreo, and Pascal Kerschke. 2019. Making a Case for (Hyper-)Parameter Tuning as Benchmark Problems. In Proceedings of the 21st Annual Conference on Genetic and Evolutionary Computation (GECCO'19) Companion. ACM. to appear.
[13]
Katharina Eggensperger, Marius Lindauer, Holger H. Hoos, Frank Hutter, and Kevin Leyton-Brown. 2018. Efficient benchmarking of algorithm configurators via model-based surrogates. Machine Learning 107, 1 (2018), 15--41.
[14]
Marcus Gallagher. 2016. Towards improved benchmarking of black-box optimization algorithms using clustering problems. Soft Computing 20, 10 (2016), 3835--3849.
[15]
Ian Goodfellow, Yoshua Bengio, and Aaron Courville. 2016. Deep Learning. MIT Press. http://www.deeplearningbook.org.
[16]
J.H. Halton. 1960. On the efficiency of certain quasi-random sequences of points in evaluating multi-dimensional integrals. Numer. Math. 2 (1960), 84--90.
[17]
J.M. Hammersley. 1960. Monte-Carlo Methods For Solving Multivariate Problems. Annals of the New York Academy of Sciences 86, 3 (1960), 844--874.
[18]
Nikolaus Hansen. 2006. The CMA Evolution Strategy: A Comparing Review. Towards a New Evolutionary Computation - Advances in the Estimation of Distribution Algorithms 192 (2006), 75 -- 102.
[19]
Nikolaus Hansen, Anne Auger, Raymond Ros, Steffen Finck, and Petr Pošík. 2010. Comparing results of 31 algorithms from the black-box optimization benchmarking BBOB-2009. In Proceedings of the 12th annual conference companion on Genetic and evolutionary computation. ACM, 1689--1696.
[20]
Pierre Hansen and Brigitte Jaumard. 1997. Cluster analysis and mathematical programming. Mathematical programming 79, 1--3 (1997), 191--215.
[21]
Trevor Hastie, Robert Tibshirani, and Jerome Friedman. 2009. The Elements of Statistical Learning: Data Mining, Inference, and Prediction (2nd ed.). Springer. http://www.springer.com/de/book/9780387848570
[22]
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep Residual Learning for Image Recognition. IEEE Conference on Computer Vision and Pattern Recognition (2016), 770--778. https://ieeexplore.ieee.org/document/7780459
[23]
Holger H. Hoos, Tomás Peitl, Friedrich Slivovsky, and Stefan Szeider. 2018. Portfolio-Based Algorithm Selection for Circuit QBFs. In CP (Lecture Notes in Computer Science), Vol. 11008. Springer, 195--209.
[24]
Eduardo Raul Hruschka, Ricardo JGB Campello, Alex A Freitas, et al. 2009. A survey of evolutionary algorithms for clustering. IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews) 39, 2 (2009), 133--155.
[25]
Alan Julian Izenman. 2008. Modern multivariate statistical techniques. Regression, classification and manifold learning (2008).
[26]
Anil K Jain, M Narasimha Murty, and Patrick J Flynn. 1999. Data clustering: a review. ACM computing surveys (CSUR) 31, 3 (1999), 264--323.
[27]
Leon Kaufman, Marc Vanden Eede, and Pierre Hansen. 1977. A Plant and Warehouse Location Problem. Journal of the Operational Research Society 28, 3 (01 Oct 1977), 547--554.
[28]
James Kennedy and Russell C. Eberhart. 1995. Particle swarm optimization. In Proceedings of the IEEE International Conference on Neural Networks. 1942--1948.
[29]
Pascal Kerschke, Holger H. Hoos, Frank Neumann, and Heike Trautmann. 2018. Automated Algorithm Selection: Survey and Perspectives. Evolutionary Computation (ECJ) (2018), 1--47.
[30]
Pascal Kerschke, Lars Kotthoff, Jakob Bossek, Holger H. Hoos, and Heike Trautmann. 2018. Leveraging TSP Solver Complementarity through Machine Learning. Evolutionary Computation 26, 4 (2018).
[31]
Pascal Kerschke and Heike Trautmann. 2018. Automated Algorithm Selection on Continuous Black-Box Problems By Combining Exploratory Landscape Analysis and Machine Learning. Evolutionary Computation (ECJ) (2018), 1--28.
[32]
Lars Kotthoff. 2014. Algorithm Selection for Combinatorial Search Problems: A Survey. AI Magazine 35, 3 (2014), 48--60.
[33]
Marius Lindauer, Holger H. Hoos, Kevin Leyton-Brown, and Torsten Schaub. 2017. Automatic construction of parallel portfolios via algorithm configuration. Artif. Intell. 244 (2017), 272--290.
[34]
Katherine Mary Malan and Andries Petrus Engelbrecht. 2013. A Survey of Techniques for Characterising Fitness Landscapes and Some Possible Ways Forward. Information Sciences (JIS) 241 (2013), 148--163.
[35]
L. Álvarez Menéndez, Francisco Javier de Cos Juez, Fernando Sánchez Lasheras, and J. A. Álvarez Riesgo. 2010. Artificial Neural Networks Applied to Cancer Detection in a Breast Screening Programme. Mathematical and Computer Modelling 52, 7--8 (2010), 983--991. https://www.sciencedirect.com/science/article/pii/S0895717710001378
[36]
Olaf Mersmann, Bernd Bischl, Heike Trautmann, Mike Preuss, Claus Weihs, and Günter Rudolph. 2011. Exploratory Landscape Analysis. In Proceedings of the 13th Annual Conference on Genetic and Evolutionary Computation (GECCO). ACM, 829--836.
[37]
Mario Andrés Muñoz Acosta, Yuan Sun, Michael Kirley, and Saman K. Halgamuge. 2015. Algorithm Selection for Black-Box Continuous Optimization Problems: A Survey on Methods and Challenges. Information Sciences (JIS) 317 (October 2015), 224--245.
[38]
Art B. Owen. 2006. Halton Sequences Avoid the Origin. SIAM Rev. 48 (2006), 487--503.
[39]
Erik Pitzer and Michael Affenzeller. 2012. A Comprehensive Survey on Fitness Landscape Analysis. In Recent Advances in Intelligent Engineering Systems, János Fodor, Ryszard Klempous, and Carmen Paz Suárez Araujo (Eds.). Springer, 161--191.
[40]
M.J.D. Powell. 1964. An efficient method for finding the minimum of a function of several variables without calculating derivatives. Comput. J. 7, 2 (1964), 155--162.
[41]
M. J. D. Powell. 1994. A Direct Search Optimization Method That Models the Objective and Constraint Functions by Linear Interpolation. Springer Netherlands, Dordrecht, 51--67.
[42]
Jérémy Rapin and Olivier Teytaud. 2019. Nevergrad. https://github.com/facebookresearch/nevergrad. (2019). Accessed: 2019-01-15, v0.1.4.
[43]
John Rischard Rice. 1976. The Algorithm Selection Problem. Advances in Computers 15 (1976), 65--118.
[44]
Brian D. Ripley. 1996. Pattern Recognition and Neural Networks. Cambridge University Press, Cambridge, UK.
[45]
Mattia Rizzini, Chris Fawcett, Mauro Vallati, Alfonso Emilio Gerevini, and Holger H. Hoos. 2017. Static and Dynamic Portfolio Methods for Optimal Planning: An Empirical Analysis. International Journal on Artificial Intelligence Tools 26, 1 (2017), 1--27.
[46]
Kate Amanda Smith-Miles. 2009. Cross-Disciplinary Perspectives on Meta-Learning for Algorithm Selection. ACM Computing Surveys (CSUR) 41 (January 2009). 1--25.
[47]
Rainer Storn and Kenneth Price. 1997. Differential Evolution: A Simple and Efficient Heuristic for Global Optimization over Continuous Spaces. J. of Global Optimization 11, 4 (Dec. 1997), 341--359.
[48]
Ponnuthurai N Suganthan, Nikolaus Hansen, Jing J Liang, Kalyanmoy Deb, Ying-Ping Chen, Anne Auger, and Santosh Tiwari. 2005. Problem definitions and evaluation criteria for the CEC 2005 special session on real-parameter optimization. KanGAL report 2005005 (2005), 2005.
[49]
Bruno Tuffin. 1998. A new permutation choice in Halton sequences. Springer New York, New York, NY, 427--435.
[50]
Vanessa Volz, Boris Naujoks, Pascal Kerschke, and Tea Tušar. 2019. Single- and Multi-Objective Game-Benchmark for Evolutionary Algorithms. In Proceedings of the 21st Annual Conference on Genetic and Evolutionary Computation (GECCO'19). ACM. to appear.
[51]
David Hilton Wolpert and William G. Macready. 1997. No Free Lunch Theorems for Optimization. IEEE Transactions on Evolutionary Computation (TEVC) 1, 1 (April 1997), 67--82.
[52]
Rui Xu and Don Wunsch. 2008. Clustering. Vol. 10. John Wiley & Sons.

Cited By

View all
  • (2025)Optimizing With Low Budgets: A Comparison on the Black-Box Optimization Benchmarking Suite and OpenAI GymIEEE Transactions on Evolutionary Computation10.1109/TEVC.2023.334678829:1(91-101)Online publication date: Feb-2025
  • (2022)Black-Box Optimization Revisited: Improving Algorithm Selection Wizards Through Massive BenchmarkingIEEE Transactions on Evolutionary Computation10.1109/TEVC.2021.310818526:3(490-500)Online publication date: Jun-2022
  • (2021)Ray Tracing-based Light Energy Prediction for Indoor Batteryless SensorsProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/34480865:1(1-27)Online publication date: 30-Mar-2021
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
GECCO '19: Proceedings of the Genetic and Evolutionary Computation Conference Companion
July 2019
2161 pages
ISBN:9781450367486
DOI:10.1145/3319619
© 2019 Association for Computing Machinery. ACM acknowledges that this contribution was authored or co-authored by an employee, contractor or affiliate of a national government. As such, the Government retains a nonexclusive, royalty-free right to publish or reproduce this article, or to allow others to do so, for Government purposes only.

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 13 July 2019

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. benchmarking
  2. machine learning
  3. open source
  4. optimization

Qualifiers

  • Research-article

Conference

GECCO '19
Sponsor:
GECCO '19: Genetic and Evolutionary Computation Conference
July 13 - 17, 2019
Prague, Czech Republic

Acceptance Rates

Overall Acceptance Rate 1,669 of 4,410 submissions, 38%

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)12
  • Downloads (Last 6 weeks)1
Reflects downloads up to 17 Feb 2025

Other Metrics

Citations

Cited By

View all
  • (2025)Optimizing With Low Budgets: A Comparison on the Black-Box Optimization Benchmarking Suite and OpenAI GymIEEE Transactions on Evolutionary Computation10.1109/TEVC.2023.334678829:1(91-101)Online publication date: Feb-2025
  • (2022)Black-Box Optimization Revisited: Improving Algorithm Selection Wizards Through Massive BenchmarkingIEEE Transactions on Evolutionary Computation10.1109/TEVC.2021.310818526:3(490-500)Online publication date: Jun-2022
  • (2021)Ray Tracing-based Light Energy Prediction for Indoor Batteryless SensorsProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/34480865:1(1-27)Online publication date: 30-Mar-2021
  • (2020)Evolving Sampling Strategies for One-Shot Optimization TasksParallel Problem Solving from Nature – PPSN XVI10.1007/978-3-030-58112-1_8(111-124)Online publication date: 31-Aug-2020
  • (2019)Single- and multi-objective game-benchmark for evolutionary algorithmsProceedings of the Genetic and Evolutionary Computation Conference10.1145/3321707.3321805(647-655)Online publication date: 13-Jul-2019
  • (2019)Making a case for (Hyper-)parameter tuning as benchmark problemsProceedings of the Genetic and Evolutionary Computation Conference Companion10.1145/3319619.3326857(1755-1764)Online publication date: 13-Jul-2019
  • (2019)Exploratory landscape analysisProceedings of the Genetic and Evolutionary Computation Conference Companion10.1145/3319619.3323389(1137-1155)Online publication date: 13-Jul-2019

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media