research-article

Configuring irace using surrogate configuration benchmarks

Authors:

Leslie Pérez Cáceres,

Patrick De Causmaecker,

Thomas StützleAuthors Info & Claims

GECCO '17: Proceedings of the Genetic and Evolutionary Computation Conference

Pages 243 - 250

https://doi.org/10.1145/3071178.3071238

Published: 01 July 2017 Publication History

Abstract

Over the recent years, several tools for the automated configuration of parameterized algorithms have been developed. These tools, also called configurators, have themselves parameters that influence their search behavior and make them malleable to different kinds of configuration tasks. The default values of these parameters are set manually based on the experience of the configurator's developers. Studying the impact of these parameters or configuring them is very expensive as it would require many executions of these tools on configuration tasks, each taking often many hours or days of computation. In this work, we tackle this problem using a meta-tuning process, based on the use of surrogate benchmarks that are much faster to evaluate. This paper studies the feasibility of this process using the popular irace configurator as the method to be meta-configured. We first study the consistency between the real and surrogate benchmarks using three measures: the prediction accuracy of the surrogate models, the homogeneity of the benchmarks and the list of important algorithm parameters. Afterwards, we use irace to configure irace on those surrogates. Experimental results indicate the feasibility of this process and a clear potential improvement of irace over its default configuration.

References

[1]

C. Ansótegui, Y. Malitsky, H. Samulowitz, M. Sellmann, and K. Tierney. Model-based genetic algorithms for algorithm configuration. IJCAI-15, pp. 733--739. IJCAI/AAAI Press, Menlo Park, CA (2015).

Digital Library

[2]

C. Ansótegui, M. Sellmann, and K. Tierney. A gender-based genetic algorithm for the automatic configuration of algorithms. In: CP 2009, LNCS, vol. 5732, pp. 142--157. Springer (2009).

Digital Library

[3]

D. Babić and F. Hutter. Spear theorem prover. In SAT'08 (2008).

[4]

T. Bartz-Beielstein, C. Lasarczyk, and M. Preuss. The sequential parameter optimization toolbox. In: Experimental Methods for the Analysis of Optimization Algorithms, pp. 337--360. Springer (2010).

[5]

A. Biedenkapp, M. Lindauer, K. Eggensperger, F. Hutter, C. Fawcett, and H. H. Hoos. Efficient parameter importance analysis via ablation with surrogates. In Proceedings of the AAAI conference, to appear. (2017)

[6]

K. Eggensperger, F. Hutter, H. H. Hoos, and K. Leyton-Brown. Efficient bench-marking of hyperparameter optimizers via surrogates. In: AAAI, pp. 1114--1120. AAAI Press (2015).

Digital Library

[7]

C. Fawcett and H. H. Hoos. Analysing differences between algorithm configurations through ablation. Journal of Heuristics, 22(4):431--458 (2016).

Digital Library

[8]

H. H. Hoos. Automated algorithm configuration and parameter tuning. In: Autonomous Search, pp. 37--71. Springer (2012).

[9]

F. Hutter, H. H. Hoos, and K. Leyton-Brown. Sequential model-based optimization for general algorithm configuration. In: LION 5, LNCS, vol. 6683, pp. 507--523. Springer (2011).

Digital Library

[10]

F. Hutter, H. H. Hoos, and K. Leyton-Brown. An efficient approach for assessing hyperparameter importance. In Proc. of the 31th Int. Conf. on Machine Learning, vol. 32, pp. 754--762 (2014).

Digital Library

[11]

F. Hutter, H. H. Hoos, K. Leyton-Brown, and T. Stützle. ParamILS: an automatic algorithm configuration framework. JAIR, 36:267--306 (2009).

[12]

F. Hutter, M. López-Ibáñez, C. Fawcett, M. T. Lindauer, H. H. Hoos, K. Leyton-Brown, and T. Stützle. AClib: a benchmark library for algorithm configuration. In:LION 8, LNCS, vol. 8426, pp. 36--40. Springer (2014).

[13]

F. Hutter, L. Xu, H. H. Hoos, and K. Leyton-Brown. Algorithm runtime prediction: Methods & evaluation. Artificial Intelligence, 206:79--111 (2014).

Digital Library

[14]

D. R. Jones, M. Schonlau, and W. J. Welch. Efficient global optimization of expensive black-box functions. Journal of Global Optimization, 13(4):455--492 (1998).

Digital Library

[15]

S. Koziel, D. E. Ciaurri, and L. Leifsson. Surrogate-based methods. Computational Optimization, Methods and Algorithms, Studies in Computational Intelligence, vol. 356, pp. 33--59. Springer (2011).

[16]

M. López-Ibáñez, J. Dubois-Lacoste, L. Pérez Cáceres, T. Stützle, and M. Birattari. The irace package: Iterated racing for automatic algorithm configuration. Operations Research Perspectives, pp. 43--58 (2016).

[17]

M. López-Ibáñez, L. Pérez Cáceres, J. Dubois-Lacoste, T. Stützle, and M. Birattari. The irace package: User guide. Tech. Rep., IRIDIA, ULB, Belgium (2016).

[18]

L. Pérez Cáceres, M. López-Ibáñez, and T. Stützle. An analysis of parameters of irace. In: EvoCOP 2014, LNCS, vol. 8600, pp. 37--48. Springer (2014).

[19]

T. Stützle. ACOTSP: A software package of various ant colony optimization algorithms applied to the symmetric traveling salesman problem, 2002.

[20]

L. Xu, F. Hutter, H. H. Hoos, and K. Leyton-Brown. SATzilla: portfolio-based algorithm selection for SAT. JAIR, 32:565--606 (2008)

Digital Library

[21]

E. Zarpas. Benchmarking SAT solvers for bounded model checking. In:Int. Conf. on Theory and Applications of Satisfiability Testing, vol. 3569, pp. 340--354 (2005).

Digital Library

Cited By

Rehbach FRehbach F(2023)Methods/ContributionsEnhancing Surrogate-Based Optimization Through Parallelization10.1007/978-3-031-30609-9_3(29-94)Online publication date: 30-May-2023
https://doi.org/10.1007/978-3-031-30609-9_3
Eryoldaş YDurmuşoglu A(2022)A Literature Survey on Offline Automatic Algorithm ConfigurationApplied Sciences10.3390/app1213631612:13(6316)Online publication date: 21-Jun-2022
https://doi.org/10.3390/app12136316
Moosbauer JBinder MSchneider LPfisterer FBecker MLang MKotthoff LBischl B(2022)Automated Benchmark-Driven Design and Explanation of Hyperparameter OptimizersIEEE Transactions on Evolutionary Computation10.1109/TEVC.2022.321133626:6(1336-1350)Online publication date: Dec-2022
https://doi.org/10.1109/TEVC.2022.3211336
Show More Cited By

Index Terms

Configuring irace using surrogate configuration benchmarks
1. Theory of computation
  1. Design and analysis of algorithms
    1. Algorithm design techniques

Recommendations

Evaluating random forest models for irace
GECCO '17: Proceedings of the Genetic and Evolutionary Computation Conference Companion

Automatic algorithm configurators can greatly improve the performance of algorithms by effectively searching the parameter space. As algorithm configuration tasks can have large parameter spaces and the execution of candidate algorithm configurations is ...
Configuring highly available clusters using hacmp 4.5
Automatic DRAM Subsystem Configuration with irace
RAPIDO '23: Proceedings of the DroneSE and RAPIDO: System Engineering for constrained embedded systems

Nowadays, DRAM subsystem configuration includes a large number of parameters, resulting in an extensive design space. Setting these parameters is a challenging step in system design as the parameter-workload interactions are complex. Since design space ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

GECCO '17: Proceedings of the Genetic and Evolutionary Computation Conference

July 2017

1427 pages

ISBN:9781450349208

DOI:10.1145/3071178

General Chair:
Peter A. N. Bosman
Centrum Wiskunde & Informatica (CWI)

Copyright © 2017 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGEVO: ACM Special Interest Group on Genetic and Evolutionary Computation

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 01 July 2017

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

COMEX
Flemish Government
Research Foundation - Flanders (FWO)

Conference

GECCO '17

Sponsor:

SIGEVO

GECCO '17: Genetic and Evolutionary Computation Conference

July 15 - 19, 2017

Berlin, Germany

Acceptance Rates

GECCO '17 Paper Acceptance Rate 178 of 462 submissions, 39%;

Overall Acceptance Rate 1,669 of 4,410 submissions, 38%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

10
Total Citations
View Citations
199
Total Downloads

Downloads (Last 12 months)5
Downloads (Last 6 weeks)1

Reflects downloads up to 05 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Rehbach FRehbach F(2023)Methods/ContributionsEnhancing Surrogate-Based Optimization Through Parallelization10.1007/978-3-031-30609-9_3(29-94)Online publication date: 30-May-2023
https://doi.org/10.1007/978-3-031-30609-9_3
Eryoldaş YDurmuşoglu A(2022)A Literature Survey on Offline Automatic Algorithm ConfigurationApplied Sciences10.3390/app1213631612:13(6316)Online publication date: 21-Jun-2022
https://doi.org/10.3390/app12136316
Moosbauer JBinder MSchneider LPfisterer FBecker MLang MKotthoff LBischl B(2022)Automated Benchmark-Driven Design and Explanation of Hyperparameter OptimizersIEEE Transactions on Evolutionary Computation10.1109/TEVC.2022.321133626:6(1336-1350)Online publication date: Dec-2022
https://doi.org/10.1109/TEVC.2022.3211336
Luo CLin JCai SChen XHe BQiao BZhao PLin QZhang HWu WRajmohan SZhang D(2021)AutoCCAGProceedings of the 43rd International Conference on Software Engineering10.1109/ICSE43902.2021.00030(201-212)Online publication date: 22-May-2021
https://dl.acm.org/doi/10.1109/ICSE43902.2021.00030
Blot APetke J(2020)Synthetic Benchmarks for Genetic ImprovementProceedings of the IEEE/ACM 42nd International Conference on Software Engineering Workshops10.1145/3387940.3392175(287-288)Online publication date: 27-Jun-2020
https://dl.acm.org/doi/10.1145/3387940.3392175
Zaefferer MRehbach F(2020)Continuous Optimization Benchmarks by SimulationParallel Problem Solving from Nature – PPSN XVI10.1007/978-3-030-58112-1_19(273-286)Online publication date: 31-Aug-2020
https://doi.org/10.1007/978-3-030-58112-1_19
Nair VAgrawal AChen JFu WMathew GMenzies TMinku LWagner MYu ZZaidman AKamei YHill E(2018)Data-driven search-based software engineeringProceedings of the 15th International Conference on Mining Software Repositories10.1145/3196398.3196442(341-352)Online publication date: 28-May-2018
https://dl.acm.org/doi/10.1145/3196398.3196442
Stützle TLópez-Ibáñez M(2018)Automated Design of Metaheuristic AlgorithmsHandbook of Metaheuristics10.1007/978-3-319-91086-4_17(541-579)Online publication date: 21-Sep-2018
https://doi.org/10.1007/978-3-319-91086-4_17
Dang NDe Causmaecker P(2018)Analysis of Algorithm Components and Parameters: Some Case StudiesLearning and Intelligent Optimization10.1007/978-3-030-05348-2_25(288-303)Online publication date: 31-Dec-2018
https://doi.org/10.1007/978-3-030-05348-2_25
Blot AKessaci MJourdan LDe Causmaecker P(2018)Adaptive Multi-objective Local Search Algorithms for the Permutation Flowshop Scheduling ProblemLearning and Intelligent Optimization10.1007/978-3-030-05348-2_22(241-256)Online publication date: 31-Dec-2018
https://doi.org/10.1007/978-3-030-05348-2_22

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten