Skip to main content
SpringerLink
Log in
Book cover

Autonomous Search pp 37–71Cite as

Automated Algorithm Configuration and Parameter Tuning

  • Chapter

Abstract

The use of automated configuration and parameter tuning techniques plays an increasingly important role in the design, evaluation and application of high-performance algorithms for difficult computational problems. This chapter provides an introduction to these techniques and gives an overview of three families of procedures for automatically optimising the performance of parameterised algorithms. Racing procedures iteratively evaluate parameter settings on problem instances from a given set and use statistical hypothesis tests to eliminate candidate configurations that are significantly outperformed by other configurations. ParamILS uses a powerful stochastic local search method to search within potentially vast spaces of candidate configurations of a given algorithm. And finally, sequential model-based optimisation (SMBO) methods build a response surface model that relates parameter settings to performance, and use this model to iteratively identify promising settings. We also briefly survey other algorithm configuration and parameter tuning procedures, as well as related approaches, such as instance-based algorithm selection and configuration.

This is a preview of subscription content, log in via an institution.

Chapter
USD   29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD   84.99
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD   109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD   109.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Learn about institutional subscriptions

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. Adenso-Diaz, B., Laguna, M.: Fine-tuning of algorithms using fractional experimental design and local search. Operations Research 54(1):99–114 (2006)

    Article  MATH  Google Scholar 

  2. Ansótegui, C., Sellmann, M., Tierney, K.: A gender-based genetic algorithm for the automatic configuration of algorithms. In: Proceedings of the 15th International Conference on Principles and Practice of Constraint Programming (CP 2009), pp. 142–157 (2009)

    Chapter  Google Scholar 

  3. Applegate, D. L., Bixby, R. E., Chvátal, V., Cook, W. J.: The Traveling Salesman Problem: A Computational Study. Princeton University Press (2006)

    MATH  Google Scholar 

  4. Audet, C., Orban, D.: Finding optimal algorithmic parameters using the mesh adaptive direct search algorithm. SIAM Journal on Optimization 17(3):642–664 (2006)

    Article  MATH  MathSciNet  Google Scholar 

  5. Balaprakash, P., Birattari, M., Stützle, T.: Improvement strategies for the F-Race algorithm: Sampling design and iterative refinement. In: Bartz-Beielstein, T., Blesa, M., Blum, C., Naujoks, B., Roli, A., Rudolph, G., Sampels, M. (eds) 4th International Workshop on Hybrid Metaheuristics, Proceedings, HM 2007, Springer Verlag, Berlin, Germany, Lecture Notes in Computer Science, vol. 4771, pp. 108–122 (2007)

    Google Scholar 

  6. Balaprakash, P., Birattari, M., Stützle, T., Dorigo, M.: Estimation-based metaheuristics for the probabilistic traveling salesman problem. Computers & OR 37(11):1939–1951 (2010)

    Article  MATH  Google Scholar 

  7. Bartz-Beielstein, T.: Experimental Research in Evolutionary Computation: The New Experimentalism. Natural Computing Series, Springer Verlag, Berlin, Germany (2006)

    MATH  Google Scholar 

  8. Bartz-Beielstein, T., Lasarczyk, C., Preuß, M.: Sequential parameter optimization. In: McKay, B., et al. (eds) Proceedings 2005 Congress on Evolutionary Computation (CEC’05), Edinburgh, Scotland, IEEE Press, vol. 1, pp. 773–780 (2005)

    Google Scholar 

  9. Bartz-Beielstein, T., Lasarczyk, C., Preuss, M.: Sequential parameter optimization toolbox, manual version 0.5, September 2008, available at http://www.gm.fh-koeln.de/imperia/md/content/personen/lehrende/bartz_beielstein_thomas/spotdoc.pdf (2008)

  10. Battiti, R., Brunato, M., Mascia, F.: Reactive Search and Intelligent Optimization. Operations Research/Computer Science Interfaces, Springer Verlag (2008)

    Google Scholar 

  11. Birattari, M., Stützle, T., Paquete, L., Varrentrapp,K.: A racing algorithm for configuring metaheuristics. In: GECCO ’02: Proceedings of the Genetic and Evolutionary Computation Conference, pp. 11–18 (2002)

    Google Scholar 

  12. Birattari, M., Yuan, Z., Balaprakash, P., Stützle, T.: F-race and Iterated F-Race: An overview. In: Bartz-Beielstein, T., Chiarandini, M., Paquete, L., Preuss, M. (eds) Experimental Methods for the Analysis of Optimization Algorithms, Springer, Berlin, Germany, pp. 311–336 (2010)

    Chapter  Google Scholar 

  13. Bűrmen, Á., Puhan, J., Tuma, T.: Grid restrained Nelder-Mead algorithm. Computational Optimization and Applications 34(3):359–375 (2006)

    Article  MathSciNet  Google Scholar 

  14. Carchrae, T., Beck, J.: Applying machine learning to low knowledge control of optimization algorithms. Computational Intelligence 21(4):373–387 (2005)

    Article  MathSciNet  Google Scholar 

  15. Chiarandini, M., Fawcett, C., Hoos, H.: A modular multiphase heuristic solver for post enrollment course timetabling (extended abstract). In: Proceedings of the 7th International Conference on the Practice and Theory of Automated Timetabling PATAT (2008)

    Google Scholar 

  16. Da Costa, L., Fialho, Á., Schoenauer, M., Sebag, M.: Adaptive Operator Selection with Dynamic Multi-Armed Bandits. In: Proceedings of the 10th Annual Conference on Genetic and Evolutionary Computation (GECCO’08), pp. 913–920 (2008)

    Chapter  Google Scholar 

  17. Fawcett, C., Hoos, H., Chiarandini, M.: An automatically configured modular algorithm for post enrollment course timetabling. Tech. Rep. TR-2009-15, University of British Columbia, Department of Computer Science (2009)

    Google Scholar 

  18. Fukunaga, A. S.: Automated discovery of composite SAT variable-selection heuristics. In: Proceedings of the 18th National Conference on Artificial Intelligence (AAAI-02), pp. 641–648 (2002)

    Google Scholar 

  19. Fukunaga, A. S.: Evolving local search heuristics for SAT using genetic programming. In: Genetic and Evolutionary Computation – GECCO-2004, Part II, Springer-Verlag, Seattle, WA, USA, Lecture Notes in Computer Science, vol. 3103, pp. 483–494 (2004)

    Chapter  Google Scholar 

  20. Gagliolo, M., Schmidhuber, J.: Dynamic algorithm portfolios. In: Amato, C., Bernstein, D., Zilberstein, S. (eds) Proceedings of the 9th International Symposium on Artificial Intelligence and Mathematics (AI-MATH-06) (2006)

    Google Scholar 

  21. Gomes, C. P., Selman, B., Crato, N., Kautz, H.: Heavy-tailed phenomena in satisfiability and constraint satisfaction problems. Journal of Automated Reasoning 24(1-2):67–100 (2000)

    Article  MATH  MathSciNet  Google Scholar 

  22. Gratch, J., Chien, S. A.: Adaptive problem-solving for large-scale scheduling problems: A case study. Journal of Artificial Intelligence Research 4:365–396 (1996)

    Google Scholar 

  23. Gratch, J., Dejong, G.: Composer: A probabilistic solution to the utility problem in speed-up learning. In: Rosenbloom, P., Szolovits, P. (eds) Proceedings of the 10th National Conference on Artificial Intelligence (AAAI-92), AAAI Press / The MIT Press, Menlo Park, CA, USA, pp. 235–240 (1992)

    Google Scholar 

  24. Guerri, A., Milano, M.: Learning techniques for automatic algorithm portfolio selection. In: Proceedings of the 16th European Conference on Artificial Intelligence (ECAI 2004), pp. 475–479 (2004)

    Google Scholar 

  25. Hansen, N.: The CMA evolution strategy: A comparing review. In: Lozano, J., Larranaga, P., Inza, I., Bengoetxea, E. (eds) Towards a new evolutionary computation. Advances on estimation of distribution algorithms, Springer, pp. 75–102 (2006)

    Google Scholar 

  26. Hansen, N., Kern, S.: Evaluating the CMA evolution strategy on multimodal test functions. In: Yao, X., et al. (eds) Parallel Problem Solving from Nature PPSN VIII, Springer, LNCS, vol. 3242, pp. 282–291 (2004)

    Chapter  Google Scholar 

  27. Hansen, N., Ostermeier, A.: Completely derandomized self-adaptation in evolution strategies. Evolutionary Computation 9(2):159–195 (2001)

    Article  Google Scholar 

  28. Hoos, H.: Computer-aided design of high-performance algorithms. Tech. Rep. TR-2008-16, University of British Columbia, Department of Computer Science (2008)

    Google Scholar 

  29. Hoos, H., Stützle, T.: Local search algorithms for SAT: An empirical evaluation. Journal of Automated Reasoning 24(4):421–481 (2000)

    Article  MATH  Google Scholar 

  30. Hoos, H., Stützle, T.: Stochastic Local Search—Foundations and Applications. Morgan Kaufmann Publishers, USA (2004)

    Google Scholar 

  31. Huang, D., Allen, T. T., Notz, W. I., Zeng, N.: Global optimization of stochastic black-box systems via sequential kriging meta-models. Journal of Global Optimization 34(3):441–466 (2006)

    Article  MATH  MathSciNet  Google Scholar 

  32. Hutter, F.: Automated configuration of algorithms for solving hard computational problems. Ph.D. thesis, University of British Columbia, Department of Computer Science, Vancouver, BC, Canada (2009)

    Google Scholar 

  33. Hutter, F., Tompkins, D. A., Hoos, H.: Scaling and Probabilistic Smoothing: Efficient Dynamic Local Search for SAT. In: Principles and Practice of Constraint Programming – CP 2002, Springer-Verlag, LNCS, vol. 2470, pp. 233–248 (2002)

    Chapter  Google Scholar 

  34. Hutter, F., Hamadi, Y., Hoos, H., Leyton-Brown, K.: Performance prediction and automated tuning of randomized and parametric algorithms. In: Principles and Practice of Constraint Programming – CP 2006, Springer-Verlag, LNCS, vol. 4204, pp. 213–228 (2006)

    Chapter  Google Scholar 

  35. Hutter F., Babić, D., Hoos, H., Hu, A. J.: Boosting verification by automatic tuning of decision procedures. In: Proc. Formal Methods in Computer-Aided Design (FMCAD’07), IEEE Computer Society Press, pp. 27–34 (2007)

    Google Scholar 

  36. Hutter, F., Hoos, H., Stützle, T.: Automatic algorithm configuration based on local search. In: Proceedings of the 22nd National Conference on Artificial Intelligence (AAAI-07), pp. 1152–1157 (2007)

    Google Scholar 

  37. Hutter, F., Hoos, H., Leyton-Brown, K., Murphy, K.: An experimental investigation of model-based parameter optimisation: SPO and beyond. In: Proceedings of the 11th Annual Conference on Genetic and Evolutionary Computation (GECCO’09), ACM, pp. 271–278 (2009)

    Chapter  Google Scholar 

  38. Hutter, F., Hoos, H., Leyton-Brown, K., Stützle, T.: ParamILS: An automatic algorithm configuration framework. Journal of Artificial Intelligence Research 36:267–306 (2009)

    MATH  Google Scholar 

  39. Hutter, F., Hoos, H., Leyton-Brown, K., Stützle, T.: ParamILS: An automatic algorithm configuration framework (extended version). Tech. Rep. TR-2009-01, University of British Columbia, Department of Computer Science (2009)

    Google Scholar 

  40. Hutter, F., Hoos, H., Leyton-Brown K.: Sequential model-based optimization for general algorithm configuration (extended version). Tech. Rep. TR-2010-10, University of British Columbia, Department of Computer Science (2010)

    Google Scholar 

  41. Hutter, F., Hoos, H., Leyton-Brown, K., Murphy, K.: Time-bounded sequential parameter optimization. In: Proceedings of the 4th International Conference on Learning and Intelligent Optimization (LION 4), Springer-Verlag, LNCS, vol. 6073, pp. 281–298 (2010)

    Google Scholar 

  42. Hutter, F., Hoos, H., Leyton-Brown, K.: Automated configuration of mixed integer programming solvers. In: Proceedings of the 7th International Conference on the Integration of AI and OR Techniques in Constraint Programming for Combinatorial Optimization Problems (CPAIOR 2010), Springer-Verlag, LNCS, vol. 6140, pp. 186–202 (2010)

    Chapter  Google Scholar 

  43. Hutter, F., Hoos, H., Leyton-Brown, K.: Extending sequential model-based optimization to general algorithm configuration. To appear in: Proceedings of the 5th International Conference on Learning and Intelligent Optimization (LION 5) (2011)

    Google Scholar 

  44. Jones, D. R., Schonlau, M., Welch, W. J.: Efficient global optimization of expensive black box functions. Journal of Global Optimization 13:455–492 (1998)

    Article  MATH  MathSciNet  Google Scholar 

  45. KhudaBukhsh, A., Xu, L., Hoos, H., Leyton-Brown, K.: SATenstein: Automatically building local search SAT solvers from components. In: Proceedings of the 21st International Joint Conference on Artificial Intelligence (IJCAI-09), pp 517–524 (2009)

    Google Scholar 

  46. Leyton-Brown, K., Nudelman, E., Andrew, G., McFadden, J., Shoham, Y.: A portfolio approach to algorithm selection. In: Rossi, F. (ed) Principles and Practice of Constraint Programming – CP 2003, Springer Verlag, Berlin, Germany, Lecture Notes in Computer Science, vol. 2833, pp. 899–903 (2003)

    Chapter  Google Scholar 

  47. Lourenço, H. R., Martin, O., Stützle, T.: Iterated local search. In: Glover, F., Kochenberger, G. (eds) Handbook of Metaheuristics, Kluwer Academic Publishers, Norwell, MA, USA, pp. 321–353 (2002)

    Google Scholar 

  48. Maron, O., Moore, A. W.: Hoeffding races: Accelerating model selection search for classification and function approximation. In: Advances in neural information processing systems 6, Morgan Kaufmann, pp. 59–66 (1994)

    Google Scholar 

  49. Nelder, J. A., Mead, R.: A simplex method for function minimization. The Computer Journal 7(4):308–313 (1965)

    MATH  Google Scholar 

  50. Nell, C. W., Fawcett, C., Hoos, H., Leyton-Brown K.: HAL: A framework for the automated design and analysis of high-performance algorithms. To appear in: Proceedings of the 5th International Conference on Learning and Intelligent Optimization (LION 5) (2011)

    Google Scholar 

  51. Nocedal, J., Wright, S. J.: Numerical Optimization, 2nd edn. Springer-Verlag (2006)

    MATH  Google Scholar 

  52. Nouyan, S., Campo, A., Dorigo, M.: Path formation in a robot swarm: Self-organized strategies to find your way home. Swarm Intelligence 2(1):1–23 (2008)

    Article  Google Scholar 

  53. Pellegrini, P., Birattari, M.: The relevance of tuning the parameters of metaheuristics. a case study: The vehicle routing problem with stochastic demand. Tech. Rep. TR/IRIDIA/2006-008, IRIDIA, Université Libre de Bruxelles, Brussels, Belgium (2006)

    Google Scholar 

  54. Pop, M., Salzberg, S. L., Shumway, M.: Genome sequence assembly: Algorithms and issues. Computer 35(7):47–54 (2002)

    Article  Google Scholar 

  55. Prasad, M. R., Biere, A., Gupta, A.: A survey of recent advances in SAT-based formal verification. International Journal on Software Tools for Technology Transfer 7(2):156–173 (2005)

    Article  Google Scholar 

  56. Rasmussen, C. E., Williams, C. K. I.: Gaussian Processes for Machine Learning. The MIT Press (2006)

    MATH  Google Scholar 

  57. Rice, J.: The algorithm selection problem. Advances in Computers 15:65–118 (1976)

    Article  Google Scholar 

  58. Rossi-Doria, O., Sampels, M., Birattari, M., Chiarandini, M., Dorigo, M., Gambardella, L. M., Knowles, J. D., Manfrin, M., Mastrolilli, M., Paechter, B., Paquete, L., Stützle, T.: A comparison of the performance of different metaheuristics on the timetabling problem. In: Burke, E. K., Causmaecker, P. D. (eds) Practice and Theory of Automated Timetabling IV, 4th International Conference, PATAT 2002, Selected Revised Papers, Springer, Lecture Notes in Computer Science, vol. 2740, pp. 329–354 (2003)

    Google Scholar 

  59. Sacks, J., Welch, W., Mitchell, T., Wynn, H.: Design and analysis of computer experiments (with discussion). Statistical Science 4:409–435 (1989)

    Article  MATH  MathSciNet  Google Scholar 

  60. Santner, T., Williams, B., Notz, W.: The Design and Analysis of Computer Experiments. Springer Verlag, New York (2003)

    MATH  Google Scholar 

  61. Schiavinotto, T., Stützle, T.: The linear ordering problem: Instances, search space analysis and algorithms. Journal of Mathematical Modelling and Algorithms 3(4):367–402 (2004)

    Article  MATH  MathSciNet  Google Scholar 

  62. Schonlau, M., Welch, W. J., Jones, D. R.: Global versus local search in constrained optimization of computer models. In: Flournoy, N., Rosenberger, W., Wong, W. (eds) New Developments and Applications in Experimental Design, vol. 34, Institute of Mathematical Statistics, Hayward, California, pp. 11–25 (1998)

    Chapter  Google Scholar 

  63. Spall, J.: Introduction to Stochastic Search and Optimization. John Wiley & Sons, Inc., New York, NY, USA (2003)

    Book  MATH  Google Scholar 

  64. Stützle, T., Hoos, H.: MAX-MIN Ant System. Future Generation Computer Systems 16(8):889–914 (2000)

    Article  Google Scholar 

  65. Thachuk, C., Shmygelska, A., Hoos, H.: A replica exchange Monte Carlo algorithm for protein folding in the hp model. BMC Bioinformatics 8(342) (2007)

    Google Scholar 

  66. Tompkins, D., Hoos, H.: Dynamic Scoring Functions with Variable Expressions: New SLS Methods for Solving SAT. In: Proceedings of the 13th International Conference on Theory and Applications of Satisfiability Testing (SAT 2010), Springer-Verlag, LNCS, vol. 6175, pp. 278–292 (2010)

    Chapter  Google Scholar 

  67. Xu, L., Hutter, F., Hoos, H., Leyton-Brown, K.: SATzilla-07: The design and analysis of an algorithm portfolio for SAT. In: Principles and Practice of Constraint Programming – CP 2007, Springer Berlin / Heidelberg, LNCS, vol. 4741, pp. 712–727 (2007)

    Chapter  Google Scholar 

  68. Xu, L., Hutter, F., Hoos, H., Leyton-Brown, K.: SATzilla: Portfolio-based algorithm selection for SAT. Journal of Artificial Intelligence Research 32:565–606 (2008)

    MATH  Google Scholar 

  69. Xu, L., Hutter, F., Hoos, H., Leyton-Brown, K.: SATzilla2009: An Automatic Algorithm Portfolio for SAT, Solver Description, SAT Competition 2009 (2009)

    Google Scholar 

  70. Xu, L., Hoos, H., Leyton-Brown, K.: Hydra: Automatically configuring algorithms for portfolio-based selection. In: Proceedings of the 24th AAAI Conference on Artificial Intelligence (AAAI-10), pp. 210–216 (2010)

    Google Scholar 

  71. Yuan, Z., Fügenschuh, A., Homfeld, H., Balaprakash, P., Stützle T., Schoch M.: Iterated greedy algorithms for a real-world cyclic train scheduling problem. In: Blesa, M., Blum, C., Cotta, C., Fernández, A., Gallardo, J., Roli, A., Sampels, M. (eds) Hybrid Metaheuristics, Lecture Notes in Computer Science, vol. 5296, Springer Berlin / Heidelberg, pp. 102–116 (2008)

    Chapter  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. University of British Columbia, 2366 Main Mall, Vancouver, BC, V6T 1Z4, Canada

    Holger H. Hoos

Authors
  1. Holger H. Hoos
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Holger H. Hoos .

Editor information

Editors and Affiliations

  1. Microsoft Research Cambridge, JJ Thomson Avenue 7, Cambridge, CB3 0FB, United Kingdom

    Youssef Hamadi

  2. Federico Santa María, Departamento de Informática, Universidad Técnica, Avenida España 1680, Valparaíso, Chile

    Eric Monfroy

  3. Faculté des Sciences, LERIA, Université d'Angers, Boulevard Lavoisier 2, Angers CX 01, 49045, France

    Frédéric Saubion

Rights and permissions

Reprints and permissions

Copyright information

© 2011 Springer-Verlag Berlin Heidelberg

About this chapter

Cite this chapter

Hoos, H.H. (2011). Automated Algorithm Configuration and Parameter Tuning. In: Hamadi, Y., Monfroy, E., Saubion, F. (eds) Autonomous Search. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-21434-9_3

Download citation

  • DOI: https://doi.org/10.1007/978-3-642-21434-9_3

  • Publisher Name: Springer, Berlin, Heidelberg

  • Print ISBN: 978-3-642-21433-2

  • Online ISBN: 978-3-642-21434-9

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation