Abstract
Most publications on surrogate models have focused either on the prediction quality or on the optimization performance. It is still unclear whether the prediction quality is indeed related to the suitability for optimization. Moreover, most of these studies only employ low-dimensional test cases. There are no results for popular surrogate models, such as kriging, for high-dimensional (n > 10) noisy problems. In this paper, we analyze both aspects by comparing different surrogate models on the noisy 22-dimensional car setup optimization problem, based on both, prediction quality and optimization performance. In order not to favor specific properties of the model, we run two conceptually different modern optimization methods on the surrogate models, CMA-ES and BOBYQA. It appears that kriging and random forests are very good modeling techniques with respect to both, prediction quality and suitability for optimization algorithms.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsPreview
Unable to display preview. Download preview PDF.
References
Biermann, D., Weinert, K., Wagner, T.: Model-based optimization revisited: Towards real-world processes. In: Michalewicz, Z., Reynolds, R.G. (eds.) Proc. 2008 IEEE Congress on Evolutionary Computation (CEC 2008), pp. 2980–2987. IEEE Press, Piscataway (2008)
Breiman, L.: Random forests. Machine Learning 45(1), 5–32 (2001)
Cardamone, L., Loiacono, D., Lanzi, P.L.: Car Setup Optimization. Competition Software Manual. Tech. Rep. 2010.1, Dipartimento di Elettronica e Informazione, Politecnico di Milano, Italy (2010)
Cressie, N.: The origins of kriging. Mathematical Geology 22(3), 239–252 (1990)
Forrester, A.I.J., Keane, A.J., Bressloff, N.W.: Design and analysis of ’noisy’ computer experiments. AIAA Journal 44(10), 2331–2339 (2006)
Ginsbourger, D., Picheny, V., Roustant, O., Richet, Y.: Kriging with heterogeneous nugget effect for the approximation of noisy simulators with tunable fidelity. In: Proc. Joint Meeting of the Statistical Society of Canada and the Société Francaise de Statistique (2008)
Hansen, N., Ostermeier, A.: Completely derandomized self-adaptation in evolution strategies. Evolutionary Computation 9(2), 159–195 (2001)
Hastie, T., Tibshirani, R.: Generalized additive models. Statistical Science 1(3), 297–318 (1986)
Hastie, T., Tibshirani, R., Friedman, J.: The Elements of Statistical Learning: Data Mining, Inference, and Prediction, 2nd edn. Springer (2009)
Horton, R.L.: The General Linear Model. McGraw-Hill, New York (1978)
Huang, D., Allen, T.T., Notz, W.I., Zheng, N.: Global optimization of stochastic black-box systems via sequential kriging meta-models. Journal on Global Optimization 34(4), 441–466 (2006)
Jin, Y.: A comprehensive survey of fitness approximation in evolutionary computation. Soft Computing 9(1), 3–12 (2005)
Jones, D.R.: A taxonomy of global optimization methods based on response surfaces. Journal of Global Optimization 21(4), 345–383 (2001)
Jones, D.R., Schonlau, M., Welch, W.J.: Efficient global optimization of expensive black-box functions. Journal of Global Optimization 13(4), 455–492 (1998)
Kemmerling, M.: Optimierung der Fahrzeugabstimmung auf Basis verrauschter Daten einer Autorennsimulation. Diplomarbeit, TU Dortmund (2010) (in German)
Kemmerling, M., Preuss, M.: Automatic adaptation to generated content via car setup optimization in torcs. In: Proceedings of the 2010 IEEE Conference on Computational Intelligence and Games, CIG 2010, pp. 131–138. IEEE (2010)
Kleijnen, J.P.C.: Kriging metamodeling in simulation: A review. European Journal of Operational Research 192(3), 707–716 (2009)
Powell, M.J.D.: The bobyqa algorithm for bound constrained optimization without derivatives. Tech. Rep. DAMTP 2009/NA06, Centre for Mathematical Sciences, University of Cambridge, UK (2009)
Preuss, M., Rudolph, G., Wessing, S.: Tuning optimization algorithms for real-world problems by means of surrogate modeling. In: Proc. 12th Annual Conference on Genetic and Evolutionary Computation (GECCO 2010), pp. 401–408. ACM, New York (2010)
Rasmussen, C.E., Williams, C.K.I.: Gaussian processes for machine learning. Springer (2006)
Sacks, J., Welch, W.J., Mitchell, T.J., Wynn, H.P.: Design and analysis of computer experiments. Statistical Science 4(4), 409–423 (1989)
Tenne, Y., Izui, K., Nishiwaki, S.: Dimensionality-reduction frameworks for computationally expensive problems. In: Fogel, G., Ishibuchi, H. (eds.) Proc. 2010 IEEE Congress on Evolutionary Computation (CEC 2010), pp. 1–8. IEEE Press, Piscataway (2010)
Wang, G.G., Shan, S.: Review of metamodeling techniques in support of engineering design optimization. Journal of Mechanical Design 129(4), 370–380 (2007)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2012 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Preuss, M., Wagner, T., Ginsbourger, D. (2012). High-Dimensional Model-Based Optimization Based on Noisy Evaluations of Computer Games. In: Hamadi, Y., Schoenauer, M. (eds) Learning and Intelligent Optimization. LION 2012. Lecture Notes in Computer Science, vol 7219. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-34413-8_11
Download citation
DOI: https://doi.org/10.1007/978-3-642-34413-8_11
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-34412-1
Online ISBN: 978-3-642-34413-8
eBook Packages: Computer ScienceComputer Science (R0)