Skip to main content
SpringerLink
Log in

Efficient global optimization algorithm assisted by multiple surrogate techniques

  • Published:
Journal of Global Optimization Aims and scope Submit manuscript

Abstract

Surrogate-based optimization proceeds in cycles. Each cycle consists of analyzing a number of designs, fitting a surrogate, performing optimization based on the surrogate, and finally analyzing a candidate solution. Algorithms that use the surrogate uncertainty estimator to guide the selection of the next sampling candidate are readily available, e.g., the efficient global optimization (EGO) algorithm. However, adding one single point at a time may not be efficient when the main concern is wall-clock time (rather than number of simulations) and simulations can run in parallel. Also, the need for uncertainty estimates limits EGO-like strategies to surrogates normally implemented with such estimates (e.g., kriging and polynomial response surface). We propose the multiple surrogate efficient global optimization (MSEGO) algorithm, which adds several points per optimization cycle with the help of multiple surrogates. We import uncertainty estimates from one surrogate to another to allow use of surrogates that do not provide them. The approach is tested on three analytic examples for nine basic surrogates including kriging, radial basis neural networks, linear Shepard, and six different instances of support vector regression. We found that MSEGO works well even with imported uncertainty estimates, delivering better results in a fraction of the optimization cycles needed by EGO.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Wang L., Shan S., Wang G.G.: Mode-pursuing sampling method for global optimization on expensive black-box functions. Eng. Optim. 36(4), 419–438 (2004)

    Article  Google Scholar 

  2. Kleijnen J.P.C., Sanchez S.M., Lucas T.W., Cioppa T.M.: A user’s guide to the brave new world of designing simulation experiments. INFORMS J. Comput. 17(3), 263–289 (2005)

    Article  Google Scholar 

  3. Sóbester A., Leary S., Keane A.: On the design of optimization strategies based on global response surface approximation models. J. Glob. Optim. 33(1), 31–59 (2005)

    Article  Google Scholar 

  4. Simpson, T.W., Toropov, V., Balabanov, V., Viana, F.A.C.: Design and analysis of computer experiments in multidisciplinary design optimization: a review of how far we have come—or not. In: 12th AIAA/ISSMO Multidisciplinary Analysis and Optimization Conference, AIAA–2008–5802. AIAA, Victoria, BC, Canada (2008)

  5. Forrester A.I.J., Keane A.J.: Recent advances in surrogate-based optimization. Prog. Aerosp. Sci. 45(1–3), 50–79 (2009)

    Article  Google Scholar 

  6. Viana, F.A.C., Gogu, C., Haftka, R.T.: Making the most out of surrogate models: tricks of the trade. In: ASME 2010 International Design Engineering Technical Conferences & Computers and Information in Engineering Conference, DETC2010–28813, ASME, Montreal, QC, Canada (2009)

  7. Jones D.R., Schonlau M., Welch W.J.: Efficient global optimization of expensive black-box functions. J. Glob. Optim. 13(4), 455–492 (1998)

    Article  Google Scholar 

  8. Jin, R., Chen, W., Sudjianto, A.: On sequential sampling for global metamodeling for in engineering design. In: ASME 2002 International Design Engineering Technical Conferences & Computers and Information in Engineering Conference, DETC2002–34092. ASME, Montreal, QC, Canada (2002)

  9. Huang D., Allen T., Notz W., Zeng N.: Global optimization of stochastic black-box systems via sequential kriging meta-models. J. Glob. Optim. 34(3), 441–466 (2006)

    Article  Google Scholar 

  10. Henkenjohann N., Kunert J.: An efficient sequential optimization approach based on the multivariate expected improvement criterion. Qual. Eng. 19(4), 267–280 (2007)

    Article  Google Scholar 

  11. Ponweiser, W., Wagner, T., Vincze, M.: Clustered multiple generalized expected improvement: a novel infill sampling criterion for surrogate models. In: IEEE Congress on Evolutionary Computation, pp. 3514–3521. IEEE, Hong Kong (2008)

  12. Jones D.R.: A taxonomy of global optimization methods based on response surfaces. J. Glob. Optim. 21(4), 345–383 (2001)

    Article  Google Scholar 

  13. Ginsbourger D., Le Riche R., Carraro L.: Kriging is well-suited to parallelize optimization. In: Tenne, Y., Goh, C. (eds) Computational Intelligence in Expensive Optimization Problems, vol. 2, pp. 131–162. Springer, Heidelberg (2010)

    Chapter  Google Scholar 

  14. Samad A., Kim K.Y., Goel T., Haftka R.T., Shyy W.: Multiple surrogate modeling for axial compressor blade shape optimization. J. Propuls. Power 24(2), 302–310 (2008)

    Article  Google Scholar 

  15. Viana, F.A.C., Haftka, R.T.: Using multiple surrogates for metamodeling. In: 7th ASMO-UK ISSMO International Conference on Engineering Design Optimization, pp. 1–18. ISSMO, Bath, UK (2008)

  16. Gorissen D., Dhaene T., de Turck F.: Evolutionary model type selection for global surrogate modeling. J. Mach. Learn. Res. 10(1), 2039–2078 (2009)

    Google Scholar 

  17. Cho Y.C., Jayaraman B., Viana F.A.C., Haftka R.T., Shyy W.: Surrogate modeling for characterizing the performance of dielectric barrier discharge plasma actuator. Int. J. Comput. Fluid Dyn. 24(7), 281–301 (2010)

    Article  Google Scholar 

  18. Gutmann H.M.: A radial basis function method for global optimization. J. Glob. Optim. 19(3), 201–227 (2001)

    Article  Google Scholar 

  19. Stein M.L.: Interpolation of Spatial Data: Some Theory for Kriging. Springer, Berlin (1999)

    Book  Google Scholar 

  20. Martin J.D., Simpson T.W.: Use of kriging models to approximate deterministic computer models. AIAA J. 43(4), 853–863 (2005)

    Article  Google Scholar 

  21. Yamamoto J.K.: An alternative measure of the reliability of ordinary kriging estimates. Math. Geol. 32(4), 489–509 (2000)

    Article  Google Scholar 

  22. den Hertog D., Kleijnen J.P.C., Siem A.Y.D.: The correct kriging variance estimated by bootstrapping. J. Oper. Res. Soc. 57(4), 400–409 (2006)

    Article  Google Scholar 

  23. Kleijnen, J.P.C., van Beers, W.C.M., van Nieuwenhuyse, I.: Expected improvement in efficient global optimization through bootstrapped kriging. J. Glob. Optim. (available online), pp. 1–15 (2011)

  24. Forrester A.I.J., Sóbester A., Keane A.J.: Engineering Design Via Surrogate Modelling: A Practical Guide. Wiley, New York (2008)

    Book  Google Scholar 

  25. Forrester A.I.J., Sóbester A., Keane A.J.: Multi-fidelity optimization via surrogate modeling. Proc. R. Soc. A Math. Phys. Eng. Sci. 463(2088), 3251–3269 (2007)

    Article  Google Scholar 

  26. Fang H., Horstemeyer M.F.: Global response approximation with radial basis functions. Eng. Optim. 38(4), 407–424 (2006)

    Article  Google Scholar 

  27. Goel T., Stander N.: Comparing three error criteria for selecting radial basis function network topology. Comput. Methods Appl. Mech. Eng. 198(27–29), 2137–2150 (2009)

    Article  Google Scholar 

  28. Berry M.W., Minser K.S.: Algorithm 798: high-dimensional interpolation using the modified Shepard method. ACM Trans. Math. Softw. 25(3), 353–366 (1999)

    Article  Google Scholar 

  29. Thacker W.I., Zhang J., Watson L.T., Birch J.B., Iyer M.A., Berry M.W.: Algorithm 905: Sheppack: modified Shepard algorithm for interpolation of scattered multivariate data. ACM Trans. Math. Softw. 37(3), 1–20 (2010)

    Article  Google Scholar 

  30. Cristianini N., Shawe-Taylor J.: An Introduction to Support Vector Machines and Other Kernel-Based Learning Methods. Cambridge University Press, Cambridge (2000)

    Book  Google Scholar 

  31. Smola A.J., Schölkopf B.: A tutorial on support vector regression. Stat. Comput. 14(3), 199–222 (2004)

    Article  Google Scholar 

  32. Viana, F.A.C., Haftka, R.T.: Importing uncertainty estimates from one surrogate to another. In: 50th AIAA/ASME/ASCE/AHS/ASC Structures, Structural Dynamics, and Materials Conference, pp. AIAA–2009–2237. AIAA, Palm Springs, CA, USA (2009)

  33. Viana F.A.C., Haftka R.T., Steffen V. Jr: Multiple surrogates: how cross-validation errors can help us to obtain the best predictor. Struct. Multidiscip. Optim. 39(4), 439–457 (2009)

    Article  Google Scholar 

  34. Lophaven, S.N., Nielsen, H.B., Søndergaard, J.: Dace—a matlab kriging toolbox. Tech. Rep. IMM–TR–2002–12, Technical University of Denmark, Denmark (2002). Available at http://www2.imm.dtu.dk/~hbn/dace/

  35. MathWorks contributors: MATLAB The language of technical computing. The MathWorks, Inc, Natick, MA, USA, version 7.0 release 14 edn. (2004)

  36. Jekabsons, G.: RBF: Radial Basis Function interpolation for MATLAB/OCTAVE. Riga Technical University, Latvia, version 1.1 edn. (2009). Available at http://www.cs.rtu.lv/jekabsons/regression.html

  37. Viana, F.A.C.: SURROGATES Toolbox User’s Guide. Gainesville, FL, USA, version 3.0 edn. (2011). Available at http://sites.google.com/site/felipeacviana/surrogatestoolbox

  38. Gunn, S.R.: Support vector machines for classification and regression. Tech. rep., University of Southampton, UK (1997). Available at http://www.isis.ecs.soton.ac.uk/resources/svminfo/

  39. Roustant, O. Ginsbourger, D., Deville, Y.: DiceKriging, DiceOptim: two R packages for the analysis of computer experiments by kriging-based metamodeling and optimization. Ecole Nationale Supérieure des Mines de Saint-Etienne, France and University of Bern, Switzerland (2010). Available at http://hal.archives-ouvertes.fr/hal-00495766/

  40. Cherkassky V., Ma Y.: Practical selection of svm parameters and noise estimation for svm regression. Neural Netw. 17(1), 113–126 (2004)

    Article  Google Scholar 

  41. Sasena, M.J.: Optimization of computer simulations via smoothing splines and kriging metamodels. Master’s thesis, University of Michigan, Ann Arbor, MI, USA (1998)

  42. Dixon, L.C.W., Szegö, G.P.: Towards Global Optimization 2. North Holland, Amsterdam (1978)

  43. McKay M.D., Beckman R.J., Conover W.J.: A comparison of three methods for selecting values of input variables from a computer code. Technometrics 21(2), 239–245 (1979)

    Google Scholar 

  44. Price K.V., Storn R.M., Lampinen J.A.: Differential Evolution: A Practical Approach to Global Optimization. Springer, New York (2005)

    Google Scholar 

  45. Storn R., Price K.: Differential evolution—a simple and efficient heuristic for global optimization over continuous spaces. J. Glob. Optim. 11(4), 341–359 (1997)

    Article  Google Scholar 

  46. Schutte J.F., Haftka R.T., Fregly B.J.: Improved global convergence probability using multiple independent optimizations. Int. J. Numer. Methods Eng. 71(6), 678–702 (2007)

    Article  Google Scholar 

  47. Viana F.A.C., Haftka R.T.: Cross validation can estimate how well prediction variance correlates with error. AIAA J. 47(9), 2266–2270 (2009)

    Article  Google Scholar 

  48. Meckesheimer M., Booker A.J., Barton R.R., Simpson T.W.: Computationally inexpensive metamodel assessment strategies. AIAA J. 40(10), 2053–2060 (2002)

    Article  Google Scholar 

  49. Picard R.R., Cook R.D.: Cross-validation of regression models. J. Am. Stat. Assoc. 79(387), 575–583 (1984)

    Article  Google Scholar 

  50. Varma S., Simon R.: Bias in error estimation when using cross-validation for model selection. BMC Bioinf. 7(91), 1290–1300 (2006)

    Google Scholar 

  51. Myers R.H.: Classical and Modern Regression with Applications. Duxbury Press, Belmont (2000)

    Google Scholar 

Download references

Author information

Author notes

  1. Felipe A. C. Viana

    Present address: GE Global Research, Niskayuna, NY, 12309, USA

Authors and Affiliations

  1. Department of Mechanical and Aerospace Engineering, University of Florida, Gainesville, FL, 32611, USA

    Felipe A. C. Viana & Raphael T. Haftka

  2. Departments of Computer Science and Mathematics, Virginia Polytechnic Institute and State University, Blacksburg, VA, 24061, USA

    Layne T. Watson

Authors
  1. Felipe A. C. Viana
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Raphael T. Haftka
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Layne T. Watson
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Felipe A. C. Viana.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Viana, F.A.C., Haftka, R.T. & Watson, L.T. Efficient global optimization algorithm assisted by multiple surrogate techniques. J Glob Optim 56, 669–689 (2013). https://doi.org/10.1007/s10898-012-9892-5

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10898-012-9892-5

Keywords

Search

Navigation