Skip to main content
SpringerLink
Log in

Optimization of black-box problems using Smolyak grids and polynomial approximations

  • Published:
Journal of Global Optimization Aims and scope Submit manuscript

Abstract

A surrogate-based optimization method is presented, which aims to locate the global optimum of box-constrained problems using input–output data. The method starts with a global search of the n-dimensional space, using a Smolyak (Sparse) grid which is constructed using Chebyshev extrema in the one-dimensional space. The collected samples are used to fit polynomial interpolants, which are used as surrogates towards the search for the global optimum. The proposed algorithm adaptively refines the grid by collecting new points in promising regions, and iteratively refines the search space around the incumbent sample until the search domain reaches a minimum hyper-volume and convergence has been attained. The algorithm is tested on a large set of benchmark problems with up to thirty dimensions and its performance is compared to a recent algorithm for global optimization of grey-box problems using quadratic, kriging and radial basis functions. It is shown that the proposed algorithm has a consistently reliable performance for the vast majority of test problems, and this is attributed to the use of Chebyshev-based Sparse Grids and polynomial interpolants, which have not gained significant attention in surrogate-based optimization thus far.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14

Similar content being viewed by others

References

  1. Kolda, T.G., Lewis, R.M., Torczon, V.: Optimization by direct search: new perspectives on some classical and modern methods. SIAM Rev. 45(3), 385–482 (2003)

    Article  MathSciNet  MATH  Google Scholar 

  2. Conn, A.R., Le Digabel, S.: Use of quadratic models with mesh-adaptive direct search for constrained black box optimization. Optim. Methods Softw. 28(1), 139–158 (2013)

    Article  MathSciNet  MATH  Google Scholar 

  3. Davis, E., Ierapetritou, M.: A kriging-based approach to MINLP containing black-box models and noise. Ind. Eng. Chem. Res. 47(16), 6101–6125 (2008)

    Article  Google Scholar 

  4. Jones, D.R., Schonlau, M., Welch, W.J.: Efficient global optimization of expensive black-box functions. J. Glob. Optim. 13(4), 455–492 (1998)

    Article  MathSciNet  MATH  Google Scholar 

  5. Regis, R.G.: Constrained optimization by radial basis function interpolation for high-dimensional expensive black-box problems with infeasible initial points. Eng. Optim. 46(2), 218–243 (2014)

    Article  MathSciNet  Google Scholar 

  6. Audet, C., Dennis Jr., J.E.: A progressive barrier for derivative-free nonlinear programming. SIAM J. Optim. 20(1), 445–472 (2009)

    Article  MathSciNet  MATH  Google Scholar 

  7. Boukouvala, F., Ierapetritou, M.G.: Derivative-free optimization for expensive constrained problems using a novel expected improvement objective function. AIChE J. 60(7), 2462–2474 (2014)

    Article  Google Scholar 

  8. Conn, A.R., Scheinberg, K., Vicente, L.N.: Introduction to Derivative-Free Optimization. SIAM, Philadelphia (2009)

    Book  MATH  Google Scholar 

  9. Echebest, N., Schuverdt, M.L., Vignau, R.P.: A derivative-free method for solving box-constrained underdetermined nonlinear systems of equations. Appl. Math. Comput. 219(6), 3198–3208 (2012)

    MathSciNet  MATH  Google Scholar 

  10. Le Thi, H.A., Vaz, A.I.F., Vicente, L.N.: Optimizing radial basis functions by d.c. programming and its use in direct search for global derivative-free optimization. Top 20(1), 190–214 (2012)

    Article  MathSciNet  MATH  Google Scholar 

  11. Rios, L.M., Sahinidis, N.V.: Derivative-free optimization: a review of algorithms and comparison of software implementations. J. Glob. Optim. 56(3), 1247–1293 (2013)

    Article  MathSciNet  MATH  Google Scholar 

  12. Boukouvala, F., Floudas, C.A.: ARGONAUT: AlgoRithms for global optimization of coNstrAined grey-box compUTational problems. Optim. Lett. 11(5), 895–913 (2017)

    Article  MathSciNet  MATH  Google Scholar 

  13. Eason, J.P., Biegler, L.T.: A trust region filter method for glass box/black box optimization. AIChE J. 62(9), 3124–3136 (2016)

    Article  Google Scholar 

  14. Amaran, S., et al.: Simulation optimization: a review of algorithms and applications. 4OR 12(4), 301–333 (2014)

    Article  MathSciNet  MATH  Google Scholar 

  15. Amaran, S., et al.: Simulation optimization: a review of algorithms and applications. Ann. Oper. Res. 240(1), 351–380 (2016)

    Article  MathSciNet  MATH  Google Scholar 

  16. Cozad, A., Sahinidis, N.V., Miller, D.C.: Learning surrogate models for simulation-based optimization. AIChE J. 60(6), 2211–2227 (2014)

    Article  Google Scholar 

  17. Jakobsson, S., et al.: A method for simulation based optimization using radial basis functions. Optim. Eng. 11(4), 501–532 (2010)

    Article  MathSciNet  MATH  Google Scholar 

  18. Quan, N., et al.: Simulation optimization via kriging: a sequential search using expected improvement with computing budget constraints. IIE Trans. 45(7), 763–780 (2013)

    Article  Google Scholar 

  19. Das, S., Suganthan, P.N.: Differential evolution: a survey of the state-of-the-art. IEEE Trans. Evol. Comput. 15(1), 4–31 (2011)

    Article  Google Scholar 

  20. Egea, J.A., Martí, R., Banga, J.R.: An evolutionary method for complex-process optimization. Comput. Oper. Res. 37(2), 315–324 (2010)

    Article  MATH  Google Scholar 

  21. Ong, Y.S., Nair, P.B., Keane, A.J.: Evolutionary optimization of computationally expensive problems via surrogate modeling. Aiaa J. 41(4), 687–696 (2003)

    Article  Google Scholar 

  22. Boukouvala, F., Misener, R., Floudas, C.A.: Global optimization advances in mixed-integer nonlinear programming, MINLP, and constrained derivative-free optimization, CDFO. Eur. J. Oper. Res. 252(3), 701–727 (2016)

    Article  MathSciNet  MATH  Google Scholar 

  23. Hooke, R., Jeeves, T.A.: Direct search solution of numerical and statistical problems. J. ACM 8(2), 212–229 (1961)

    Article  MATH  Google Scholar 

  24. Nedler, J.A., Mead, R.: A simplex method for function minimization. Comput. J. 7, 308–313 (1965)

    Article  MathSciNet  MATH  Google Scholar 

  25. Booker, A.J., et al.: A rigorous framework for optimization of expensive functions by surrogates. Struct. Multidiscip. Optim. 17(1), 1–13 (1999)

    Article  MathSciNet  Google Scholar 

  26. Boukouvala, F., Ierapetritou, M.G.: Surrogate-based optimization of expensive flowsheet modeling for continuous pharmaceutical manufacturing. J. Pharm. Innov. 8(2), 131–145 (2013)

    Article  Google Scholar 

  27. Caballero, J.A., Grossmann, I.E.: An algorithm for the use of surrogate models in modular flowsheet optimization. AIChE J. 54(10), 2633–2650 (2008)

    Article  Google Scholar 

  28. Ciaurri, D.E., Mukerji, T., Durlofsky, L.J.: Derivative-free optimization for oil field operations. In: Yang, X.-S., Koziel, S. (eds.) Computational Optimization and Applications in Engineering and Industry, pp. 19–55. Springer, Berlin (2011)

    Chapter  Google Scholar 

  29. Egea, J.A., et al.: Scatter search for chemical and bio-process optimization. J. Glob. Optim. 37(3), 481–503 (2007)

    Article  MathSciNet  MATH  Google Scholar 

  30. Torn, A., Zilinskas, A.: Global Optimization. Springer, New York (1989)

    Book  MATH  Google Scholar 

  31. Jones, D.R.: A taxonomy of global optimization methods based on response surfaces. J. Glob. Optim. 21(4), 345–383 (2001)

    Article  MathSciNet  MATH  Google Scholar 

  32. Powell, M.J.D.: A direct search optimization method that models the objective and constraint functions by linear interpolation. In: Gomez, S., Hennart, J.-P. (eds.) Advances in Optimization and Numerical Analysis, p. 51–67. Springer, Dordrecht (1994)

  33. Powell, M.J.: The BOBYQA Algorithm for Bound Constrained Optimization Without Derivatives. Cambridge NA Report NA2009/06. University of Cambridge, Cambridge (2009)

  34. Wilson, Z.T., Sahinidis, N.V.: The ALAMO approach to machine learning. Comput. Chem. Eng. 106, 785–795 (2017)

    Article  Google Scholar 

  35. Davis, E., Ierapetritou, M.: A kriging based method for the solution of mixed-integer nonlinear programs containing black-box functions. J. Glob. Optim. 43(2–3), 191–205 (2009)

    Article  MathSciNet  MATH  Google Scholar 

  36. Palmer, K., Realff, M.: Optimization and validation of steady-state flowsheet simulation metamodels. Chem. Eng. Res. Des. 80(7), 773–782 (2002)

    Article  Google Scholar 

  37. Boukouvala, F., Hasan, M.F., Floudas, C.A.: Global optimization of general constrained grey-box models: new method and its application to constrained PDEs for pressure swing adsorption. J. Global Optim. 67(1–2), 3–42 (2017)

    Article  MathSciNet  MATH  Google Scholar 

  38. Gutmann, H.-M.: A radial basis function method for global optimization. J. Glob. Optim. 19(3), 201–227 (2001)

    Article  MathSciNet  MATH  Google Scholar 

  39. Muller, J., Shoemaker, C.A., Piche, R.: SO-MI: a surrogate model algorithm for computationally expensive nonlinear mixed-integer black-box global optimization problems. Comput. Oper. Res. 40(5), 1383–1400 (2013)

    Article  MathSciNet  MATH  Google Scholar 

  40. Regis, R.G., Shoemaker, C.A.: A quasi-multistart framework for global optimization of expensive functions using response surface models. J. Glob. Optim. 56(4), 1719–1753 (2013)

    Article  MathSciNet  MATH  Google Scholar 

  41. Eason, J., Cremaschi, S.: Adaptive sequential sampling for surrogate model generation with artificial neural networks. Comput. Chem. Eng. 68, 220–232 (2014)

    Article  Google Scholar 

  42. Fahmi, I., Cremaschi, S.: Process synthesis of biodiesel production plant using artificial neural networks as the surrogate models. Comput. Chem. Eng. 46, 105–123 (2012)

    Article  Google Scholar 

  43. Henao, C.A., Maravelias, C.T.: Surrogate-based process synthesis. Comput. Aided Chem. Eng. 28, 1129–1134 (2010)

    Article  Google Scholar 

  44. Martelli, E., Amaldi, E.: PGS-COM: a hybrid method for constrained non-smooth black-box optimization problems: Brief review, novel algorithm and comparative evaluation. Comput. Chem. Eng. 63, 108–139 (2014)

    Article  Google Scholar 

  45. Novak, E., et al.: Smolyak/sparse grid algorithms. Tractability Multivar. Probl. Std. Inf. Funct. 12, 320–397 (2010)

    Google Scholar 

  46. Plaskota, L., Wasilkowski, G.W.: Smolyak’s algorithm for integration and L-1-approximation of multivariate functions with bounded mixed derivatives of second order. Numer. Algorithms 36(3), 229–246 (2004)

    Article  MathSciNet  MATH  Google Scholar 

  47. Bungartz, H.J., Dirnstorfer, S.: Multivariate quadrature on adaptive sparse grids. Computing 71(1), 89–114 (2003)

    Article  MathSciNet  MATH  Google Scholar 

  48. Bungartz, H.J., Dirnstorfer, S.: Higher order quadrature on sparse grids. In: Computational Science—Iccs 2004. In: Bubak, M., et al. (Ed.) Proceedings, p. 394–401 (2004)

  49. Bungartz, H.-J., Pfluger, D., Zimmer, S.: Adaptive sparse grid techniques for data mining. Springer, Berlin (2008)

    Book  Google Scholar 

  50. Harding, B.: Adaptive Sparse Grids and Extrapolation Techniques. In: Garcke, J., Pfluger, D. (eds.) Sparse Grids and Applications—Stuttgart 2014. Lecture Notes in Computational Science and Engineering, vol 109. Springer, Berlin (2016)

  51. Jiang, Y., Xu, Y.S.: B-spline quasi-interpolation on sparse grids. J. Complex. 27(5), 466–488 (2011)

    Article  MathSciNet  MATH  Google Scholar 

  52. Pfluger, D., Peherstorfer, B., Bungartz, H.J.: Spatially adaptive sparse grids for high-dimensional data-driven problems. J. Complex. 26(5), 508–522 (2010)

    Article  MathSciNet  MATH  Google Scholar 

  53. Sickel, W., Ullrich, T.: Spline interpolation on sparse grids. Appl. Anal. 90(3–4), 337–383 (2011)

    Article  MathSciNet  MATH  Google Scholar 

  54. Zenger, C.: Sparse grids. In: Hackbusch, W. (ed.) Parallel algorithms for partial differential equations. Proceedings of the Sixth GAMM Seminar. Notes on numerical fluid mechanics. Braunschweig, Verlag Vieweg, vol 31 (1991)

  55. Novak, E., Ritter, K.: Global optimization using hyperbolic cross points. In: Floudas, C.A., Pardalos, P.M. (eds.) State of the Art in Global Optimization: Computational Methods and Applications. Springer, Boston (1996)

    Google Scholar 

  56. Smolyak, S.A.: Quadrature and interpolation formulas for tensor products of certain classes of functions. In: Dokl. Akad. Nauk SSSR (1963)

  57. Gerstner, T., Griebel, M.: Numerical integration using sparse grids. Numer. Algorithms 18(3–4), 209–232 (1998)

    Article  MathSciNet  MATH  Google Scholar 

  58. Dung, D.: Sampling and cubature on sparse grids based on a B-spline quasi-interpolation. Found. Comput. Math. 16(5), 1193–1240 (2016)

    Article  MathSciNet  MATH  Google Scholar 

  59. Tang, J.J., et al.: Dimension-adaptive sparse grid interpolation for uncertainty quantification in modern power systems: probabilistic power flow. IEEE Trans. Power Syst. 31(2), 907–919 (2016)

    Article  Google Scholar 

  60. Peherstorfer, B., et al.: Selected recent applications of sparse grids. Numer. Math. Theory Methods Appl. 8(1), 47–77 (2015)

    Article  MathSciNet  MATH  Google Scholar 

  61. Gajda, P.: Smolyak’s algorithm for weighted L-1-approximation of multivariate functions with bounded rth mixed derivatives over R-d. Numer. Algorithms 40(4), 401–414 (2005)

    Article  MathSciNet  MATH  Google Scholar 

  62. Xu, G.Q.: On weak tractability of the Smolyak algorithm for approximation problems. J. Approx. Theory 192, 347–361 (2015)

    Article  MathSciNet  MATH  Google Scholar 

  63. Wasilkowski, G.W., Wozniakowski, H.: Explicit cost bounds of algorithms for multivariate tensor product problems. J. Complex. 11(1), 1–56 (1995)

    Article  MathSciNet  MATH  Google Scholar 

  64. Valentin, J., Pfluger, D.: Hierarchical gradient-based optimization with b-splines on sparse grids. In: Garcke, J., Pfluger, D. (Eds.) Sparse Grids and Applications—Stuttgart 2014. Lecture Notes in Computational Science and Engineering, vol 109. Springer, Cham (2016)

  65. Grimstad, B., Sandnes, A.: Global optimization with spline constraints: a new branch-and-bound method based on B-splines. J. Glob. Optim. 65(3), 401–439 (2016)

    Article  MathSciNet  MATH  Google Scholar 

  66. Hulsmann, M., Reith, D.: SpaGrOW-A derivative-free optimization scheme for intermolecular force field parameters based on sparse grid methods. Entropy 15(9), 3640–3687 (2013)

    Article  MathSciNet  MATH  Google Scholar 

  67. Sankaran, S.: Stochastic optimization using a sparse grid collocation scheme. Probab. Eng. Mech. 24(3), 382–396 (2009)

    Article  Google Scholar 

  68. Chen, P., Quarteroni, A.: A new algorithm for high-dimensional uncertainty quantification based on dimension-adaptive sparse grid approximation and reduced basis methods. J. Comput. Phys. 298, 176–193 (2015)

    Article  MathSciNet  MATH  Google Scholar 

  69. Pronzato, L., Müller, W.G.: Design of computer experiments: space filling and beyond. Stat. Comput. 22(3), 681–701 (2012)

    Article  MathSciNet  MATH  Google Scholar 

  70. McKay, M.D., Beckman, R.J., Conover, W.J.: A comparison of three methods for selecting values of input variables in the analysis of output from a computer code. Technometrics 21(2), 239–245 (1979)

    MathSciNet  MATH  Google Scholar 

  71. Davis, E., Ierapetritou, M.: A centroid-based sampling strategy for kriging global modeling and optimization. AIChE J. 56(1), 220–240 (2010)

    Google Scholar 

  72. Owen, A.B.: Orthogonal arrays for computer experiments, integration and visualization. Stat. Sin. 2(2), 439–452 (1992)

    MathSciNet  MATH  Google Scholar 

  73. Garud, S.S., Karimi, I.A., Kraft, M.: Smart sampling algorithm for surrogate model development. Comput. Chem. Eng. 96, 103–114 (2017)

    Article  Google Scholar 

  74. Fornberg, B., Zuev, J.: The Runge phenomenon and spatially variable shape parameters in RBF interpolation. Comput. Math Appl. 54(3), 379–398 (2007)

    Article  MathSciNet  MATH  Google Scholar 

  75. Runge, C.: Über empirische Funktionen und die Interpolation zwischen äquidistanten Ordinaten. Zeitschrift für Mathematik und Physik 46(224–243), 20 (1901)

    MATH  Google Scholar 

  76. Xiang, S., Chen, X., Wang, H.: Error bounds for approximation in Chebyshev points. Numer. Math. 116(3), 463–491 (2010)

    Article  MathSciNet  MATH  Google Scholar 

  77. Barthelmann, V., Novak, E., Ritter, K.: High dimensional polynomial interpolation on sparse grids. Adv. Comput. Math. 12(4), 273–288 (2000)

    Article  MathSciNet  MATH  Google Scholar 

  78. Davis, P.J., Rabinowitz, P.: Methods of Numerical Integration. Dover Publications (2007)

  79. Judd, K.L., et al.: Smolyak method for solving dynamic economic models: lagrange interpolation, anisotropic grid and adaptive domain. J. Econ. Dyn. Control 44, 92–123 (2014)

    Article  MathSciNet  Google Scholar 

  80. Trefethen, N.: Six myths of polynomial interpolation and quadrature. Math. Today 47, 184–188 (2011)

    MathSciNet  Google Scholar 

Download references

Acknowledgements

The authors would like to acknowledge the contribution of the late Professor Christodoulos A Floudas who introduced the authors to this area. CAF was a visionary and pioneer in the areas of global optimization, computational biology, energy systems, grey-box optimization and machine learning. This work was initiated when CAK and FB were postdoctoral associates in the laboratory of CAF, but was completed after his passing when they joined Georgia Institute of Technology.

Author information

Authors and Affiliations

  1. Wallace H. Coulter Department of Biomedical Engineering, Georgia Institute of Technology, Atlanta, USA

    Chris A. Kieslich

  2. School of Chemical and Biomolecular Engineering, Georgia Institute of Technology, Atlanta, USA

    Fani Boukouvala

  3. Texas A&M Energy Institute, Texas A&M University, College Station, USA

    Christodoulos A. Floudas

  4. Department of Chemical Engineering, Texas A&M University, College Station, USA

    Christodoulos A. Floudas

Authors
  1. Chris A. Kieslich
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Fani Boukouvala
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Christodoulos A. Floudas
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Fani Boukouvala.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Kieslich, C.A., Boukouvala, F. & Floudas, C.A. Optimization of black-box problems using Smolyak grids and polynomial approximations. J Glob Optim 71, 845–869 (2018). https://doi.org/10.1007/s10898-018-0643-0

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10898-018-0643-0

Keywords

Search

Navigation