Skip to main content
SpringerLink
Log in

Data-driven spatial branch-and-bound algorithms for box-constrained simulation-based optimization

  • Published:
Journal of Global Optimization Aims and scope Submit manuscript

Abstract

The ability to use complex computer simulations in quantitative analysis and decision-making is highly desired in science and engineering, at the same rate as computation capabilities and first-principle knowledge advance. Due to the complexity of simulation models, direct embedding of equation-based optimization solvers may be impractical and data-driven optimization techniques are often needed. In this work, we present a novel data-driven spatial branch-and-bound algorithm for simulation-based optimization problems with box constraints, aiming for consistent globally convergent solutions. The main contribution of this paper is the introduction of the concept data-driven convex underestimators of data and surrogate functions, which are employed within a spatial branch-and-bound algorithm. The algorithm is showcased by an illustrative example and is then extensively studied via computational experiments on a large set of benchmark problems.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14

Similar content being viewed by others

Data availability

My manuscript data will be available as supplementary materials.

References

  1. Amaran, S., Sahinidis, N.V., Sharda, B., Bury, S.J.: Simulation optimization: a review of algorithms and applications. 4OR 12(4), 301–333 (2014). https://doi.org/10.1007/s10288-014-0275-2

    Article  MathSciNet  MATH  Google Scholar 

  2. Gosavi, A.: Background. In: Gosavi, A. (ed.) Simulation-Based Optimization: Parametric Optimization Techniques and Reinforcement Learning, pp. 1–12. Springer, US, Boston, MA (2015)

    MATH  Google Scholar 

  3. Caballero, J.A., Grossmann, I.E.: An algorithm for the use of surrogate models in modular flowsheet optimization. AIChE J. 54(10), 2633–2650 (2008). https://doi.org/10.1002/aic.11579

    Article  Google Scholar 

  4. Henao, C.A., Maravelias, C.T.: Surrogate-based superstructure optimization framework. AIChE J. 57(5), 1216–1232 (2011). https://doi.org/10.1002/aic.12341

    Article  Google Scholar 

  5. Bhosekar, A., Ierapetritou, M.: Advances in surrogate based modeling, feasibility analysis, and optimization: a review. Comput. Chem. Eng. 108, 250–267 (2018). https://doi.org/10.1016/j.compchemeng.2017.09.017

    Article  Google Scholar 

  6. McBride, K., Sundmacher, K.: Overview of surrogate modeling in chemical process engineering. Chem. Ing. Tec. 91(3), 228–239 (2019). https://doi.org/10.1002/cite.201800091

    Article  Google Scholar 

  7. Bajaj, I., Hasan, M.M.F.: Deterministic global derivative-free optimization of black-box problems with bounded Hessian. Optim. Lett. (2019). https://doi.org/10.1007/s11590-019-01421-0

    Article  MATH  Google Scholar 

  8. Kieslich, C.A., Boukouvala, F., Floudas, C.A.: Optimization of black-box problems using Smolyak grids and polynomial approximations. J. Global Optim. (2018). https://doi.org/10.1007/s10898-018-0643-0

    Article  MathSciNet  MATH  Google Scholar 

  9. Cozad, A., Sahinidis, N.V., Miller, D.C.: Learning surrogate models for simulation-based optimization. AIChE J. 60(6), 2211–2227 (2014). https://doi.org/10.1002/aic.14418

    Article  Google Scholar 

  10. Schweidtmann, A.M., Mitsos, A.: Deterministic global optimization with artificial neural networks embedded. J. Optim. Theory Appl. 180(3), 925–948 (2019). https://doi.org/10.1007/s10957-018-1396-0

    Article  MathSciNet  MATH  Google Scholar 

  11. Garud, S.S., Mariappan, N., Karimi, I.A.: Surrogate-based black-box optimisation via domain exploration and smart placement. Comput. Chem. Eng. 130, 106567 (2019). https://doi.org/10.1016/j.compchemeng.2019.106567

    Article  Google Scholar 

  12. Nelder, J.A., Mead, R.: A simplex method for function minimization. Comput. J. 7(4), 308–313 (1965). https://doi.org/10.1093/comjnl/7.4.308

    Article  MathSciNet  MATH  Google Scholar 

  13. Torczon, V.: On the convergence of pattern search algorithms. SIAM J. Optim. 7(1), 1–25 (1997). https://doi.org/10.1137/s1052623493250780

    Article  MathSciNet  MATH  Google Scholar 

  14. Lewis, R.M., Torczon, V.: Pattern search algorithms for bound constrained minimization. SIAM J. Optim. 9(4), 1082–1099 (1999). https://doi.org/10.1137/s1052623496300507

    Article  MathSciNet  MATH  Google Scholar 

  15. Audet, C., Dennis, J.E.: Mesh adaptive direct search algorithms for constrained optimization. SIAM J. Optim. 17(1), 188–217 (2006). https://doi.org/10.1137/040603371

    Article  MathSciNet  MATH  Google Scholar 

  16. Jones, D.R.: Direct global optimization algorithm direct global optimization algorithm. In: Floudas, C.A., Pardalos, P.M. (eds.) Encyclopedia of Optimization, pp. 725–735. Springer, US, Boston, MA (2009)

    Chapter  Google Scholar 

  17. Jones, D.R., Perttunen, C.D., Stuckman, B.E.: Lipschitzian optimization without the Lipschitz constant. J. Optim. Theory Appl. 79(1), 157–181 (1993). https://doi.org/10.1007/bf00941892

    Article  MathSciNet  MATH  Google Scholar 

  18. Huyer, W., Neumaier, A.: Global optimization by multilevel coordinate search. J. Global Optim. 14(4), 331–355 (1999). https://doi.org/10.1023/a:1008382309369

    Article  MathSciNet  MATH  Google Scholar 

  19. Kirkpatrick, S., Gelatt, C.D., Vecchi, M.P.: Optimization by simulated annealing. Science 220(4598), 671–680 (1983). https://doi.org/10.1126/science.220.4598.671

    Article  MathSciNet  MATH  Google Scholar 

  20. Reeves, C.R.: Feature article-genetic algorithms for the operations researcher. INFORMS J. Comput. 9(3), 231–250 (1997). https://doi.org/10.1287/ijoc.9.3.231

    Article  MATH  Google Scholar 

  21. Whitley, D.: A genetic algorithm tutorial. Stat. Comput. 4(2), 65–85 (1994). https://doi.org/10.1007/bf00175354

    Article  Google Scholar 

  22. Kennedy, J., Eberhart, R.: Particle swarm optimization. In: Proceedings of ICNN'95-International Conference on Neural Networks, vol. 1944, pp. 1942–1948, 27 November-1 December 1995

  23. Rios, L.M., Sahinidis, N.V.: Derivative-free optimization: a review of algorithms and comparison of software implementations. J. Global Optim. 56(3), 1247–1293 (2013). https://doi.org/10.1007/s10898-012-9951-y

    Article  MathSciNet  MATH  Google Scholar 

  24. Conn, A.R., Scheinberg, K., Vicente, L.N.: Introduction to Derivative-free Optimization. Society for Industrial and Applied Mathematics (SIAM, 3600 Market Street, Floor 6, Philadelphia, PA 19104), (2009)

  25. Boukouvala, F., Misener, R., Floudas, C.A.: Global optimization advances in mixed-integer nonlinear programming, MINLP, and constrained derivative-free optimization CDFO. Eur. J. Oper. Res. 252(3), 701–727 (2016). https://doi.org/10.1016/j.ejor.2015.12.018

    Article  MathSciNet  MATH  Google Scholar 

  26. Larson, J., Menickelly, M., Wild, S.M.: Derivative-free optimization methods. Acta Numer 28, 287–404 (2019). https://doi.org/10.1017/S0962492919000060

    Article  MathSciNet  MATH  Google Scholar 

  27. Audet, C., Hare, W.: The beginnings of DFO algorithms. In: Derivative-Free and Blackbox Optimization. pp. 33–54. Springer, Cham (2017)

  28. Cox, D.D., John, S.: A statistical method for global optimization. In: [Proceedings] 1992 IEEE International Conference on Systems, Man, and Cybernetics, vol. 1242, pp. 1241–1246, 18–21 October 1992

  29. Jones, D.R., Schonlau, M., Welch, W.J.: Efficient global optimization of expensive black-box functions. J. Global Optim. 13(4), 455–492 (1998). https://doi.org/10.1023/A:1008306431147

    Article  MathSciNet  MATH  Google Scholar 

  30. Huyer, W., Neumaier, A.: SNOBFIT-stable noisy optimization by branch and fit. ACM Trans. Math. Softw. 35(2), 1–25 (2008). https://doi.org/10.1145/1377612.1377613

    Article  MathSciNet  Google Scholar 

  31. Thebelt, A., Kronqvist, J., Mistry, M., Lee, R.M., Sudermann-Merx, N., Misener, R.: ENTMOOT: a framework for optimization over ensemble tree models. arXiv e-prints (2020).

  32. Boukouvala, F., Floudas, C.A.: ARGONAUT: algorithms for global optimization of constrained grey-box computational problems. Optim. Lett. 11(5), 895–913 (2017). https://doi.org/10.1007/s11590-016-1028-2

    Article  MathSciNet  MATH  Google Scholar 

  33. Müller, J., Park, J., Sahu, R., Varadharajan, C., Arora, B., Faybishenko, B., Agarwal, D.: Surrogate optimization of deep neural networks for groundwater predictions. J. Global Optim. (2020). https://doi.org/10.1007/s10898-020-00912-0

    Article  MATH  Google Scholar 

  34. Forrester, A.I.J., Keane, A.J.: Recent advances in surrogate-based optimization. Prog. Aerosp. Sci. 45(1), 50–79 (2009). https://doi.org/10.1016/j.paerosci.2008.11.001

    Article  Google Scholar 

  35. Barton, R.R., Meckesheimer, M.: Chapter 18 metamodel-based simulation optimization. In: Henderson, S.G., Nelson, B.L. (eds.) Handbooks in Operations Research and Management Science, vol. 13. pp. 535–574. Elsevier (2006)

  36. Eason, J., Cremaschi, S.: Adaptive sequential sampling for surrogate model generation with artificial neural networks. Comput. Chem. Eng. 68, 220–232 (2014). https://doi.org/10.1016/j.compchemeng.2014.05.021

    Article  Google Scholar 

  37. Tawarmalani, M., Sahinidis, N.V.: A polyhedral branch-and-cut approach to global optimization. Math. Program. 103(2), 225–249 (2005). https://doi.org/10.1007/s10107-005-0581-8

    Article  MathSciNet  MATH  Google Scholar 

  38. McKay, M.D., Beckman, R.J., Conover, W.J.: Comparison of three methods for selecting values of input variables in the analysis of output from a computer code. Technometrics 21(2), 239–245 (1979). https://doi.org/10.1080/00401706.1979.10489755

    Article  MathSciNet  MATH  Google Scholar 

  39. Hüllen, G., Zhai, J., Kim, S.H., Sinha, A., Realff, M.J., Boukouvala, F.: Managing uncertainty in data-driven simulation-based optimization. Comput. Chem. Eng. (2019). https://doi.org/10.1016/j.compchemeng.2019.106519

    Article  Google Scholar 

  40. Misener, R., Floudas, C.A.: ANTIGONE: algorithms for continuous/integer global optimization of nonlinear equations. J. Global Optim. 59(2), 503–526 (2014). https://doi.org/10.1007/s10898-014-0166-2

    Article  MathSciNet  MATH  Google Scholar 

  41. Androulakis, I.P., Maranas, C.D., Floudas, C.A.: αBB: A global optimization method for general constrained nonconvex problems. J. Global Optim. 7(4), 337–363 (1995). https://doi.org/10.1007/bf01099647

    Article  MathSciNet  MATH  Google Scholar 

  42. Tawarmalani, M., Sahinidis, N.V.: Global optimization of mixed-integer nonlinear programs: a theoretical and computational study. Math. Program. 99(3), 563–591 (2004). https://doi.org/10.1007/s10107-003-0467-6

    Article  MathSciNet  MATH  Google Scholar 

  43. Azar, M.G., Dyer, E.L., K, K.P., #246, rding: Convex relaxation regression: black-box optimization of smooth functions by learning their convex envelopes. Paper presented at the Proceedings of the Thirty-Second Conference on Uncertainty in Artificial Intelligence, Jersey City, New Jersey, USA

  44. Xu, W.L., Nelson, B.L.: Empirical stochastic branch-and-bound for optimization via simulation. IIE Trans. 45(7), 685–698 (2013). https://doi.org/10.1080/0740817X.2013.768783

    Article  Google Scholar 

  45. Sergeyev, Y.D., Kvasov, D.E.: A deterministic global optimization using smooth diagonal auxiliary functions. Commun. Nonlinear Sci. Numer. Simul. 21(1), 99–111 (2015). https://doi.org/10.1016/j.cnsns.2014.08.026

    Article  MathSciNet  MATH  Google Scholar 

  46. McCormick, G.P.: Computability of global solutions to factorable nonconvex programs: part I—convex underestimating problems. Math. Program. 10(1), 147–175 (1976). https://doi.org/10.1007/BF01580665

    Article  MATH  Google Scholar 

  47. Duran, M.A., Grossmann, I.E.: An outer-approximation algorithm for a class of mixed-integer nonlinear programs. Math. Program. 36(3), 307–339 (1986). https://doi.org/10.1007/BF02592064

    Article  MathSciNet  MATH  Google Scholar 

  48. Vapnik, V.: The Nature of Statistical Learning Theory. Springer, New York (1999)

    MATH  Google Scholar 

  49. Smola, A.J., Schölkopf, B.: A tutorial on support vector regression. Stat. Comput. 14(3), 199–222 (2004). https://doi.org/10.1023/b:stco.0000035301.49549.88

    Article  MathSciNet  Google Scholar 

  50. Oliphant, T.E.: A Guid to Numpy. Trelgol Publishing, USA (2006)

  51. Walt, S.V.D., Colbert, S.C., Varoquaux, G.: The Numpy array: a structure for efficient numerical computation. Comput. Sci. Eng. 13(2), 22–30 (2011). https://doi.org/10.1109/MCSE.2011.37

    Article  Google Scholar 

  52. Hart, W.E., Laird, C., Watson, J.-P., Woodruff, D.L.: Pyomo-Optimization Modeling in Python. Springer (2012)

    Book  Google Scholar 

  53. Baudin, M., Christopoulou, M., Collette, Y., Martinez, J.-M.: pyDOE: The experimental design package for python. In

  54. Pedregosa, F., Ga, #235, Varoquaux, l., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Prettenhofer, P., Weiss, R., Dubourg, V., Vanderplas, J., Passos, A., Cournapeau, D., Brucher, M., Perrot, M., #201, Duchesnay, d.: Scikit-learn: machine learning in Python. J. Mach. Learn. Res. 12, 2825–2830 (2011)

  55. Zhai, J., Boukouvala, F.: Nonlinear variable selection algorithms for surrogate modeling. AIChE J. (2019). https://doi.org/10.1002/aic.16601

    Article  Google Scholar 

  56. Guzman, Y.A.: Theoretical Advances in Robust Optimization, Feature Selection, and Biomarker Discovery. Academic dissertations (Ph.D.), Princeton University (2016)

  57. Bound-constrained programs. http://www.minlp.com/nlp-and-minlp-test-problems. 2019

Download references

Acknowledgements

The authors acknowledge financial support from the National Science Foundation (NSF-1805724) (JZ, FB), RAPID (FB) and Georgia Institute of Technology Startup Funding (JZ, FB).

Funding

National Science Foundation (NSF-1805724), RAPID and Georgia Institute of Technology Georgia Tech Startup Funding.

Author information

Authors and Affiliations

  1. School of Chemical and Biomolecular Engineering, Georgia Institute of Technology, 311 Ferst Dr., Atlanta, GA, 30332, USA

    Jianyuan Zhai & Fani Boukouvala

Authors
  1. Jianyuan Zhai
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Fani Boukouvala
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Fani Boukouvala.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Appendix

Appendix

1.1 Appendix 1: List of lower dimensional problems

See Table

Table 3 List of lower dimensional problems
Full size table

3.

1.2 Appendix 2: List of high dimensional problems

See Table

Table 4 List of higher dimenional problems
Full size table

4.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zhai, J., Boukouvala, F. Data-driven spatial branch-and-bound algorithms for box-constrained simulation-based optimization. J Glob Optim 82, 21–50 (2022). https://doi.org/10.1007/s10898-021-01045-8

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10898-021-01045-8

Keywords

Search

Navigation