Skip to main content
Log in

Filtering Bayesian optimization approach in weakly specified search space

  • Regular Paper
  • Published:
Knowledge and Information Systems Aims and scope Submit manuscript

Abstract

Bayesian optimization (BO) has recently emerged as a powerful and flexible tool for hyper-parameter tuning and more generally for the efficient global optimization of expensive black-box functions. Systems implementing BO have successfully solved difficult problems in automatic design choices and machine learning hyper-parameters tunings. Many recent advances in the methodologies and theories underlying Bayesian optimization have extended the framework to new applications and provided greater insights into the behavior of these algorithms. Still, these established techniques always require a user-defined space to perform optimization. This pre-defined space specifies the ranges of hyper-parameter values. In many situations, however, it can be difficult to prescribe such spaces, as a prior knowledge is often unavailable. Setting these regions arbitrarily can lead to inefficient optimization—if a space is too large, we can miss the optimum with a limited budget, and on the other hand, if a space is too small, it may not contain the optimum point that we want to get. The unknown search space problem is intractable to solve in practice. Therefore, in this paper, we narrow down to consider specifically the setting of “weakly specified” search space for Bayesian optimization. By weakly specified space, we mean that the pre-defined space is placed at a sufficiently good region so that the optimization can expand and reach to the optimum. However, this pre-defined space need not include the global optimum. We tackle this problem by proposing the filtering expansion strategy for Bayesian optimization. Our approach starts from the initial region and gradually expands the search space. We develop an efficient algorithm for this strategy and derive its regret bound. These theoretical results are complemented by an extensive set of experiments on benchmark functions and two real-world applications which demonstrate the benefits of our proposed approach.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

Notes

  1. A trivial range of is too large and defect the purpose of Bayesian optimization by easily exceeding the evaluation budget.

  2. https://www.sfu.ca/~ssurjano.

  3. https://github.com/ntienvu/ACML2016_BNMC.

  4. http://www.thermocalc.com.

References

  1. Avettand-Fènoël M-N, Taillard R (2016) Effect of a pre or postweld heat treatment on microstructure and mechanical properties of an aa2050 weld obtained by ssfsw. Mater Des 89:348–361

    Article  Google Scholar 

  2. Balachandran PV, Xue D, Theiler J, Hogden J, Lookman T (2016) Adaptive strategies for materials design using uncertainties. Sci Rep 6:19660

    Article  Google Scholar 

  3. Bogunovic I, Scarlett J, Krause A, Cevher V (2016) Truncated variance reduction: a unified approach to bayesian optimization and level-set estimation. In: Advances in neural information processing systems, pp 1507–1515

  4. Breiman L (2001) Random forests. Mach Learn 45(1):5–32

    Article  MATH  Google Scholar 

  5. Brochu E, Cora VM, De Freitas N (2010) A tutorial on bayesian optimization of expensive cost functions, with application to active user modeling and hierarchical reinforcement learning. arXiv preprint arXiv:1012.2599

  6. Adam DB (2011) Convergence rates of efficient global optimization algorithms. J Mach Learn Res 12:2879–2904

    MathSciNet  MATH  Google Scholar 

  7. Contal E, Buffoni D, Robicquet A, Vayatis N (2013) Parallel gaussian process optimization with upper confidence bound and pure exploration. In: Machine learning and knowledge discovery in databases. Springer, pp 225–240

  8. Nguyen TD, Gupta S, Rana S, Nguyen V, Venkatesh S, Deane KJ, Sanders PG (2016) Cascade Bayesian optimization. In: Australasian joint conference on artificial intelligence. Springer, pp 268–280

  9. Frazier PI, Wang J (2016) Bayesian optimization for materials design. In: Information science for materials discovery and design. Springer, pp 45–75

  10. Freitas ND, Zoghi M, Smola AJ (2012) Exponential regret bounds for gaussian process bandits with deterministic observations. In: Proceedings of the 29th international conference on machine learning (ICML-12), pp 1743–1750

  11. González J, Dai Z, Hennig P, Lawrence ND (2016) Batch Bayesian optimization via local penalization. In: International conference on artificial intelligence and statistics, pp 648–657

  12. Gonzalez J, Osborne M, Lawrence N (2016) Glasses: relieving the myopia of Bayesian optimisation. In: Proceedings of the 19th international conference on artificial intelligence and statistics, pp 790–799

  13. Hennig P, Schuler CJ (2012) Entropy search for information-efficient global optimization. J Mach Learn Res 13:1809–1837

    MathSciNet  MATH  Google Scholar 

  14. Hernández-Lobato JM, Hoffman MW, Ghahramani Z (2014) Predictive entropy search for efficient global optimization of black-box functions. In: Advances in neural information processing systems, pp 918–926

  15. Hernández-Lobato JM, Requeima J, Pyzer-Knapp EO, Aspuru-Guzik A (2017) Parallel and distributed thompson sampling for large-scale accelerated exploration of chemical space. In: International conference on machine learning, pp 1470–1479

  16. Hoffman M, Brochu E, de Freitas N (2011) Portfolio allocation for bayesian optimization. In: Proceedings of the 27th conference on uncertainty in artificial intelligence. AUAI Press, pp 327–336

  17. Hutter F, Hoos HH, Leyton-Brown K (2011) Sequential model-based optimization for general algorithm configuration. In: Learning and intelligent optimization. Springer, pp 507–523

  18. Donald RJ (2001) A taxonomy of global optimization methods based on response surfaces. J Glob Optim 21(4):345–383

    Article  MathSciNet  MATH  Google Scholar 

  19. Donald RJ, Perttunen CD, Stuckman BE (1993) Lipschitzian optimization without the lipschitz constant. J Optim Theory Appl 79(1):157–181

    Article  MathSciNet  MATH  Google Scholar 

  20. Kampmann R, Wagner R (1983) Kinetics of precipitation in metastable binary alloys-theory and applications to cu-1.9 at% ti and ni-14 at% al. In: Proceedings of the 2nd acta-scripta metallurgica conference on decomposition of alloys: the early stages, pp 91–103

  21. Le T, Nguyen K, Nguyen V, Nguyen TD, Phung D (2017) Gogp: fast online regression with gaussian processes. In: IEEE 17th international conference on data mining (ICDM)

  22. Ph Lequeu KP, Smith KP, Daniélou A (2010) Aluminum-copper-lithium alloy 2050 developed for medium to thick plate. J Mater Eng Perform 19(6):841–847

    Article  Google Scholar 

  23. Li C, Gupta S, Rana S, Nguyen V, Venkatesh S, Shilton A (2017) High dimensional bayesian optimization using dropout. In: Proceedings of the 26th international joint conference on artificial intelligence, pp 2096–2102

  24. Li C, Rana S, Gupta S, Nguyen V, Venkatesh S (2017) Bayesian optimization with monotonicity information. In: Workshop on Bayesian optimization at neural information processing systems (NIPSW)

  25. Mockus J, Tiesis V, Zilinskas A (1978) The application of bayesian methods for seeking the extremum. Towards Glob Optim 2(117–129):2

    MATH  Google Scholar 

  26. Nguyen V, Gupta S, Rana S, Li C, Venkatesh S (2016) A Bayesian nonparametric approach for multi-label classification. In: Proceedings of The 8th Asian conference on machine learning (ACML), pp 254–269

  27. Nguyen V, Gupta S, Rana S, Li C, Venkatesh S (2016) Think globally, act locally: a local strategy for Bayesian optimization. In: Workshop on Bayesian optimization at neural information processing systems (NIPSW)

  28. Nguyen V, Gupta S, Rana S, Li C, Venkatesh S (2017) Bayesian optimization in weakly specified search space. In: IEEE 17th international conference on data mining (ICDM)

  29. Nguyen V, Gupta S, Rana S, Li C, Venkatesh S (2017) Predictive variance reduction search. In: Workshop on Bayesian optimization at neural information processing systems (NIPSW)

  30. Nguyen V, Gupta S, Rana S, Li C, Venkatesh S (2017) Regret for expected improvement over the best-observed value and stopping condition. In: Proceedings of The 9th Asian conference on machine learning (ACML), pp 279–294

  31. Nguyen V, Rana S, Gupta SK, Li C, Venkatesh S (2016) Budgeted batch Bayesian optimization. In: 16th International conference on data mining (ICDM), pp 1107–1112

  32. Joaquin Q-C, Edward RC (2005) A unifying view of sparse approximate gaussian process regression. J Mach Learn Res 6:1939–1959

    MathSciNet  MATH  Google Scholar 

  33. Rana S, Li C, Gupta S, Nguyen V, Venkatesh S (2017) High dimensional Bayesian optimization with elastic gaussian process. In: Proceedings of the 34th international conference on machine learning (ICML), pp 2883–2891

  34. Rasmussen CE (2004) Gaussian processes in machine learning. Advanced lectures on machine learning. Springer, Berlin, Heidelberg, pp 63–71

    Chapter  Google Scholar 

  35. Ratschek H, Voller RL (1990) Global optimization over unbounded domains. SIAM J Control Optim 28(3):528–539

    Article  MathSciNet  MATH  Google Scholar 

  36. Shahriari B, Bouchard-Cote A, de Freitas N (2016) Unbounded Bayesian optimization via regularization. In: Proceedings of the 19th international conference on artificial intelligence and statistics, pp 1168–1176

  37. Shahriari B, Swersky K, Wang Z, Adams RP, de Freitas N (2016) Taking the human out of the loop: a review of Bayesian optimization. Proc IEEE 104(1):148–175

    Article  Google Scholar 

  38. Snoek J, Larochelle H, Adams RP (2012) Practical Bayesian optimization of machine learning algorithms. In: Advances in neural information processing systems, pp 2951–2959

  39. Snoek J, Rippel O, Swersky K, Kiros R, Satish N, Sundaram N, Patwary M , Prabhat M, Adams R (2015) Scalable bayesian optimization using deep neural networks. In: Proceedings of the 32nd international conference on machine learning, pp 2171–2180

  40. Springenberg JT, Klein A, Falkner S, Hutter F (2016) Bayesian optimization with robust bayesian neural networks. In: Advances in neural information processing systems, pp 4134–4142

  41. Srinivas N, Krause A, Kakade S, Seeger M (2010) Gaussian process optimization in the bandit setting: No regret and experimental design. In: Proceedings of the 27th international conference on machine learning, pp 1015–1022

  42. Thornton C, Hutter F, Hoos HH, Leyton-Brown K (2013) Auto-weka: combined selection and hyperparameter optimization of classification algorithms. In: Proceedings of the 19th ACM SIGKDD international conference on Knowledge discovery and data mining. ACM, pp 847–855

  43. Wang Z, Zhou B, Jegelka S (2016) Optimization as estimation with gaussian processes in bandit settings. In: Proceedings of the 19th international conference on artificial intelligence and statistics, pp 1022–1031

  44. Wang Z, de Freitas N (2014) Theoretical analysis of bayesian optimisation with unknown Gaussian process hyper-parameters. arXiv preprint arXiv:1406.7758

  45. Xue D, Balachandran PV, Hogden J, Theiler J, Xue D, Lookman T (2016) Accelerated search for materials with targeted properties by adaptive design. Nat Commun 7:11241

    Article  Google Scholar 

Download references

Acknowledgements

This research was partially funded by the Australian Government through the Australian Research Council (ARC) and the Telstra-Deakin Centre of Excellence in Big Data and Machine Learning. Prof Venkatesh is the recipient of an ARC Australian Laureate Fellowship (FL170100006).

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Vu Nguyen.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Nguyen, V., Gupta, S., Rana, S. et al. Filtering Bayesian optimization approach in weakly specified search space. Knowl Inf Syst 60, 385–413 (2019). https://doi.org/10.1007/s10115-018-1238-2

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10115-018-1238-2

Keywords

Navigation