Abstract
Bayesian optimization (BO) has recently emerged as a powerful and flexible tool for hyper-parameter tuning and more generally for the efficient global optimization of expensive black-box functions. Systems implementing BO have successfully solved difficult problems in automatic design choices and machine learning hyper-parameters tunings. Many recent advances in the methodologies and theories underlying Bayesian optimization have extended the framework to new applications and provided greater insights into the behavior of these algorithms. Still, these established techniques always require a user-defined space to perform optimization. This pre-defined space specifies the ranges of hyper-parameter values. In many situations, however, it can be difficult to prescribe such spaces, as a prior knowledge is often unavailable. Setting these regions arbitrarily can lead to inefficient optimization—if a space is too large, we can miss the optimum with a limited budget, and on the other hand, if a space is too small, it may not contain the optimum point that we want to get. The unknown search space problem is intractable to solve in practice. Therefore, in this paper, we narrow down to consider specifically the setting of “weakly specified” search space for Bayesian optimization. By weakly specified space, we mean that the pre-defined space is placed at a sufficiently good region so that the optimization can expand and reach to the optimum. However, this pre-defined space need not include the global optimum. We tackle this problem by proposing the filtering expansion strategy for Bayesian optimization. Our approach starts from the initial region and gradually expands the search space. We develop an efficient algorithm for this strategy and derive its regret bound. These theoretical results are complemented by an extensive set of experiments on benchmark functions and two real-world applications which demonstrate the benefits of our proposed approach.










Similar content being viewed by others
Notes
A trivial range of is too large and defect the purpose of Bayesian optimization by easily exceeding the evaluation budget.
References
Avettand-Fènoël M-N, Taillard R (2016) Effect of a pre or postweld heat treatment on microstructure and mechanical properties of an aa2050 weld obtained by ssfsw. Mater Des 89:348–361
Balachandran PV, Xue D, Theiler J, Hogden J, Lookman T (2016) Adaptive strategies for materials design using uncertainties. Sci Rep 6:19660
Bogunovic I, Scarlett J, Krause A, Cevher V (2016) Truncated variance reduction: a unified approach to bayesian optimization and level-set estimation. In: Advances in neural information processing systems, pp 1507–1515
Breiman L (2001) Random forests. Mach Learn 45(1):5–32
Brochu E, Cora VM, De Freitas N (2010) A tutorial on bayesian optimization of expensive cost functions, with application to active user modeling and hierarchical reinforcement learning. arXiv preprint arXiv:1012.2599
Adam DB (2011) Convergence rates of efficient global optimization algorithms. J Mach Learn Res 12:2879–2904
Contal E, Buffoni D, Robicquet A, Vayatis N (2013) Parallel gaussian process optimization with upper confidence bound and pure exploration. In: Machine learning and knowledge discovery in databases. Springer, pp 225–240
Nguyen TD, Gupta S, Rana S, Nguyen V, Venkatesh S, Deane KJ, Sanders PG (2016) Cascade Bayesian optimization. In: Australasian joint conference on artificial intelligence. Springer, pp 268–280
Frazier PI, Wang J (2016) Bayesian optimization for materials design. In: Information science for materials discovery and design. Springer, pp 45–75
Freitas ND, Zoghi M, Smola AJ (2012) Exponential regret bounds for gaussian process bandits with deterministic observations. In: Proceedings of the 29th international conference on machine learning (ICML-12), pp 1743–1750
González J, Dai Z, Hennig P, Lawrence ND (2016) Batch Bayesian optimization via local penalization. In: International conference on artificial intelligence and statistics, pp 648–657
Gonzalez J, Osborne M, Lawrence N (2016) Glasses: relieving the myopia of Bayesian optimisation. In: Proceedings of the 19th international conference on artificial intelligence and statistics, pp 790–799
Hennig P, Schuler CJ (2012) Entropy search for information-efficient global optimization. J Mach Learn Res 13:1809–1837
Hernández-Lobato JM, Hoffman MW, Ghahramani Z (2014) Predictive entropy search for efficient global optimization of black-box functions. In: Advances in neural information processing systems, pp 918–926
Hernández-Lobato JM, Requeima J, Pyzer-Knapp EO, Aspuru-Guzik A (2017) Parallel and distributed thompson sampling for large-scale accelerated exploration of chemical space. In: International conference on machine learning, pp 1470–1479
Hoffman M, Brochu E, de Freitas N (2011) Portfolio allocation for bayesian optimization. In: Proceedings of the 27th conference on uncertainty in artificial intelligence. AUAI Press, pp 327–336
Hutter F, Hoos HH, Leyton-Brown K (2011) Sequential model-based optimization for general algorithm configuration. In: Learning and intelligent optimization. Springer, pp 507–523
Donald RJ (2001) A taxonomy of global optimization methods based on response surfaces. J Glob Optim 21(4):345–383
Donald RJ, Perttunen CD, Stuckman BE (1993) Lipschitzian optimization without the lipschitz constant. J Optim Theory Appl 79(1):157–181
Kampmann R, Wagner R (1983) Kinetics of precipitation in metastable binary alloys-theory and applications to cu-1.9 at% ti and ni-14 at% al. In: Proceedings of the 2nd acta-scripta metallurgica conference on decomposition of alloys: the early stages, pp 91–103
Le T, Nguyen K, Nguyen V, Nguyen TD, Phung D (2017) Gogp: fast online regression with gaussian processes. In: IEEE 17th international conference on data mining (ICDM)
Ph Lequeu KP, Smith KP, Daniélou A (2010) Aluminum-copper-lithium alloy 2050 developed for medium to thick plate. J Mater Eng Perform 19(6):841–847
Li C, Gupta S, Rana S, Nguyen V, Venkatesh S, Shilton A (2017) High dimensional bayesian optimization using dropout. In: Proceedings of the 26th international joint conference on artificial intelligence, pp 2096–2102
Li C, Rana S, Gupta S, Nguyen V, Venkatesh S (2017) Bayesian optimization with monotonicity information. In: Workshop on Bayesian optimization at neural information processing systems (NIPSW)
Mockus J, Tiesis V, Zilinskas A (1978) The application of bayesian methods for seeking the extremum. Towards Glob Optim 2(117–129):2
Nguyen V, Gupta S, Rana S, Li C, Venkatesh S (2016) A Bayesian nonparametric approach for multi-label classification. In: Proceedings of The 8th Asian conference on machine learning (ACML), pp 254–269
Nguyen V, Gupta S, Rana S, Li C, Venkatesh S (2016) Think globally, act locally: a local strategy for Bayesian optimization. In: Workshop on Bayesian optimization at neural information processing systems (NIPSW)
Nguyen V, Gupta S, Rana S, Li C, Venkatesh S (2017) Bayesian optimization in weakly specified search space. In: IEEE 17th international conference on data mining (ICDM)
Nguyen V, Gupta S, Rana S, Li C, Venkatesh S (2017) Predictive variance reduction search. In: Workshop on Bayesian optimization at neural information processing systems (NIPSW)
Nguyen V, Gupta S, Rana S, Li C, Venkatesh S (2017) Regret for expected improvement over the best-observed value and stopping condition. In: Proceedings of The 9th Asian conference on machine learning (ACML), pp 279–294
Nguyen V, Rana S, Gupta SK, Li C, Venkatesh S (2016) Budgeted batch Bayesian optimization. In: 16th International conference on data mining (ICDM), pp 1107–1112
Joaquin Q-C, Edward RC (2005) A unifying view of sparse approximate gaussian process regression. J Mach Learn Res 6:1939–1959
Rana S, Li C, Gupta S, Nguyen V, Venkatesh S (2017) High dimensional Bayesian optimization with elastic gaussian process. In: Proceedings of the 34th international conference on machine learning (ICML), pp 2883–2891
Rasmussen CE (2004) Gaussian processes in machine learning. Advanced lectures on machine learning. Springer, Berlin, Heidelberg, pp 63–71
Ratschek H, Voller RL (1990) Global optimization over unbounded domains. SIAM J Control Optim 28(3):528–539
Shahriari B, Bouchard-Cote A, de Freitas N (2016) Unbounded Bayesian optimization via regularization. In: Proceedings of the 19th international conference on artificial intelligence and statistics, pp 1168–1176
Shahriari B, Swersky K, Wang Z, Adams RP, de Freitas N (2016) Taking the human out of the loop: a review of Bayesian optimization. Proc IEEE 104(1):148–175
Snoek J, Larochelle H, Adams RP (2012) Practical Bayesian optimization of machine learning algorithms. In: Advances in neural information processing systems, pp 2951–2959
Snoek J, Rippel O, Swersky K, Kiros R, Satish N, Sundaram N, Patwary M , Prabhat M, Adams R (2015) Scalable bayesian optimization using deep neural networks. In: Proceedings of the 32nd international conference on machine learning, pp 2171–2180
Springenberg JT, Klein A, Falkner S, Hutter F (2016) Bayesian optimization with robust bayesian neural networks. In: Advances in neural information processing systems, pp 4134–4142
Srinivas N, Krause A, Kakade S, Seeger M (2010) Gaussian process optimization in the bandit setting: No regret and experimental design. In: Proceedings of the 27th international conference on machine learning, pp 1015–1022
Thornton C, Hutter F, Hoos HH, Leyton-Brown K (2013) Auto-weka: combined selection and hyperparameter optimization of classification algorithms. In: Proceedings of the 19th ACM SIGKDD international conference on Knowledge discovery and data mining. ACM, pp 847–855
Wang Z, Zhou B, Jegelka S (2016) Optimization as estimation with gaussian processes in bandit settings. In: Proceedings of the 19th international conference on artificial intelligence and statistics, pp 1022–1031
Wang Z, de Freitas N (2014) Theoretical analysis of bayesian optimisation with unknown Gaussian process hyper-parameters. arXiv preprint arXiv:1406.7758
Xue D, Balachandran PV, Hogden J, Theiler J, Xue D, Lookman T (2016) Accelerated search for materials with targeted properties by adaptive design. Nat Commun 7:11241
Acknowledgements
This research was partially funded by the Australian Government through the Australian Research Council (ARC) and the Telstra-Deakin Centre of Excellence in Big Data and Machine Learning. Prof Venkatesh is the recipient of an ARC Australian Laureate Fellowship (FL170100006).
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Nguyen, V., Gupta, S., Rana, S. et al. Filtering Bayesian optimization approach in weakly specified search space. Knowl Inf Syst 60, 385–413 (2019). https://doi.org/10.1007/s10115-018-1238-2
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10115-018-1238-2