Abstract
Bayesian optimization, which offers efficient parameter search, suffers from high computation cost if the parameters have high dimensionality because the search space expands and more trials are needed. One existing solution is an embedding method that enables the search to be restricted to a low-dimensional subspace, but this method works well only when the number of embedding dimensions closely match the that of effective dimensions, which affects the function value. However, in practical situations, the number of effective dimensions is unknown, and embedding into a low dimensional subspace to save computation cost often results in a search in a lower dimensional subspace than the effective dimensions. This study proposes a Bayesian optimization method that uses random embedding to remain efficient even if the embedded dimension is lower than the effective dimensions. By conducting parallel search in an initially low dimensional space and performing multiple cycles in which the search space is incrementally improved, the optimum solution can be efficiently found. An experiment on benchmark problems shows the effectiveness of the proposed method.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
We also checked the processing time of each method. PSRE and SRE have similar times, while RE is slower and BO is about ten times slower than PSRE.
References
Snoek, J., Larochelle, H., Adams, R.P.: Practical Bayesian optimization of machine learning algorithms. In: Advances in Neural Information Processing Systems (NIPS), pp. 2951–2959. Curran Associates Inc. (2012)
Kiyotake, H., Kohjima, M., Matsubayashi, T., Toda, H.: Multi agent flow estimation based on Bayesian optimization with time delay and low dimensional parameter conversion. In: Miller, T., Oren, N., Sakurai, Y., Noda, I., Savarimuthu, B.T.R., Cao Son, T. (eds.) PRIMA 2018. LNCS (LNAI), vol. 11224, pp. 53–69. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-03098-8_4
Brochu, E., Cora, V.M., de Freitas, N.: A tutorial on Bayesian optimization of expensive cost functions, with application to active user modeling and hierarchical reinforcement learning (2010). http://arXiv.org/abs/1012.2599
Jones, D., Schonlau, M., Welch, W.: Efficient global optimization of expensive black-box functions. J. Global Optim. 13(4), 455–492 (1998)
Rasmussen, C.E., Williams, C.K.: Gaussian Process for Machine Learning. MIT Press, Cambridge (2006)
Wang, Z., Hutter, F., Zoghi, M., Matheson, D., de Freitas, N.: Bayesian optimization in a billion dimensions via random embeddings. J. Artif. Intell. Res. (JAIR) 55, 361–387 (2016)
Kandasamy, K., Krishnamurthy, A., Schneider, J., Poczos, B.: Parallelised Bayesian optimisation via Thompson sampling. In: Proceedings of the Twenty-First International Conference on Artificial Intelligence and Statistics (AISTATS), vol. 84, pp. 133–142. PMLR (2018)
Kandasamy, K., Schneider, J., Poczos, B.: High dimensional Bayesian optimisation and bandits via additive models. In: Proceedings of the 32nd International Conference on Machine Learning (ICML), vol. 37, pp. 295–304 (2015)
Rolland, P., Scarlett, J., Bogunovic, I., Cevher, V.: High dimensional Bayesian optimization via additive models with overlapping groups. In: International Conference on Artificial Intelligence and Statistics (AISTATS), pp. 298–307 (2018)
Qian, H., Hu, Y., Yu, Y.: Derivative-free optimization of high-dimensional non-convex functions by sequential random embeddings. In: Proceedings of the Twenty-Fifth International Joint Conference on Artificial Intelligence (IJCAI), pp. 1946–1952. AAAI Press, New York (2016)
Bull, A.D.: Convergence rates of efficient global optimization algorithms. J. Mach. Learn. Res. 12(Oct), 2879–2904 (2011)
Vazquez, E., Bect, J.: Convergence properties of the expected improvement algorithm with fixed mean and covariance functions. J. Stat. Plann. Infer. 140(11), 3088–3095 (2010)
Mockus, J.: Bayesian Approach to Global Optimization: Theory and Applications, vol. 37. Springer, Dordrecht (2012)
Davis, L.: Handbook of Genetic Algorithms. Van Nostrand Reinhold, New York (1991)
Hansen, N., Muller, S.D., Koumoutsakos, P.: Reducing the time complexity of the derandomized evolution strategy with covariance matrix adaptation (CMA-ES). Evol. Comput. 11(1), 1–18 (2003)
Hansen, N., Auger, A., Ros, R., Finck, S., Posik, P.: Comparing results of 31 algorithms from the black-box optimization benchmarking BBOB-2009. In: Proceedings of the 12th Annual Conference Companion on Genetic and Evolutionary Computation (GECCO), pp. 1689–1696. ACM, New York (2010)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this paper
Cite this paper
Yokoyama, N., Kohjima, M., Matsubayashi, T., Toda, H. (2020). Efficient Bayesian Optimization Based on Parallel Sequential Random Embeddings. In: Palaiahnakote, S., Sanniti di Baja, G., Wang, L., Yan, W. (eds) Pattern Recognition. ACPR 2019. Lecture Notes in Computer Science(), vol 12046. Springer, Cham. https://doi.org/10.1007/978-3-030-41404-7_32
Download citation
DOI: https://doi.org/10.1007/978-3-030-41404-7_32
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-41403-0
Online ISBN: 978-3-030-41404-7
eBook Packages: Computer ScienceComputer Science (R0)