Abstract
The hyperparameter optimization of a random forest (RF) is a discrete black-box optimization problem that aims to find the settings of the hyperparameters that optimize an overall out-of-bag (OOB) performance measure of the RF. This problem is computationally expensive for high-dimensional data involving many predictors and a large number of data points. This paper explores the use of surrogate-based approaches, particularly radial basis function (RBF) methods and Bayesian optimization techniques, to optimize the hyperparameters of a classification RF. The surrogates approximate the functional relationship between the hyperparameters and the overall OOB performance of the RF, and they are used to guide the search for a global optimum for the hyperparameter optimization problem. While Bayesian optimization methods have been used to tune the hyperparameters of a classification RF, RBF methods have rarely been used for this task, if at all. We compare the performance of the B-CONDOR-RBF algorithm and a Bayesian optimization method with global and local discrete random search methods to tune the discrete hyperparameters of an RF on 10 classification data sets that involve up to 753 predictor variables and up to 19K data points. The global variant of B-CONDOR-RBF obtained better overall OOB prediction error than the discrete random search methods and two Bayesian optimization algorithms given a limited budget of only 100 function evaluations. Furthermore, a local variant of B-CONDOR-RBF outperformed the random search methods and achieved comparable performance with the Bayesian optimization algorithms on the same problems.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Archetti, F., Candelieri, A.: Bayesian Optimization and Data Science. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-24494-1
Bergstra, J., Bengio, Y.: Random search for hyper-parameter optimization. J. Mach. Learn. Res. 13, 281–305 (2012)
Breiman, L.: Random forests. Mach. Learn. 45, 5–32 (2001)
Dua, D., Graff, C.: UCI Machine Learning Repository. University of California, School of Information and Computer Science, Irvine (2019). http://archive.ics.uci.edu/ml
Ilievski, I., Akhtar, T., Feng, J., Shoemaker, C.: Efficient hyperparameter optimization for deep learning algorithms using deterministic RBF surrogates. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 31, no. 1 (2017)
Jones, D.R., Schonlau, M., Welch, W.J.: Efficient global optimization of expensive black-box functions. J. Glob. Optim. 13, 455–492 (1998)
Joy, T.T., Rana, S., Gupta, S., Venkatesh, S.: Hyperparameter tuning for big data using Bayesian optimisation. In: 2016 23rd International Conference on Pattern Recognition (ICPR), pp. 2574–2579 (2016). https://doi.org/10.1109/ICPR.2016.7900023
Kuhn, M., Johnson, K.: Applied Predictive Modeling. Springer, New York (2013). https://doi.org/10.1007/978-1-4614-6849-3
Kuhn, M., Johnson, K.: AppliedPredictiveModeling: Functions and Data Sets for ‘Applied Predictive Modeling’. R package version 1.1-7 (2018). https://CRAN.R-project.org/package=AppliedPredictiveModeling
Picheny, V., Ginsbourger, D., Roustant, O.: DiceOptim: Kriging-Based Optimization for Computer Experiments. R package version 2.1.1 (2021). https://CRAN.R-project.org/package=DiceOptim
Powell, M.J.D.: The theory of radial basis function approximation in 1990. In: Light, W. (ed.) Advances in Numerical Analysis, Volume 2: Wavelets, Subdivision Algorithms and Radial Basis Functions, pp. 105–210. Oxford University Press, Oxford (1992)
Probst, P., Wright, M.N., Boulesteix, A.-L.: Hyperparameters and tuning strategies for random forest. WIREs Data Min. Knowl. Discov. 9(3), e1301 (2019)
R Core Team: R: A language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria (2021). https://www.R-project.org/
Rasmussen, C.E., Williams, C.K.I.: Gaussian Processes for Machine Learning (Adaptive Computation and Machine Learning). The MIT Press (2006)
Regis, R.G.: Stochastic radial basis function algorithms for large-scale optimization involving expensive black-box objective and constraint functions. Comput. Oper. Res. 38(5), 837–853 (2011)
Regis, R.G.: Hyperparameter tuning of random forests using radial basis function models. In: Nicosia, G., et al. (eds.) LOD 2022. LNCS, vol. 13810, pp. 309–324. Springer, Cham (2023). https://doi.org/10.1007/978-3-031-25599-1_23
Roustant, O., Ginsbourger, D., Deville, Y.: DiceKriging, DiceOptim: two R packages for the analysis of computer experiments by kriging-based metamodeling and optimization. J. Stat. Softw. 51(1), 1–55 (2012)
RStudio Team: RStudio: Integrated Development for R. RStudio, PBC, Boston, MA (2020). http://www.rstudio.com/
Schonlau, M.: Computer experiments and global optimization. Ph.D. thesis, University of Waterloo, Canada (1997)
Turner, R., et al.: Bayesian optimization is superior to random search for machine learning hyperparameter tuning: analysis of the black-box optimization challenge 2020. In: Proceedings of the NeurIPS 2020 Competition and Demonstration Track, in Proceedings of Machine Learning Research, vol. 133, pp. 3–26 (2021)
Vu, K.K., D’Ambrosio, C., Hamadi, Y., Liberti, L.: Surrogate-based methods for black-box optimization. Int. Trans. Oper. Res. 24, 393–424 (2017)
Wright, M.N., Ziegler, A.: Ranger: a fast implementation of random forests for high dimensional data in C++ and R. J. Stat. Softw. 77(1), 1–17 (2017)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Regis, R.G. (2023). Radial Basis Function and Bayesian Methods for the Hyperparameter Optimization of Classification Random Forests. In: Gervasi, O., et al. Computational Science and Its Applications – ICCSA 2023 Workshops. ICCSA 2023. Lecture Notes in Computer Science, vol 14105. Springer, Cham. https://doi.org/10.1007/978-3-031-37108-0_33
Download citation
DOI: https://doi.org/10.1007/978-3-031-37108-0_33
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-37107-3
Online ISBN: 978-3-031-37108-0
eBook Packages: Computer ScienceComputer Science (R0)