A New Baseline for Automated Hyper-Parameter Optimization

Abstract

Finding the optimal hyper-parameters values for a given problem is essential for most machine learning algorithms. In this paper, we propose a novel hyper-parameter optimization algorithm that is very simple to implement and still competitive with the state-of-the-art L-SHADE variant of Differential Evolution. While the most common method for hyper-parameter optimization is a combination of grid and manual search, random search has recently shown itself to be more effective and has been proposed as a baseline against which to measure other methods. In this paper, we compare three optimization algorithms, namely, the state-of-the-art L-SHADE algorithm, the random search algorithm, and our novel and simple adaptive random search algorithm. We find that our simple adaptive random search strategy is capable of finding parameters that achieve results comparable to the state-of-the-art L-SHADE algorithm, both of which achieve significantly better performance than random search when optimizing the hyper-parameters of the state-of-the-art XGBoost algorithm for 11 datasets. Because of the significant performance increase of our simple algorithm when compared to random search, we propose this as the new go-to method for tuning the hyper-parameters of machine learning algorithms when desiring a simple-to-implement algorithm, and also propose to use this algorithm as a new baseline against which other strategies should be measured.