Local learning regularization networks for localized regression

Abstract

Local learning algorithms use a neighborhood of training data close to a given testing query point in order to learn the local parameters and create on-the-fly a local model specifically designed for this query point. The local approach delivers breakthrough performance in many application domains. This paper considers local learning versions of regularization networks (RN) and investigates several options for improving their online prediction performance, both in accuracy and speed. First, we exploit the interplay between locally optimized and globally optimized hyper-parameters (regularization parameter and kernel width) each new predictor needs to optimize online. There is a substantial reduction of the operation cost in the case we use two globally optimized hyper-parameters that are common to all local models. We also demonstrate that this global optimization of the two hyper-parameters produces more accurate models than the other cases that locally optimize online either the regularization parameter, or the kernel width, or both. Then by comparing Eigenvalue decomposition (EVD) with Cholesky decomposition specifically for the local learning training and testing phases, we also reveal that the Cholesky-based implementations are faster that their EVD counterparts for all the training cases. While EVD is suitable for validating cost-effectively several regularization parameters, Cholesky should be preferred when validating several neighborhood sizes (the number of k-nearest neighbors) as well as when the local network operates online. Then, we exploit parallelism in a multi-core system for these local computations demonstrating that the execution times are further reduced. Finally, although the use of pre-computed stored local models instead of the online learning local models is even faster, this option deteriorates the performance. Apparently, there is a substantial gain in waiting for a testing point to arrive before building a local model, and hence the online local learning RNs are more accurate than their pre-computed stored local models. To support all these findings, we also present extensive experimental results and comparisons on several benchmark datasets.

Appendix

For all the algorithms, we use a lot of caching to speed up the process. The training phase of each case must also find the best number of neighbors, denoted as k _best. We search for the best k number in the grid {δL, 2δL,…, L _max} where L _max is the maximum candidate number of neighbors. A local distance matrix maintains the cached distances between the neighbor points. Based on this matrix, a cached local kernel matrix is created once for every candidate σ _m value. Thus, for each L _max, σ _m and λ _l value only one Cholesky is computed for the kernel matrix. Then, progressively the Cholesky back substitution solves for the local weights of all the k candidate values. All four local RN cases use the minimum global training errors to find the best global parameters.

In the training phase, there are 3 loops one inside another. One loop iterates through the candidate width values σ _m , another loop iterates through the candidate k-neighbor values and another iterates through the candidate regularization values λ _l . The ordering of the loops is important. For the best and fastest ordering in the EVD implementations the loop that tests the widths σ _m must be first, followed by a second loop that iterates through the candidate k-neighbor values which inside has the last loop that validates the candidate regularization values λ _l . In the Cholesky implementations, the fastest ordering of computations again has first the loop for the widths σ _m , but now the second loop must iterate through the candidate regularization values λ _l , and the loop for the candidate k-neighbor values must be the third.