A new perspective for Minimal Learning Machines: A lightweight approach

Highlights

• We propose a new procedure to train Minimal Learning Machines.

• Our proposal do not rely on any assumption regarding the data.

• Our proposal builds a regularized system that imposes sparseness.

• We evaluated both proposals on well-known state-of-the-art datasets.

Abstract

This paper introduces a new procedure to train Minimal Learning Machines (MLM) for regression tasks. Besides that, we propose a new prediction process in MLM. A well-known drawback concerning the (original) MLM model formulation is the lack of sparseness.The most recent efforts on this problem strongly rely on the selection of reference points before training and prediction steps in MLM, all based on some supposition regarding the data. In the opposite direction, here, we explore another formulation of MLM in which we do not rely on any assumption regarding the data for prior selection. Instead, our proposal, named Lightweight Minimal Learning Machine (LW-MLM), builds a regularized system that imposes sparseness. We thrive in such a sparse criterion, not by selection but instead using a piece of weighted information into the model. We validate the contributions of this paper through four types of experiments to evaluate different aspects of our proposal: the prediction error performance, the goodness-of-fit of estimated vs. measured values, the norm values which are related to the sparsity, and finally, the prediction error in high dimensional settings. Based on the results, we show that LW-MLM is a valid alternative since achieved similar or higher accuracy rates against other variants being all seen as statistically equivalent.

Introduction

The Minimal Learning Machine (MLM, [4]) is a supervised method used for classification and regression tasks. The training step is based on a multi-response linear regression model that finds the mapping between distances computed from the input and output spaces. In such a model, we obtain the outputs from new inputs (not seen in the training step) by a linear system resolution with an optimization process in the output space. Also, in the original proposal, predicting new inputs becomes an onerous task since such a computation (might) involves all samples in the training set, namely, Reference points (RPs). However, the MLM exhibits an equivalent or even better generalization performance in comparison to other models [4].

A well-known drawback concerning the (original) MLM model formulation is the lack of sparseness. Sparseness is related to express the induced model in terms of a relatively small number of samples in both training and prediction procedures. As the induced model lacks sparseness, it becomes intractable to deal with large-size datasets since the computational complexity in the prediction scales according to the size of the training set. Another known drawback, this time related to the goodness-of-fit of estimated values, is the lack of a term besides the error during the learning. In this case, regularization in the model can discourage the learning of a more complex or flexible model, to avoid the risk of overfitting.

In the most recent and adopted formulation of MLM, named Random-MLM [4], a lower rank linear system is obtained by randomly selecting samples as a manner to also impose some sparsity in both training and prediction alongside regularization. However, in this approach, randomness can discard potentially useful data that could be important for the induction process and might end up impairing the MLM generalization performance. Thus, a proper manner of imposing some sparsity and regularization over the random selection is indeed desirable due to the reasons as the ones mentioned earlier.

The most recent efforts (Section 2) on this problem strongly rely on RPs selections before training and prediction steps in MLM. To do it so, such efforts rely on any supposition regarding the data. In the opposite direction, here, we intend to explore another formulation of MLM in which we do not rely on any supposition regarding the data for prior selection. Instead, we rely on the supposition that the whole known geometric structure of data is learned while it encourages us to discard some RPs only in the out-of-sample prediction procedure to provide a more compact and faster model.

From the sparsity point-of-view, such a process of selecting RPs and computing the mapping between distance matrices is treated as a sparse resolution problem of linear systems with multiple responses, which unfortunately is an NP-Hard problem. In this setting, we posed the MLM model in two mains constraints to solve this problem: (1) reconstruction constraint (error term), and (2) model complexity constraint (norm term). The reconstruction constraint requires that the recovered mapping should be consistent with input and output spaces. Additionally, the model complexity constraint assumes that one can represent such a mapping sparsely in an appropriately chosen overcomplete dictionary and that one can recover their sparse representations from it. For sparse learning strategies, we refer the reader to the following works [6], [7], [20] and for some applications related to image super-resolution  [21], order preservation [14], and multitask learning in image emotion distribution prediction  [23], all sparse representation-based.

Stated that, this work tackles the problem of imposing sparsity and regularization in MLM by proposing a novel formulation based on a pattern relevance. Our proposal, named “Lightweight” MLM (LW-MLM, Section 3) is mainly based on a ranking criterion that accounts for the matching between the input and output local distance measurements, indicating which points correspond to the most linear part of the target function. The main idea and principal contribution is to build a regularized system by pattern as a manner to impose sparseness not by selection but using weighted information into the model. Differently from other approaches such as the Regularized Least-Squares version of MLM (wMLM), ours do not work at the error term, but instead in the norm of the solution obtained by the distance regression.

We validate the contributions of this paper through four types of experiments to evaluate different aspects of LW-MLM, namely, the prediction error performance, the goodness-of-fit of estimated vs. measured values, the norm values which are related to the sparsity and, finally, the prediction error in high dimensional feature spaces (Section 4). Based on the results, we show that LW-MLM is a valid alternative since achieved similar or higher accuracy rates against other variants, all with low variance, being all seen as statistically equivalent (Section 5). Also, from empirical analysis, we support that, due to such a formulation in LW-MLM, some problems can take advantage of the regularization approach by pattern while other models do not perform precisely.