Training regression ensembles by sequential target correction and resampling

Abstract

Ensemble methods learn models from examples by generating a set of hypotheses, which are then combined to make a single decision. We propose an algorithm to construct an ensemble for regression estimation. Our proposal generates the hypotheses sequentially using a simple procedure whereby the target map to be learned by the base learner at each step is modified as a function of the previous step error. We state a theorem that relates the overall upper error bound of the composite hypothesis obtained within this procedure to the training errors of the individual hypotheses. We also demonstrate that the proposed procedure results in a learning functional that enforces a weighted form of Negative Correlation with respect to previous hypotheses. Additionally, we incorporate resampling to allow the ensemble to control the impact of highly influential data points, showing that this component significantly improves its ability to generalize from the known examples. We describe experiments performed to evaluate our technique on real and synthetic datasets using neural networks as base learners. These results show that our technique exhibits considerably better prediction errors than the Negative Correlation (NC) method and that its performance is very competitive with that of the Bagging and AdaBoost algorithms for regression estimation.

Introduction

Ensemble methods [31], [14], [27], [5] have been demonstrated to be a powerful and flexible way to improve the performance of a base learning algorithm in a variety of machine learning scenarios, including classification [45], [41], [19], regression [7], [8], novelty detection [32], times series forecasting [46] and clustering [10], [37]. The basic idea consists of extracting a model from data by combining a set of simple models that are constructed and organized to achieve a desired goal. The Bagging [4], AdaBoost [34], Mixture of Experts [17], Stacking [44] and Negative Correlation [22] algorithms, as well as their many variations, are well-known examples of this class of methods.

The concept of diversity is commonly used by the ensemble community to denote the differences among the individual components to be combined. Because the replication of multiple exact copies of the same model does not provide an advantage over the use of a single instance of such a model, much research has been directed to the definition of strategies to measure and generate diversity in a useful way [6], [20]. The most commonly investigated method to promote diversity among the models in an ensemble is probably the manipulation of the training data used to build the individual models. The Bagging [4] and AdaBoost [34] algorithms, for example, build these models by resampling the original dataset in a first step and then training a base learning algorithm with the obtained datasets. Another investigated approach to achieve diversity in the ensemble is to use different learning functionals to construct the component models, changing the individual learning goals in terms of the original learning task and the state of the other components in the system. Negative Correlation (NC) [7], [22] for example, trains each model with a learning functional that explicitly controls the tradeoff between the individual accuracy and the contribution to a measure of diversity in the ensemble. In [7] Brown et al. showed that NC is competitive with Bagging and AdaBoost in regression estimation scenarios.

This paper presents a new algorithm for building ensembles in regression problems, where the composite predictor is obtained by following a stage-wise additive modeling scheme. At each step, the target map to be learned by the base learner is modified as a function of the previous step error by using a very simple analytic rule. We show that this method is equivalent to using a learning functional similar to that employed by NC [7], [22] to measure and generate diversity in the ensemble. However, in NC, the diversity is measured as a kind of correlation between individual predictions. Specifically, the correlations of the differences between the individual and composite predictions are used to encourage diversity. Our functional, in contrast, uses the differences between the individual predictions and the original target map. This way of measuring diversity in the system encourages only those differences among the individual hypotheses that compensate for the current errors of the ensemble. On the other hand, this functional weights the correlations between the actual and past hypotheses such that the correlations with the more recent hypotheses have a greater weight.

Our experiments show that, compared with Negative Correlation, the proposed procedure can considerably increase the ability of the ensemble to improve the performance of the base learner. On the theoretical side, we provide two results that explicitly relate the overall upper error bound of the composite hypothesis obtained within this procedure to the training errors of the individual hypotheses, thereby demonstrating how the former can be monotonically improved as the number of predictors in the ensemble increases. We also discuss how the stage-wise scheme to modify the target map to be learned by the base learner could increase the leverage effect of highly influential points in the regression, thereby compromising the generalization ability of the ensemble. To address this issue we propose the use of resampling as a preliminary step to generate the individual components. In [13], experimental evidence is presented to support the hypothesis that Bagging stabilizes prediction by equalizing the influence of training examples. In many situations, highly influential points are outliers, and their down-weighting could help to produce more robust predictions. As with Bagging, we show that resampling the dataset before each learning step allows the ensemble to control the impact of highly influential points, which, in turn, improves its ability to generalize from examples.

This article is organized as follows. In Section 2 we state some basic definitions about supervised learning from examples. We then present some background on ensemble learning in Section 3, discussing the key ideas behind diversity generation, Negative Correlation and resampling. In Section 4, we introduce the model proposed for ensemble learning and show that it has the above properties. In Section 5 we present the experimental results. Finally, Section 6 provides the conclusions of this work.