Model combination in neural-based forecasting

Abstract

This paper discusses different ways of combining neural predictive models or neural-based forecasts. The proposed approaches consider Gaussian radial basis function networks, which can be efficiently identified and estimated through recursive/adaptive methods. The usual framework for linearly combining estimates from different models is extended, to cope with the case where the forecasting errors from those models are correlated. A prefiltering methodology is proposed, addressing the problems raised by heavily nonstationary time series. Moreover, the paper discusses two approaches for decision-making from forecasting models: either inferring decisions from combined predictive estimates, or combining prescriptive solutions derived from different forecasting models.

Introduction

Time series forecasting is a common goal in data mining applications, where most often the recorded data is indexed in time, and the variables or attributes have distributional properties and correlation effects that are nonstationary in time. Without sacrificing predictive accuracy, the models to be used should be not too complex, should be both flexible and robust, and the methods to estimate those models should be efficient. Therefore, it is convenient to depart from the classic point of view of identifying a single, “clearly best” model, which might require a high computational burden for its identification and optimization.

Most references in the literature on neural-based forecasting follow that traditional paradigm, usually referring to the application of multilayer perceptrons (see [28] for a review). These are highly nonlinear models requiring optimization in a high-dimensional space of a nonlinear least-squares cost function, inevitably with many local optima (see [18] for a comprehensive introduction to the optimization of neural networks). Parameter updating in those models, given new data, is cumbersome and, most importantly, their direct application to nonstationary data is inadequate (see [23], [24]).

This paper seeks to discuss alternative approaches where, while still using supervised neural models, one can achieve good, if not better forecasting performance, through efficient recursive estimation and adaptive identification methods. All the parametric predictive models here proposed—neural or otherwise—are based on time-varying linear parameters that can be efficiently estimated in a recursive manner. Furthermore, we concur with the viewpoint that two or more suboptimal models, linearly composed or linearly combined, may, in general, constitute a better alternative to the optimization of a single neural model in terms of predictive accuracy, efficiency and robustness.

In most of the literature about time series forecasting and neural supervised learning, a classic paradigm is followed, where all effort is directed to the identification and estimation of a single model, in some sense optimal within a class of many possible models, different in structure, in size or in parameterization. The rationale behind this paradigm is the assumption that a “best” model can be conveniently identified for a given problem.

In real-world problems, the true model is likely to be unknown and some choices and assumptions have to be made such that the problem under study can be acceptably modelled and the underlying optimization problem solved. During this process, there are some issues that might be hard to sort out, such as choosing appropriate model selection criteria. In particular, if the chosen model is too complex or overparameterized, it can learn the noise intrinsic to the data, thus causing poor generalization performance, i.e., producing poor results when applied to new data (see, e.g., [3]).

As alternatives to the classic paradigm, several approaches have been proposed, where multiple models are explored and combined. This tends to minimize the implicit risk in taking into account just one model, even it is optimized, and that is limited in its capabilities with respect to the characteristics of the data and to the problem itself.

One may identify three fundamental ways of combining models, to yield a final estimate:

• model mixing, where the chosen estimate is computed as a combination of estimates from different models;

• model synthesis, where the chosen estimate is obtained from the (linear) combination of different, partial models, estimated in conjunction or in sequence; and,

• model switching, where the chosen estimate is selected from the estimates of different models.

The main goal of the present work is to propose some guidelines for combining neural models or neural-based forecasts. While most of the optimization problems reported here are related to time series forecasting, some may be adapted to other problems, such as classification or clustering. The main models used in the paper are Gaussian radial basis function networks, a type of supervised neural networks that may be conveniently used as filtering models, and not just as strictly regressive models (Section 2). We propose some new ideas of using the model mixing approach (Section 3), provided the data is reasonably stationary, or the model synthesis approach (Section 4), when the data is clearly nonstationary. As prediction is just a means for supporting decision-making, in Section 5 we discuss whether one should combine the predictive models or the prescriptive ones.