Fast meta-models for local fusion of multiple predictive models

Abstract

Fusing the outputs of an ensemble of diverse predictive models usually boosts overall prediction accuracy. Such fusion is guided by each model's local performance, i.e., each model's prediction accuracy in the neighborhood of the probe point. Therefore, for each probe we instantiate a customized fusion mechanism. The fusion mechanism is a meta-model, i.e., a model that operates one level above the object-level models whose predictions we want to fuse. Like these models, such a meta-model is defined by structural and parametric information.

In this paper, we focus on the definition of the parametric information for a given structure. For each probe point, we either retrieve or compute the parameters to instantiate the associated meta-model. The retrieval approach is based on a CART-derived segmentation of the probe's state space, which contains the meta-model parameters. The computation approach is based on a run-time evaluation of each model's local performance in the neighborhood of the probe. We explore various structures for the meta-model, and for each structure we compare the pre-compiled (retrieval) or run-time (computation) approaches.

We demonstrate this fusion methodology in the context of multiple neural network models. However, our methodology is broadly applicable to other predictive modeling approaches. This fusion method is illustrated in the development of highly accurate models for emissions, efficiency, and load prediction in a complex power plant. The locally weighted fusion method boosts the predictive performance by 30–50% over the baseline single model approach for the various prediction targets. Relative to this approach, typical fusion strategies that use averaging or globally weighting schemes only produce a 2–6% performance boost over the same baseline.

Introduction

Developing accurate predictive models is a critical task in many applications. On one hand, there is a broad application of predictive models for pattern analysis and classification. On the other hand, an accurate predictive model for systems and processes is increasingly critical to the efficient design, utility, maintainability, and management of complex engineered systems. These models are typically utilized to realize the inherent transfer relationship between the inputs and outputs of a system. Here we use “system” to represent any entity for which some of its attributes (outputs) are influenced by other attributes (inputs) through an inherent functional mapping. Such a system model is often a necessary step in the optimization or management of certain behaviors of the system. Only when an accurate model is developed we can reliably approximate the behavior of the system of interest. Modeling techniques can typically be classified into physics-based first principles methods (e.g. lumped parameter models) or data-driven methods (e.g. neural network models).

In general, given a system input $\overrightarrow{x}$, and a system response $t(\overrightarrow{x})$ we construct a system model $f(\overrightarrow{x})$. The model is always an approximation to the underlying system, and includes an error component $\epsilon (\overrightarrow{x})$, where $t(\overrightarrow{x})=f(\overrightarrow{x})+\epsilon (\overrightarrow{x})$. In some applications, $f(\overrightarrow{x})$ may be derived through physics-based approaches such as finite elements or computational fluid dynamics. Typically, data-driven modeling is applied for modeling complex systems for which the physics is not well understood, or for which a physics-based model is very difficult to develop and maintain at the accuracy levels required of the application.

Regression and other generalized linear modeling techniques are well known in the statistics literature used to build data-driven models [1]. Neural networks are nonlinear universal functional approximators often used for generalized regression. Similar to other data-driven modeling techniques, a neural network is trained and validated using historical data to learn a regression function $f(\overrightarrow{x})$. The prediction uncertainty of a neural network is due to several factors:

• data noise $\epsilon (\overrightarrow{x})$;

• model parameter misspecification caused by

  • variation due to randomly sampling the training set,

  • non-deterministic training results, and

  • varying initial conditions during training; and

• model structure misspecification, e.g. not enough neurons.

While the issues mentioned above are common in data-driven models, they are inevitable in physics-based modeling as well due to simplification of the model structure and uncertainties involved in parameter estimations.

Given the fact that there are errors existing in the models one derives, research work has been conducted to calibrate these errors. In the neural network domain, prediction error has been studied in the context of computing confidence intervals for neural networks [2], [3], [4]. These studies include techniques from the field of nonlinear regression, for instance, the so-called “delta method” and “sandwich method” [5], and the “bootstrap method” [6]. The bootstrap method takes advantage of computational resources to provide a good estimation of the variance due to model parameter misspecification without assuming a normal probability distribution for the error term.

Researchers often ask the question of how one can mitigate these errors in the derived predictive models. Researchers have tried to tackle this problem from different angles, among which fusion of multiple models becomes an increasingly attractive and effective strategy. Given the various uncertainties associated with model accuracy, an ensemble of diverse models could complement each other. A fusion of these multiple models could generate an overall model with lower uncertainty and higher accuracy than is possible with a singleton model. A principal advantage of combining multiple models is the avoidance of loss of information that might result if only the best performing model is selected. Finding the best way of exploitation of the information contained in an ensemble of imperfect estimators is central to the idea of fusing multiple models for improved performance. This is the main focus of this paper.

In our previous work [24], we introduced a method for fusing multiple models based on their measured local performance in the neighborhood of interest. In our experiments we obtained promising results, between 18% and 38% over the average of 30 neural network predictors, which we used as our baseline. However, identifying local performance online could be a costly procedure that might not be suitable when response time is an issue. In this paper, we present different ways of performing such local fusion. In particular, we propose the use of classification and regression tree (CART) [25] to create and represent a segmentation of the probe's state space. This partition contains the parameters needed to instantiate the corresponding meta-model.

To appropriately compare this new approach with our previous work, we use the same object-level models (multiple neural networks) and the same application (emissions, efficiency, and load predictions in a complex power plant) as in [24]. We show that this local fusion method in general is very effective for boosting model prediction accuracy. Although the fusion strategy is demonstrated in the context of multiple neural network models, it can be easily applied to other predictive modeling techniques as well.

In Section 2, we will briefly review related work in the literature, while in Section 3 we describe the general methodology and process for generating and fusing multiple neural network predictors and different fusion strategies; In Section 4, we present the results from a set of experiments based on prediction of emissions, efficiency, and load in a complex real-world power plant, while in Section 5 we present our conclusions describe potential future work.