Error estimation properties of Gaussian process models in stochastic simulations

Abstract

The theoretical relationship between the prediction variance of a Gaussian process model (GPM) and its mean square prediction error is well known. This relationship has been studied for the case when deterministic simulations are used in GPM, with application to design of computer experiments and metamodeling optimization. This article analyzes the error estimation of Gaussian process models when the simulated data observations contain measurement noise. In particular, this work focuses on the correlation between the GPM prediction variance and the distribution of prediction errors over multiple experimental designs, as a function of location in the input space. The results show that the error estimation properties of a Gaussian process model using stochastic simulations are preserved when the signal-to-noise ratio in the data is larger than 10, regardless of the number of training points used in the metamodel. Also, this article concludes that the distribution of prediction errors approaches a normal distribution with a variance equal to the GPM prediction variance, even in the presence of significant bias in the GPM predictions.

Introduction

A recurring question in metamodeling is how to assess the prediction accuracy of the model. Even though a validation step is typically performed as part of the metamodel construction, the question remains as to how accurate is the prediction of the model at an untried sample point. Assume that the response of an expensive simulation  f can be described as y_tr(x) = f(x), where the subscript tr denotes the true mean response of the simulation at x. The approximate model of the expensive simulation, $\hat{\text{f}}$, predicts the mean response at x as: $y_{\mathit{app}}(\mathbf{x})=\hat{\text{f}}(\mathbf{x})$. Therefore, the mean square prediction error of the approximate model at x is:

$${\delta }^2(\mathbf{x})=[y_{\mathit{tr}}(\mathbf{x})-y_{\mathit{app}}(\mathbf{x}){]}^2$$

In the scenario of expensive simulations, a user may have limited resources (i.e. limited number of simulations) for the training and validation steps of the metamodel. This situation is more difficult when the user is approximating stochastic simulations, since each observed response y from the simulation is corrupted by a measurement noise η around the mean response y(x) = y_tr(x) + η. Because of this scenario, researchers have looked for alternatives to obtain an estimation of the prediction error that do not require additional evaluations from the expensive simulation.

Gaussian process modeling (GPM) is one of the most popular methods for constructing approximate models, not only because of its flexibility and good prediction results, but also because it has its own error estimation on the GPM prediction. According to the theory of GPM, when the GPM structure is completely known (that is, the true GPM parameter set, the true local correlation structure and the true regression functions in the model are known), the GPM prediction variance is an error estimator of the mean square prediction error. In practice, the user has to make decisions about the GPM structure, incurring an additional model uncertainty factor that alters this theoretical property. Many of the applications of GPM implement a “plug-in” version of the Gaussian process, using an estimated parameter set. Santner et al. (2003) called this version of GPM the empirical best linear unbiased predictor. Several authors already mentioned this situation, suggesting that the GPM prediction variance of the empirical GPM version is an underestimator of the “true” GPM prediction variance (Cressie, 1993, den Hertog et al., 2006). These issues raise questions about how to estimate the error in Gaussian process models in practice.

Error estimation measures are useful for the assessment of approximate models. In many applications, the success of approximate models depends on the accuracy in the error estimation. Error estimators are used to quantify the level of uncertainty or “trust” in the prediction of an approximate model, therefore indicating particular regions in the input space where additional samples are required. Some examples include improvement of design of experiments in the creation of approximate models (Hernandez and Grover, 2010), and optimization of time-consuming computer simulations via black-box models (Kleijnen et al., 2010).

Early findings about error estimation of GPM were made by Meckesheimer et al. (2002). They evaluate leave-k-out cross-validation strategies as a procedure to assess the accuracy of low-order polynomial functions, radial basis functions and kriging models over the design space. Goel et al. (2009) presented a detailed study on error estimation, evaluating response surfaces and kriging models for six classical benchmark examples in the statistics field. As a result of this last study, Goel et al. concluded that local evaluations of GPM prediction variance can be used for global error estimation of Gaussian process models. As relevant as these results are in the error estimation of GPM, the studies were limited to predictions of deterministic simulations.

The presence of noise in the observations incorporates an additional element in the GPM prediction that has been discussed previously in the literature. Kleijnen and coworkers have worked extensively in the use of kriging models for random simulations (Kleijnen et al., 2010, Kleijnen and van Beers, 2005, van Beers and Kleijnen, 2003, van Beers and Kleijnen, 2008), using replicates at each sample point to calculate sample means, and then treating those values as deterministic outcomes in the GPM construction. In the area of error estimation, the same group implemented a parametric bootstrapping approach to calculate the mean square prediction error of the GPM (den Hertog et al., 2006), but it was not used as an error estimator in the approximate model and it was only employed with deterministic simulations. A similar implementation of this bootstrapping approach was also used to evaluate the uncertainty of time-course experimental data in cell signaling pathways and network topology of time-series gene expression data (Kirk and Stumpt, 2009). Ankenman et al. (2010) extended Kleijnen’s GPM for random simulations with their stochastic kriging model, which models the intrinsic uncertainty, or noise, in the simulations with an additional variance parameter for each sample point. Different from these papers, where the major interest was the GPM mean prediction, the work presented here focuses on the GPM prediction variance and its role as an error estimator of the approximate model when stochastic observations are used in GPM.