Kriging metamodel with modified nugget-effect: The heteroscedastic variance case

Abstract

Metamodels are commonly used to approximate and analyze simulation models. However, in cases where the simulation output variances are non-zero and not constant, many of the current metamodels which assume homogeneity, fail to provide satisfactory estimation. In this paper, we present a kriging model with modified nugget-effect adapted for simulations with heterogeneous variances. The new model improves the estimations of the sensitivity parameters by explicitly accounting for location dependent non-constant variances and smoothes the kriging predictor’s output accordingly. We look into the effects of stochastic noise on the parameter estimation for the classic kriging model that assumes deterministic outputs and note that the stochastic noise increases the variability of the classic parameter estimation. The nugget-effect and proposed modified nugget-effect stabilize the estimated parameters and decrease the erratic behavior of the predictor by penalizing the likelihood function affected by stochastic noise. Several numerical examples suggest that the kriging model with modified nugget-effect outperforms the kriging model with nugget-effect and the classic kriging model in heteroscedastic cases.

Introduction

Computer simulation is commonly used in industry as a tool to aid in studying the system’s characteristics and behaviors. It is especially useful in system optimization problems, where the costs can be greatly reduced by running experiments on the simulation models instead of the real systems. As the complexity of the simulation model increases, the computing cost of running experiments on the simulation model becomes much higher. Metamodels have been applied as simplified approximations to the complex simulation model; see Kleijnen, 1987, Kleijnen, 1998. Replacing the simulation model with a metamodel in expensive experiments can increase the efficiency and lower the computing costs. A review of metamodel applications in engineering can be found in Simpson, Peplinski, Koch, and Allen (2001). Among the different types of metamodels available, the spatial correlation model, also known as the kriging model, is one of the more promising metamodels as it is more flexible than regression models and not as complicated and time consuming as artificial intelligence (AI) techniques; see Li, Ng, Xie, and Goh (2010) for a comparative study. The kriging model was originally developed in the field of geo-statistics; see Matheron (1963). It was first introduced into Design and Analysis of Computer Experiment (DACE) by Sacks, Welch, Mitchell, and Wynn (1989) and Sacks, Schiller, and Welch (1989). Recently, there is an increasing interest in adopting kriging metamodels in industrial engineering problems and applications (e.g. Ankenman et al., 2010, Huang et al., 2006, Sakata et al., 2007, Wang et al., 2008).

The kriging model is very suitable for deterministic simulation problems. It is attractive for its interpolating characteristic, providing predictions with the same values as the observations. For example, in Gupta, Yu, Xu, and Reinikainen (2006), the kriging metamodel is adopted for its interpolating characteristic. For stochastic simulations where the responses at the same location vary (for example in a simulation of a queueing system), the interpolation characteristic of kriging models becomes less desirable. In order to model the random fluctuations in stochastic situations, the nugget-effect is introduced. The term “nugget” is borrowed from geo-statistics, referring to the unexpected nugget of gold found in a mining process. According to Cressie (1993, p. 127), the nugget-effect in geo-statistics is caused by two factors: micro-scale variation and measurement error. In this article, we assume that the system studied can be modeled as an L₂-continuous random process (see Cressie, 1993, p. 112), and hence the nugget-effect studied here is purely caused by the random measurement error (or random noise).

The nugget-effect in kriging assumes second-order stationarity and is typically used to model white noise effect. Most kriging publications assume that the variance of the random error is homogeneous and the kriging model with nugget-effect is sufficient to solve the problem. However, there are many real world situations where the homoscedastic assumption does not hold. These include queueing systems and networks which can be found in many industrial engineering problems. When applying the homoscedastic kriging model in a heteroscedastic case, the fit can be poor, especially when the sample size is small. We illustrate the noisy applications with the simple function displayed in Fig. 1.

The test function consists of a second-order signal function and a noisy function with step variance.

$$y=S(x)+\epsilon =x^2+\epsilon$$

where ε indicates the random noise component, with variance ${\sigma }_{\epsilon }^2=0.083$ when x∈[−5, 2), and ${\sigma }_{\epsilon }^2=8.3$ when x ∈ [2, 5]. In Fig. 1, the solid line indicates the signal function y = x², and the dots are the noisy observations of the signal function y = x² + ε.

In the traditional application of kriging in stochastic simulations, replications are taken at each observation point and the averages of the replicates at each point are used as the inputs to the model. Kleijnen (2008, p. 92) recommends at least n ⩾ 2 replications to be taken equally at each observation point when no prior knowledge on the variance forms is available, otherwise, the simulation exercise may be meaningless due to the variability in the data. In this test function example, we assume that a budget for only 76 runs is available. Based on this, we spread 19 points from −5 to 5, taking four replicates at each point. The averages of the four replicates at each of the 19 points are used as the inputs of the model. The solid line in Fig. 2 plots the fit of the traditional deterministic ordinary kriging (OK) model.

With limited replications and input points, the ordinary kriging model’s predictor output is poor with obvious fluctuations away from the true function when x < −4 and x > 1. Because the traditional ordinary kriging model is designed under deterministic assumptions, random noise can cause an ill fit and result in disappointing predictions. We note that the predictor output will improve as more replications and observation points are taken. However, in many practical applications of simulation, the computer model can be complicated and time consuming to run (see Gramacy and Lee, 2009, Gupta et al., 2006), limiting the number of observation points and replications that can be taken.

Considering the kriging model with nugget-effect which has a homogenous variance assumption, we pool the sample variances at the 19 observation points to estimate the nugget-effect. The predictor output adopting this model is plotted as the dashed line in Fig. 2.

As seen in Fig. 2, the nugget-effect predictor’s output is smoother than the OK predictor. However, in the region x ∈ [2, 5] where the variance is higher, the fit is poor compared with the fit in the region x∈[−5, 2). This indicates that the nugget-effect model can still be inadequate as the heterogeneous variance can have an impact on local predictions. Moreover, due to the homogeneous noise assumptions of this model, there is no clear method to estimate the nugget-effect under these heterogeneous conditions.

This same phenomenon occurs in the simulation of the M/M/1 queue, one of the most basic queueing models. Van Beers and Kleijnen (2003) proposed a detrending approach to model out the trend in the data using least squares methods and then apply the deterministic ordinary kriging model to the detrended data. Two alternative methods were later proposed by Kleijnen and Van Beers (2005) to improve the application of kriging in stochastic problems: the replication method and the studentization method. The replication method proposes that the heteroscedastic problem can be converted into a homoscedastic problem by taking appropriate replications at all the observation locations. This method requires a sequential design with sufficient computing resources to run all the replications. For example, in Fig. 1, the number of replicates needed in the region with higher variance should be 100 times larger than the number of replicates in the region with lower variance (because the variance is 100 times bigger in the former region) in order to convert the heteroscedastic case into a homoscedastic case. In the study of the M/M/1 queue system for the case where the computing budget is limited, both the OK and nugget-effect model with the application of this replication method can still be inadequate. The studentization method is developed on the basis of the detrended kriging approach. The main idea is to model the trend in the data and then standardize the detrended data. It is an intuitive method to handle inputs with different variances. However, in their numerical examples, this method did not improve much over the OK model. This is due to the amplification of the uncertainty in the estimation of the signal function and variance in the transformation of the predictor, especially when the sample size is small.

As seen in Fig. 2, both the OK model and the nugget-effect model perform poorly when dealing with heteroscedastic data. In this paper, we relax the stationarity assumption on the covariance process and propose the kriging model with modified nugget-effect to model heteroscedastic observations. This model follows the basic framework of the kriging model with nugget-effect, but extends it by taking the sample variance as an additional input to provide variance information. This method has two main benefits: first, the new model retains the original simple structure of the kriging model with nugget-effect, and second, the computing resources needed for computing the sample variance can be significantly lower than the requirement of the replication method. The sample variance is used as an additional input variable and it can reduce the impact of the heterogeneous variance on the local prediction by penalizing the data with higher variance. In the numerical experiments shown in this paper, the modified nugget-effect model’s performance in the heteroscedastic case is consistently better than the OK model and nugget-effect model.

In this paper, the proposed new kriging model form is based on an extension of the nugget-effect model. Ankenman, Nelson and Staum (2010) recently proposed an alternative stochastic kriging model for stochastic simulations. Although the mathematical predictor forms of both models are equivalent (as will be seen in Section 2), our initial assumptions differ in that our proposed modified nugget-effect model is developed from the traditional nugget-effect model, extending it to treat the additional noise component ε as a non stationary component of the random process. The stochastic kriging model is developed based on the deterministic kriging model, and considers the additional noise component ε as the intrinsic uncertainty of the simulation itself. Ankenman et al., 2010, Chen et al., 2010 go onto look at the effects of common random numbers on the model and describe experimental design strategies under the stochastic model. In this paper, our focus differs in that we study in detail the effects and influence of stochastic noise on the traditional deterministic ordinary kriging model and nugget-effect model, looking more deeply into the effects on parameter estimation and characteristics of the likelihood function. We also compare in detail the prediction performances of the three models, providing insights on when each model form is sufficient and adequate.

This article is organized as follows: In Section 2, we develop the proposed modified nugget-effect model. We then address the issues of parameter estimation and error measurement and further study the effects of stochastic noise on the traditional models as well as illustrate how the modified nugget-effect model mitigates this problem. In Section 3, we study the prediction performance and characteristics of the proposed model. Then in Section 4, the performance of the modified nugget-effect model is illustrated with several numerical experiments and a case study. Comparisons with the traditional kriging model and the nugget-effect model are given, and finally, comparisons with the studentization method are also made.