skip to main content
research-article

Enhancing Response Predictions with a Joint Gaussian Process Model for Stochastic Simulation Models

Published: 05 February 2020 Publication History

Abstract

The stochastic Gaussian process model has been widely used in stochastic simulation metamodeling. In practice, the performance of this model can be largely affected by the noise in the observations. In this article, we propose an approach to mitigate the impact of the noisy observations by jointly modeling the response of interest with a correlated but less-noisy auxiliary response. The main idea is to leverage on and learn from the correlated and more accurate response to improve the prediction. To achieve this, we extend the existing deterministic multi-response model for stochastic simulation to jointly model the two responses, use some simplified examples to show the benefit of the proposed model, and investigate the input estimation of this model. Quantile prediction is used to illustrate the efficiency of the proposed approach by jointly modeling it with the expectation, which typically has a less noisy estimator compared with that of the quantile. Several numerical examples are then conducted, and the results show that the joint model can provide better performance. These promising results illustrate the potential of this joint model especially in situations where the response of interest is much noisier or when observations are scarce. We further propose a two-stage design approach based on the multi-response model to more efficiently utilize limited computing budget to improve predictions. We also see from these designs the benefits of the joint model, where the more accurate auxiliary response observations can be used to improve the response of interest.

Supplementary Material

a4-wang-supp.pdf (wang.zip)
Supplemental movie, appendix, image and software files for, Enhancing Response Predictions with a Joint Gaussian Process Model for Stochastic Simulation Models

References

[1]
Mauricio A. Álvarez and Neil D. Lawrence. 2011. Computationally efficient convolved multiple output Gaussian processes. J. Mach. Learn. Res. 12, May (2011), 1459--1500.
[2]
Bruce Ankenman, Barry L. Nelson, and Jeremy Staum. 2010. Stochastic kriging for simulation metamodeling. Op. Res. 58, 2 (2010), 371--382.
[3]
Søren Asmussen and Peter W. Glynn. 2007. Stochastic Simulation: Algorithms and Analysis. Vol. 57. Springer Science 8 Business Media.
[4]
R. Raj Bahadur. 1966. A note on quantiles in large samples. Ann. Math. Stat. 37, 3 (1966), 577--580.
[5]
Jennifer M. Bekki, Xi Chen, and Demet Batur. 2014. Steady-state quantile parameter estimation: An empirical comparison of stochastic kriging and quantile regression. In Proceedings of the Winter Simulation Conference. IEEE Press, 3880--3891.
[6]
Leo Breiman and Jerome H. Friedman. 1997. Predicting multivariate responses in multiple linear regression. J. Royal Stat. Soc.: Series B (Stat. Meth.) 59, 1 (1997), 3--54.
[7]
E. Jack Chen. 2009. Metamodels for estimating quantiles of systems with one controllable parameter. Simulation 85, 5 (2009), 307--317.
[8]
Xi Chen, Bruce E. Ankenman, and Barry L. Nelson. 2012. The effects of common random numbers on stochastic kriging metamodels. ACM Trans. Mod. Comput. Simul. 22, 2 (2012), 7.
[9]
Xi Chen, Bruce E. Ankenman, and Barry L. Nelson. 2013. Enhancing stochastic kriging metamodels with gradient estimators. Op. Res. 61, 2 (2013), 512--528.
[10]
Xi Chen and Kyoung-Kuk Kim. 2014. Stochastic kriging with biased sample estimates. ACM Trans. Mod. Comput. Simul. 24, 2 (2014), 8.
[11]
Xi Chen and Kyoung-Kuk Kim. 2016. Efficient VaR and CVaR measurement via stochastic kriging. INFORMS J. Comput. 28, 4 (2016), 629--644.
[12]
Stefano Conti and Anthony O’Hagan. 2010. Bayesian emulation of complex multi-output and dynamic computer models. J. Stat. Plann. Infer. 140, 3 (2010), 640--651.
[13]
Noel Cressie. 2015. Statistics for Spatial Data. John Wiley 8 Sons.
[14]
Carla Currin, Toby Mitchell, Max Morris, and Don Ylvisaker. 1991. Bayesian prediction of deterministic functions, with applications to the design and analysis of computer experiments. J. Amer. Statist. Assoc. 86, 416 (1991), 953--963.
[15]
Sophie Dabo-Niang and Baba Thiam. 2010. Robust quantile estimation and prediction for spatial processes. Stat. Prob. Lett. 80, 17 (2010), 1447--1458.
[16]
Thomas E. Fricker, Jeremy E. Oakley, and Nathan M. Urban. 2013. Multivariate Gaussian process emulators with nonseparable covariance structures. Technometrics 55, 1 (2013), 47--56.
[17]
Michel Goulard and Marc Voltz. 1992. Linear coregionalization model: Tools for estimation and choice of cross-variogram matrix. Math. Geol. 24, 3 (1992), 269--286.
[18]
Abhishek Gupta, Yu Ding, Leon Xu, and Tommi Reinikainen. 2006. Optimal parameter selection for electronic packaging using sequential computer simulations. J. Manuf. Sci. Eng. 128, 3 (2006), 705--715.
[19]
Xuming He. 1997. Quantile curves without crossing. Amer. Stat. 51, 2 (1997), 186--192.
[20]
Andres F. Hernandez and Martha A. Grover. 2013. Error estimation properties of Gaussian process models in stochastic simulations. Euro. J. Op. Res. 228, 1 (2013), 131--140.
[21]
Dave Higdon. 2002. Space and space-time modeling using process convolutions. In Quantitative Methods for Current Environmental Issues. Springer, 37--56.
[22]
L. Jeff Hong and Guangwu Liu. 2009. Simulating sensitivities of conditional value at risk. Manag. Sci. 55, 2 (2009), 281--293.
[23]
V. Roshan Joseph. 2006. Limit kriging. Technometrics 48, 4 (2006), 458--466.
[24]
Andre G. Journel and Ch. J. Huijbregts. 1978. Mining Geostatistics. Academic Press.
[25]
Jack P. C. Kleijnen. 2008. Design and Analysis of Simulation Experiments. Vol. 20. Springer.
[26]
Jack P. C. Kleijnen and Ehsan Mehdad. 2014. Multivariate versus univariate kriging metamodels for multi-response simulation models. Eur. J. Op. Res. 236, 2 (2014), 573--582.
[27]
Roger Koenker. 1984. A note on L-estimates for linear models. Stat. Prob. Lett. 2, 6 (1984), 323--325.
[28]
Roger Koenker. 2005. Quantile Regression. Number 38. Cambridge University Press.
[29]
John McFarland, Sankaran Mahadevan, Vicente Romero, and Laura Swiler. 2008. Calibration and uncertainty analysis for computer simulations with multivariate output. AIAA J. 46, 5 (2008), 1253--1265.
[30]
Toby Mitchell, Max Morris, and Donald Ylvisaker. 1994. Asymptotically optimum experimental designs for prediction of deterministic functions given derivative information. J. Stat. Plann. Infer. 41, 3 (1994), 377--389.
[31]
Marvin K. Nakayama. 2014. Confidence intervals for quantiles using sectioning when applying variance-reduction techniques. ACM Trans. Mod. Comput. Simul. 24, 4 (2014), 19.
[32]
Richard E. Nance and Robert G. Sargent. 2002. Perspectives on the evolution of simulation. Op. Res. 50, 1 (2002), 161--172.
[33]
Szu Hui Ng and Jun Yin. 2012. Bayesian kriging analysis and design for stochastic simulations. ACM Trans. Mod. Comput. Simul. 22, 3 (2012), 17.
[34]
Tuan Pham and Michael Wagner. 1999. Filtering noisy images using kriging. In Proceedings of the 5th International Symposium on Signal Processing and Its Applications (ISSPA’99), Vol. 1. IEEE, 427--430.
[35]
Jonathan Rougier. 2008. Efficient emulators for multivariate deterministic functions. J. Computat. Graph. Stat. 17, 4 (2008), 827--843.
[36]
Thomas J. Santner, Brian J. Williams, and William I. Notz. 2013. The Design and Analysis of Computer Experiments. Springer Science 8 Business Media.
[37]
Andrew F. Seila. 1982. A batching approach to quantile estimation in regenerative simulations. Manag. Sci. 28, 5 (1982), 573--581.
[38]
Jay M. Ver Hoef and Ronald Paul Barry. 1998. Constructing and fitting models for cokriging and multivariable spatial prediction. J. Stat. Plann. Infer. 69, 2 (1998), 275--294.
[39]
Songhao Wang, Szu Hui Ng, and William Benjamin Haskell. 2018. Quantile simulation optimization with stochastic co-kriging model. In Proceedings of the Winter Simulation Conference (WSC’18). IEEE, 2119--2130.
[40]
Jun Yin, Szu Hui Ng, and Kien Ming Ng. 2009. A study on the effects of parameter estimation on kriging model’s prediction error in stochastic simulations. In Proceedings of the Winter Simulation Conference, M. D. Rossetti, R. R. Hill, B. Johansson, A. Dunkin, and R. G. Ingalls (Eds.). Institute of Electrical and Electronics Engineers, Inc., 674--685.
[41]
Jun Yin, Szu Hui Ng, and Kien Ming Ng. 2011. Kriging metamodel with modified nugget-effect: The heteroscedastic variance case. Comput. Industr. Eng. 61, 3 (2011), 760--777.

Cited By

View all
  • (2022)Analyzing Stochastic Computer Models: A Review with OpportunitiesStatistical Science10.1214/21-STS82237:1Online publication date: 1-Feb-2022

Index Terms

  1. Enhancing Response Predictions with a Joint Gaussian Process Model for Stochastic Simulation Models

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image ACM Transactions on Modeling and Computer Simulation
    ACM Transactions on Modeling and Computer Simulation  Volume 30, Issue 1
    January 2020
    165 pages
    ISSN:1049-3301
    EISSN:1558-1195
    DOI:10.1145/3382041
    Issue’s Table of Contents
    Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 05 February 2020
    Accepted: 01 August 2019
    Revised: 01 April 2019
    Received: 01 October 2017
    Published in TOMACS Volume 30, Issue 1

    Permissions

    Request permissions for this article.

    Check for updates

    Author Tags

    1. Multi-response simulation metamodeling
    2. quantile prediction

    Qualifiers

    • Research-article
    • Research
    • Refereed

    Funding Sources

    • Ministry of Education, Singapore

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)15
    • Downloads (Last 6 weeks)0
    Reflects downloads up to 30 Jan 2025

    Other Metrics

    Citations

    Cited By

    View all
    • (2022)Analyzing Stochastic Computer Models: A Review with OpportunitiesStatistical Science10.1214/21-STS82237:1Online publication date: 1-Feb-2022

    View Options

    Login options

    Full Access

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    HTML Format

    View this article in HTML Format.

    HTML Format

    Figures

    Tables

    Media

    Share

    Share

    Share this Publication link

    Share on social media