Skip to main content
SpringerLink
Log in

Computer experiment designs for accurate prediction

  • Published:
Statistics and Computing Aims and scope Submit manuscript

Abstract

Computer experiments using deterministic simulators are sometimes used to replace or supplement physical system experiments. This paper compares designs for an initial computer simulator experiment based on empirical prediction accuracy; it recommends designs for producing accurate predictions. The basis for the majority of the designs compared is the integrated mean squared prediction error (IMSPE) that is computed assuming a Gaussian process model with a Gaussian correlation function. Designs that minimize the IMSPE with respect to a fixed set of correlation parameters as well as designs that minimize a weighted IMSPE over the correlation parameters are studied. These IMSPE-based designs are compared with three widely-used space-filling designs. The designs are used to predict test surfaces representing a range of stationary and non-stationary functions. For the test conditions examined in this paper, the designs constructed under IMSPE-based criteria are shown to outperform space-filling Latin hypercube designs and maximum projection designs when predicting smooth functions of stationary appearance, while space-filling and maximum projection designs are superior for test functions that exhibit strong non-stationarity.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13

Similar content being viewed by others

References

  • Audze, P., Eglais, V.: New approach for planning out of experiments. Probl. Dyn. Strengths 35, 104–107 (1977)

    Google Scholar 

  • Ba, S., Joseph, V.R.: Composite gaussian process models for emulating expensive functions. Ann. Appl. Stat. 6(4), 1838–1860 (2012)

    Article  MathSciNet  MATH  Google Scholar 

  • Ba, S., Joseph, V.R.: MaxPro: maximum projection designs (2015). https://CRAN.R-project.org/package=MaxPro, r package version 3.1-2

  • Bates, R.A., Riccomagno, E., Schwabe, R., Wynn, H.P.: Lattices and dual lattices in experimental design for Fourier models. In: Proceedings of Workshop on “Quasi-Monte Carlo Methods and Their Applications”, pp 1–14 (1995)

  • Bates, S.J., Sienz, J., Toropov, V.V.: Formulation of the optimal latin hypercube design of experiments using a permutation genetic algorithm. Adv. Eng. Soft. 34, 493–506 (2003)

    Article  Google Scholar 

  • Bechhofer, R.E., Santner, T.J., Goldsman, D.M.: Design and Analysis of Experiments for Statistical Selection, Screening, and Multiple Comparisons. Wiley, Hoboken (1995)

    Google Scholar 

  • Bursztyn, D., Steinberg, D.M.: Comparison of designs for computer experiments. J. Stat. Plann. Inference 136, 1103–1119 (2006)

    Article  MathSciNet  MATH  Google Scholar 

  • Currin, C., Mitchell, T.J., Morris, M.D., Ylvisaker, D.: Bayesian prediction of deterministic functions, with applications to the design and analysis of computer experiments. J. Am. Stat. Ass. 86, 953–963 (1991)

    Article  MathSciNet  Google Scholar 

  • van Dam, E., den Hertog, D., Husslage, B., Rennen, G.: Space-filling designs. http://www.spacefillingdesigns.nl/. Accessed March 2013 (2013)

  • Draguljić, D., Santner, T.J., Dean, A.M.: Non-collapsing spacing-filling designs for bounded polygonal regions. Technometrics 54, 169–178 (2012)

    Article  MathSciNet  Google Scholar 

  • Efron, B.: Frequentist accuracy of bayesian estimates. J. R. Stat. Soc. B (2014). doi:10.1111/rssb.12080

  • Fogelson, A., Kuharsky, A., Yu, H.: Computational modeling of blood clotting: coagulation and three-dimensional platelet aggregation. Polymer and Cell Dynamics: Multicsale Modeling and Numerical Simulations, pp. 145–154. Birkhaeuser-Verlag, Basel (2003)

    Chapter  Google Scholar 

  • Forrester, A., Sobester, A., Keane, A.: Engineering design via surrogate modelling: a practical guide. Wiley, Chichester (2008)

    Book  Google Scholar 

  • Givens, G., Hoeting, J.: Computational Statistics. Wiley, Hoboken (2012)

    Book  MATH  Google Scholar 

  • Hajagos, J.G.: Modeling uncertainty in population biology: How the model is written does matter. In: Hanson, K.M., Hemez, F.M. (eds) Proceedings of the SAMO 2004 Conference on Sensitivity Analysis. http://library.lanl.gov/ccw/samo2004/, Los Alamos National Laboratory, Los Alamos, pp. 363–368 (2005)

  • Higdon, D., Kennedy, M., Cavendish, J., Cafeo, J., Ryne, R.: Combining field data and computer simulations for calibration and prediction. SIAM J. Sci. Comput. 26, 448–466 (2004)

    Article  MathSciNet  MATH  Google Scholar 

  • Johnson, M.E., Moore, L.M., Ylvisaker, D.: Minimax and maximin distance designs. J. Stat. Plann. Inference 26, 131–148 (1990)

    Article  MathSciNet  Google Scholar 

  • Johnson, R.T., Montgomery, D.C., Jones, B.: An empirical study of the prediction performance of space-filling designs. Int. J. Exp. Des. Process Optim. 2(1), 1–18 (2011)

    Article  Google Scholar 

  • Joseph, V.R., Gul, E., Ba, S.: Maximum projection designs for computer experiments. Biometrika 102(2), 371–380 (2015)

    Article  MathSciNet  MATH  Google Scholar 

  • Kennedy, J., Eberhart, R.: Particle swarm optimization. In: Neural Networks, 1995. Proceedings., IEEE International Conference on, vol 4, pp. 1942–1948 (1995)

  • Kincaid, D., Cheney, E.: Numerical Analysis: Mathematics of Scientific Computing. American Mathematical Society, Providence (2002)

    MATH  Google Scholar 

  • Leatherman, E.R., Dean, A.M., Santner, T.J.: Computer experiment designs via particle swarm optimization. In: Melas, V., Mignani, S., Monari, P., Salmaso, L. (eds.) Topics in Statistical Simulation: Research Papers from the \(7^{th}\) International Workshop on Statistical Simulation, vol. 114, pp. 309–317. Springer, Berlin (2014a)

  • Leatherman, E.R., Guo, H., Gilbert, S.L., Hutchinson, I.D., Maher, S.A., Santner, T.J.: Using a statistically calibrated biphasic finite element model of the human knee joint to identify robust designs for a meniscal substitute. J. Biomech. Eng. 136(7), 071,007 (2014b)

  • Liefvendahl, M., Stocki, R.: A study on algorithms for optimization of latin hypercubes. J. Stat. Plann. Inference 136, 3231–3247 (2006)

    Article  MathSciNet  MATH  Google Scholar 

  • MacDonald, B., Ranjan, P., Chipman, H., et al.: Gpfit: an r package for fitting a gaussian process model to deterministic simulator outputs. J. Stat. Soft. 64, i12 (2015)

    Article  Google Scholar 

  • MATLAB Parametric Empirical Kriging (MPErK) (2013) T.J. Santner Group, The Ohio State University

  • McKay, M.D., Beckman, R.J., Conover, W.J.: A comparison of three methods for selecting values of input variables in the analysis of output from a computer code. Technometrics 21, 239–245 (1979)

  • Montgomery, G.P., Truss, L.T.: Combining a statistical design of experiments with formability simulations to predict the formability of pockets in sheet metal parts. In: Technical Report 2001-01-1130, Society of Automotive Engineers (2001)

  • Morokoff, W.J., Caflisch, R.E.: Quasi-monte carlo integration. J. Comput. Phys. 122, 218–230 (1995)

    Article  MathSciNet  MATH  Google Scholar 

  • Morris, M.D., Mitchell, T.J.: Exploratory designs for computational experiments. J. Stat. Plann. Inference 43, 381–402 (1995)

    Article  MATH  Google Scholar 

  • Nekkanty, S.: Characterization of damage and optimization of thin film coatings on ductile substrates. PhD thesis, Department of Mechanical Engineering, The Ohio State University, Columbus, Ohio, USA (2009)

  • Niederreiter, H.: Random Number Generation and Quasi-Monte Carlo Methods. SIAM, Philadelphia (1992)

    Book  MATH  Google Scholar 

  • Ong, K., Santner, T., Bartel, D.: Robust design for acetabular cup stability accounting for patient and surgical variability. J. Biomech. Eng. 130(031), 001 (2008)

    Google Scholar 

  • Pronzato, L., Müller, W.G.: Design of computer experiments: space filling and beyond. Stat. Comput. 22, 681–701 (2012)

    Article  MathSciNet  MATH  Google Scholar 

  • R Core Team (2016) R: A Language and Environment for Statistical Computing. R Foundation for Statistical Computing, Vienna, Austria. https://www.R-project.org/

  • Sacks, J., Schiller, S.B., Welch, W.J.: Design for computer experiments. Technometrics 31, 41–47 (1989a)

    Article  MathSciNet  Google Scholar 

  • Sacks, J., Welch, W.J., Mitchell, T.J., Wynn, H.P.: Design and analysis of computer experiments. Stat. Sci. 4, 409–423 (1989b)

    Article  MathSciNet  MATH  Google Scholar 

  • Silvestrini, R.T., Montgomery, D.C., Jones, B.: Comparing computer experiments for the Gaussian process model using integrated prediction variance. Qual. Eng. 25, 164–174 (2013)

    Article  Google Scholar 

  • Trosset, M.W.: The krigifier: a procedure for generating pseudorandom nonlinear objective functions for computational experimentation. In: Technical Report 35, Institute for Computer Applications in Science and Engineering, NASA Langley Research Center (1999)

  • Upton, M.L., Guilak, F., Laursen, T.A., Setton, L.A.: Finite element modeling predictions of region-specific cell-matrix mechanics in the meniscus. Biomech. Model. Mechanobiol. 5, 140–149 (2006)

    Article  Google Scholar 

  • Villarreal-Marroquín, M.G., Svenson, J.D., Sun, F., Santner, T.J., Dean, A., Castro, J.M.: A comparison of two metamodel-based methodologies for multiple criteria simulation optimization using an injection molding case study. J. Poly. Eng. 33, 193–209 (2013)

    Google Scholar 

  • Welch, W.J.: Aced: algorithms for the construction of experimental designs. Am. Stat. 39, 146 (1985)

    Article  Google Scholar 

  • Xiong, Y., Chen, W., Apley, D., Ding, X.: A non-stationary covariance-based kriging method for metamodelling in engineering design. Int. J. Numer. Methods Eng. 71(6), 733–756 (2007). doi:10.1002/nme.1969

    Article  MATH  Google Scholar 

  • Yang, X.S.: Engineering Optimization: An Introduction with Metaheuristic Applications, 1st edn. Wiley Publishing, Hoboken (2010)

    Book  Google Scholar 

Download references

Acknowledgements

The authors wish to thank the referees for their suggestions which led to the improvement of this paper. This research was sponsored, in part, by an allocation of computing time from the Ohio Supercomputer Center, and by the National Science Foundation under Agreements DMS-0806134, DMS-1310294, and DMS-1564395 (The Ohio State University). Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.

Author information

Authors and Affiliations

  1. West Virginia University, Morgantown, WV, 26506, USA

    Erin R. Leatherman

  2. The Ohio State University, Columbus, OH, 43210, USA

    Thomas J. Santner & Angela M. Dean

Authors
  1. Erin R. Leatherman
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Thomas J. Santner
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Angela M. Dean
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Erin R. Leatherman.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (pdf 2165 KB)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Leatherman, E.R., Santner, T.J. & Dean, A.M. Computer experiment designs for accurate prediction. Stat Comput 28, 739–751 (2018). https://doi.org/10.1007/s11222-017-9760-8

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11222-017-9760-8

Keywords

Search

Navigation