Skip to main content
Springer Nature Link
Log in

Validation of metamodels in simulation: a new metric

  • Original Article
  • Published:
Engineering with Computers Aims and scope Submit manuscript

Abstract

Metamodels are used to provide more efficient predictions than the underlying simulation models do, but at the price of reduced prediction accuracy. Statistics used to quantify this prediction accuracy include the root-mean square error (RMSE), the coefficient of determination R-square, and the average absolute error (AAE). Such statistics depend on the average prediction accuracy over the validation sample; i.e., these metrics are sensitive to the size of the validation sample. This article, therefore, introduces a new metric, called the Model acceptability score (MAS). Preliminary results indicate that MAS is less sensitive to the validation sample size. The article focuses on deterministic simulation, which is used in various engineering disciplines, e.g., electronic engineering.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16

Similar content being viewed by others

References

  1. Wang G, Shan S (2007) Review of metamodeling techniques in support of engineering design optimization. ASME J Mech Des 129:370–380

    Article  Google Scholar 

  2. Tang B (1993) Orthogonal array-based Latin hypercubes. J Am Stat Assoc 88:1392–1397

    Article  MATH  Google Scholar 

  3. Park J (1994) Optimal Latin-hypercube designs for computer experiments. J Stat Plan Inference 39:95–111

    Article  MATH  Google Scholar 

  4. Ye K, Li W, Sudjianto A (2000) Algorithmic construction of optimal symmetric Latin hypercube designs. J Stat Plan Inferences 90:145–159

    Article  MathSciNet  MATH  Google Scholar 

  5. Fang K, Lin D, Winker P, Zhang Y (2000) Uniform design: theory and application. Technometrics 39:237–248

    Article  MathSciNet  Google Scholar 

  6. Angün E, Kleijnin J, Hertog D, Gürkan G (2002) Response surface methodology revisited. Proceedings IEEE 2002 Winter Simulation Conference, pp 377–383

  7. Martin JD, Simpson TW (2005) Use of kriging models to approximate deterministic computer models. AIAA J 43:853–863

    Article  Google Scholar 

  8. Friedman J (1991) Multivariate adaptive regression splines. Ann Stat 19:1–141

    Article  MATH  Google Scholar 

  9. Shin M, Sargent R, Goel A (2002) Gaussian radial basis functions for simulation metamodeling. Proceedings IEEE 2002 Winter Simulation Conference, pp 483–488

  10. Clarke S, Griebsch J, Simpson T (2005) Analysis of support vector regression for approximation of complex engineering analyses. J Mech Des 127:1077–1087

    Article  Google Scholar 

  11. Simpson T, Peplinski J, Koch P, Allen J (2001) Metamodels for computer-based engineering design: survey and recommendations. Eng Comput 17:129–150

    Article  MATH  Google Scholar 

  12. http://www.mae.ufl.edu/haftka

  13. Barton (1999) Graphical Methods for the Design of Experiments, Springer, New York

  14. Sargent R (2004) Validation and verification of simulation models. Proceedings 2004 Winter Simulation Conference, pp 13–24

  15. Kleijnen J, Deflandre D (2006) Validation of regression metamodels in simulation: bootstrap approach. Eur J Oper Res 170:120–131

    Article  MathSciNet  MATH  Google Scholar 

  16. Hamad H, Al-Hamdan S (2007) Discovering metamodels’ quality-of-fit via graphical techniques. Eur J Oper Res 178:543–559

    Article  MATH  Google Scholar 

  17. Hamad H, Al-Smadi A (2007) Space partitioning in engineering design via metamodel acceptance score distribution. Eng Comput 23:175–185

    Article  Google Scholar 

  18. Hamad H (2006) A new metric for measuring metamodels quality-of-fit for deterministic simulations. Proceedings 2006 Winter Simulation Conference, pp 882–888

  19. Guinta A, Wojtkiewicz S, Eldred M (2003) Overview of modern design of experiment methods for computational sciences Proceedings. 41st Aerospace Sciences Meeting and Exhibit, AIAA, Washington, pp 649

  20. Palmer K, Tsui K-L (2001) A minimum bias Latin hypercube design. IIE Trans 33:793–808

    Google Scholar 

Download references

Acknowledgments

The author would like to thank Professor J. P. C. Kleijnen for his very helpful comments, suggestions, improvements, and corrections to the original manuscript.

Author information

Authors and Affiliations

  1. Electronic Engineering Department, Hijjawi College of Engineering Technology, Yarmouk University, Irbid, Jordan

    Husam Hamad

Authors
  1. Husam Hamad
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Husam Hamad.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Hamad, H. Validation of metamodels in simulation: a new metric. Engineering with Computers 27, 309–317 (2011). https://doi.org/10.1007/s00366-010-0200-z

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00366-010-0200-z

Keywords