skip to main content
article

Calculation of confidence intervals for simulation output

Published: 01 October 2004 Publication History

Abstract

This article is concerned with the calculation of confidence intervals for simulation output that is dependent on two sources of variability. One, referred to as <i>simulation variability</i>, arises from the use of random numbers in the simulation itself; and the other, referred to as <i>parameter variability</i>, arises when the input parameters are unknown and have to be estimated from observed data. Three approaches to the calculation of confidence intervals are presented--the traditional asymptotic normality theory approach, a bootstrap approach and a new method which produces a conservative approximation based on performing just two simulation runs at carefully selected parameter settings. It is demonstrated that the traditional and bootstrap approaches provide similar degrees of accuracy and that whilst the new method may sometimes be very conservative, it can be calculated in a small fraction of the computational time of the exact methods.

References

[1]
Apostol, T. M. 1974. Mathematical Analysis. Addison-Wesley, Reading, Mass.
[2]
Chang, C. S., Heidelberger, P., and Shahabuddin, P. 1995. Fast simulation of packet loss rates in a shared buffer communications switch. ACM Trans. Model. Comput. Simul. 5, 4, 306--325.
[3]
Cheng, R. C. H. and Holland, W. 1997. Sensitivity of computer simulation experiments to errors in input data. J. Statist. Comput. Simul. 57, 219--241.
[4]
Cheng, R. C. H. and Holland, W. 1998. Two-point methods for assessing variability in simulation output. J. Statist. Comput. Simul. 60, 183--205.
[5]
Cox, D. R. and Hinkley, D. V. 1974. Theoretical Statistics. Chapman and Hall, London, England.
[6]
Davison, A. C. and Hinkley, D. V. 1997. Bootstrap Methods and Their Application. Cambridge University Press, Cambridge, England.
[7]
Efron, B. 1982. The Jacknife, the Bootstrap and Other Resampling Plans. CBMS-NSF Regional Conference Series in Applied Mathematics. SIAM.
[8]
Efron, B. and Tibshirani, R. J. 1993. An Introduction to the Bootstrap. Chapman & Hall, New York.
[9]
Kleinrock, L. 1976. Queueing Systems : Volume 2. Computer Applications. Wiley, New York.
[10]
L'Ecuyer, P. and Perron, G. 1994. On the convergence rates of IPA and FDC derivative estimators. Oper. Res. 42, 463--656.
[11]
Lütkepohl, H. 1993. Introduction to Multiple Time Series Analysis. Springer-Verlag, Berlin, Germany.
[12]
Rao, C. R. 1973. Linear Statistical Inference and Its Applications. Wiley, New York.

Cited By

View all
  • (2025)Quantifying and combining uncertainty for improving the behavior of Digital Twin Systemsat - Automatisierungstechnik10.1515/auto-2024-003673:2(81-99)Online publication date: 3-Feb-2025
  • (2024)A Shrinkage Approach to Improve Direct Bootstrap Resampling Under Input UncertaintyINFORMS Journal on Computing10.1287/ijoc.2022.004436:4(1023-1039)Online publication date: 1-Jul-2024
  • (2024)Taming uncertainty with MDE: an historical perspectiveSoftware and Systems Modeling10.1007/s10270-024-01227-4Online publication date: 28-Oct-2024
  • Show More Cited By

Index Terms

  1. Calculation of confidence intervals for simulation output

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image ACM Transactions on Modeling and Computer Simulation
    ACM Transactions on Modeling and Computer Simulation  Volume 14, Issue 4
    October 2004
    99 pages
    ISSN:1049-3301
    EISSN:1558-1195
    DOI:10.1145/1029174
    Issue’s Table of Contents

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 01 October 2004
    Published in TOMACS Volume 14, Issue 4

    Permissions

    Request permissions for this article.

    Check for updates

    Author Tags

    1. δ-Method
    2. parameter variability
    3. simulation variability
    4. two-point method
    5. uncertainty analysis

    Qualifiers

    • Article

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)27
    • Downloads (Last 6 weeks)0
    Reflects downloads up to 13 Feb 2025

    Other Metrics

    Citations

    Cited By

    View all
    • (2025)Quantifying and combining uncertainty for improving the behavior of Digital Twin Systemsat - Automatisierungstechnik10.1515/auto-2024-003673:2(81-99)Online publication date: 3-Feb-2025
    • (2024)A Shrinkage Approach to Improve Direct Bootstrap Resampling Under Input UncertaintyINFORMS Journal on Computing10.1287/ijoc.2022.004436:4(1023-1039)Online publication date: 1-Jul-2024
    • (2024)Taming uncertainty with MDE: an historical perspectiveSoftware and Systems Modeling10.1007/s10270-024-01227-4Online publication date: 28-Oct-2024
    • (2023)Input Uncertainty Quantification via Simulation BootstrappingProceedings of the Winter Simulation Conference10.5555/3643142.3643451(3693-3704)Online publication date: 10-Dec-2023
    • (2023)Estimation of Expectations and Variance Components in Two-Level Nested Simulation ExperimentsAppliedMath10.3390/appliedmath30300313:3(582-600)Online publication date: 7-Aug-2023
    • (2023)Uncertainty Quantification and Exploration for Reinforcement LearningOperations Research10.1287/opre.2023.243672:4(1689-1709)Online publication date: 2-Mar-2023
    • (2023)Uncertainty-aware Simulation of Adaptive SystemsACM Transactions on Modeling and Computer Simulation10.1145/358951733:3(1-19)Online publication date: 13-May-2023
    • (2023)Input Uncertainty Quantification Via Simulation Bootstrapping2023 Winter Simulation Conference (WSC)10.1109/WSC60868.2023.10407968(3693-3704)Online publication date: 10-Dec-2023
    • (2022)Distributional Input UncertaintyProceedings of the Winter Simulation Conference10.5555/3586210.3586430(2617-2628)Online publication date: 11-Dec-2022
    • (2022)Cheap Bootstrap for Input Uncertainty QuantificationProceedings of the Winter Simulation Conference10.5555/3586210.3586404(2318-2329)Online publication date: 11-Dec-2022
    • Show More Cited By

    View Options

    Login options

    Full Access

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Figures

    Tables

    Media

    Share

    Share

    Share this Publication link

    Share on social media