How Many Times Should One Run a Computational Simulation?

Abstract

This chapter is an attempt to answer the question “how many runs of a computational simulation should one do,” and it gives an answer by means of statistical analysis. After defining the nature of the problem and which types of simulation are mostly affected by it, the article introduces statistical power analysis as a way to determine the appropriate number of runs. Two examples are then produced using results from an agent-based model. The reader is then guided through the application of this statistical technique and exposed to its limits and potentials.

Appendices

Further Reading

Details on several power measures can be found in Cohen (1988) and Liu (2014). Specific information on ABM and power are in Secchi and Seri (2017).

Appendix

11.2.1 Number of Runs Calculations

The following is the R code for a function that calculates the number of runs for the configuration of parameters (G, here G) and effect size (f, here ES), given 1 − β = 0.95, α = 0.01:

n.runs <- function(G, ES) {   return(14.091 ⋆ Ĝ(-0.640) ⋆ EŜ(-1.986)) }

In the case discussed in Exercise 1 above, the numbers are:

n.runs(3, 0.25) [1]  109.465

The same analysis using the exact function of the package pwr on power analysis (see Champely et al. 2016) is:

pwr.anova.test(f=0.25, k=3, power=0.95,                     sig.level=0.01)

and yields n = 111.677.

11.2.2 Effect Size for ANOVA vs OLS Regression

In the text we have used a one-way ANOVA test to estimate the number of runs, taking 1 − β = 0.95, α = 0.01 and a given effect size f. However, we then used regression analysis to study the differences between under-, correctly-, and over-powered models.

Since there is transformation between the parameters of ANOVA and OLS regression, it is possible to connect the way effect size is calculated in the first to the second.

As mentioned in the text of the chapter, the effect size for ANOVA is:

$$\displaystyle \begin{aligned}f = \sqrt{\frac{ n \sum_{j=1}^G \left(\bar{x}_{j}-\bar{x}\right)^2 }{ \sum_{j=1}^G \sum_{i=1}^n \left(x_{ij}-\bar{x}_{j}\right)^2 }} \end{aligned}$$

The quantity under the square root is the SSB divided by the Sum of Squares Within (SSW) or, in Cohen’s terms, $f = \frac {\sigma _m}{\sigma }$ (Cohen 1992). The effect size for regression is, according to Cohen (1992), $f^2 = \frac {R^2}{1 - R^2}$. It is easy to demonstrate that:

$$\displaystyle \begin{aligned}f^2 = \frac{R^2}{1 - R^2} = \frac{\mathrm{SSB}}{\mathrm{SSR}} \end{aligned}$$

where the SSW in a one-way ANOVA is comparable to the Sum of Squares of Residuals (SSR) in an OLS regression with exactly the same dependent and independent variables.