Why Analyze Replications?
A stochastic system may generate a long sequence of dependent observations within each run. For example, a hypothetical claims-processing model might produce a daily workload whose level today is related to its level yesterday. Treating every simulated day as an independent observation can then understate uncertainty. A direct alternative is to run the system independently several times and calculate one summary for each run.
The independent replication is the unit of statistical analysis. The summary might be a run total, a run average, or another specified output. If the runs are generated independently under the same model and settings, their summaries are independent observations of the same target quantity. Dependence among observations within a run does not, by itself, create dependence between the run summaries.
The definition of \(Z_j\) must match the question. If the target is expected total cost over a fixed one-year horizon, each \(Z_j\) should be that run's one-year total. If the target is expected daily cost, each \(Z_j\) might instead be the average daily cost for a run of the same length. These targets are not interchangeable: one is a total and the other is an average.
Estimating Uncertainty Across Runs
Suppose \(Z_1,\ldots,Z_R\) are independent replications with a common finite variance. Their sample variance and the estimated standard error of the mean are
Here \(S_Z\) measures variability among complete run summaries, while \(\widehat{\operatorname{SE}}(\bar Z)\) measures estimated Monte Carlo uncertainty in their average. It is not generally valid to substitute the standard deviation of individual within-run observations for \(S_Z\), because those observations may be dependent and are not the independent units used in this calculation.
If the replication-level outputs are normally distributed, the usual two-sided \(100(1-\alpha)\%\) confidence interval for their mean is exact:
where \(t_{\nu,p}\) denotes the \(p\)-quantile of a Student \(t\) distribution with \(\nu\) degrees of freedom. If the outputs are not normally distributed, this interval is generally an approximation, often justified when the number of independent replications is sufficiently large and the variance is finite. With few runs, strong skewness or heavy tails can make a normal-based approximation unreliable.
Why the Replication-Based Standard Error Works
The reason the standard error uses \(R\) independent summaries is seen by considering their average. If each \(Z_j\) has variance \(\sigma_Z^2\), independence makes every covariance between distinct runs zero. Therefore,
Replacing the unknown \(\sigma_Z^2\) with the sample variance \(S_Z^2\) gives the estimated standard error \(S_Z/\sqrt{R}\). Notice that this argument requires independence across replications, not independence of observations within each replication. The within-run dependence is already reflected in the variability of each whole-run summary.
Specify the output quantity for one complete run, including its horizon and whether it is a total, average, or another summary.
Use independent random streams or an equivalent design so that one run's random outcomes do not determine another's.
Record \(Z_j\) for each replication, retaining the same definition and simulation settings across runs.
Compute \(\bar Z\), \(S_Z\), and \(S_Z/\sqrt{R}\); use a t interval with \(R-1\) degrees of freedom when its assumptions are appropriate.
Worked Examples
Worked Example: Daily Averages From Six Independent Runs
A hypothetical service system is simulated for the same long horizon in six independent runs. For each run, the recorded output is its average daily workload. The six run averages are 7, 8, 9, 10, 11, and 15 workload units per day. Assume these replication-level averages are independent and normally distributed. Estimate the expected average daily workload and give a 90% t confidence interval, using \(t_{5,0.95}=2.015\).
Method. Treat the six run averages—not the individual days inside the runs—as the six observations. Their sum is \(7+8+9+10+11+15=60\), so
The deviations from 10 are \(-3,-2,-1,0,1,5\). The squared deviations are \(9,4,1,0,1,25\), which sum to 40. Thus,
The confidence-interval margin is \(2.015(1.1547005)=2.3267216\). The endpoints are \(10-2.3267216=7.6732784\) and \(10+2.3267216=12.3267216\). To three decimal places, the estimated mean is 10.000 and the 90% interval is \([7.673,12.327]\) workload units per day.
Check. The interval is centered at 10, and its half-width is approximately 2.327, matching the calculated margin. The common mistake avoided is rounding the endpoints incorrectly or treating the many daily observations within each run as additional independent replications.
Worked Example: Uncertainty in a Run Total
A hypothetical financial-risk simulation is run independently three times, with the same one-year horizon and model settings. The output from each run is its total annual payment, in thousands of dollars. The observed totals are 30, 36, and 42. Assume normal replication-level totals and use \(t_{2,0.95}=2.920\) for a 90% interval for the expected annual total.
Method. The sample mean total is
thousand dollars. The deviations are \(-6,0,6\), and the squared deviations sum to \(36+0+36=72\). The sample variance is divided by \(R-1=2\), giving
thousand dollars. The margin is \(2.920(6/\sqrt{3})=10.1152\) thousand dollars. Therefore, the interval is \([36-10.1152,\ 36+10.1152]=[25.8848,\ 46.1152]\) thousand dollars, or \([25.885,\ 46.115]\) thousand dollars to three decimals.
Check. The sample standard deviation is 6, but the standard error is smaller by a factor of \(\sqrt{3}\); it is not 3. The interval is centered at 36 and has half-width 10.1152. The trap is dividing the standard deviation by 3 instead of by \(\sqrt{3}\).
Worked Example: Correlated Daily Outputs Within Each Run
A hypothetical inventory model produces daily costs that tend to persist from one day to the next. Four independent runs use the same horizon, and each run's output is its average daily cost, in dollars. The four run averages are 3.2, 3.8, 4.0, and 5.0. Assume the run averages are independent and normally distributed, and use \(t_{3,0.975}=3.182\) for a 95% interval.
Method. The run-average sum is \(3.2+3.8+4.0+5.0=16.0\), so the estimate is \(\bar Z=16/4=4.0\) dollars per day. The deviations are \(-0.8,-0.2,0,1.0\); their squares sum to \(0.64+0.04+0+1.00=1.68\). Hence,
The margin is \(3.182(0.3741657)=1.1906\) dollars per day. The 95% interval is \([4.0-1.1906,\ 4.0+1.1906]=[2.8094,\ 5.1906]\), or \([2.81,\ 5.19]\) dollars per day to the nearest cent.
Check. The calculation uses four independent run averages even if each run contains hundreds of mutually dependent daily costs. Computing a standard error from all daily costs as if they were independent would generally use the wrong variance calculation. This example avoids that error by making the complete run average the analysis unit.
Assumptions and Interpretation
A replication-based interval addresses simulation error for the specified model and run-level estimand. It does not automatically account for uncertainty in model parameters, data, or assumptions. Those sources of uncertainty require additional treatment. Likewise, the interval only describes the target actually represented by \(Z_j\): an interval for average daily output is not an interval for a run total.
| Condition or choice | Why it matters |
|---|---|
| Independent runs | Justifies the zero cross-run covariance used in the standard-error derivation. |
| Same design and output definition | Ensures the summaries estimate a common quantity rather than a mixture of different targets. |
| Finite variance | Needed for the usual variance and standard-error argument. |
| Normal run summaries, or enough runs for an approximation | Supports use of the t interval; with few highly skewed summaries, coverage may be poor. |
A practical review should ask what counts as one replication, whether runs are truly independent, whether all runs use the same horizon, and whether the reported standard error was calculated across the run summaries. If the system's outputs are time-dependent within a run, that fact is a reason to be careful about the analysis unit, not a reason to count each time point as an independent replication.
- Wrong unit: treating every daily observation as independent when the daily outputs are correlated. Correction: form one defined summary per independent run and calculate variability across those summaries.
- Wrong divisor: reporting \(S_Z/R\) as the standard error. Correction: the standard error is \(S_Z/\sqrt{R}\).
- Wrong variance denominator: calculating the sample variance with \(R\) rather than \(R-1\). Correction: use \(R-1\) for the usual unbiased sample variance.
- Wrong target: combining run totals with run averages as if they were the same output. Correction: define one estimand and construct every replication summary consistently.
Check Your Understanding
For each problem, identify the replication-level analysis and check the arithmetic before interpreting the result.
- Four independent hypothetical runs produce average weekly outputs of 4, 6, 8, and 10 units. Find the estimated mean, sample variance, and standard error across runs. (Answer: mean \(=7\), sample variance \(=20/3=6.6667\), standard error \(=\sqrt{5/3}=1.2910\) units.)
- Five independent runs of a hypothetical annual-payment model produce totals of 18, 20, 22, 24, and 26 thousand dollars. Calculate the estimated mean total and its standard error. (Answer: mean \(=22\) thousand dollars, standard error \(=\sqrt{2}=1.4142\) thousand dollars.)
- Three independent normal run summaries are 2, 5, and 8. Using \(t_{2,0.975}=4.303\), calculate a 95% t interval for their expected value. (Answer: mean \(=5\), standard error \(=\sqrt{3}=1.7321\), interval approximately \([-2.453,\ 12.453]\).)
- A hypothetical system has 50 dependent daily measurements in each of four independent runs. The run averages are 2, 4, 6, and 8. Find the standard error for the estimated expected run average, and state the correct number of independent observations for this calculation. (Answer: \(1.2910\) units; use four independent run averages, not 200 daily measurements.)
- Five independent simulated run-level costs, in dollars, are 7, 7, 10, 10, and 11. Using \(t_{4,0.95}=2.132\), estimate the mean and give a 90% t interval. (Answer: mean \(=9.00\), standard error \(=\sqrt{0.7}=0.8367\), interval approximately \([7.216,\ 10.784]\) dollars.)