Interpreting Variation Across Simulation Runs

Why Repeated Runs Can Differ

A simulation estimates operating characteristics by generating many hypothetical trial paths and summarizing their outcomes. Even when the design, scenario, and code are unchanged, a second run will usually produce somewhat different estimates because it uses different random draws. The key question is whether the observed difference is plausible Monte Carlo variation or a sign that something about the runs is not comparable.

This tutorial focuses on interpreting differences across repeated runs. The earlier material on Monte Carlo error and simulation failures covers basic estimation uncertainty and failure diagnosis; here the emphasis is on comparing runs, accounting for replication counts, checking random streams, and keeping scenario-level differences visible.

The intended conditions matter. Two numbers should be compared as repeated estimates of the same quantity only when the design, scenario assumptions, outcome definition, analysis rule, and any weighting or filtering are aligned. If one run uses a different treatment effect, nuisance parameter, stopping rule, or definition of a successful rejection, the difference is not merely run-to-run Monte Carlo variation.

Quantifying Differences Between Independent Runs

Suppose each simulated trial produces a binary outcome \(I_i\), equal to 1 if the specified event occurs and 0 otherwise. For example, the event might be rejection of \(H_0\) under the null scenario. With \(M\) independent simulated trials, the estimated event probability is \(\hat{p}=X/M\), where \(X\) is the number of events. Under the binomial model, its Monte Carlo standard error is approximately \(\sqrt{\hat{p}(1-\hat{p})/M}\).

For two runs using independent random streams, the difference between their estimates has variance equal to the sum of their variances. This follows because the variance of a difference is the sum of the variances when the estimates are independent:

Dividing the observed difference by this estimated standard error gives a useful standardized discrepancy. It is a diagnostic scale, not an automatic test of whether the software is correct. A small value means the observed difference is modest relative to the expected simulation noise; a large value warrants investigation, but does not by itself identify the cause.

Replication counts must be included in the comparison. Estimates from runs of different sizes can have noticeably different precision, even when their point estimates are similar. Treating the counts as equal can exaggerate or understate how surprising a discrepancy is.

Derivation and the Role of Pairing

For an estimated event probability, each replicate is a Bernoulli random variable with variance \(p(1-p)\). The mean of \(M\) independent Bernoulli variables therefore has variance \(p(1-p)/M\). Replacing the unknown \(p\) with its estimate gives the familiar estimated Monte Carlo variance. For two independent runs, if \(D=\hat{p}_1-\hat{p}_2\), then

because independence makes the covariance zero. That last equality is not valid when the runs are deliberately paired using common random numbers. In a paired design, the same simulated underlying trial conditions are used to compare two methods or designs. The variance of the paired difference includes a covariance term, and that term can reduce or increase uncertainty.

For paired outcomes \(Y_{1i}\) and \(Y_{2i}\), define \(D_i=Y_{1i}-Y_{2i}\) for each matched simulated trial. The estimated mean difference is \(\bar{D}\), and its standard error is \(s_D/\sqrt{M}\), where \(s_D\) is the sample standard deviation of the paired differences. This is the appropriate calculation when the matching is intentional and valid; calculating a standard error as if the two estimates were independent discards the pairing information.

Separate Run Variation From Scenario Variation

A scenario matrix may include different treatment effects, nuisance assumptions, or operational conditions. Variation in estimates across those scenarios is not the same as variation across repeated runs of a fixed scenario. Keep scenario-specific estimates visible before calculating an overall summary. If an overall target is needed, state the weights: a weighted average represents the chosen mix of scenarios, not a universal operating characteristic.

Random-Stream Checks and Practical Diagnostics

A run comparison is interpretable only if its random-number generation was managed as intended. A seed supports reproducibility, but a seed value alone does not demonstrate that parallel jobs used distinct, non-overlapping streams. Nor does changing a seed guarantee independence if stream initialization or parallel allocation is faulty. Record enough information to identify the run, scenario, replication count, and stream allocation, and check that the intended configuration was used consistently.

When estimates differ more than expected, investigate in a structured order:

• Align the target. Confirm that the scenarios, stopping rules, analysis definitions, units, and any exclusions or weights match.
• Check the denominator. Use the number of valid completed replicates for the estimate, and separately account for failed or incomplete replicates.
• Check stream handling. Confirm that repeated runs are reproducible when the same configuration and seed are intentionally reused, and that independent or parallel runs receive the intended distinct streams.
• Inspect scenario-level results. Look for a discrepancy concentrated in one scenario rather than relying only on a pooled average that could conceal it.
• Compare with the appropriate uncertainty. Use independent-run calculations for independent streams and paired differences when common random numbers create matched outcomes.

Patterns can guide diagnosis. A small discrepancy that shrinks as replication counts increase is consistent with Monte Carlo noise. A large, repeatable discrepancy tied to one scenario, a changed denominator, or a particular stream allocation calls for checking the run setup and summaries. A changing estimate across runs is not itself proof of a defect; conversely, identical output from a reused stream is not evidence from an independent replication.

Common Mistakes and Limitations

A common mistake is to say, “The estimates differ, so the design or code must have changed.” That reasoning ignores Monte Carlo uncertainty. The correction is to first verify comparability, then scale the observed difference by an uncertainty estimate appropriate to the replication counts and stream relationship. The opposite mistake is also possible: labeling every discrepancy “random noise” without checking scenario definitions, completed replication counts, and stream allocation.

Another pitfall is pooling all replicates from several runs without checking whether they estimate the same quantity. Pooling is sensible for independent replicates of a common scenario and outcome definition, with the total number of valid replicates used as the denominator. It is misleading if scenarios differ, if some runs use different rules, or if duplicated random streams cause replicates to be counted more than once. When runs share random numbers, estimates are correlated, so formulas that assume independent runs may misstate uncertainty.

Finally, a standardized discrepancy is a summary, not a diagnosis. It cannot identify whether a problem arose from the scenario configuration, stream management, implementation, or an unusual random draw. Monte Carlo uncertainty also concerns the numerical precision of a simulation estimate; it does not cover uncertainty in assumptions about the trial, the appropriateness of the simulated model, or the relevance of the scenarios.