Diagnosing Monte Carlo Error and Simulation Failures

Why Monte Carlo Diagnostics Matter

A simulation estimates an operating characteristic by generating many trial paths under specified assumptions and summarizing what happens. For example, the estimated Type I error is the fraction of paths under \(H_0\) that reject \(H_0\). That fraction is not usually the exact probability specified by the simulation model: it has Monte Carlo error because it is computed from a finite number of random paths.

A small discrepancy from a target may therefore be ordinary simulation noise. But a smooth-looking estimate is not proof that the simulation is correct, and an error message is not a harmless inconvenience. The diagnostic task is to quantify the uncertainty expected from finite replication, inspect whether estimates stabilize as paths accumulate, and investigate warnings and failed paths before deciding what the results support.

Monte Carlo Error for an Operating Characteristic

Suppose each simulated path produces a binary indicator \(X_r\), equal to 1 if a specified event occurs and 0 otherwise. For a Type I error simulation, the event might be rejection of \(H_0\) when \(H_0\) is true. For a power simulation, it might be rejection of \(H_0\) under a specified alternative. If the paths are generated independently under the same scenario, then \(X_r\) are independent Bernoulli random variables with event probability \(p\). The estimate is \(\hat{p}=\sum_{r=1}^{R}X_r/R\).

The Bernoulli variance is \(p(1-p)\). Replacing the unknown \(p\) with its estimate gives the usual estimated MCSE:

When the number of events and non-events is sufficiently large, an approximate 95% Monte Carlo confidence interval is \(\hat{p}\pm1.96\widehat{SE}_{MC}(\hat{p})\). It describes the precision of the simulated estimate under the assumed simulation model. It does not account for uncertainty about whether that model, its nuisance parameters, or its implementation represents the intended design.

The normal approximation can be poor when \(R\hat{p}\) or \(R(1-\hat{p})\) is small, especially for rare events or estimates near 0 or 1. In those cases, use a binomial interval designed for small counts, such as a Wilson or exact interval, rather than relying on an interval that can extend outside the probability range. The interval method does not fix a poor simulation model or a defective implementation.

A Short Derivation of the MCSE

For independent paths, \(\hat{p}\) is the average of \(R\) Bernoulli indicators. The variance of an average of independent variables is the variance of one variable divided by \(R\). Therefore:

Taking the square root gives the standard deviation of \(\hat{p}\), namely \(\sqrt{p(1-p)/R}\). Since \(p\) is unknown, substituting \(\hat{p}\) yields the estimated MCSE. This calculation relies on independent simulated paths generated under the same scenario. Repeated interim looks within one path are not independent simulation replicates: the path, not each look, is the unit contributing one simulated trial outcome.

Precision, Convergence Patterns, and What They Can Show

A useful first check is whether the MCSE is small enough for the decision being made. A result can be numerically close to a target but too imprecise to distinguish an acceptable design from an unacceptable one. Conversely, a modest difference from the target may be unremarkable when the MCSE is large. The practical standard for adequate precision should be set in relation to the design question, not inferred after seeing a favorable estimate.

Cumulative estimates plot or tabulate \(\hat{p}\) after increasing numbers of paths. Batch estimates divide the generated paths into non-overlapping groups and calculate a separate estimate for each group. These views can reveal large jumps, persistent drift, unusual batch discrepancies, or a result that depends strongly on a small portion of the simulation. They are diagnostic summaries, not proof that the code is correct or that a theoretical limit has been reached. Random estimates naturally fluctuate, and a flat-looking segment can occur by chance.

When the simulated outcome is a numerical quantity rather than an event indicator, such as the sample size at stopping, calculate its mean across paths and estimate the MCSE as \(s/\sqrt{R}\), where \(s\) is the sample standard deviation across independent paths. This summary requires finite variance for the usual interpretation and can be sensitive to a highly skewed distribution. For stopping look or sample size distributions, the quantity and summary should be stated clearly; the event-probability formula should not be applied to a numerical outcome as though it were binary.

Warnings and Failed Replicates

A warning may indicate a harmless numerical event, a boundary case that the method is designed to handle, or a failure that changes the simulated outcome. Its meaning depends on where it occurs and what values the code returns. Record the warning type, scenario, replicate identifier, and relevant inputs and outputs. Then determine whether that path produced a valid outcome under the prespecified simulation procedure. Suppressing warnings without investigating them removes diagnostic information.

A failed replicate is not automatically a non-event. If a path fails before the operating-characteristic outcome is computed, silently removing it and using only successful paths estimates an event rate conditional on successful computation. That can differ from the intended probability across all simulated trial paths, particularly if failures occur more often for particular data patterns. If an outcome is genuinely defined to include a particular event, such as a planned non-convergence outcome, that definition must be part of the simulation target rather than an after-the-fact recoding.

Investigate whether failures cluster by scenario, interim look, boundary proximity, or unusual generated data. Check whether the failure rate is negligible for a justified reason, whether paths can be reproduced, and whether failure handling was defined before summarizing results. If a path is rerun, retain its original identity and document why and how it was rerun. A low overall failure fraction does not establish that the estimate is unaffected if failures are selective.

A Practical Diagnostic Sequence

Comparisons and Limitations

An MCSE or Monte Carlo interval is more informative than reporting only a rounded estimate, because it shows how much finite replication could plausibly move the result. Increasing \(R\) usually decreases the MCSE at a rate proportional to \(1/\sqrt{R}\); for example, quadrupling \(R\) approximately halves it for the same event probability. This improvement can be computationally costly, and it addresses only simulation precision.

Cumulative and batch checks complement, but do not replace, the MCSE. They may expose unstable patterns or anomalous groups that a single overall interval conceals, but they cannot certify the simulation procedure. Likewise, a small MCSE cannot rule out coding errors, incorrect assumptions, or selective failures. Independent software verification and formal testing are separate safeguards. For a rare event, the ordinary normal interval may be unreliable and the event count may remain too small for a useful estimate even when \(R\) seems large; use a suitable interval and consider whether the design question requires greater precision.

Common Pitfall

Wrong reasoning: “The simulated error rate is 4.7%, below the 5% target, so the design meets the target.” This treats a finite-replication estimate as exact. It may also ignore failures or assume that the simulated model is correct.

Correction: Report the estimate with its MCSE or interval and state the scenario, replication count, and treatment of failed paths. Judge whether the result is precise enough for the decision, then separately assess whether warnings, failures, or model choices undermine its interpretation. A Monte Carlo interval is not a guarantee that the design is correct; it quantifies only one source of uncertainty.