Monte Carlo Precision in Type I Error Validation

Monte Carlo Uncertainty in a Simulated Type I Error Rate

A simulation can estimate how often an adaptive test rejects \(H_0\) when the null model is true. But the estimated rate is itself random: another set of simulated trials could produce a different number of rejections. Monte Carlo precision describes this simulation-to-simulation variability. It is separate from uncertainty about whether the simulated model represents the scientific setting adequately.

The earlier material on simulating Type I error across adaptation rules concerns how to generate trial outcomes and apply the complete test procedure. Here, take that procedure as specified and quantify the sampling uncertainty in its simulated rejection rate. The target is the Type I error probability for a particular null distribution, design, adaptation rule, and rejection criterion—not a universal guarantee over every possible null parameter value.

This binomial model requires independent replicates generated under the same specified null scenario, with the same analysis rule applied to each. Any randomization used by the trial procedure should be generated as part of each replicate when it belongs to the design. If the simulated trials share random inputs or are otherwise dependent, the binomial standard error below does not automatically apply. Correctly simulating the specified null scenario is also essential: a narrow interval cannot repair an incorrectly coded adaptation rule or an inappropriate null model.

Standard Error and Confidence Interval

Under the binomial model, \(E(\hat{\pi})=\pi\) and \(\operatorname{Var}(\hat{\pi})=\pi(1-\pi)/M\). Since \(\pi\) is unknown, estimate the standard deviation of \(\hat{\pi}\) by substituting \(\hat{\pi}\):

This is the Monte Carlo standard error (MCSE). It measures typical variation in the estimated rejection proportion across repeated simulations of size \(M\), under the stated binomial assumptions. It is not the standard error of a treatment effect estimated within one simulated trial.

A simple approximate interval is \(\hat{\pi}\pm z_{1-\gamma/2}\widehat{SE}_{MC}\), where \(1-\gamma\) is the confidence level. This Wald interval is convenient when \(M\hat{\pi}\) and \(M(1-\hat{\pi})\) are not small, but it can extend beyond zero or one and performs poorly near those boundaries. The Wilson score interval is a useful alternative. Put \(z=z_{1-\gamma/2}\); its center and half-width are:

The Wilson interval is \([C-H,C+H]\). It follows by inverting the binomial score test: retain candidate values \(\pi\) for which the observed proportion is within \(z\) estimated standard errors of \(\pi\), and solve that inequality for \(\pi\). Unlike the Wald interval, the resulting limits stay within \([0,1]\). An exact binomial interval is another option, particularly when the count of rejections is very small; exact intervals can be conservative because the binomial count is discrete.

Choosing a Simulation Size for Desired Precision

Before running a large validation, it is often useful to determine how many replicates are needed for a target margin of error. For a two-sided confidence level \(1-\gamma\), a normal-approximation interval has half-width approximately

Solving for \(M\), and substituting a planning value \(q\) for the unknown \(\pi\), gives

Round the result up to a whole number. The choice of \(q\) should be justified by the anticipated Type I error rate or a pilot simulation. Since \(q(1-q)\leq 1/4\), using \(q=0.5\) gives a conservative size for the approximate interval’s standard error when no plausible rate is available. For a test intended to have Type I error near \(0.05\), planning around \(q=0.05\) is usually less conservative but requires confidence that this is a reasonable planning value.

Rare Rejections, Comparisons, and Failure Modes

When no simulated trial rejects, the plug-in MCSE is zero. That is a computational consequence of substituting \(\hat{\pi}=0\), not evidence that the underlying Type I error probability is zero. For example, if \(K=0\), a two-sided 95% exact binomial interval has lower limit zero and upper limit \(1-0.025^{1/M}\). With \(M=2{,}000\), the upper limit is approximately \(0.001843\). The simulation is consistent with a small positive rejection probability; it does not establish that the probability vanishes.

The standard error formula assumes independent replicates and a fixed simulation size. Repeatedly checking the estimate and stopping as soon as the interval appears sufficiently narrow can invalidate ordinary fixed-\(M\) confidence-interval coverage. Set \(M\) in advance, or use an interval and stopping rule specifically justified for sequential Monte Carlo. Likewise, if results across several adaptation rules are presented as a family of simultaneous claims, separate 95% intervals do not automatically provide 95% simultaneous coverage.

These calculations address random error from finite simulation. They do not diagnose all errors in the validation. A coding error, an incorrectly generated null distribution, or failure to include all parts of the adaptation algorithm can make the estimate precise but wrong for the intended procedure. The binomial calculations also apply to a particular simulated null configuration. Establishing control across a composite null may require examining appropriate parameter values or justifying why the selected scenario is conservative.

Common Mistake and Practical Checks

Wrong reasoning: “The simulated rate is \(0.0493\), close to \(0.05\), so the test has been shown to control Type I error at \(0.05\).” The point estimate alone ignores Monte Carlo variability and applies only to the simulated null scenario.

Correction: Report \(K\), \(M\), \(\hat{\pi}\), an MCSE or confidence interval, and the null scenario and test procedure used. Compare the interval with the nominal level as a description of simulation uncertainty, not as a substitute for checking the simulation design. If the interval is too wide for the intended assessment, increase the prespecified simulation size. If the result is surprising, verify that each replicate is independent, the adaptation and stopping rules are implemented as intended, and rejection is defined consistently with the test.

For a target half-width \(d\), the approximate planning equation also shows the basic cost of greater precision: \(M\) grows proportionally to \(1/d^2\). Halving the desired half-width therefore requires about four times as many replicates, all else equal. This makes explicit why a very precise validation can require substantial computation even when each simulated trial is inexpensive.