Confidence Intervals for Simulated Error Rates

Why Put an Interval Around a Simulated Error Rate?

A simulation estimates an operating characteristic by repeating a trial-generation and decision procedure under a specified scenario. For example, under a global-null scenario, a replicate may be labelled an error if the simulated trial rejects at least one true null hypothesis. Across many replicates, the proportion labelled as errors estimates the scenario’s familywise error probability. That proportion is itself random: a different set of simulation replicates would generally produce a different value.

A confidence interval describes this Monte Carlo uncertainty. It does not account for every uncertainty in a MAMS design, and it does not establish that the simulation model or code is correct. Its interpretation is conditional on the specified scenario, the simulated data-generating model, and the rule used to classify each replicate.

Definition: Suppose $R$ independent simulation replicates are generated under the same scenario. Let $X_i=1$ if replicate $i$ has the event of interest and $X_i=0$ otherwise. If each replicate has the same event probability $p$, then $X=\sum_{i=1}^R X_i$ has a binomial distribution, $X\sim\operatorname{Binomial}(R,p)$. The simulated event-rate estimate is $\hat{p}=X/R$.

For familywise error, define the event before simulation begins. A replicate is an error if the procedure makes at least one false rejection among the hypotheses that are true in the specified scenario. Thus a replicate contributes one binary outcome, even if more than one true null is rejected. Other simulated probabilities, such as the probability of a particular decision, can be handled in the same way when each replicate yields one binary indicator for that event.

The binomial model requires independent replicates with a common event probability $p$. Dependence among decisions within one replicate does not invalidate this model: those decisions are combined into that replicate’s single event indicator. But if the replicate indicators are dependent, or if the simulation settings change from replicate to replicate so their event probabilities differ, an ordinary binomial interval may not apply.

Wilson Score Intervals

A Wilson interval is a practical default for a binomial proportion. Unlike the simple Wald interval, it behaves sensibly near the boundaries zero and one. For a two-sided confidence level $1-\alpha$, let $z_{1-\alpha/2}$ be the corresponding standard normal quantile. The Wilson interval is:

\frac{ \hat{p}+\dfrac{z_{1-\alpha/2}^{\,2}}{2R} \ \pm\ z_{1-\alpha/2} \sqrt{\dfrac{\hat{p}(1-\hat{p})}{R} +\dfrac{z_{1-\alpha/2}^{\,2}}{4R^2}} }{ 1+\dfrac{z_{1-\alpha/2}^{\,2}}{R} }

The plus sign gives the upper endpoint and the minus sign gives the lower endpoint. For a 95% interval, $z_{0.975}\approx1.96$. The endpoints are proportions; multiply by 100 to express them as percentages.

Justification: Under the binomial model, the standard error of $\hat{p}$, when evaluated at a candidate value $p$, is $\sqrt{p(1-p)/R}$. Inverting the approximate score condition $\left|\hat{p}-p\right|\le z_{1-\alpha/2}\sqrt{p(1-p)/R}$ means retaining the values of $p$ that satisfy $$ R(\hat{p}-p)^2\le z_{1-\alpha/2}^{\,2}p(1-p). $$ Collecting terms gives a quadratic inequality in $p$. Solving its equality gives the Wilson endpoints shown above. Because $p$ appears in the standard error, this procedure does not simply place a fixed standard error around $\hat{p}$.

Worked Example: A Simulated Familywise Error Rate

In a hypothetical global-null simulation, 28 of $R=1{,}000$ independent replicates result in at least one false rejection. Estimate the familywise error probability and calculate a 95% Wilson interval.

The point estimate is $\hat{p}=28/1{,}000=0.028$, or 2.8%. With $z=1.96$, the denominator in the Wilson formula is $1+3.8416/1{,}000=1.003842$. The center is $$ \frac{0.028+3.8416/(2{,}000)}{1.003842} =\frac{0.0299208}{1.003842} \approx 0.02981. $$ The half-width is $$ \frac{1.96\sqrt{(0.028)(0.972)/1{,}000+3.8416/(4{,}000{,}000)}}{1.003842} = \frac{1.96\sqrt{0.0000281764}}{1.003842} \approx 0.01036. $$

Therefore, the 95% Wilson interval is approximately $0.02981\pm0.01036$, or $(0.01944,\,0.04017)$. In this hypothetical simulation, the estimated familywise error probability is 2.8%, with a 95% interval from about 1.94% to 4.02%. This interval reflects simulation variability under the specified global-null model; it is not a probability statement that the fixed value $p$ lies in this particular interval.

Exact Binomial Intervals and Boundary Counts

The Clopper–Pearson interval uses the binomial distribution directly rather than a normal approximation. Its endpoints are binomial-tail quantiles, equivalently beta-distribution quantiles. Write $Q_{\mathrm{Beta}(a,b)}(u)$ for the $u$ quantile of a beta distribution with shape parameters $a$ and $b$. For $0<X<R$, the two-sided exact interval is:

\left( Q_{\mathrm{Beta}(X,\,R-X+1)}(\alpha/2),\ Q_{\mathrm{Beta}(X+1,\,R-X)}(1-\alpha/2) \right).

If $X=0$, the lower endpoint is 0 and the upper endpoint is $1-(\alpha/2)^{1/R}$. If $X=R$, the lower endpoint is $(\alpha/2)^{1/R}$ and the upper endpoint is 1. These boundary formulas avoid beta distributions with a zero shape parameter. The interval has coverage at least its nominal level under the fixed-$R$ binomial model, though discreteness often makes it conservative: its actual coverage can exceed the stated level.

Worked Example: No Errors Observed

A hypothetical simulation produces zero familywise errors in $R=80$ independent global-null replicates. Find the two-sided 95% Clopper–Pearson interval.

Here $X=0$, so $\hat{p}=0/80=0$. The exact lower endpoint is 0. The upper endpoint is $$ 1-(0.025)^{1/80} =1-\exp\left(\frac{\ln(0.025)}{80}\right) =1-\exp(-0.04611) \approx 1-0.9549 =0.04506. $$

The interval is approximately $(0,\,0.0451)$, or 0% to 4.51%. Observing no errors in 80 replicates does not show that the event probability is zero. Under the binomial model, values up to about 4.51% remain inside this two-sided 95% exact interval.

A zero count is a boundary case for interval construction, not evidence that the simulation has identified a zero probability. Wilson intervals also remain defined at this boundary. For example, with $X=0$, the Wilson upper endpoint is $z^2/(R+z^2)$, and its lower endpoint is 0. The exact and Wilson intervals need not have the same endpoints; their different coverage properties are part of the choice between them.

Worked Example: Comparing Wilson and Wald Intervals

Suppose a hypothetical simulation has $X=16$ error events among $R=200$ independent replicates. Calculate the Wilson interval and compare it with the Wald interval.

The estimate is $\hat{p}=16/200=0.080$. For a 95% Wilson interval, the denominator is $1+3.8416/200=1.019208$, and the center is $$ \frac{0.080+3.8416/400}{1.019208} =\frac{0.089604}{1.019208} \approx 0.08792. $$ The half-width is $$ \frac{1.96\sqrt{(0.080)(0.920)/200+3.8416/(4\cdot200^2)}}{1.019208} = \frac{1.96\sqrt{0.00039201}}{1.019208} \approx 0.03808. $$ Thus the Wilson interval is approximately $(0.04984,\,0.1260)$, or 4.98% to 12.60%.

The Wald interval uses $\hat{p}\pm1.96\sqrt{\hat{p}(1-\hat{p})/R}$. Here its standard error is $\sqrt{0.080(0.920)/200}=\sqrt{0.000368}\approx0.01918$, giving $(0.0424,\,0.1176)$. Both intervals are inside $[0,1]$ in this example, but the Wald interval can have poor coverage, especially for small counts or estimates near a boundary. It is not the preferred general-purpose interval.

Choosing and Interpreting an Interval

Wilson and Clopper–Pearson intervals are both defensible choices for simulated binomial rates. Wilson intervals are usually shorter and have good approximate coverage; Clopper–Pearson intervals provide at least the nominal coverage under the binomial model, often at the cost of being wider. The Wald interval is simple but can extend below 0 or above 1 and may undercover. For a planned analysis, specify the interval method and confidence level so results are comparable across scenarios.

Method	Basis	Practical feature	Important caution
Wilson score	Inversion of an approximate binomial score test	Usually performs well and stays within 0 and 1	Coverage is approximate, not guaranteed to be at least nominal for every $p$ and $R$
Clopper–Pearson exact	Binomial tail probabilities	Coverage is at least the nominal level under the model	Can be conservative and wider than necessary
Wald	Estimate plus or minus a normal-approximation margin	Easy to calculate	Can have poor coverage and invalid endpoints, especially near 0 or 1

A 95% confidence interval procedure has 95% coverage in repeated applications under its stated conditions: across repetitions of the simulation with the same fixed $p$, approximately 95% of intervals from a nominal 95% procedure contain $p$, subject to the method’s coverage properties. The interval from one run does not assign a 95% probability to $p$. It is also not a measure of uncertainty about whether the specified MAMS model represents a real trial, or whether the software correctly implements the intended rules.

Diagnostics, Failure Modes, and Limitations

Before interpreting an interval, check how the event count was created. Verify that every replicate used the intended scenario and decision rule, that an event was counted once per replicate, and that the denominator $R$ includes only valid replicates as planned. Unexpectedly invalid or omitted replicates can change the estimand if their exclusion is related to the event. Also check that replicates are independent and have a common event probability. Shared outcomes or decisions within a simulated trial are compatible with the binomial setup when each trial contributes one indicator; dependence between simulation replicates is a different issue.

Ordinary Wilson and Clopper–Pearson intervals assume a fixed number of independent replicates. Repeatedly checking the interval and stopping simulation when it first becomes sufficiently narrow is not generally covered by the usual fixed-sample interpretation. The resulting stopping rule can affect coverage. Likewise, combining counts from scenarios with different event probabilities does not generally produce a binomial count with one common $p$.

The interval quantifies Monte Carlo error only. If important nuisance parameters, accrual assumptions, or analysis rules are uncertain, a narrow interval for one scenario does not resolve that uncertainty. Nor can a confidence interval detect a coding error shared by all replicates. Independent code checks and scenario validation address different problems.

Common pitfall: “We observed zero errors, so the familywise error probability is zero.” The conclusion does not follow: the observed rate is zero, but the underlying probability can be positive. Report an interval, state the replicate count, and interpret the upper endpoint as a bound from the chosen interval procedure under its assumptions.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions.

In 500 independent replicates, 10 have at least one false rejection under a global-null scenario. What is the simulated rate, and what is the random variable used for a binomial interval?
Why can several dependent hypothesis decisions within a single replicate still lead to a binomial model for the simulated familywise error count?
For $X=0$ in $R=100$ replicates, calculate the upper endpoint of the two-sided 95% Clopper–Pearson interval using $1-(0.025)^{1/R}$.
State one advantage of the Wilson interval over the Wald interval and one advantage of the Clopper–Pearson interval over Wilson.

Solutions. (1) The rate is $10/500=0.020$, or 2.0%. For each replicate, define $X_i=1$ if at least one true null is falsely rejected and 0 otherwise; the total $X=\sum X_i$ is binomial when replicate indicators are independent with a common probability. (2) The familywise event is one binary outcome per replicate, so dependence among decisions within that replicate is absorbed into its indicator. (3) The upper endpoint is $1-(0.025)^{1/100}=1-\exp(\ln(0.025)/100)\approx1-\exp(-0.03689)\approx0.03622$, or 3.62%; the interval is $(0,\,0.03622)$. (4) Wilson intervals generally avoid invalid boundaries and perform better than Wald intervals near 0 or 1. Clopper–Pearson intervals have at least the nominal coverage under the binomial model, while Wilson coverage is approximate.