Rare-Event Error-Rate Estimation

Why Rare Errors Need Deliberate Planning

A simulation can estimate a false-rejection probability only by generating enough trial replicates for false rejections to occur often enough to be informative. When the probability is small, a seemingly large number of replicates may still produce only a handful of events—or none. In that situation, reporting the simulated proportion without considering its Monte Carlo uncertainty can give an unjustifiably precise impression.

For a MAMS design, a replicate may count as a false rejection when the trial’s decision rule rejects at least one true null hypothesis under a specified scenario. Under a global-null scenario, this event estimates the global-null familywise error rate. For strong error control, the corresponding error event and rate must be evaluated under each relevant partial-null scenario as well. The scenario defines the probability being estimated; pooling scenarios generally estimates a different quantity.

Definition: In $n$ independent simulation replicates under a fixed scenario, let $X_i=1$ if replicate $i$ has the specified false-rejection event and $X_i=0$ otherwise. If the event probability is $p$, then $X_i\sim\operatorname{Bernoulli}(p)$, and the total event count $K=\sum_{i=1}^{n}X_i$ has a binomial distribution with parameters $n$ and $p$. The crude Monte Carlo estimate is $\hat{p}=K/n$.

This model assumes that replicates are generated independently from the same specified scenario and that the event is defined consistently. Dependence among treatment comparisons within a simulated trial is allowed: the replicate-level indicator records the trial-wide event, and independence is required across simulation replicates. The simulation code must also implement the intended trial design and decision rule. More replicates cannot correct a wrongly specified scenario or a coding error.

From Replicate Counts to Precision

Since $X_i$ is Bernoulli, its mean is $p$ and its variance is $p(1-p)$. Independence across replicates gives $\operatorname{Var}(\hat{p})=p(1-p)/n$. The Monte Carlo standard error (MCSE) is therefore estimated by substituting $\hat{p}$ for the unknown $p$.

\operatorname{MCSE}(\hat{p}) \approx \sqrt{\frac{\hat{p}(1-\hat{p})}{n}}, \qquad \operatorname{CV}(\hat{p}) \approx \frac{\operatorname{MCSE}(\hat{p})}{\hat{p}} = \sqrt{\frac{1-\hat{p}}{n\hat{p}}}.

The coefficient of variation (CV) expresses the MCSE relative to the estimated rate. For planning, replace $\hat{p}$ with a plausible value $p_0$. If $p_0$ is small, the expected number of events is $\lambda=np_0$, and the relative MCSE is approximately $1/\sqrt{\lambda}$. Thus, achieving about 10% relative MCSE requires roughly 100 expected events; about 5% requires 400. These are planning approximations, not guarantees of a particular interval width.

Planning result: For a target relative MCSE $r$ at a plausible error rate $p_0$, choose $$ n \geq \frac{1-p_0}{p_0r^2}. $$ For a target absolute MCSE $\delta$, choose $$ n \geq \frac{p_0(1-p_0)}{\delta^2}. $$ Round the result up to a whole number of replicates. For small $p_0$, the relative-precision formula is approximately $n\geq 1/(p_0r^2)$.

Relative and absolute precision answer different questions. A rate of 0.001 estimated with MCSE 0.0001 has about 10% relative MCSE, but its absolute MCSE is 0.0001. An absolute tolerance of 0.001 would be uninformative for assessing that rate, whereas a tolerance of 0.0001 may require many replicates. Select the precision target according to the decision the estimate must support.

Worked Example: Planning for a Very Small Rate

Suppose a design is expected to have a familywise false-rejection probability near 0.001. Plan for a relative MCSE no greater than 10%, using the crude Monte Carlo estimator.

Set $p_0=0.001$ and $r=0.10$. The planning formula gives $$ n \geq \frac{1-0.001}{0.001(0.10)^2} =\frac{0.999}{0.00001}=99{,}900. $$ Thus, plan for at least 99,900 independent replicates; choosing 100,000 gives an expected event count of $100{,}000(0.001)=100$.

At $n=100{,}000$, the MCSE at the planning value is $$ \sqrt{\frac{0.001(0.999)}{100{,}000}} =\sqrt{0.00000000999} \approx 0.000100. $$ This is approximately 10.0% of 0.001. The calculation describes simulation precision if the assumed rate is about right; it does not establish that the true rate is 0.001. After running the simulation, report the observed event count and estimate, and assess uncertainty using an appropriate interval for a binomial rate.

What Few or Zero Events Mean

For a rare event, the count $K$, not only the proportion $K/n$, is central to interpretation. If the true rate is small, the count is approximately Poisson with mean $np$, which explains why expected event counts govern relative precision. The approximation is most useful when $n$ is large and $p$ is small; the binomial model remains the defining model here.

A zero count is possible even when the underlying rate is nonzero. In that case $\hat{p}=0$, but the plug-in MCSE is also zero, which must not be read as perfect precision. The plug-in formula fails to convey the uncertainty at the boundary. A binomial confidence interval, such as the interval methods covered in the earlier material on confidence intervals for simulated error rates, communicates that uncertainty. For example, with zero events in $n$ independent replicates, a one-sided 95% exact upper bound is $1-0.05^{1/n}$, approximately $3/n$ for large $n$. This is an upper bound, not evidence that the rate equals zero.

Worked Example: Interpreting a Zero Count

A hypothetical simulation runs 20,000 independent replicates under a fixed null scenario and observes no false rejections. What can be concluded about the rate?

The estimate is $\hat{p}=0/20{,}000=0$. It is not valid to conclude that the error rate is zero or that its MCSE is truly zero. Using the one-sided exact 95% upper bound for a zero count, $$ p_{\mathrm{upper}}=1-0.05^{1/20{,}000} \approx 0.0001498. $$ The simulation is compatible with rates up to about 0.000150 at this confidence level. The approximation $3/n$ gives $3/20{,}000=0.000150$, essentially the same result. If the goal were to make this upper bound smaller, the simulation would need substantially more replicates.

For comparison, if the true rate were 0.0001, the probability of seeing zero events in 20,000 replicates would be $$ (1-0.0001)^{20{,}000}\approx 0.1353. $$ So a zero count would not be surprising at that rate. The calculation illustrates why “no observed errors” is not equivalent to “no errors occur.”

A Replicate Count Is a Design Choice

Planning begins with the precision question, not with a conventional round number of simulations. The steps below translate a plausible rate into a count, then check whether the resulting event count is adequate for the intended report.

Define the event and scenario.
Specify exactly what constitutes a false rejection and which null configuration is being simulated. Keep distinct scenario-specific error rates distinct.

Choose a plausible planning rate.
Use design knowledge, an analytic calculation, or a pilot estimate. If the rate is uncertain, examine more than one plausible value rather than relying on a single optimistic guess.

Set the precision target.
Decide whether the purpose requires a relative MCSE, an absolute MCSE, or an interval whose upper bound is below a specified threshold.

Calculate and review the replicate count.
Use the relevant planning formula, round up, and calculate $np_0$. If the expected count is very small, ordinary normal approximations may be unreliable and the probability of zero events may be substantial.

Worked Example: Choosing a Count for 5% Relative MCSE

A hypothetical design has a plausible false-rejection rate of 0.002. The simulation team wants a relative MCSE of at most 5% at that planning value. Find a suitable count and check its expected event count.

Here $p_0=0.002$ and $r=0.05$. Therefore, $$ n \geq \frac{1-0.002}{0.002(0.05)^2} =\frac{0.998}{0.000005} =199{,}600. $$ Choose 200,000 replicates. The expected event count is $200{,}000(0.002)=400$.

The MCSE at the planning value is $$ \sqrt{\frac{0.002(0.998)}{200{,}000}} =\sqrt{0.00000000998} \approx 0.0000999, $$ which is about $0.0000999/0.002=0.0499$, or 4.99% relative MCSE. This meets the stated planning target. If the actual rate is substantially smaller, the event count and precision will also be smaller; the target is conditional on the planning value.

Diagnostics and Alternatives

After simulation, compare the observed count with the count anticipated at the planning rate. A large discrepancy does not by itself signal an error—the count is random—but it should prompt a check of the scenario, event definition, and implementation. Also verify that the replicates represent the intended sampling process and that no replicate has been silently excluded. If the observed count is low, avoid presenting a narrow normal-approximation interval without checking its suitability; exact or other binomial interval methods are generally preferable for sparse counts.

Crude Monte Carlo is direct and broadly applicable: each independent trial replicate contributes one event indicator. Its limitation is inefficiency when the event is extremely rare. An analytical calculation can be more efficient or exact when the model permits it, but may not capture a complex MAMS decision rule. Specialized rare-event simulation or variance-reduction methods may reduce computational cost, but they require additional assumptions and careful validation; their estimates are not automatically the simple count $K/n$. Whichever approach is used, report the estimand, scenario, estimator, uncertainty, and computational method.

Mistake: “We ran 10,000 replicates and observed no false rejections, so the rate is zero.” Correction: The estimate is zero, but uncertainty remains; quantify an upper bound and consider whether the replicate count can resolve rates of practical interest.
Mistake: “The observed MCSE is zero, so the estimate is exact.” Correction: With zero events the plug-in MCSE is uninformative at the boundary; use a binomial interval or a planned upper-bound calculation.
Mistake: “A large total number of replicates guarantees precision.” Correction: Precision depends on the event probability as well as $n$. Check expected and observed event counts for each scenario-specific rate.

Key takeaway: For a rare error rate, plan around the expected number of false-rejection events, $np_0$, as well as the total replicate count. Roughly 100 expected events correspond to about 10% relative MCSE, while zero observed events still permit a nonzero error rate.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions.

At a planning rate of 0.0005, approximately how many replicates are needed for 10% relative MCSE?
If the rate is 0.004 and $n=25{,}000$, what is the expected event count and approximate relative MCSE?
Why is a plug-in MCSE of zero after observing zero events misleading?
Under what circumstances should error-rate estimates from different null scenarios be reported separately?

Solutions. (1) For small $p_0$, $n\approx 1/(p_0r^2)=1/[0.0005(0.10)^2]=200{,}000$, corresponding to about 100 expected events. (2) The expected count is $25{,}000(0.004)=100$. The relative MCSE is approximately $\sqrt{(1-0.004)/100}=\sqrt{0.00996}\approx0.0998$, or 9.98%. (3) The plug-in estimate sets $\hat{p}=0$, so the formula returns zero, but a finite simulation with no events does not rule out a positive underlying rate; use a binomial interval or upper bound. (4) Report them separately when they represent distinct estimands, such as global-null and partial-null error rates relevant to strong error control. Pooling them changes the quantity being estimated unless a specific mixture probability is the intended estimand.