Why Low Event Rates Need Special Attention
When a clinical outcome is rare, a trial can contain many participants but very few events. That sparsity affects more than the precision of an effect estimate: it can change how often a trial crosses its decision threshold, how an estimator behaves near zero, and how often an interval procedure covers the true event rate. Operating characteristics therefore need to be checked under data-generating scenarios that include low event rates, rather than inferred from large-sample approximations alone.
The central task is to repeatedly generate trial data under a specified scenario, apply the planned analysis and decision rule to each trial, and summarize the resulting decisions and estimates. For binary outcomes, the scenario must specify the event probability in each arm, the sample size or allocation, and any relevant dependence or stratification. The simulation must reproduce the outcome-generating process and the analysis that the trial will actually use.
For a simple starting model, suppose each participant in an arm has a binary outcome, with 1 denoting an event and 0 denoting no event. If outcomes are independent and have the same event probability \(p\), then the event count \(X\) among \(n\) participants follows \(X\sim\operatorname{Binomial}(n,p)\). In a two-arm trial, this assumption can be made separately for each arm, using probabilities \(p_T\) and \(p_C\). A more complex trial may require varying risks, clustering, covariates, or other features; the simulation assumptions must then represent those features rather than silently treating all outcomes as identical independent Bernoulli observations.
Event Counts and Approximation Failure
The sample event proportion is \(\hat{p}=X/n\). Under the binomial model, \(E(\hat{p})=p\) and \(\operatorname{Var}(\hat{p})=p(1-p)/n\). These identities do not imply that the sampling distribution of \(\hat{p}\) is well approximated by a normal distribution. When \(np\) is small, the count is discrete, often concentrated at zero or one, and the estimate can pile up at the boundary \(\hat{p}=0\). The standard error formula can still be correct while a normal approximation based on it is poor.
Worked Example: Probability of At Most One Event
Suppose, hypothetically, that 100 participants have independent binary outcomes, each with event probability \(p=0.01\). The expected count is \(np=1\), so zero or one event is not an unusual form of data sparsity. Calculate the probability of observing at most one event.
Under the stated independent, common-risk model, \(X\sim\operatorname{Binomial}(100,0.01)\). Therefore,
Thus the probability is approximately 0.7358, or 73.58%. The displayed components round separately to 0.3660 and 0.3697, whose displayed sum is 0.7357; the final result is rounded from the unrounded terms, not from that rounded sum.
For comparison, a normal approximation to \(X\) has mean 1 and standard deviation \(\sqrt{0.99}\approx0.9950\). Applying a continuity correction gives approximately \(P(-0.5<X<1.5)\), which standardizes to \(P(-1.508<Z<0.503)\), or about 0.626. This is far from the exact probability 0.7358. Here the expected event count is too small for that approximation to represent the count distribution well.
This calculation also illustrates why diagnostics should examine the distribution of event counts, not just the total sample size. Useful summaries include the fraction of replicates with zero events in either arm, the fraction with no events in the entire trial, and the frequency of very small cell counts. A large nominal sample does not eliminate sparse-data problems if the underlying risk is sufficiently low.
Decision Rates Under a Rare-Event Scenario
A decision rate is the proportion of simulated trials that satisfy a prespecified decision rule. The rule, prior, analysis, and threshold must be fixed before interpreting this proportion. For example, a Bayesian rule might declare the treatment arm favorable when the posterior probability that its event risk is lower than the control risk exceeds 0.95. The rule is evaluated separately on each simulated data set; a decision rate is not obtained by applying the rule once to average data.
Worked Example: Simulated False-Decision Rate
Consider a hypothetical calibration run with 10,000 simulated trials. In every trial, each arm has 100 participants, the event outcomes are independent within and between arms, and both true event probabilities are \(p_T=p_C=0.01\). Suppose the planned analysis uses independent \(\operatorname{Beta}(1,1)\) priors for the two event probabilities and declares treatment favorable when \(P(p_T<p_C\mid D)>0.95\), where \(D\) denotes that trial's data. Assume the simulation records 615 favorable decisions.
Because the two true event probabilities are equal, a favorable decision is a false favorable decision under this scenario. The estimated decision rate is
or 6.15%. This is an estimate of the rule's false favorable decision probability under the specified scenario, not a claim that the posterior probability of the null hypothesis is 6.15%. The posterior threshold used to make each decision and the repeated-sampling frequency of those decisions are different quantities.
The simulation itself has finite Monte Carlo uncertainty. For \(R\) independent simulated trials and a binary outcome \(I_r\) indicating whether replicate \(r\) meets the decision rule, the estimator is \(\hat{\theta}=R^{-1}\sum_{r=1}^{R}I_r\). Since each \(I_r\) is Bernoulli with success probability \(\theta\), its variance is \(\theta(1-\theta)\). Independence across replicates then gives \(\operatorname{Var}(\hat{\theta})=\theta(1-\theta)/R\). Estimating \(\theta\) by \(\hat{\theta}\) gives the Monte Carlo standard error
The estimated Monte Carlo standard error is about 0.24 percentage points. It measures simulation noise in estimating the operating characteristic; it is not the standard error of a treatment-effect estimate in an individual trial. With rare events, many replicates may contain no events or only a few events. Those replicates still count in the decision-rate denominator and must not be discarded unless the planned analysis explicitly treats them otherwise.
Estimator Behavior and Interval Coverage
For a single arm under the binomial model, the maximum-likelihood estimate \(\hat{p}=X/n\) is unbiased, but unbiasedness does not ensure that its distribution is approximately normal or that a particular interval procedure behaves well. With zero events, the estimate is exactly zero; a Wald interval formed as \(\hat{p}\pm1.96\sqrt{\hat{p}(1-\hat{p})/n}\) then has zero width. It fails to express uncertainty about a small, nonzero event probability. Other approximate intervals can also have poor coverage or behave awkwardly near the parameter boundary.
A Bayesian credible interval is an interval containing a specified fraction of the posterior distribution given the data and prior. Its repeated-sampling coverage is a separate property: under a fixed true \(p\), it is the probability across repeated data sets that the interval contains that \(p\). A 95% posterior credible interval does not, by definition, have 95% frequentist coverage at every fixed event rate. If coverage is an operating characteristic of interest, it should be assessed across simulated trials generated at specified true rates.
Worked Example: Assessing Coverage at a Rare Event Rate
Suppose a hypothetical simulation generates 10,000 independent data sets, each with \(n=100\) Bernoulli outcomes and true event probability \(p=0.01\). For each data set, the analyst uses a \(\operatorname{Beta}(1,1)\) prior and calculates the equal-tailed 95% posterior credible interval for \(p\). Assume 9,816 of the simulated intervals contain the true value 0.01.
The estimated repeated-sampling coverage is
or 98.16% in this illustrative run. Its Monte Carlo standard error is
Thus the reported coverage is about 98.16%, with Monte Carlo standard error about 0.13 percentage points. This example concerns coverage under the stated true rate and data-generating model. It does not mean that the posterior probability of \(p=0.01\) is 98.16%, nor does it establish coverage at other event rates, sample sizes, or prior choices. In a calibration study, the interval construction and prior are part of the procedure being evaluated and should be applied consistently to every replicate.
For a sparse binary endpoint, exact binomial calculations can check event-count probabilities and estimator properties when their assumptions apply. Simulation is useful for evaluating a complete analysis pipeline, including a Bayesian decision rule, interval construction, stratification, and other planned trial features. Asymptotic methods may be convenient, but their adequacy should be demonstrated in the relevant low-rate scenarios rather than assumed from general large-sample reasoning.
Diagnostics, Alternatives, and Limitations
Low-rate calibration is most informative when it makes the source of sparsity visible and separates uncertainty about the trial procedure from Monte Carlo error. Inspect at least the event-count distribution in each arm, the frequency of zero-event and very-low-event trials, and the estimates and intervals produced in those trials. Check that the analysis succeeds for boundary data; numerical failures, undefined estimates, or silent exclusions can distort operating characteristics. If trial outcomes are stratified or otherwise dependent, verify that the simulation preserves the design and dependence structure.
| Approach | Useful role | Low-rate concern |
|---|---|---|
| Exact binomial calculation | Checks event-count probabilities under independent, common-risk Bernoulli outcomes | Does not by itself evaluate a complicated trial analysis or decision pipeline |
| Repeated trial simulation | Evaluates decisions, estimators, and intervals under a specified full scenario | Has Monte Carlo error and depends on correctly specified simulation assumptions |
| Large-sample normal approximation | Can be a convenient calculation when its approximation is adequate | May be inaccurate with few events, boundary estimates, or sparse cells |
A common mistake is to say, “The trial has 100 participants per arm, so the normal approximation is reliable.” The relevant event information is not determined by \(n\) alone: with \(p=0.01\), the expected count in an arm of 100 is only one, and the probability of zero events is \(0.99^{100}\approx0.3660\). The correction is to examine the event-count distribution under each scenario, use a procedure appropriate for sparse outcomes, and calibrate the complete analysis at those low rates.
Another limitation is that a reassuring result in one scenario is not a guarantee of performance elsewhere. Coverage or decision rates can change with the event probabilities, allocation, sample size, prior, or outcome model. A simulation estimates performance only for the scenarios and implementation it actually evaluates. Report the scenario assumptions, number of replicates, estimated rate, and Monte Carlo uncertainty so that the evidence can be judged in context.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions below.
- If \(n=200\) and \(p=0.005\), what is the expected event count, and why does that alone not establish that a normal approximation is reliable?
- A simulation makes 80 favorable decisions in 2,000 trials under a null scenario. Calculate the estimated decision rate.
- In one sentence, distinguish a 95% posterior credible interval from 95% repeated-sampling coverage.
- Why can the Wald interval have zero width when no events are observed?
- Name two diagnostics that should be examined when calibrating a rare binary endpoint.
Solutions. (1) The expected count is \(np=200(0.005)=1\). A small expected count suggests substantial mass at very low counts, so the distribution may be discrete and poorly approximated by a normal distribution. (2) The estimated rate is \(80/2000=0.04\), or 4%. (3) A 95% credible interval contains 95% posterior probability conditional on the data and prior, whereas 95% repeated-sampling coverage means that the procedure contains the fixed true value in 95% of repeated data sets under a specified scenario. (4) With \(\hat{p}=0\), the Wald standard error \(\sqrt{\hat{p}(1-\hat{p})/n}\) is zero, so both interval endpoints equal zero. (5) Suitable diagnostics include the proportion of replicates with zero events in either arm and the distribution of event counts; checking interval coverage or numerical failures is also useful.