Why Put an Interval Around a Simulated Rate?
A simulated operating characteristic is estimated from a finite set of trial replicates. For example, a simulation may estimate the probability that a design declares efficacy under a specified scenario, or the probability that a trial stops for a safety rule. The reported proportion is not generally the exact probability: it also reflects Monte Carlo variation from using a finite number of replicates. A confidence interval makes that simulation uncertainty visible.
The interval concerns the event probability under the specified simulation scenario and implementation. It does not describe uncertainty in the scenario assumptions, model parameters, or clinical effect unless those are explicitly incorporated into the simulation. It is also not an interval for the proportion of patients who experience an event in a future trial. The introductory material on operating characteristics distinguishes the design’s performance probability from observed outcomes in any one trial; here the task is to quantify the simulation error in estimating that probability.
The binomial model requires that the event definition is fixed before counting and that replicate event indicators are independent and have the same event probability \(p\). In practice, this means using the same design, scenario, and event rule for each replicate, with independent random generation across replicates. Each replicate contributes one binary result: event or no event. A design can be adaptive within a replicate; that does not violate the model, provided the replicate-level outcomes are independent across replicates and generated under the same scenario.
If a simulation uses unequal weights, reuses dependent random numbers across replicates, or changes the scenario while counting, \(K\) need not have a binomial distribution. An ordinary binomial interval is then not justified without additional methods. Similarly, if simulation is stopped or extended according to the observed event count, a fixed-\(n\) interval need not retain its stated coverage. The calculations below presume a fixed replication count.
Wilson Score Confidence Interval
A common choice is the Wilson score interval. For a two-sided confidence level \(1-\alpha\), let \(z_{1-\alpha/2}\) be the corresponding standard normal quantile. The interval is obtained by inverting the score test for a binomial probability. It remains within \([0,1]\), including when \(K=0\) or \(K=n\), and usually behaves better than the elementary Wald interval.
The score-test inversion explains both the formula and its boundary behavior. For a proposed value \(p_0\), the score statistic is \((\hat{p}-p_0)/\sqrt{p_0(1-p_0)/n}\). The values not rejected by a two-sided score test satisfy
Multiplying through and collecting powers of \(p_0\) gives
The accepted values lie between the roots of this quadratic. Applying the quadratic formula and expressing the roots as center plus or minus half-width gives the Wilson formula above. Since the test parameter is constrained to be a probability, the construction respects the range from zero to one; it does not rely on a symmetric normal interval centered at \(\hat{p}\).
Worked Example: A Simulated Efficacy Probability
Suppose 200 independent trial replicates are generated under one fixed hypothetical scenario. In 74 replicates, the prespecified efficacy criterion is met. The estimated probability is \(\hat{p}=74/200=0.370\). The fixed scenario, event definition, and independent replicate generation satisfy the binomial assumptions.
For a 95% Wilson interval, use \(z=1.96\) and \(z^2=3.8416\). The center is
The half-width is
Thus the 95% Wilson interval is approximately \(0.37245\pm0.06633\), or \((0.3061,\,0.4388)\). In context, the simulated estimate is 37.0%, and the interval is compatible with event probabilities from about 30.6% to 43.9% under this scenario and simulation procedure. In repeated simulations of 200 independent replicates, Wilson intervals constructed this way have coverage close to the nominal 95%; the interval does not assign a 95% probability to the fixed \(p\).
Counts at or Near a Boundary
When no replicate has the event, \(\hat{p}=0\). That does not establish that the event probability is zero: a positive but small \(p\) can produce no events in a finite simulation. For \(K=0\), the Wilson lower endpoint is zero, while the upper endpoint is positive. At the other boundary, \(K=n\), the Wilson upper endpoint is one and the lower endpoint is below one. These features are important when reporting rare failure, safety, or success events.
Worked Example: No Events in 50 Replicates
In a hypothetical simulation of 50 independent replicates under one fixed scenario, a specified failure event occurs zero times. Therefore \(K=0\), \(n=50\), and \(\hat{p}=0\). The binomial assumptions hold if each replicate independently used the same scenario and the same failure definition.
For a 95% Wilson interval, the center is \(3.8416/(2\cdot50)\) divided by \(1+3.8416/50\), and the half-width is the same quantity. Equivalently, the lower endpoint is zero and the upper endpoint is
The interval is therefore \((0,\,0.07135)\), or \((0,\,0.0714)\) when rounded to four decimal places. The simulation observed no failures, but the data do not rule out a failure probability of several percent. This is Monte Carlo uncertainty about the failure probability under the chosen scenario, not proof that the design is risk-free.
A useful alternative for boundary counts is the exact Clopper–Pearson interval. Its endpoints are obtained by inverting equal-tailed binomial tests: for an interior count, the lower endpoint \(p_L\) solves \(P_{p_L}(X\ge K)=\alpha/2\), and the upper endpoint \(p_U\) solves \(P_{p_U}(X\le K)=\alpha/2\), where \(X\sim\operatorname{Binomial}(n,p)\). At \(K=0\), the lower endpoint is zero and the upper endpoint solves \((1-p_U)^n=\alpha/2\). Exact intervals have coverage at least the nominal level, but can be conservative and wider than necessary.
Worked Example: Exact Interval When the Event Is Unobserved
Use the same hypothetical count, \(K=0\) in \(n=50\) independent replicates, and construct a two-sided 95% Clopper–Pearson interval. The upper endpoint satisfies \(P_{p_U}(X=0)=0.025\), or \((1-p_U)^{50}=0.025\). Solving,
The exact interval is approximately \((0,\,0.07112)\). It differs slightly from the Wilson interval \((0,\,0.07135)\). Both express substantial uncertainty despite zero observed events; the exact construction guarantees at least 95% coverage for the binomial model, whereas the Wilson interval is an approximate score-based interval.
When \(K\) is close to \(n\), the same issue arises at the upper boundary. For instance, if 78 of 80 hypothetical replicates meet a fixed criterion, then \(\hat{p}=0.975\). The 95% Wilson center and half-width are
The resulting interval is approximately \((0.9134,\,0.9931)\). Although the observed rate is 97.5%, the lower endpoint shows that an appreciably smaller underlying event probability remains compatible with only 80 replicates. The endpoints stay within the allowable probability range.
Choosing and Reporting an Interval
The Wald interval, \(\hat{p}\pm z\sqrt{\hat{p}(1-\hat{p})/n}\), is simple but can extend below zero or above one. Worse, if \(K=0\) or \(K=n\), its estimated standard error is zero, yielding a zero-width interval despite the finite replication count. This makes it an unsuitable default for simulated event rates, especially near the boundaries.
Wilson is often a practical default: it is straightforward, remains in the probability range, and generally has more reliable coverage than Wald without being as conservative as exact intervals. Clopper–Pearson is useful when guaranteed at-least-nominal coverage is a priority, including for very sparse events, with the trade-off that intervals may be wider. No method repairs an inadequate number of replicates: if the interval is too wide for the intended decision, more independent simulation replicates may be needed.
| Method | Strength | Limitation |
|---|---|---|
| Wald | Simple formula and familiar form | Can leave \([0,1]\), have poor coverage near boundaries, or collapse to zero width at \(K=0\) or \(K=n\) |
| Wilson score | Respects probability bounds and usually has good practical coverage | Coverage is approximate rather than guaranteed to be at least nominal for every \(p\) and \(n\) |
| Clopper–Pearson exact | Coverage is at least the nominal level under the binomial model | Can be conservative and comparatively wide |
Always report the event definition, scenario, \(K\), \(n\), estimated rate, interval method, confidence level, and endpoints. For example: “Under the specified hypothetical null scenario, 74 of 200 independent replicates met the efficacy criterion; the estimated rate was 37.0%, with a 95% Wilson interval of 30.6% to 43.9%.” This lets a reader distinguish a simulation estimate from the target operating characteristic and assess whether its Monte Carlo precision is adequate.
Diagnostics and Common Pitfalls
- Reading zero observed events as zero probability. The wrong reasoning is that \(K=0\) proves \(p=0\). The correction is that finite simulation can miss a rare event; report an interval with a positive upper endpoint.
- Using Wald automatically. A symmetric normal interval may cross the probability boundaries or collapse at an observed rate of zero or one. Use an interval designed for a binomial probability, such as Wilson or exact.
- Confusing simulation uncertainty with design uncertainty. A narrow interval says the simulation has estimated the event probability precisely under the specified model. It does not show that the scenario assumptions are clinically correct or that model uncertainty is small.
- Applying a binomial interval to dependent replicates. If replicate event indicators are correlated, the binomial variance and coverage calculations may not apply. Check how random numbers and replicate generation are organized before interpreting the interval.
- Reporting an interval without identifying its target. State the scenario and event explicitly; a confidence interval for a failure probability under one scenario is not an interval for all possible operating characteristics.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- In 400 independent replicates under one fixed hypothetical scenario, an event occurs 96 times. What is \(\hat{p}\), and what probability does it estimate?
- Why is a zero-width Wald interval inappropriate when no events occur in a finite simulation?
- For zero events in 50 replicates, write the equation defining the 95% exact upper endpoint and give its approximate value.
- A researcher changes the scenario halfway through a run but combines all event counts and applies a binomial interval. What assumption is at risk, and why?
- What does a narrow confidence interval around a simulated efficacy probability establish, and what does it not establish?
Solutions. (1) \(\hat{p}=96/400=0.240\); it estimates the probability of the defined event for one replicate under that fixed scenario and simulation procedure. (2) Wald substitutes \(\hat{p}=0\) into its standard error, giving \(z\sqrt{0(1-0)/n}=0\); that reflects a defective boundary approximation, not certainty that \(p=0\). (3) The equation is \((1-p_U)^{50}=0.025\), so \(p_U=1-0.025^{1/50}\approx0.07112\). (4) The common event probability is at risk: different scenarios can have different probabilities, so the combined indicators need not be identically distributed binomial trials. (5) A narrow interval indicates low Monte Carlo uncertainty for the specified scenario and model. It does not establish that the scenario or model is clinically realistic, nor does it quantify uncertainty across scenarios.