Confidence Intervals For Simulated Operating Characteristics

What a Confidence Interval for a Simulation Estimates

A simulated operating characteristic is a probability defined by a trial procedure and a data-generating scenario. Examples include the probability that a decision rule declares success when the null scenario is true, or the probability of success under a specified treatment effect. A simulation estimates this probability using a finite number of trial replicates, so the estimate itself has Monte Carlo uncertainty.

The interval in this tutorial describes uncertainty from having run a finite simulation. It is not an interval for individual patient outcomes, and it does not quantify uncertainty about whether the chosen data-generating model represents future patients. Patient-level variation is already part of the simulated trial process that defines the operating characteristic. The interval instead reflects variation in the estimated probability that would arise if the simulation were independently repeated with the same number of replicates.

The binomial model requires that each replicate produce one binary outcome, that the replicates be independent, and that they share the same success probability \(\theta\). This usually means holding the trial procedure and the scenario fixed while independently generating each simulated trial. Any randomness within a trial—such as patient responses, enrollment, or a randomized treatment allocation—is included in the replicate outcome. It does not make the replicate-level outcome non-binary.

The target \(\theta\) is conditional on the scenario as specified. If uncertain scenario parameters are drawn anew for each replicate, the interval can still describe the Monte Carlo precision of the resulting average probability, provided the replicate outcomes remain independent and identically distributed. But it does not isolate uncertainty about the parameter values from uncertainty induced by averaging over those draws. The interpretation must match the simulation design.

The Wilson Score Confidence Interval

A useful default for a binomial probability is the Wilson score interval. It behaves well for probabilities near zero or one and for modest numbers of events, unlike the simple Wald interval. For a two-sided confidence level \(1-\alpha\), let \(z_{1-\alpha/2}\) be the corresponding standard normal quantile. The Wilson interval is:

The plus and minus signs give the upper and lower endpoints, respectively. The interval is bounded between zero and one. It is often described as a score interval because it is obtained by inverting a test based on the standardized difference between the observed proportion and a candidate value of \(\theta\).

The “approximately” matters: the binomial count is discrete, so a procedure based on a normal score approximation need not have exactly its nominal coverage for every \(M\) and \(\theta\). Wilson intervals generally have more reliable coverage than Wald intervals, but they are not exact intervals.

Why the Wilson Formula Has This Form

The score construction starts with a candidate value \(\theta\) and asks whether the observed replicate proportion \(\hat{\theta}\) is too far from it. Under the binomial model, the estimated variance of \(\hat{\theta}\) when testing that candidate is \(\theta(1-\theta)/M\). Retaining \(\theta\), rather than replacing it with \(\hat{\theta}\) in the standard error, gives the score inequality:

To obtain the interval, square both sides, multiply by the positive quantity \(M\theta(1-\theta)\), and collect terms in \(\theta\):

The candidate values satisfying this quadratic inequality lie between its two roots. Applying the quadratic formula and dividing numerator and denominator by \(M\) gives the Wilson endpoints shown above. This inversion also explains why the interval is centered near, but not generally exactly at, \(\hat{\theta}\): the denominator and the added terms account for the binomial variance depending on the candidate probability.

Worked Examples

Alternatives, Diagnostics, and Failure Modes

The Wald interval uses \(\hat{\theta}\pm z\sqrt{\hat{\theta}(1-\hat{\theta})/M}\). It is easy to calculate but can have poor coverage, especially with few events or when \(\hat{\theta}\) is near zero or one; it can even extend below zero or above one. In the zero-event example, its estimated standard error is zero, so it gives the misleading interval \((0,0)\). This is a reason not to use it as the default for simulated operating characteristics.

An exact Clopper–Pearson interval is obtained by inverting binomial tests and has coverage at least the nominal level under the binomial model. Its coverage can be conservative, meaning that it may cover more often than the stated confidence level, and its intervals are often wider than Wilson intervals. It is a useful alternative when a conservative exact procedure is required or when sparse outcomes make the approximation of a score interval a concern. Neither method corrects a simulation design that violates the binomial assumptions.

Before interpreting an interval, check the replicate-level data and the simulation design. Each replicate should contribute exactly one binary result to the count, and the total \(M\) should be the number of valid independent replicates for that scenario. If replicates share randomness in a way that makes their outcomes dependent, or if weighted outcomes are used, the ordinary binomial interval is not justified by the count alone. If replicates use different success probabilities, their sum is generally not binomial with one common \(\theta\). A calculation that treats such outcomes as identically distributed can understate or otherwise misrepresent Monte Carlo uncertainty.

Another important limitation is the scope of the target. A narrow interval indicates that the simulation has estimated \(\theta\) precisely for the scenario and procedure used; it does not establish that the scenario is clinically plausible, that all relevant scenarios were considered, or that a prior and decision rule are appropriate. Those questions require scientific and design judgments beyond binomial Monte Carlo uncertainty. In Bayesian trial calibration, the interval is a confidence interval for a simulation probability, not a posterior credible interval for a model parameter.

In practice, report the event count \(K\), the number of replicates \(M\), the estimate \(K/M\), the interval method, and the interval endpoints. State the scenario and the replicate-level event in context. Reporting just the estimated percentage hides how much it may change through simulation noise; reporting just the interval hides the information in the observed count and total.