Estimating An Operating Characteristic From Replicates

From Trial Decisions to Operating-Characteristic Estimates

An operating characteristic describes how a trial procedure behaves over repeated hypothetical trials under specified conditions. For example, it may be the probability that a Bayesian decision rule declares success when the treatment effect has a particular value. That probability is a property of the trial procedure, the scenario, and the data-generating mechanism—not a probability calculated from the results of one actual trial.

Often the probability is difficult to calculate directly. Simulation provides a practical estimate: generate many trial replicates under the same specified scenario, apply the same trial procedure to each, and record whether a chosen event occurs. The event can be success, an early stop, a safety rule being triggered, or another clearly defined trial-level outcome.

Definition: Let $A$ be a specified event for a complete trial replicate under a fixed scenario and trial procedure. The operating characteristic of interest is $\theta=P(A)$, where the probability is taken over the randomness in the data-generating mechanism and any other random components included in the trial procedure. For $M$ simulated replicates, define $Y_r=1$ if event $A$ occurs in replicate $r$, and $Y_r=0$ otherwise. The Monte Carlo estimate is $$ \hat{\theta}_{MC}=\frac{1}{M}\sum_{r=1}^{M}Y_r. $$

The estimate is the fraction of simulated trials in which the event occurred. If 764 of 1,000 replicates meet a prespecified success rule, the estimate is $764/1000=0.764$, or 76.4%. It is an estimate of the probability under the specified scenario, not an exact value merely because the simulation software returned a precise decimal.

What Must Be Fixed Before Averaging

An indicator average answers a precise question only when the event and the conditions defining its probability are clear. Before generating replicates, specify the scenario, the data-generating mechanism, the trial procedure, and the event being tallied. A scenario may include the true treatment and control response rates, the distribution of a continuous outcome, or assumptions about recruitment and missing data. The procedure includes the design and analysis choices that turn the generated data into a trial decision.

The event must be defined at the level of a complete replicate. For example, “the posterior probability exceeds 0.95” is not itself enough unless the posterior quantity, threshold, analysis population, timing, and any other rule details are specified. If success means that the posterior probability of a clinically meaningful benefit exceeds a cutoff, the event is that the specified analysis of a replicate’s data satisfies that decision rule. It is not the posterior probability itself.

The basic interpretation assumes that each replicate is generated under the same scenario and procedure and that the replicate outcomes are independent draws from the specified simulation distribution. Random components such as treatment allocation or interim stopping should be simulated according to the mechanism defined for the trial. They are part of the probability being estimated when they are part of the procedure. The decision rule must be applied consistently; changing it partway through the simulations means the replicates no longer estimate one common operating characteristic.

Conditional on the fixed scenario, procedure, and event definition, each $Y_r$ is a Bernoulli random variable with probability $\theta$: it equals one with probability $\theta$ and zero with probability $1-\theta$. The Monte Carlo estimate is therefore a sample proportion. It summarizes the simulated replicates, not uncertainty about an individual participant’s outcome and not uncertainty about the treatment effect in any one replicate.

Why the Indicator Average Estimates the Probability

The justification follows directly from the definition of an indicator. Since $Y_r$ is one when $A$ occurs and zero otherwise, its expected value equals the probability of $A$. Averaging many such indicators estimates that expectation.

E(Y_r)=1\cdot P(A)+0\cdot P(A^c)=P(A)=\theta.

If all $M$ replicates are generated under the same conditions, linearity of expectation gives

E(\hat{\theta}_{MC}) = E\left(\frac{1}{M}\sum_{r=1}^{M}Y_r\right) = \frac{1}{M}\sum_{r=1}^{M}E(Y_r) = \frac{1}{M}\sum_{r=1}^{M}\theta = \theta.

Thus, the indicator average is unbiased for the specified probability under these conditions. If the replicates are also independent, the law of large numbers says that the average approaches $\theta$ as $M$ increases. This does not make a finite-run estimate equal to $\theta$: a different set of simulated replicates can produce a different proportion. Quantifying the simulation variability or constructing an interval for the operating characteristic requires additional tools.

Worked Example: Estimating the Probability of Trial Success

Suppose a hypothetical trial is evaluated under a specified scenario in which the true response rate is 0.35 in control and 0.50 in treatment. Each simulated trial uses the same design and Bayesian analysis, and success is defined in advance as the posterior probability that the treatment response rate exceeds the control response rate being greater than 0.975. In $M=200$ independently simulated trial replicates, 146 meet this success rule. Estimate the probability of success under this scenario.

For replicate $r$, let $Y_r=1$ if its analysis satisfies the prespecified rule, and $Y_r=0$ otherwise. The tally is $\sum_{r=1}^{200}Y_r=146$. Therefore,

\hat{\theta}_{MC} = \frac{1}{200}\sum_{r=1}^{200}Y_r = \frac{146}{200} = 0.730.

The estimated probability of success is 0.730, or 73.0%, under this data-generating scenario and trial procedure. This is a Monte Carlo estimate based on 200 replicates; it is not an exact probability and is not the posterior probability from any one of the trials. The calculation is appropriate if the 200 replicates used the stated scenario, were generated independently, and all applied the same success rule.

Worked Examples with Different Events

Worked Example: Estimating a False-Positive Probability

Consider a hypothetical Bayesian superiority trial under a null scenario in which treatment and control have the same true response rate, 0.40. Suppose the trial declares success when the posterior probability of a higher treatment response rate exceeds 0.95. In 500 independently generated replicates under this null scenario, the rule declares success in 21. Estimate the probability of a success declaration under this scenario.

Here event $A$ is “the specified decision rule declares success,” and the data-generating scenario has equal true response rates. The event occurs 21 times, so

\hat{\theta}_{MC} = \frac{21}{500} = 0.042.

The estimated probability of a success declaration under this particular null scenario is 0.042, or 4.2%. It is appropriate to describe this as the simulated false-positive probability for the stated scenario and decision rule. The estimate does not establish that the probability is exactly 4.2%, nor does this single scenario automatically summarize performance across every parameter value in a composite null hypothesis.

Worked Example: Estimating an Early-Stopping Probability

In a hypothetical two-stage trial, the protocol includes an interim rule that stops for futility if the predictive probability of final success is below 0.10. Under one fixed scenario, 84 of 300 independently simulated trials stop for futility at the interim analysis. Estimate the probability of stopping for futility under this scenario.

Define $Y_r=1$ when replicate $r$ meets the protocol’s interim futility stopping rule, and zero otherwise. The event occurs 84 times, giving

\hat{\theta}_{MC} = \frac{84}{300} = 0.280.

The estimated probability that the trial stops for futility at the interim analysis is 0.280, or 28.0%, under the specified scenario and protocol. This tally should include only replicates that reach the interim decision point and satisfy the stated rule. If the simulation instead tallied all early stops, including safety stops, it would estimate a different event.

Diagnostics, Comparisons, and Limitations

The arithmetic is simple, but an apparently plausible proportion can still answer the wrong question. Verify that each counted unit is a whole simulated trial replicate, that every replicate corresponds to the intended scenario, and that the event is applied identically. Check the raw event count against the total number of replicates: the estimate must lie between zero and one, and it equals zero or one only if the event was absent or present in every simulated replicate. A reported percentage should also agree with the count and denominator after rounding.

A useful audit is to inspect a few replicate-level records. Confirm that the recorded indicator matches the trial outputs and decision rule, including relevant details such as analysis population, interim timing, and handling of missing observations. If a replicate can reach success through more than one route, specify whether the event is the union of those routes; if the protocol requires multiple criteria, specify whether all must hold. Ambiguity in the event definition cannot be repaired by increasing the number of replicates.

This approach differs from an analytic calculation, which derives the probability directly from a mathematical model when that is tractable. Simulation is useful when the complete design and decision rule make a direct calculation difficult, but it inherits any errors in the simulated data-generating mechanism or implementation. It also differs from evaluating the posterior probability within one observed trial: that posterior is conditional on that trial’s data and model, whereas an operating characteristic averages a trial-level event across repeated data sets generated under a specified scenario.

Quantity	What it describes	How it is obtained here
Posterior probability in one trial	Uncertainty about a parameter given that trial’s observed data and model	Calculated from that replicate’s data
Operating characteristic	Probability that a specified trial-level event occurs under a scenario and procedure	Target probability $P(A)$
Monte Carlo estimate	Approximation to the operating characteristic from simulated replicates	Number of event occurrences divided by number of replicates

The main limitation of the indicator average is that it is tied to the exact scenario, event, and implementation used to produce it. It does not automatically generalize to other treatment effects, missingness assumptions, or decision cutoffs. Nor does a large number of replicates protect against using an incorrect scenario or coding the rule incorrectly. The estimate is only as relevant as the assumptions and trial procedure it represents.

Common Mistakes

Calling the estimate exact. A simulated proportion is based on a finite set of random replicates. Report it as an estimate, with the event, scenario, and number of replicates stated.
Averaging the wrong quantity. To estimate the probability of a trial-level event, use a zero-or-one indicator for each replicate. Averaging posterior probabilities across trials answers a different question.
Mixing scenarios or procedures. If replicates use different true parameter values or decision rules, their pooled fraction generally does not estimate the operating characteristic for any one of those conditions. Separate scenario-specific estimates unless a mixture is explicitly the target.
Changing the event after seeing results. Selecting or redefining the event based on the simulated tally changes the question and can make the reported estimate difficult to interpret. Define the decision event before evaluating replicates.
Treating a scenario-specific result as universal. An estimate under one null value or treatment effect does not by itself describe performance over a range of values.

Key takeaway: To estimate an operating characteristic, define one trial-level event under fixed simulation conditions, record its indicator for every replicate, and divide the total number of event occurrences by the number of replicates. Interpret the result as a Monte Carlo estimate of the specified probability.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions.

A success rule is applied in 400 simulated trials, and 318 meet it. What is the Monte Carlo estimate of the success probability?
In 250 replicates under a specified null scenario, 9 trials declare success. State the estimated probability and its context.
Why is the average posterior probability from each replicate generally not the same target as the proportion of replicates that declare success?
A simulation combines 100 replicates at one treatment effect with 100 at another, then reports one pooled success proportion. What is the main interpretive problem?

Solutions. (1) The estimate is $318/400=0.795$, or 79.5%, for the specified scenario and success rule. (2) The estimate is $9/250=0.036$, or 3.6%; it estimates the probability of a success declaration under that particular null scenario and trial procedure. (3) A posterior probability is a numerical summary of parameter uncertainty conditional on one replicate’s data. A success indicator records whether that replicate crosses a decision threshold. Averaging those posterior probabilities estimates an average posterior quantity, while averaging the indicators estimates the probability of the decision event. (4) The pooled proportion combines two data-generating scenarios. Unless the target is explicitly the stated mixture of those scenarios, it cannot be interpreted as the operating characteristic at either treatment effect separately.