Estimating Operating Characteristics From Replicates

Why Replicates Are the Unit of Evaluation

A clinical trial simulation typically generates many complete trial data sets under a specified design and scenario. Each simulated trial is a replicate: it includes the simulated participants, the planned conduct and analysis, and the resulting estimate or decision. Repeating that process lets us ask how the design and analysis perform across the trial outcomes that could arise under the scenario.

The quantity of interest is usually a feature of the distribution of trial-level results. Examples include the proportion of trials that meet a prespecified success rule, the average estimation error across trials, or the proportion of trials in which a decision rule selects a particular action. The replicate—not the individual participant—is therefore the basic unit for these summaries.

Definition: An operating characteristic is a performance quantity defined for a trial design and analysis under a specified simulation scenario. A replicate-level summary is a value calculated from the results of one complete simulated trial, such as its success indicator, parameter estimate, or selected action. An operating characteristic is commonly estimated by aggregating that summary over replicates.

Before calculating an operating characteristic, specify the scenario, the quantity being evaluated, and how it is calculated in each replicate. For example, “success” might mean that a prespecified test is statistically significant in the intended direction, or that an estimated treatment effect exceeds a decision threshold. Those rules are not interchangeable. A success criterion should be defined before examining the simulated results, so the reported rate has a clear meaning.

The usual interpretation assumes that each replicate is generated under the same specified scenario and design, and that replicates are independently generated. The simulated workflow should also match the intended trial process: the same eligibility rules, conduct assumptions, analysis method, and decision rule must be applied consistently. If different scenarios or designs are mixed, a single average may describe that mixture rather than any one scenario.

Success Rates and Bias From Replicate Summaries

For \(R\) replicates, define \(I_r=1\) if replicate \(r\) meets the specified success rule and \(I_r=0\) otherwise. The estimated success rate is the mean of these indicators. It estimates the probability that a simulated trial under the stated scenario will meet that rule.

\widehat{P}(\text{success})=\frac{1}{R}\sum_{r=1}^{R} I_r

This formula is appropriate when every replicate has a defined success outcome and each simulated trial is intended to have equal weight. The success rule must say how to treat analysis failures or other outcomes that prevent a trial from meeting the rule. If such replicates are silently omitted, the resulting fraction estimates success conditional on being included—not necessarily the chance of success for a simulated trial.

Bias concerns an estimator of a specified parameter. Let \(\theta\) be the true parameter value under the scenario, and let \(\hat{\theta}_r\) be the estimate from replicate \(r\). The replicate-level estimation error is \(\hat{\theta}_r-\theta\); the average of these errors estimates the estimator’s bias under that scenario.

\widehat{\operatorname{Bias}}(\hat{\theta}) = \frac{1}{R}\sum_{r=1}^{R}(\hat{\theta}_r-\theta) = \left(\frac{1}{R}\sum_{r=1}^{R}\hat{\theta}_r\right)-\theta

The bias is signed: a positive value indicates that estimates average above the true value, and a negative value indicates that they average below it. It is not the same as the typical size of an individual estimation error; positive and negative errors can cancel in the average. The parameter, estimator, analysis population, scenario, and handling of unusable estimates therefore need to be stated when reporting bias.

Worked Example: Estimating a Trial Success Rate

Suppose 200 independent hypothetical trial replicates are simulated under one scenario. Success is defined in advance as a positive estimated treatment effect with a two-sided test result of \(p<0.05\). In 132 replicates, both conditions are met.

Each replicate contributes one indicator: \(I_r=1\) for those 132 successes and \(I_r=0\) for the other 68 replicates. Thus:

\widehat{P}(\text{success}) = \frac{132(1)+68(0)}{200} = \frac{132}{200} = 0.660

The estimated success rate is 66.0% for this scenario and this particular success rule. It does not mean that 66.0% of participants respond, nor does it mean that the treatment effect is 66.0%. It summarizes how often an entire simulated trial meets the rule. If the rule had instead required the estimated effect to exceed a clinically relevant threshold, the success indicators—and therefore the operating characteristic—could differ.

Worked Example: Estimating Bias Across Replicates

In a hypothetical scenario, the true exposure-response parameter is \(\theta=1.20\) mg/L. Five simulated trials produce estimates of 1.10, 1.25, 1.30, 1.05, and 1.20 mg/L. The estimates and their errors are:

\begin{aligned} \text{Errors} &= (-0.10,\ 0.05,\ 0.10,\ -0.15,\ 0.00)\ \text{mg/L} \\ \widehat{\operatorname{Bias}}(\hat{\theta}) &= \frac{-0.10+0.05+0.10-0.15+0.00}{5}\ \text{mg/L} \\ &= \frac{-0.10}{5}\ \text{mg/L} = -0.020\ \text{mg/L} \end{aligned}

Equivalently, the mean estimate is \(5.90/5=1.18\) mg/L, so the estimated bias is \(1.18-1.20=-0.020\) mg/L. Across these five replicates, the estimator averages 0.020 mg/L below the scenario’s true parameter. This small hypothetical set illustrates the calculation; it is not enough to establish that the estimator has negligible bias in general.

Decision Probabilities and the Meaning of a Summary

Many simulated trials end in a decision rather than a single parameter estimate: advance a candidate, modify a regimen, or stop development, for instance. Define the decision rule first, then record the action from each replicate. For action \(a\), define \(J_{r,a}=1\) if replicate \(r\) selects \(a\), and \(J_{r,a}=0\) otherwise. Its estimated decision probability is the fraction of replicates selecting that action.

\widehat{P}(\text{decision}=a) = \frac{1}{R}\sum_{r=1}^{R}J_{r,a}

If the decision categories are mutually exclusive and exhaustive, their estimated probabilities sum to one. If a replicate can select multiple actions, or can produce no action, the categories need to reflect that structure; otherwise, the reported probabilities may not be interpretable as a complete decision distribution.

A key choice is what to summarize within each replicate before aggregating. Consider a trial-level response rate: first calculate the rate for each trial, then average those rates to give every trial equal weight. Pooling all responders and participants across trials instead weights trials by their participant counts. The two calculations answer different questions when trial sizes vary.

Summary	Calculation	Interpretation
Mean replicate-level rate	Calculate each trial’s rate, then average trial rates	Expected rate in a randomly selected replicate when replicates receive equal weight
Pooled participant rate	Sum events across trials and divide by the total participants	Rate for a randomly selected participant among the pooled simulated participants; larger trials receive more weight
Replicate success rate	Count trials meeting a trial-level rule and divide by the number of trials	Probability that a simulated trial meets that rule under the scenario

Worked Example: Equal-Weight and Pooled Rates

Suppose two hypothetical simulated trials have different sizes. In Trial A, 5 of 10 participants respond; in Trial B, 40 of 100 respond. To estimate the mean replicate-level response rate, calculate each trial’s rate and average them:

\frac{1}{2}\left(\frac{5}{10}+\frac{40}{100}\right) = \frac{1}{2}(0.50+0.40) = 0.45

The equal-weight mean replicate-level rate is 45.0%. The pooled participant rate is:

\frac{5+40}{10+100} = \frac{45}{110} = 0.4091

The pooled rate is 40.9%. The difference occurs because Trial B is larger and therefore contributes more participants to the pooled calculation, while the replicate-level mean gives each trial equal weight. Neither calculation is universally preferable; the choice depends on the target quantity. Pooling participant records does not, by itself, estimate the probability that a trial meets a trial-level success rule.

Worked Example: Estimating Development Decision Probabilities

A hypothetical simulation contains 100 replicates. A prespecified decision rule assigns each trial to exactly one of three actions: advance, modify, or stop. The resulting counts are 58, 27, and 15, respectively.

The estimated probabilities are:

\begin{aligned} \widehat{P}(\text{advance}) &= \frac{58}{100}=0.58,\\ \widehat{P}(\text{modify}) &= \frac{27}{100}=0.27,\\ \widehat{P}(\text{stop}) &= \frac{15}{100}=0.15. \end{aligned}

They sum to \(0.58+0.27+0.15=1.00\), as expected for mutually exclusive and exhaustive actions. Under this scenario and decision rule, 58% of simulated trials select advance, 27% select modify, and 15% select stop. These are probabilities of actions across simulated trials, not probabilities that any action is correct. Assessing whether the action is appropriate requires defining what “correct” means under the scenario.

Checks, Alternatives, and Limitations

A useful report makes the target and calculation reproducible. State the scenario and design, the number of replicates, the replicate-level quantity, the aggregation rule, and the units where applicable. For a probability or rate, give the numerator and denominator or otherwise make clear what counted as an event. For bias, state the true parameter value and identify how unavailable estimates were handled. For a decision probability, describe the decision categories and rule.

Several common checks help reveal an unclear or inconsistent summary:

Check the denominator. Confirm that the denominator is the intended set of replicates. If failures are excluded, state what the resulting conditional rate describes; if failure counts as not meeting the success rule, apply that rule consistently.
Check the unit of analysis. Make sure that a trial-level operating characteristic is calculated from trial-level summaries. Treating all simulated participants as one large trial can obscure trial-to-trial variation and change the weighting.
Check the definition of success or action. A rate has no unique interpretation without its rule. A test-based criterion and a clinically meaningful threshold may yield different success rates.
Check units and scale. Bias is expressed in the units of the parameter; a probability is dimensionless. A percentage should be identified as a percentage, not as a parameter estimate.
Check consistency across scenarios. Apply the same rule when comparing scenarios unless a deliberate change is part of the question. If scenarios are pooled, state the weighting and recognize that the result describes the mixture.

The mean estimate across replicates is often the appropriate summary for bias, but it can conceal a wide or asymmetric distribution of estimates. A bias near zero does not imply that individual estimates are close to the truth: positive and negative errors can cancel. Likewise, the fraction of successful trials is tied to its particular decision threshold and analysis procedure; it is not a general measure of all aspects of design quality.

An alternative to an equal-weight mean of replicate-specific rates is a pooled participant-level rate, as shown above. Another alternative for a continuous trial-level outcome is to report a median or quantile across replicates instead of a mean. These alternatives are not merely different calculation shortcuts: they summarize different features or apply different weights. Choose the summary to match the scientific question, and label it precisely. Point estimates from a finite number of simulations also vary across simulation runs; quantifying that simulation variability is a separate step.

Common pitfall: “There were 45 responders among 110 simulated participants, so the trial success probability was 40.9%.” This treats participants pooled across trials as though they represented trial-level successes. The correction is to define a success indicator for each complete trial and calculate the fraction of trials meeting the prespecified rule. A pooled response rate may be useful for a participant-level target, but it does not answer the trial success question.

A Practical Workflow

Specify the scenario and target.
State which design and assumptions are held fixed, and whether the target is a trial success probability, estimator bias, or a decision probability.

Define one summary per replicate.
Set the success indicator, estimation error, or decision category before aggregating results. Define how failures and missing results count.

Aggregate at the intended level.
Average the replicate-level indicators or errors, giving replicates the intended weights. Do not substitute a pooled participant calculation for a trial-level summary.

Report the interpretation and checks.
Give the numerator and denominator where relevant, units for estimates and bias, and enough detail to distinguish the summary from plausible alternatives.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions.

In 80 hypothetical replicates, 52 meet a prespecified success rule. What is the estimated success rate, and what does it describe?
A parameter’s true value is 4.0 mg/L, and the mean estimate across replicates is 4.3 mg/L. What is the estimated bias, with units and direction?
Why can the mean of trial-specific response rates differ from a pooled response rate?
Three mutually exclusive decision actions occur in 30, 12, and 8 of 50 replicates. What are their estimated probabilities, and do they sum to one?

Solutions. (1) \(52/80=0.650\), or 65.0%; it estimates the probability that a simulated trial under the specified scenario meets that success rule. (2) \(4.3-4.0=0.3\) mg/L, indicating an estimated average bias of 0.3 mg/L above the true value. (3) The mean of trial-specific rates weights trials equally, whereas pooling participants weights trials according to their participant counts; unequal trial sizes can therefore produce different results. (4) The probabilities are \(30/50=0.60\), \(12/50=0.24\), and \(8/50=0.16\); they sum to \(1.00\).