Estimating Operating Characteristics Across Replicates

From Simulated Trials to Design-Level Summaries

A single simulated adaptive trial shows one possible course of events under a design: which participants were enrolled, how assignments changed, and what final decision was reached. It does not, by itself, show how the design tends to behave. To assess that behavior, simulate many trials under the same specified conditions and summarize the outcomes across replicates.

The essential distinction is between a replicate-level outcome and an operating characteristic. A replicate-level outcome belongs to one simulated trial, such as its final recommendation, total sample size, or whether a prespecified event occurred. An operating characteristic describes the distribution or probability of such outcomes over repeated trials generated by the design in a specified scenario.

Definition: Fix a trial design, a data-generating scenario, and a replicate-level quantity \(Y\). The operating characteristic for \(Y\) is a feature of the distribution of \(Y\) over hypothetical repetitions of the trial under those fixed conditions. Examples include the probability of recommending a particular treatment, the mean total sample size, and the distribution of trial duration.

The scenario matters: it specifies the outcome-generating conditions against which the design is evaluated. The adaptive algorithm also matters, because its rules determine how each trial proceeds as data accumulate. The earlier material on design assurance, executable algorithms, data-generating scenarios, and simulating one adaptive trial replicate provides the ingredients for this repeated-trial assessment.

Define What Each Replicate Contributes

Suppose \(K\) simulated trials are run under one fixed scenario and one fixed design. Label the replicates \(k=1,\ldots,K\). For each replicate, record the outcome needed for the question at hand, denoted \(Y_k\). The records must be defined consistently: for example, total sample size should include the same participant types in every replicate, and “successful recommendation” should use the same decision rule in every replicate.

For an event of interest, define an indicator \(I_k\) that equals 1 when the event occurs in replicate \(k\), and 0 otherwise. If the event probability under the specified design and scenario is \(p\), then \(I_k\) has probability \(p\) of equaling 1. The estimated event probability is the fraction of replicates in which the event occurred:

\hat{p}_{\mathrm{sim}}=\frac{1}{K}\sum_{k=1}^{K} I_k =\frac{\text{number of replicates meeting the event definition}}{K}.

The subscript here distinguishes this simulation-based estimate from a participant-level sample proportion. Its denominator is the number of simulated trials, not the number of participants enrolled across all trials. For a numerical replicate-level outcome, such as total sample size \(N_k\), the corresponding across-replicate mean is:

\bar{N}_{\mathrm{sim}}=\frac{1}{K}\sum_{k=1}^{K}N_k.

These summaries answer different questions. An event fraction estimates how often a specified event occurs; an average describes the mean of a numerical outcome. Other questions may call for a distribution, such as the counts or proportions of replicates ending in each recommendation. The summary should follow the scientific question, rather than being selected simply because it is easy to calculate.

Why the Event Fraction Estimates an Event Probability

Under repeated simulation, each \(I_k\) is a Bernoulli random variable with probability \(p\) of equaling 1. If replicates are generated independently under the same design and scenario, then each has expectation \(E(I_k)=p\). Linearity of expectation gives:

E(\hat{p}_{\mathrm{sim}}) =E\left(\frac{1}{K}\sum_{k=1}^{K}I_k\right) =\frac{1}{K}\sum_{k=1}^{K}E(I_k) =\frac{1}{K}\sum_{k=1}^{K}p =p.

Thus, the replicate fraction targets the event probability under the specified simulation conditions. A finite set of replicates will not generally produce the exact probability: the estimate changes with the simulated results. The derivation also shows why the setup matters. If replicates use different scenarios, or do not follow the same intended design algorithm, their event indicators need not share a common probability \(p\), and the simple fraction no longer summarizes one clearly defined operating characteristic.

Worked Example: Estimating a Recommendation Rate

In a hypothetical evaluation, the event is defined in advance as “the trial recommends Treatment A.” Ten independently simulated trials use the same specified scenario and the same adaptive design. Their final decisions are:

A, A, B, A, none, B, A, A, B, A.

Here “none” means the algorithm makes no treatment recommendation. Six of the ten replicates recommend Treatment A, so the replicate-level indicator values for that event sum to 6. The estimated probability is:

\hat{p}_{\mathrm{sim}} =\frac{6}{10} =0.60.

The estimate is therefore 0.60, or 60% of simulated trials. In context, it estimates the probability that this design recommends Treatment A under the specified scenario. It does not mean that 60% of participants received Treatment A, and it does not establish that the recommendation is clinically correct. Those are different quantities requiring different replicate-level definitions.

Summarize Numerical Outcomes Across Trials

An adaptive trial can stop at different times or enroll different numbers of participants in different replicates. If the question concerns expected resource use, the total sample size from each replicate is one observation in the across-replicate summary. The mean is not obtained by pooling all participants and treating them as though they came from one trial; it is calculated from the replicate-level totals.

Worked Example: Mean Total Sample Size

Eight hypothetical replicates of the same design under the same scenario have total sample sizes of 48, 50, 42, 50, 46, 50, 44, and 50 participants. The mean sample size across replicates is:

\bar{N}_{\mathrm{sim}} =\frac{48+50+42+50+46+50+44+50}{8} =\frac{380}{8} =47.5\text{ participants}.

Thus, the estimated mean total sample size is 47.5 participants per simulated trial under these conditions. The fractional value is appropriate for an average even though every individual trial has an integer sample size. The four replicates with 50 participants do not imply that the trial always enrolls 50; the replicate-level values show that the total varied from 42 to 50 in this set.

A mean alone can conceal variation. Here, a useful accompanying description is the observed range, 42 to 50 participants, or a table of how often each total occurred. Which additional summary is useful depends on whether the design question concerns average resource use, the chance of reaching a cap, or another feature of the distribution.

Denominators Define the Question

Rates can have different meanings depending on their denominator. For example, the unconditional probability of a correct recommendation treats every simulated trial as eligible, including trials that make no recommendation. A conditional rate instead restricts attention to replicates meeting a stated condition. Neither is inherently preferable; the condition must be explicit, and the result must not be described as though it used a different denominator.

Worked Example: Unconditional and Conditional Success

In ten hypothetical simulated trials, the target treatment is correctly recommended in six replicates. Two additional replicates make an incorrect recommendation, and two make no recommendation. Define “correct recommendation” as recommending the target treatment.

For the unconditional rate, every simulated trial is in the denominator:

\frac{6\text{ correct recommendations}}{10\text{ simulated trials}}=0.60.

This estimates the probability of a correct recommendation per simulated trial, counting no recommendation as not correct. If the question is instead the fraction correct among trials that make any recommendation, there are eight such trials:

\frac{6\text{ correct recommendations}}{8\text{ trials with a recommendation}}=0.75.

The conditional rate is 0.75 among trials with a recommendation. It does not estimate the probability of a correct recommendation in all simulated trials. Reporting the numerator and denominator, or naming the condition directly, prevents these two quantities from being confused.

Practical Workflow and Diagnostics

A reliable summary starts before simulation, with an explicit link between the design question and each replicate-level record. A compact workflow is:

State the target quantity.
Specify the event, numerical outcome, or distributional feature and its units. Define decision categories and any conditions that determine eligibility for a denominator.

Fix the simulation conditions.
Use the same scenario, design rules, stopping rules, and outcome definitions for the replicate set intended to estimate one operating characteristic.

Record one result per trial.
Store the relevant outcome for every replicate, including explicit categories for no recommendation or algorithm failure when they are possible.

Summarize and check.
Calculate the specified fraction, mean, or distribution summary; verify counts and denominators against the replicate records before interpreting it.

Several checks help identify a faulty summary. First, verify that the number of recorded outcomes equals the intended number of replicates, and that event counts sum to that number when categories are mutually exclusive and exhaustive. Second, check whether missing results or failed runs were silently dropped. If a replicate does not yield the intended output, its handling should be specified; excluding it without explanation can change the target quantity. Third, confirm that quantities use consistent units and definitions, particularly when adaptive stopping makes trial size or duration vary.

Also inspect whether the simulation actually represents one design and scenario. A changed stopping rule, different scenario parameters, or inconsistent implementation can make the pooled summary difficult to interpret. Finally, distinguish variability among simulated trial outcomes from variation in the summary caused by having a finite number of replicates. The first is part of the design’s operating behavior; the second means the reported estimate is an approximation based on a limited simulation run. Formal assessment of simulation error requires additional tools.

Choosing a Summary Without Losing Its Meaning

Question	Replicate-level record	Across-replicate summary
How often does a defined event occur?	Indicator: event occurred or did not occur	Fraction of replicates with the event
What is average resource use?	Total sample size or duration	Mean of the replicate-level values
Which decisions does the design make?	Final decision category	Counts or proportions in each category

A single simulated trial cannot substitute for this repeated-trial summary: it represents one realization, not a probability or average. Conversely, a summary can hide information if it is reported alone. For instance, a mean sample size does not show the range, and an overall recommendation rate does not reveal how often the algorithm makes no recommendation unless that category is also reported.

The main alternatives are analytical calculations, simulation, or a combination. An analytical result can be useful when the design and outcome structure permit a tractable calculation. Simulation is particularly useful when adaptive decisions make the full distribution difficult to derive, because the algorithm can be executed under specified scenarios. Simulation does not remove the need for clear assumptions: its conclusions describe the simulated design and data-generating conditions, not every possible clinical setting. Results across different scenarios should therefore remain identified by scenario rather than being combined without a defined rationale.

Common Mistakes

Using participants as the denominator for a trial-level event. For the probability of a trial making a recommendation, count simulated trials, not enrolled participants.
Confusing one replicate with an operating characteristic. One trial may illustrate a possible path; repeated replicates are needed to estimate a probability or average over trial repetitions.
Changing the denominator without changing the description. A rate among trials with recommendations is conditional and differs from the rate among all trials.
Pooling incompatible replicates. Mixing scenarios or algorithm versions can produce a number that does not correspond to the operating characteristic of a single specified design condition.
Reporting a mean as though it describes every trial. An average does not imply that each trial has that outcome; inspect the replicate-level distribution when variation matters.

Key takeaway: Define one outcome per simulated trial, hold the design and scenario fixed for the intended summary, and choose an across-replicate statistic that answers the stated question. Always name the denominator and interpret the result as an estimate under those simulation conditions.

Check Your Understanding

Answer each question by identifying the replicate-level quantity and the appropriate across-replicate summary. Short solutions follow.

In 20 simulated trials, a specified safety event occurs in 3. What is the estimated event probability, and what is its denominator?
Five simulated trials enroll 30, 36, 42, 36, and 46 participants. Calculate the mean sample size.
In 12 trials, 9 make a recommendation and 6 of those recommendations are correct. Give the unconditional correct-recommendation rate and the conditional rate among trials making a recommendation.
Why should replicates generated under two different data-generating scenarios not ordinarily be combined into one event fraction?

Solutions. 1. The replicate-level quantity is the event indicator. The estimate is \(3/20=0.15\), with 20 simulated trials as the denominator. 2. The replicate-level outcomes are total sample sizes, so the mean is \((30+36+42+36+46)/5=190/5=38\) participants per trial. 3. The unconditional rate is \(6/12=0.50\); the conditional rate is \(6/9\approx0.667\) among trials making a recommendation. 4. The event probability may differ by scenario, so pooling produces a mixture whose interpretation depends on the proportion of replicates assigned to each scenario, rather than estimating the characteristic under either one fixed scenario.