Benchmarking Against Fixed Designs

Why Benchmark an Adaptive Design?

An adaptive design is not informative simply because it changes as data accrue. Its value must be judged against a credible alternative: what would a prespecified, non-adaptive design do under the same conditions? Benchmarking answers that question by comparing designs under common data-generation scenarios and with common performance measures.

The comparison should isolate the design strategy, not differences in assumptions. If an adaptive design is simulated with one treatment-effect model and its fixed comparator with another, a performance difference cannot be attributed cleanly to adaptation. Likewise, comparing adaptive power with fixed-design expected sample size does not answer a coherent comparative question. The scenario, outcome definitions, analysis conventions, and metrics must be aligned before interpreting results.

Choosing a Suitable Fixed Comparator

“Fixed” does not mean there is only one possible comparator. A conventional design might use a fixed allocation ratio, a fixed sample size, and a prespecified final analysis. The comparator should be plausible for the clinical question and trial constraints, not chosen merely because it makes the adaptive design look favorable or unfavorable.

Two complementary comparisons are often useful. A common-maximum comparison gives both designs the same maximum sample size or resource ceiling. It asks how their performance differs within a shared limit, even if the adaptive design stops early or changes allocation. A resource-matched comparison selects or calibrates the fixed design to have a similar expected sample size, duration, or other resource use. It asks whether one strategy achieves different operating performance at a similar expected cost. These comparisons answer different questions; neither should be presented as the other.

A well-specified benchmark comparison states the fixed design’s sample size, allocation, analysis rule, and any other features relevant to its operating characteristics. It also states what can change in the adaptive design, such as allocation or stopping, and what remains common, such as the endpoint definition and final estimand. If the adaptive design has a maximum sample size, reporting it alongside the fixed design’s sample size helps readers understand the resource envelope.

The same scenario grid should be applied to both designs. A scenario specifies the assumed data-generating conditions, such as treatment effects, control response, nuisance parameters, recruitment conditions, and departures from the primary model. Under each scenario, use the same outcome-generating assumptions for both designs. A design may assign participants differently or stop at a different time; those differences are part of the design being evaluated, not a reason to change the underlying scenario.

Matched Performance Measures

Choose measures that answer the comparison question and define them identically for both designs. Common examples include type I error under a null scenario, power under an alternative, bias and interval coverage for an estimand, expected sample size, and probability of selecting the correct treatment or action. State the direction of a favorable difference: for example, higher power may be desirable, while lower expected sample size may be desirable, but neither is automatically preferable without considering the other measures.

For each scenario, report each design’s estimate and the difference between them. An aggregate can be useful, but it should not replace scenario-specific results: a favorable average may conceal poor performance in an important setting. If scenario weights are used, report how the aggregate is formed and retain the underlying scenario results for context. Scenario weighting and average performance are especially useful when the intended target is performance across a specified mixture of conditions; they do not establish that performance is acceptable in every condition.

Simulation can compare the designs efficiently by pairing their results across Monte Carlo replicates. In replicate \(r\), generate a common underlying set of potential trial outcomes under the specified scenario, then apply each design’s rules to those outcomes. The resulting trial paths can differ because allocation, stopping, and analysis may differ. The paired comparison uses the same replicate’s outcomes as a matched unit; it does not require the two designs to produce identical trial data.

Paired Differences and Monte Carlo Uncertainty

Let \(Y_{Ar}\) be a performance outcome for the adaptive design and \(Y_{Fr}\) the corresponding outcome for the fixed design in replicate \(r\), under one fixed scenario. Define the replicate-level difference \(D_r=Y_{Ar}-Y_{Fr}\). For \(R\) independent simulation replicates, the estimated mean difference is \(\bar{D}\), the mean of the \(D_r\) values. For a binary outcome such as rejection, \(\bar{D}\) estimates the difference in rejection probabilities.

Here \(s_D\) is the sample standard deviation of the replicate-level differences. The usual Monte Carlo standard error formula applies when the replicates are independent and generated under the stated simulation mechanism. Pairing often reduces the standard error because shared replicate-level difficulty affects both designs in similar ways. It is important to calculate the standard error from the differences, rather than treating the two estimates as independent when their simulation results were paired.

The estimator is justified directly: each replicate difference has expectation equal to the difference in the designs’ scenario-specific mean performance. Therefore, by linearity of expectation, the average of the differences estimates that target. The standard error quantifies simulation variability in that estimate, not uncertainty about which data-generating scenario is clinically correct.

A confidence interval based on \(\bar{D}\) summarizes Monte Carlo uncertainty in the estimated comparison, subject to the simulation assumptions and an appropriate approximation for the replicate differences. It does not account for uncertainty in scenario inputs unless that uncertainty is explicitly represented in the simulation. The distinction between Monte Carlo error and design uncertainty is essential when interpreting a small apparent advantage.

Worked Comparisons Across Resources and Scenarios

Diagnostics, Limitations, and Common Mistakes

Before interpreting a benchmark, check that both designs use the same scenario inputs, outcome definitions, estimand, and performance-measure rules. Confirm that the fixed comparator is fully specified and credible, and that each design is evaluated with the same number of independent simulation replicates or that any difference is accounted for. Review scenario-level results as well as aggregates. If paired differences are highly variable or a small estimated advantage is comparable to its Monte Carlo standard error, increase simulation precision or report the uncertainty clearly rather than presenting a noisy ranking as decisive.

A benchmark remains conditional on the scenarios, models, implementation, and analysis procedures used. A fixed comparator can be poorly chosen, while a common maximum sample size can conceal differences in average resource use. Conversely, matching expected sample sizes can obscure differences in maximum exposure or operational burden. Reporting multiple complementary comparisons can clarify these issues, but no summary removes the need to state what each comparison holds constant.

• Wrong reasoning: “The adaptive design has higher power in its own simulation, so it is better.” Correction: Compare it with a credible fixed design under the same data-generating assumptions and using the same power definition.
• Wrong reasoning: “The two power estimates have separate standard errors, so their difference’s standard error follows by treating them as independent.” Correction: If replicates were paired, calculate the standard error from the replicate-level differences; the estimates are generally correlated.
• Wrong reasoning: “The weighted average is favorable, so no scenario needs separate inspection.” Correction: The aggregate describes only its specified weighting; examine scenario-specific performance for important conditions.
• Wrong reasoning: “A lower expected sample size proves the adaptive design is more efficient.” Correction: Interpret resource use alongside relevant efficacy, error-rate, estimation, and operational measures.