Frame A Simulation Comparison

Frame the Comparison Before Simulating

A simulation comparing response-adaptive randomization with a prespecified allocation is only interpretable if the comparison is defined in advance. The central question is not simply whether one design “does better.” It is: which design strategies are being compared, for which trial population and conditions, at what trial sample size, and with respect to which prespecified quantities? If those choices are vague, a numerical result can be calculated without answering a clear design question.

This tutorial focuses on framing. It does not prescribe how to generate trial data or implement an allocation algorithm. Instead, it sets out the design contrast and the targets that a later simulation will need to represent. The same principles apply whether the response-adaptive rule uses accumulating outcomes to update allocation probabilities or the comparator uses a fixed allocation ratio.

Specify the Two Strategies and the Target Population

Name both strategies concretely. For example, one strategy may use a response-adaptive allocation rule, while the comparator assigns participants according to a prespecified 1:1 ratio. “Adaptive versus standard” is not enough: “standard” could refer to different allocation ratios, eligibility rules, follow-up schedules, or analyses. The intended contrast should make clear which feature changes and which features do not.

The target population is the population to which the simulated trial is intended to apply. Describe it using the characteristics that matter to the question: for instance, adults meeting stated eligibility criteria at participating clinics, with a defined clinical condition and a specified range of baseline risk. Also state the setting and calendar or follow-up context when these affect interpretation. A broad phrase such as “patients with the condition” may conceal important limits on who is represented.

A simulation comparison needs trial conditions as well as a population label. Specify the treatments being compared, the primary endpoint and its measurement time, how outcomes affected by discontinuation or rescue treatment will be handled, and the planned analysis. These choices help define the outcome being summarized. They should be common across the two strategies unless a deliberate, separately stated contrast changes them. In particular, if the analysis differs between strategies, the comparison is no longer solely a comparison of allocation rules.

The target population and conditions are not merely background descriptions. They determine what the comparison means. A result for a highly selected population with frequent outcome events need not describe the design trade-off in a broader population with rarer events. The design contrast should therefore be stated within a clearly defined target context, rather than presented as a universal ranking of allocation strategies.

Define Trial Sample Size Precisely

For a design comparison, the trial sample size \(n\) is the number of participants enrolled in each hypothetical trial. Unless the question explicitly concerns sample-size selection, use the same \(n\) for both allocation strategies. Otherwise, a difference in performance could reflect a difference in enrollment as well as a difference in allocation.

This \(n\) is distinct from the number of simulated trials. If a simulation repeats a hypothetical trial many times, each repetition has its own trial sample size \(n\); the number of repetitions controls how precisely the simulation estimates an average over repeated trials. Those are different quantities and should never be reported as if they were interchangeable.

Also say whether \(n\) means the planned maximum enrollment or the number enrolled before a stopping rule. If the designs can stop at different times, comparing them at the same maximum enrollment does not necessarily mean they have the same realized number of participants. That may be part of the intended design contrast, but it must be stated rather than left ambiguous. A comparison intended to isolate allocation should not quietly allow enrollment or stopping rules to vary as well.

Choose the Primary Estimands

There are two levels of estimand to keep separate. A treatment estimand describes the treatment effect of interest in the target population—for example, a difference in response probabilities between A and B at 12 weeks. A design-comparison estimand describes how a prespecified summary of trial performance differs between the two allocation strategies. A design-comparison estimand does not replace the clinical treatment estimand; it asks how the design performs when that treatment question is studied.

For a precise definition, let \(D_A\) denote the adaptive design and \(D_C\) the prespecified comparator. Let \(O_D\) represent the relevant result from a hypothetical trial run under design \(D\), and let \(g(O_D)\) be the prespecified trial-level quantity of interest. Under a stated set of trial conditions, the design-comparison estimand can be written as

The expectations refer to the distribution of hypothetical trial results under the respective designs and the specified conditions. The definition is incomplete until \(g\), the population, the trial size, the endpoint, the analysis, and the conditions are stated. The direction also matters: with this ordering, a positive \(\Delta\) means the expected value of \(g\) is higher under the adaptive design. Whether that is desirable depends on what \(g\) represents.

For example, if \(g\) is an indicator that the prespecified test rejects under a particular alternative, its expectation is the probability of rejection, and \(\Delta\) is a difference in power. If \(g\) is an indicator of rejection when the null hypothesis is true, the contrast concerns the rejection probability under that null condition. If \(g\) is a participant-level outcome summary, the contrast concerns the expected value of that summary under the two strategies. These are distinct questions. Naming one as primary prevents a simulation from producing many outputs without a clear principal conclusion.

Why the Contrast Needs Matching Conditions

The basic justification for holding conditions fixed is a difference-in-expectations comparison. If the same outcome definition, population, trial size, and analysis apply to both strategies, then \(\Delta\) targets the difference associated with changing the allocation strategy under those conditions. If another feature also changes, the contrast combines its effect with the allocation difference.

For example, write \(m(D,C)\) for the expected value of a chosen trial-level quantity under design \(D\) and conditions \(C\). The intended contrast at common conditions \(C\) is

If the adaptive strategy is instead assessed at conditions \(C_A\) and the comparator at different conditions \(C_C\), the computed quantity is

This second difference cannot generally be attributed to allocation alone: the conditions also differ. Matching does not mean that every design feature must be identical. Allocation is intentionally different, and consequences of that difference—such as arm-specific sample sizes—belong to the contrast. The point is to identify intentional differences and avoid accidental ones.

The primary estimand should be fixed before examining results. If a team chooses the outcome summary, scenario, or subgroup only after seeing which comparison looks favorable, the resulting headline no longer represents a prespecified primary question. Secondary summaries can still be useful, but they should be labeled as secondary or exploratory rather than substituted for the original target.

Diagnostics, Alternatives, and Limitations

Several warning signs indicate that a comparison has not been framed tightly enough:

• The comparator is unnamed or underspecified. State its prespecified allocation probabilities and any other distinguishing features needed to identify it.
• The target population is too broad to interpret. Specify eligibility, setting, and any population features central to the question.
• The trial size is ambiguous. State whether \(n\) is maximum or fixed enrollment and distinguish it from the number of simulated repetitions.
• The primary estimand is only a label. “Power” or “patient benefit” is not enough without defining the event or outcome, conditions, and direction of the contrast.
• Unintended design features vary. Differences in follow-up, endpoint, analysis, or enrollment can prevent attribution to allocation alone.

A principal alternative is to compare complete design packages rather than isolate the allocation rule. That can be appropriate when the practical decision is between packages as they would actually be used. The estimand must then describe the package-level contrast, and the conclusion should not claim that allocation alone caused the difference. Another option is to define several contrasts for distinct target populations or trial sizes. These answer multiple scoped questions; they do not replace the need to state a primary contrast.

No single framing establishes that a strategy is best in every setting. The result applies to the chosen target population, trial conditions, sample size, comparator, and primary estimand. If those conditions are not representative of the intended use, a precisely calculated contrast can still be practically uninformative. Furthermore, different primary estimands can favor different designs: greater probability of rejection, for example, is not the same objective as a more favorable participant-level outcome.