Why Matching Conditions Matters
A simulation comparison is informative only if it isolates a meaningful difference between designs. Suppose an adaptive allocation rule is compared with fixed 1:1 allocation. If the adaptive design also enrolls more participants, uses different eligibility criteria, or stops under a different rule, an observed difference cannot be attributed to allocation alone. The comparison has changed several ingredients at once.
The goal is not to force both designs to produce identical data. Their allocations, observed outcomes, and realized stopping times may differ because those features are consequences of the designs. Rather, the goal is to give the designs the same trial conditions and data-generating scenario, changing only the design feature being studied and any consequences that follow from it.
Specify the Conditions Before Comparing Designs
A simulation scenario describes the trial environment in which each design operates. Before running simulations, write down the conditions that must be common to all comparators. In this tutorial, the central conditions are eligibility, sample size, outcome distributions, and stopping conditions. Other trial features—such as covariate distributions, follow-up, and missingness—should also be held constant when they are not part of the question.
- Eligibility: Use the same inclusion and exclusion criteria and the same source population for generating candidate participants. If eligibility depends on measured characteristics, generate those characteristics in the same way for every comparator.
- Sample size: Set the same planned number of randomized participants or the same maximum number. If recruitment failures or ineligibility occur, define whether the target refers to screened or randomized participants and apply that definition consistently.
- Outcomes: Use the same outcome model, parameter values, follow-up schedule, and outcome-assessment rules. The allocation design may change which participants are assigned to each arm and therefore which outcomes are observed in each arm; it should not silently change the underlying outcome mechanism.
- Stopping: Use the same prespecified stopping rule, including interim analysis times, thresholds, and maximum sample size. Designs can stop at different times because their data differ; the rule itself should not be changed between comparators unless stopping is the feature under study.
The outcome mechanism must be appropriate to the scientific scenario. For example, under a null scenario, treatment may have no effect on the outcome distribution; under an alternative scenario, arm-specific outcome distributions may differ according to prespecified treatment effects. In either case, use the same scenario parameters for every design being compared. If outcomes depend on baseline covariates, generate those covariates from the same distribution and use the same outcome model in each arm and design, apart from the specified treatment effects.
The simulation must also respect the allocation process: an adaptive design may use accrued outcomes to update assignment probabilities, while a fixed design follows its prespecified allocation rule. That intended difference is the object of comparison, not a mismatch. Where a simulation models potential outcomes or uses a response model conditional on treatment and covariates, specify that model consistently; it should not change just because a participant would be assigned by a different design.
A Simple Justification for Matching
Let \(D\) denote a simulated trial data set, \(d\) denote a design, and \(g(D)\) denote any summary used to describe the trial produced by that design. The purpose here is to clarify what is being compared, not to select a particular summary. Under a prespecified scenario distribution \(P\), the design-specific quantity is \(E_P[g(D_d)]\), the average of that summary over trials generated under design \(d\) in scenario \(P\). The controlled comparison is the difference
The shared \(P\) represents the common eligibility, outcome, and trial conditions. If the adaptive design is instead run under \(P_A\) and the fixed design under \(P_F\), the observed contrast can be decomposed by adding and subtracting the fixed-design expectation under \(P_A\):
The first bracket is a design contrast under one scenario. The second is a scenario mismatch: it reflects changing the environment for the fixed design from \(P_A\) to \(P_F\). Unless that second term is known to be zero, the full contrast does not isolate allocation design. Matching conditions removes this avoidable ambiguity.
Worked Example: Matching the Sample-Size Target
Suppose a simulation compares an adaptive two-arm design with a fixed 1:1 design. The scientific question is the effect of allocation rule, and the intended target is 120 randomized participants in each trial. The adaptive implementation is set to enroll 120, but the fixed-design code is mistakenly set to enroll 150.
Identify the mismatch. The allocation rule and sample size both differ. Even if the outcome model and eligibility criteria match, a comparison can reflect the extra 30 participants as well as the allocation rule.
Correct the setup. Set both designs to a target of 120 randomized participants. If the adaptive design assigns 72 participants to arm A and 48 to arm B in a particular replicate, that imbalance is allowed: it is a consequence of adaptive allocation, not a change in the total target. The fixed design will target 60 per arm.
Check the implementation. For each design, confirm that the stopping condition for reaching the sample-size target is based on the same count of randomized participants. A target of 120 screened candidates in one program and 120 randomized participants in the other would still be a mismatch.
Worked Comparisons of Eligibility, Outcomes, and Stopping
Worked Example: Matching Eligibility and the Eligible Population
Consider a hypothetical source population in which 1,000 candidates are generated for each simulated trial. Under a common rule, 80% meet the trial’s eligibility criteria, so the expected number eligible is \(1{,}000(0.80)=800\). The design comparison concerns adaptive versus fixed allocation among eligible participants.
Suppose the adaptive-design program excludes candidates with a specified baseline condition, but the fixed-design program does not. If that condition is present in 10% of the source population and is independent of the original eligibility indicator for this simplified example, then the fixed program has an expected \(800\) eligible candidates, while the adaptive program has an expected \(800(0.90)=720\) who also pass the additional exclusion. The designs are no longer operating on the same eligible population.
To match the conditions, use one eligibility function in both programs. In this example, that means either applying the additional exclusion to both or removing it from both, according to the intended trial protocol. Then generate candidates from the same source population and apply the same eligibility rule before either allocation method is used. The value 800 is an expectation, not a guarantee that every simulated replicate will contain exactly 800 eligible candidates.
Worked Example: Matching the Outcome-Generating Mechanism
Suppose a hypothetical binary-outcome scenario has a control-arm event probability of 0.20 and a treatment-arm event probability of 0.15. The intended comparison is adaptive allocation versus fixed 1:1 allocation under this same alternative scenario. For a group of 100 participants assigned to control, the expected event count is \(100(0.20)=20\); for 100 assigned to treatment, it is \(100(0.15)=15\).
Now suppose the adaptive-design program uses these probabilities, but the fixed-design program accidentally uses 0.20 in both arms. The fixed comparator is now being evaluated under a null scenario, not the intended alternative. Any contrast mixes a change in allocation rule with a change in treatment effect.
The correction is to define one scenario specification with the two probabilities and use it in both programs. Random outcomes will not generally produce exactly 20 and 15 events in every replicate; the expected counts simply follow from the specified probabilities and group sizes. If arm sizes differ under adaptive allocation, expected event counts should be calculated using those arm sizes and the same arm-specific probabilities.
Worked Example: Matching a Stopping Rule
Consider a hypothetical trial with interim checks after 60 and 80 randomized participants and a maximum of 100. At either check, the trial stops early if a prespecified statistic \(Z\) is at least 2.0; otherwise it continues to the next check or to 100. The comparison is adaptive versus fixed allocation, not one stopping policy versus another.
Both design implementations should use the same check points, threshold, and maximum. For illustration, suppose a replicate produces \(Z=1.7\) at 60 participants and \(Z=2.1\) at 80. The trial continues at the first look because \(1.7<2.0\), then stops at the second because \(2.1\geq2.0\). These decisions follow the stated rule. In another replicate, either design might stop at a different look because its observed data produce different values of \(Z\).
It would be incorrect to force both designs to stop at the same realized sample size if their observed data lead to different values of \(Z\); that would override the stopping rule. It would also be incorrect to use a threshold of 2.0 for the adaptive design and 2.5 for the fixed design. Match the rule, not necessarily the realized stopping time.
Implementation Checks and Common Failure Modes
A short design-comparison specification can prevent many avoidable errors. Before running the full simulation, list each common condition and record where it is implemented for every comparator. For example:
| Condition | What to specify | What to verify |
|---|---|---|
| Eligibility | Source population and inclusion or exclusion rules | Both designs apply the same rules before allocation |
| Sample size | Target or maximum, and whether the count is screened or randomized | Both programs use the same count and stopping basis |
| Outcomes | Outcome model, parameters, and assessment timing | Each comparator calls the same scenario specification |
| Stopping | Interim times, decision rule, and maximum | Rules match even when realized stopping times differ |
Useful diagnostics include printing the scenario settings at the start of each run, checking the number of candidates and eligible participants, and recording the planned and realized sample sizes. Compare the definitions of functions that generate outcomes and apply eligibility; duplicated code can drift apart even when it began identically. A discrepancy in these checks is evidence of a setup or implementation problem, not a design result.
A common mistake is to equate fairness with identical realized trial data: “The designs must have the same arm sizes and stop after the same number of participants.” That is wrong when arm sizes or stopping times are allowed to respond to the design. The correction is to match the rules that generate the trial conditions and let each design produce its own valid allocation and data. Conversely, allowing different eligibility or outcome rules merely because they are convenient does not make a comparison fair.
There are two legitimate comparison goals that should not be confused. A controlled design comparison holds the scenario fixed to isolate the effect of a design feature. A policy-bundle comparison may deliberately compare packages that differ in several features, such as allocation and stopping policy. The latter can answer a useful practical question, but its conclusion concerns the whole package; it cannot attribute differences to allocation alone. State which goal is intended before interpreting a contrast.
Matching conditions does not guarantee that a simulation is otherwise adequate. A scenario may omit important clinical features, an outcome model may be unrealistic, or an allocation procedure may be incorrectly implemented. Matching also does not require that every design have the same realized number of participants under a shared stopping rule. It supplies a controlled basis for the comparison; it does not by itself establish that the scenario is comprehensive or the code is correct.
State which feature differs between designs and which features must remain common.
Specify the eligible population, sample-size target or maximum, outcome mechanism, and stopping conditions.
Make each comparator use the same scenario specification while retaining its own allocation rule.
Check eligibility counts, sample-size definitions, outcome parameters, and stopping-rule definitions for unintended differences.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions below.
- A fixed-allocation comparator enrolls 90 randomized participants, while an adaptive comparator enrolls 100. The only intended difference is allocation. What should be changed?
- One comparator applies an exclusion criterion that removes 5% of otherwise eligible candidates; the other does not. Give a correction and explain why the mismatch matters.
- Both designs use the same early-stopping rule but stop at different realized sample sizes. Is that necessarily a failure to match conditions? Explain.
- A study intentionally compares an adaptive design with a more generous follow-up schedule against a fixed design with shorter follow-up. What kind of contrast is this, and what can it establish?
Solutions. (1) Set both designs to the same target, such as 100 randomized participants, and use the same definition of the count. Otherwise, sample size as well as allocation differs. (2) Apply the same exclusion criterion to both designs, using the same source population. Otherwise, the eligible populations and potentially their outcome distributions differ, so the contrast does not isolate allocation. (3) No. If both designs use the same prespecified rule, different observed data can trigger stopping at different looks. The rule must match; the realized stopping time need not. (4) This is a policy-bundle comparison. It can compare the combined packages under the specified scenario, but it cannot attribute a difference specifically to allocation or follow-up without further comparisons that separate those features.