Why Matching Trial Conditions Matters
A comparison between fixed randomization and response-adaptive randomization is informative only when it isolates the difference of interest: how treatment assignments are generated. If the adaptive design also enrolls more participants, follows them longer, or uses a different analysis, then a difference in results cannot automatically be attributed to its allocation rule. The comparison has changed several things at once.
The aim here is not to decide which design is preferable on patient or inferential grounds. It is to set up a fair comparison. That requires naming the target population, outcome, time horizon, stopping rule, and analysis before contrasting the allocation mechanisms.
Let \(A_i\) denote the treatment assigned to participant \(i\), with treatment options A and B. In a fixed design, the probability of assigning A is a prespecified constant \(q\), so \(P(A_i=\mathrm{A}\mid H_{i-1})=q\). In a response-adaptive design, that probability may depend on the history \(H_{i-1}\) available before participant \(i\) is assigned, including accrued outcomes if the rule permits them. The two rules may therefore produce different allocation proportions, even when they enroll the same number of participants.
A matched comparison should specify the same distribution of participant characteristics, treatment effects, outcome variability, calendar-time conditions, and missingness or follow-up processes for both designs. In a simulation, these are common inputs to the two design evaluations. In a comparison of actual trials, matching may be difficult: differences in sites, eligibility, clinical practice, or calendar time can remain even if the nominal sample sizes match.
What Must Be Held Constant
A practical comparison begins by writing down the conditions that both designs share. “Same sample size” alone is not enough. For example, one design could have a fixed enrollment target and the other could stop when a precision or event-count target is reached. Even if their average enrollment happens to be similar, their stopping rules differ.
- Population and eligibility: Use the same target population and eligibility criteria.
- Treatments and outcome: Keep treatment definitions, outcome definition, measurement procedures, and outcome time point the same.
- Enrollment and follow-up: Match the planned sample size or stopping rule, accrual period, and length and completeness of follow-up.
- Other trial procedures: Keep co-interventions, monitoring, and outcome ascertainment comparable.
- Analysis: Apply the same prespecified estimand and analysis method, including how missing outcomes and relevant time trends are handled.
The estimand is the precise quantity the analysis is intended to describe. For example, it might be the difference in the proportion with a specified binary outcome by day 30 between treatment A and treatment B in the eligible trial population. Changing the outcome time from day 30 to day 90 changes the estimand; it is not a minor adjustment to the same comparison.
A Simple Allocation-Mix Derivation
One reason to avoid comparing unqualified pooled outcome rates is that the rates depend on the allocation mix. Suppose, for illustration, that every participant has an observed binary outcome and that the success proportions within treatments are \(p_A\) and \(p_B\). If a fraction \(q\) of participants is assigned to A, the pooled success proportion is
This follows by counting expected successes: among \(n\) participants, approximately \(nq p_A\) successes arise in A and \(n(1-q)p_B\) in B. Dividing their sum by \(n\) gives the expression above. If the treatment-specific success proportions are unchanged but the allocation fraction changes from \(q_0\) to \(q_1\), then
Thus, the pooled rate can differ solely because the proportion assigned to each treatment differs. It is not, by itself, a clean summary for attributing a difference to the allocation mechanism. This calculation assumes that the treatment-specific proportions are comparable across the designs; if participant mix or calendar conditions differ, even that assumption may fail.
Worked Example: Same Sample Size, Different Allocation Mix
Consider two hypothetical 100-participant trials with complete, binary outcomes. Suppose both have the same treatment-specific success proportions: 30 of 50 participants assigned to A succeed, and 20 of 50 assigned to B succeed, in a balanced fixed design. In a response-adaptive design, suppose 70 participants are assigned to A and 30 to B, with the same success proportions within treatment.
In the fixed design, the pooled success proportion is \( (30+20)/100=0.50 \). In the adaptive design, the corresponding expected counts under the stated proportions are \(70(0.60)=42\) for A and \(30(0.40)=12\) for B. The pooled proportion is \( (42+12)/100=0.54 \).
The difference is \(0.54-0.50=0.04\), or 4 percentage points. The allocation fractions are \(q_0=0.50\) and \(q_1=0.70\), while \(p_A-p_B=0.60-0.40=0.20\). The derivation gives \( (0.70-0.50)(0.20)=0.04 \), the same result.
This example does not show that either allocation rule is superior. It shows that comparing pooled rates answers a question about the mixture of treatments as well as outcomes. A matched comparison must state whether the target is a treatment-specific contrast, a pooled trial outcome, or another quantity, and use that target consistently.
Worked Comparisons of Trial Conditions
Worked Example: Unequal Sample Sizes
Suppose a hypothetical fixed-design trial enrolls 100 participants and records 60 successes, while a response-adaptive trial enrolls 120 and records 78 successes. The crude proportions are \(60/100=0.600\) and \(78/120=0.650\), a difference of 0.050, or 5 percentage points.
That arithmetic is correct, but the comparison does not isolate allocation: sample size differs, and the extra participants may have been enrolled at different calendar times or under different conditions. The crude difference alone cannot establish that response-adaptive assignment produced the higher proportion.
A more informative design comparison would set the same enrollment target, for example 100 participants per design, use the same follow-up and outcome definition, and apply the same analysis. If enrollment cannot be matched because the question concerns different stopping rules, then stopping rule is another design feature under study and should be described as such, not attributed to allocation alone.
Worked Example: Unequal Outcome Follow-Up
In two hypothetical trials, 40 participants are assigned to each treatment. Suppose the numbers with an event by day 30 are 8 in A and 7 in B. The day-30 event proportions are \(8/40=0.200\) and \(7/40=0.175\), giving a difference of \(0.200-0.175=0.025\), or 2.5 percentage points.
By day 90, suppose 12 participants in A and 8 in B have experienced the event. The day-90 proportions are \(12/40=0.300\) and \(8/40=0.200\), a difference of \(0.100\), or 10 percentage points.
Both calculations are internally consistent, but they answer different time-specific questions. If the fixed design reports day-30 events and the adaptive design reports day-90 events, their difference is confounded with follow-up time. To compare allocation rules, both designs must use the same outcome horizon and comparable ascertainment. If some participants have not yet reached that horizon, the analysis must address incomplete follow-up rather than silently treating their status as known.
Worked Example: Same Data, Different Calendar-Time Analysis
Suppose outcomes change over calendar time, perhaps because clinical practice changes. In a hypothetical dataset, the success counts are as follows:
| Period | Treatment A | Treatment B |
|---|---|---|
| Early | 18/30 = 0.600 | 12/20 = 0.600 |
| Late | 8/20 = 0.400 | 9/30 = 0.300 |
| All participants | 26/50 = 0.520 | 21/50 = 0.420 |
The crude difference is \(0.520-0.420=0.100\), or 10 percentage points. But the treatments have different period mixes: A has 30 early and 20 late participants, while B has 20 early and 30 late participants. To illustrate a common-period standardization, give early and late periods equal weight. A's standardized proportion is \( (0.600+0.400)/2=0.500 \); B's is \( (0.600+0.300)/2=0.450 \). The standardized difference is \(0.500-0.450=0.050\), or 5 percentage points.
The crude and standardized results differ because they use different period weights. Equal weighting is only appropriate if it represents the intended target; it is not automatically the correct analysis. The lesson for a design comparison is to choose the target and analysis in advance and apply them consistently. If the allocation rule changes which treatment is assigned at different calendar times, calendar time may be particularly important to check.
How to Build and Check a Matched Comparison
For a simulation, define a single scenario and evaluate both allocation rules under it. Keep the same sample size, outcome-generating conditions, follow-up, stopping rule, and analysis. Repeat the comparison over many simulated trials. Where feasible, use paired simulation replicates: use the same underlying participant characteristics and potential outcome draws for the fixed and adaptive runs in a replicate. This can make the contrast less affected by unrelated simulation noise. The allocation paths still differ, and the pairing does not remove assumptions built into the simulated scenario.
For observed trials, matching is less controllable. Investigators should check whether eligibility, calendar period, outcome measurement, follow-up, and analysis are genuinely comparable. Statistical adjustment may address measured differences, but it does not guarantee that unmeasured differences have been removed. A design comparison with important unmatched conditions should be described as a comparison of the whole trial packages, rather than evidence about allocation alone.
Name the population, outcome, time horizon, and summary to be compared.
Specify eligibility, treatments, sample size or stopping rule, follow-up, and analysis.
Describe the fixed and response-adaptive assignment rules, including what information the adaptive rule uses.
Look for any additional changes in participant mix, calendar conditions, outcome measurement, or analysis.
Useful diagnostics include a side-by-side checklist of planned and achieved sample sizes, follow-up completeness, outcome definitions, and analysis procedures. Also inspect enrollment and assignment over calendar time, and verify that outcome denominators are defined in the same way. In simulation, confirm that both designs were run under the same scenario and that code implements the stated stopping and analysis rules. A mismatch is not necessarily an error, but it changes what the comparison can support.
Common Mistake and Limitations
Common mistake: “The adaptive trial had a higher pooled success proportion, so the adaptive allocation rule caused better outcomes.” This reasoning treats a pooled proportion as if allocation were the only difference. The pooled proportion can change when treatment allocation fractions change, as the derivation showed; different sample sizes, follow-up, participant mix, or analyses can also affect the comparison.
Correction: First specify the target quantity and match the non-allocation conditions. Then use a common analysis and describe any conditions that could not be matched. A matched comparison can isolate allocation more clearly, but it does not prove that all real-world differences have been controlled. In particular, simulation conclusions depend on the scenario used, and observational comparisons may retain unmeasured differences.
A matched comparison is a design discipline, not a guarantee that two trials are identical in every respect. It makes the intended contrast explicit and helps prevent a change in sample size, follow-up, or analysis from being mistaken for an allocation effect. Questions about patient benefit or inferential efficiency require additional criteria beyond matching trial conditions.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- Two designs use the same outcome definition and analysis, but one enrolls 80 participants and the other enrolls 100. What prevents this from being a matched comparison of allocation rules alone?
- In a binary-outcome trial, \(p_A=0.70\), \(p_B=0.50\), and the fraction assigned to A changes from \(q_0=0.40\) to \(q_1=0.60\). By how much does the pooled success proportion change if the treatment-specific proportions stay fixed?
- One trial measures an outcome at day 30 and another at day 60. Why is comparing their event proportions not a comparison under a common outcome condition?
- A crude analysis gives an A-minus-B difference of 8 percentage points, while a prespecified calendar-time-standardized analysis gives 3 percentage points. What should be reported, and what does the difference indicate?
Solutions. (1) Sample size differs, and possibly enrollment duration or stopping conditions as well; those differences must be matched or explicitly included as part of the comparison. (2) The change is \( (q_1-q_0)(p_A-p_B)=(0.60-0.40)(0.70-0.50)=0.20(0.20)=0.040 \), or 4 percentage points. (3) The two proportions concern different follow-up horizons and therefore different time-specific outcomes. (4) Report the prespecified standardized result for its stated target, and present the crude result as a distinct summary if useful. The discrepancy indicates that period composition affects the crude comparison; it does not by itself identify which summary is appropriate for every target.