Why Interim Randomization Tests Need the Allocation History
In a response-adaptive trial, the probability of assigning the next participant to treatment may depend on earlier assignments and responses. At a prespecified interim look, a test that treats the observed allocation as if it came from a fixed-ratio design can therefore use the wrong reference distribution. The relevant randomization distribution must reflect the trial’s actual sequential assignment rule.
This tutorial focuses on testing the sharp null of no treatment effect for any participant at a prespecified look. The central operation is to hold the outcomes fixed under that null, then replay the allocation algorithm through the look. Each replay uses its own simulated allocation history to determine subsequent assignment probabilities. A look is prespecified when its analysis time or information threshold is set in advance; the trial is not stopped merely because a test result is favorable.
The target distribution is the distribution of the chosen test statistic under the trial’s assignment mechanism, through the prespecified look, under the sharp null. It is not generally the distribution obtained by freely permuting treatment labels. In a replay, the response values are fixed, but treatment labels are reassigned; any treatment-specific response summaries used by the algorithm must be recalculated from the simulated history.
Assumptions and the Path Probability
An exact test requires a fully specified assignment mechanism through the look. In particular, the probability \(\pi_i\) must be known for every history that can arise in a replay. The sharp null must hold, and the outcomes must be defined consistently for all assignments—for example, there is no interference between participants. The enrollment schedule and interim look must also be prespecified, and the responses used to update the rule must be available at the times the rule says they are used. If response delays, missingness, or additional randomization affect assignment, they must be represented in the mechanism being replayed.
For a particular assignment path \(\mathbf a=(a_1,\ldots,a_t)\), its probability is the product of the sequential probabilities along that path. Each factor is calculated using the history created by the path so far, not automatically using the observed trial history.
Here \(\mathbf a_{i-1}\) is the simulated assignment history through participant \(i-1\). Under the sharp null, the response values in that history are fixed, while any allocation-dependent quantities are updated for the simulated assignments. For a statistic \(T_t\), an exact two-sided randomization p-value is the total probability of paths whose statistic is at least as extreme as the observed statistic, using the same extremeness rule for observed and replayed paths.
Worked Example: Enumerating an Adaptive Distribution at One Look
Consider a hypothetical trial with two participants and a look after both responses are recorded. The sharp-null response values are \(Y_1=6\) and \(Y_2=2\). The first participant is assigned to treatment with probability \(0.5\). For participant 2, the rule assigns treatment with probability \(0.75\) if participant 1 was assigned treatment and \(Y_1\geq 5\); otherwise it assigns treatment with probability \(0.25\). The observed path is \((A_1,A_2)=(1,1)\). Use the Horvitz–Thompson contrast \(T_2=\frac{1}{2}\sum_{i=1}^{2}\{A_iY_i/\pi_i-(1-A_i)Y_i/(1-\pi_i)\}\).
Under the sharp null, \(Y_1=6\) is fixed in every replay. Thus \(\pi_2=0.75\) after \(A_1=1\), and \(\pi_2=0.25\) after \(A_1=0\). Multiplying the relevant probabilities gives:
| Path | Path probability | \(T_2\) |
|---|---|---|
| (1, 1) | \(0.5\times0.75=0.375\) | \((6/0.5+2/0.75)/2=7.333\) |
| (1, 0) | \(0.5\times0.25=0.125\) | \((6/0.5-2/0.25)/2=2.000\) |
| (0, 1) | \(0.5\times0.25=0.125\) | \((-6/0.5+2/0.25)/2=-2.000\) |
| (0, 0) | \(0.5\times0.75=0.375\) | \((-6/0.5-2/0.75)/2=-7.333\) |
The observed statistic is \(7.333\). The paths at least as extreme in absolute value are \((1,1)\) and \((0,0)\), whose probabilities sum to \(0.375+0.375=0.750\). The exact two-sided p-value is therefore \(0.750\). It is not \(2/4=0.500\): the four paths do not have equal probabilities under this adaptive rule.
Why the Replay Gives a Valid Sharp-Null Test
Under the sharp null, all responses that enter the analysis or allocation rule are fixed, regardless of assignment. The only randomness in the repeated trial, conditional on those responses and other fixed inputs, is the randomization specified by the design. Replaying that design therefore generates the null distribution of the statistic. The observed path itself is one draw from this distribution.
For a discrete distribution, the probability of obtaining a statistic at least as extreme as the observed one is a valid (possibly conservative) p-value when ties are included. To see why, order the possible outcomes from most extreme to least extreme, grouping ties together. For any cutoff \(u\), the event that the tail probability assigned to the observed outcome is at most \(u\) can include only outcomes whose cumulative probability in this ordering is at most \(u\). The probability of that event is consequently at most \(u\). Thus, under the sharp null, \(P(p\leq u)\leq u\) for \(0\leq u\leq1\). Including ties can make the test conservative; random tie-breaking can produce an exactly uniform p-value when that convention is appropriate.
The result depends on replaying the actual mechanism. If the rule changes its treatment probability based on accumulated treatment-specific responses, a replay with a different assignment path may have different treatment-specific summaries and therefore different future probabilities. The observed responses remain fixed under the null, but the simulated history used to decide assignments does not.
Worked Example: Applying Look-Specific Tests at Prespecified Analyses
Suppose the hypothetical two-participant trial above has prespecified looks after participant 1 and participant 2. At the first look, the observed assignment is \(A_1=1\) and the response is \(Y_1=6\). The statistic \(T_1=A_1Y_1/0.5-(1-A_1)Y_1/0.5\) is 12. Under the sharp null, the first assignment has two possible paths: treatment gives \(T_1=12\) with probability \(0.5\), and control gives \(T_1=-12\) with probability \(0.5\). For a two-sided test, both are equally extreme, so the first-look p-value is \(1.000\).
At the second look, suppose the observed path remains \((1,1)\). The full replay from the beginning gives the second-look p-value \(0.750\), as calculated above. The first-look result is not carried forward by conditioning on the observed first assignment: the second-look test uses the randomization distribution through participant 2.
These are look-specific p-values, not by themselves a guarantee of a chosen familywise error rate across both looks. For example, if a simple Bonferroni procedure is used for two looks at overall \(\alpha=0.05\), each look is compared with \(0.05/2=0.025\). Neither p-value is below \(0.025\), so neither look rejects. A different multiplicity procedure may use different thresholds, but it must account for the prespecified repeated analyses.
Exact Enumeration and Monte Carlo Replay
If the number of possible paths is manageable, enumerate them and sum their probabilities. For longer trials, the number of paths can grow exponentially, so a practical alternative is to simulate paths from the known adaptive mechanism. For each simulated path, apply the rule sequentially, hold sharp-null responses fixed, update the simulated history, and recompute the statistic at the same prespecified look.
State the sharp null, the interim sample or information point, the statistic, and the two-sided or one-sided extremeness rule.
Record the assignment probability at each step as a function of the history available to the allocation rule.
Generate assignments using the rule, update each simulated history, and keep the null response values fixed.
For exact enumeration, sum the probabilities of at-least-as-extreme paths. For simulation, estimate that tail probability and report its Monte Carlo uncertainty.
Worked Example: Estimating the Tail Probability by Simulation
Return to the two-participant example, whose exact two-sided p-value is \(0.750\). Suppose 10,000 independent paths are simulated from its adaptive assignment rule, and 7,486 simulated statistics are at least as extreme as the observed statistic. A common simulation estimate, including the observed path as an additional draw for the rank calculation, is
An approximate Monte Carlo standard error is \(\sqrt{0.7486(1-0.7486)/10000}=0.00434\). This measures simulation error in estimating the randomization tail probability; it is not a confidence interval for a treatment effect. The simulated estimate is close to the exact value \(0.750\), as expected, but a different run would generally produce a slightly different estimate.
Diagnostics, Alternatives, and Limitations
A replay should be checked as carefully as any implementation of a sequential design. For each simulated step, verify that the probability used is the one implied by that simulated history, that it lies between 0 and 1, and that the probabilities of the available assignment options sum to 1. For short designs, compare the simulation with exact enumeration. Also check that every rule input is available at the correct time and that the simulated data structure updates treatment-specific summaries rather than silently retaining the observed trial’s summaries.
Several approaches answer different questions. Exact enumeration is transparent and removes Monte Carlo error but becomes infeasible when there are many paths. Monte Carlo replay approximates the same design-based distribution and is scalable, but requires reporting enough simulation detail to assess its numerical precision. A simple permutation distribution is appropriate only when it matches the assignment mechanism being tested; it generally does not match an adaptive rule with history-dependent probabilities. Studentized tests for weak null hypotheses address a different null and do not become exact sharp-null tests merely by being used at an interim look.
The sharp-null test does not, by itself, test only whether the average treatment effect is zero. A sharp null specifies every participant’s treatment effect; a weak null about an average effect allows individual effects to differ. In addition, a look-specific randomization p-value does not automatically adjust for repeated testing, and this fixed-look analysis does not justify stopping at an unplanned time. Exactness also relies on correct knowledge of the allocation algorithm and on representing all relevant randomization and response-observation features. If those conditions fail, the computed distribution may not describe the trial that was run.
Common Mistake
- Wrong reasoning: “At the interim look, I should hold the observed allocation history fixed and randomize only the next assignment.” For a cumulative statistic, fixing the past assignments fixes much of the statistic and usually removes the variation the test is meant to represent.
- Correction: Replay the allocation process from its beginning through the prespecified look, unless a specifically justified conditional test is being constructed. Preserve the sharp-null response values, but let each simulated allocation history determine subsequent probabilities.
- Wrong reasoning: “All treatment-label permutations are equally likely.” This is false when the adaptive probabilities vary with history.
- Correction: Weight paths by their sequential assignment probabilities, or simulate paths from the actual rule.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- Under the sharp null, what is fixed in a replay, and what must be allowed to change?
- In the two-participant example, why is the probability of path \((0,1)\) equal to \(0.125\)?
- Why does enumerating the four paths with equal weight give the wrong reference distribution in that example?
- What does a look-specific randomization p-value establish, and what does it not establish when there are multiple prespecified looks?
- Name two checks that can reveal an error in a Monte Carlo replay.
Solutions. (1) The response values are fixed by the sharp null, along with other fixed design inputs; assignments and any allocation-dependent history must change across replays. (2) The first assignment is control with probability \(0.5\); after that history, the treatment probability for participant 2 is \(0.25\), so the path probability is \(0.5\times0.25=0.125\). (3) The adaptive rule gives paths probabilities \(0.375, 0.125, 0.125,\) and \(0.375\), not \(0.25\) each, so equal weighting does not represent the trial’s mechanism. (4) It gives a tail probability under the sharp null for that look’s statistic; it does not by itself control error across multiple looks. (5) Check that each step uses the probability implied by the simulated history and that assignment probabilities sum to one; for a small design, also compare simulated results with exact enumeration.