Why replay the allocation rule?
When treatment probabilities change in response to accumulating trial information, the observed assignment is not generally one of a set of equally likely allocations. Randomization-based inference can still use the trial’s actual randomization, but it must respect that unequal probability structure. In particular, simulating assignments as if every treatment sequence were equally likely can produce an incorrect reference distribution.
The key device is to test a candidate treatment effect by imputing the outcomes that would have been seen under each possible assignment path, then replaying the allocation rule along that path. Inverting these tests gives a confidence set. The exactness of this approach depends on assumptions about potential outcomes, not merely on having recorded the treatment probabilities.
Assumptions and the sharp effect model
Consider a fixed set of \(N\) trial participants, each assigned either control (\(A_i=0\)) or treatment (\(A_i=1\)). Let \(Y_i(0)\) and \(Y_i(1)\) denote participant \(i\)’s potential outcomes under control and treatment. The observed outcome is \(Y_i=A_iY_i(1)+(1-A_i)Y_i(0)\). A randomization test can be made exact for a candidate effect \(\tau_0\) under the sharp additive-effect model
This is a sharp assumption: it determines both potential outcomes for every participant once the candidate value and observed data are given. For the observed participant, the missing outcome is imputed as \(Y_i(0)=Y_i-\tau_0 A_i\) and \(Y_i(1)=Y_i(0)+\tau_0\). These imputations let us calculate outcomes under alternative assignment paths.
The randomization guarantee also requires that the assignment mechanism be correctly specified and used as conducted. Its conditional treatment probabilities must be known for each possible history, including any dependence on earlier assignments, observed outcomes, covariates, or prespecified adaptation rules. If enrollment, stopping, or other design decisions affect which assignments can occur, those rules must also be included when replaying paths. Potential outcomes are treated as fixed, and one participant’s assignment does not change another participant’s potential outcome (no interference). The test statistic and its tail definition must be specified consistently for the candidate being tested.
Positivity is not needed for the validity of a test over paths the design actually permits. However, if the allocation mechanism gives zero probability to assignments needed to distinguish effects, the data cannot provide randomization-based information about those contrasts. An interval may then be very wide or unbounded. A sharp constant effect is stronger than a null statement about an average effect: if effects vary across participants, the missing potential outcomes cannot generally be imputed from one candidate average.
Weighted tests and their justification
For a candidate \(\tau_0\), choose a statistic \(T\) that measures evidence against that candidate. For a two-sided test, one common choice is to compare the absolute statistic with its observed absolute value. For every possible assignment path \(a\), impute the outcomes under \(\tau_0\), generate the path’s outcomes, and calculate \(T(a;\tau_0)\). The adaptive assignment rule is replayed at every step: the probability of a later assignment is evaluated using the history that would have occurred on that particular path.
Write \(q(a)\) for the probability of path \(a\) under this replay. The inclusive-tail p-value is the total probability of paths at least as extreme as the observed path:
Here \(I\{\cdot\}\) equals 1 when the stated condition holds and 0 otherwise. The path probabilities need not be equal. The p-value is the probability, under the candidate sharp model and the actual assignment mechanism, of obtaining a statistic at least as extreme as the observed one. It is not the probability that the candidate effect is true.
The coverage justification follows from a weighted-tail argument. Under a true candidate sharp effect, the observed path is drawn with probabilities \(q(a)\). Group paths into distinct statistic values, ordered from most to least extreme; for ties, group the tied values together. The inclusive-tail probability at each successive group is the cumulative sum of that group’s and all more-extreme groups’ probabilities. The event that this cumulative probability is at most \(\alpha\) consists of an initial portion of the ordered groups whose total probability is at most \(\alpha\). Thus the probability of rejecting the true candidate at level \(\alpha\) is no more than \(\alpha\). This argument uses path probabilities, not a uniform-rank claim. Randomizing rejection at a boundary can attain exact size; without that extra randomization, the test is generally conservative.
Worked Example: Unequal Path Probabilities
Suppose an adaptive design has three possible statistic magnitudes, ordered from most to least extreme: 4, 2, and 1. Their assignment probabilities are 0.10, 0.25, and 0.65. Calculate the inclusive-tail p-value when the observed magnitude is 2, and assess rejection at \(\alpha=0.05\).
The paths with magnitude at least 2 have total probability \(0.10+0.25=0.35\). Therefore \(p=0.35\), and the candidate is not rejected at the 0.05 level. If the observed magnitude were 4, its p-value would be 0.10, still above 0.05. The observed statistic’s rank is not uniform: the most extreme rank occurs with probability 0.10, not \(1/3\). The weighted-tail calculation, rather than a rank argument, is what supports the size guarantee.
For comparison, if the most extreme value had probability 0.02 and all remaining probability were less extreme, rejecting only at that value would have size 0.02. This is at most 0.05 but does not attain 0.05; the discreteness of the randomization distribution makes the test conservative.
Replaying an adaptive trial
A practical implementation enumerates all assignment paths when feasible, or samples paths from the known mechanism when enumeration is too large. Each simulated path must update its own history. Holding the observed sequence of probabilities fixed while changing assignments is generally wrong: a different early outcome may lead to a different later allocation probability.
Under the constant additive model, use the candidate to impute both potential outcomes for each participant from the observed outcome and assignment.
At each assignment, apply the conditional probability specified by the mechanism to the history on that path. Generate the corresponding outcomes from the imputed potential outcomes.
Compute the chosen statistic on each path and sum the probabilities of paths at least as extreme as observed.
Repeat over candidate effects and retain those with \(p(\tau_0)>\alpha\). The retained values form a confidence set.
Worked Example: Replaying an Outcome-Adapted Allocation
Consider two participants. The first is assigned treatment with probability 0.5. After the first outcome is observed, the second participant is assigned treatment with probability 0.8 if that outcome is at least 7, and with probability 0.2 otherwise. Hypothetical observed data are \(A_1=1,Y_1=8\) and \(A_2=0,Y_2=5\). Test the candidate constant effect \(\tau_0=2\) using \(T=\sum_{i=1}^2(2A_i-1)Y_i\), with two-sided extremeness measured by \(|T|\).
Under the candidate, the imputed control outcomes are \(Y_1(0)=8-2=6\) and \(Y_2(0)=5\), so the treatment outcomes are 8 and 7. Replay all four paths. If \(A_1=1\), the first outcome is 8 and the probability of treating participant 2 is 0.8. The paths \((1,1)\) and \((1,0)\) therefore have probabilities \(0.5(0.8)=0.4\) and \(0.5(0.2)=0.1\). If \(A_1=0\), the first outcome is 6, below 7, so the corresponding probabilities for \((0,1)\) and \((0,0)\) are \(0.5(0.2)=0.1\) and \(0.5(0.8)=0.4\).
The path statistics are \(T(1,1)=8+7=15\), \(T(1,0)=8-5=3\), \(T(0,1)=7-6=1\), and \(T(0,0)=-6-5=-11\). The observed path is \((1,0)\), with \(|T_{\mathrm{obs}}|=3\). Its inclusive two-sided tail includes \((1,1)\), \((1,0)\), and \((0,0)\), whose total probability is \(0.4+0.1+0.4=0.9\). Thus the p-value for \(\tau_0=2\) is 0.9. This small example is deliberately coarse: it illustrates how the later allocation probability changes with the path’s first outcome, not how to obtain a narrow interval.
Inverting tests into a confidence set
For each candidate value, perform the test using that value’s imputations and reference distribution. Retaining candidates with p-values above \(\alpha\) yields a confidence set with randomization coverage of at least \(1-\alpha\) under the constant additive-effect model, over repetitions of the assignment mechanism for the fixed potential outcomes. This is a repeated-procedure guarantee, not a probability statement that a particular realized interval contains the parameter.
Worked Example: Reading an Inverted Set
Suppose a hypothetical fixed trial has 12 participants and a prespecified adaptive allocation algorithm. The observed treatment-minus-control mean difference in a biomarker is 4.0 mg/dL. The analysis enumerates paths using the recorded conditional assignment probabilities and tests constant additive effects with a two-sided statistic. At \(\alpha=0.05\), the weighted tail at candidate \(\tau_0=0\) is \(31/1024=0.0303\), so zero is rejected. At \(\tau_0=4\) mg/dL it is \(287/1024=0.2803\), so that candidate is retained. At \(\tau_0=8\) mg/dL it is \(42/1024=0.0410\), so that candidate is rejected.
These calculations show how weighted path probabilities determine inclusion; the observed mean difference alone does not determine the interval. In a complete analysis, the test is evaluated over all candidate values, not just these three. Suppose the full inversion for this hypothetical data set retains the connected range from 1.2 to 6.4 mg/dL and rejects values outside it. The resulting 95% confidence set is then reported as 1.2 to 6.4 mg/dL. Its interpretation is that the procedure has at least 95% coverage under the sharp constant-effect model and the specified assignment mechanism. The endpoints are not a 95% probability statement about this particular interval.
Diagnostics, alternatives, and limitations
Before trusting a result, verify that path probabilities sum to one, that every adaptive update was recomputed on each path, and that the observed path receives its actual probability. Check that the statistic is defined for every permitted path; if it is not, prespecify a suitable alternative rather than silently dropping paths. With Monte Carlo sampling instead of full enumeration, report the simulation method and assess Monte Carlo error, especially when a p-value is near \(\alpha\). A useful diagnostic is to compare simulated allocation histories with the documented conditional randomization probabilities.
Randomization-based inference differs from an interval based on a parametric outcome model: its reference distribution comes from the known assignment design, not an assumed outcome distribution. It also differs from a naive bootstrap that resamples observed participants without reproducing the allocation algorithm. Those approaches answer different questions and do not automatically preserve the adaptive randomization distribution. The advantage here is design-based validity under the stated sharp model; the cost is that the model is restrictive and exact enumeration can be computationally demanding.
A confidence set obtained by inversion need not be a single interval. Discrete tests can produce gaps or conservative endpoints, so taking the smallest interval containing the set may include candidate values that were not themselves retained. Report whether the result is the full inverted set or its interval hull. Most importantly, a sharp constant-effect interval is not automatically valid for a heterogeneous treatment effect, an average treatment effect, or an effect in a selected subgroup. Those targets require assumptions and methods appropriate to them; the randomization calculation alone does not supply the missing potential outcomes.
- Unequal probabilities are not equal ranks. Do not treat the observed statistic’s rank as uniform when adaptive paths have unequal probabilities. Sum path probabilities in the inclusive tail.
- Do not freeze the observed allocation probabilities. Later probabilities may change when an alternative path changes earlier assignments or outcomes; replay the rule using that path’s history.
- Do not confuse sharp and average nulls. The candidate constant effect imputes every missing potential outcome. An average-effect statement does not generally do so.
- Do not overstate coverage. The guarantee is under the assumed potential-outcome model and assignment mechanism; it is not a posterior probability for the realized interval.
Check Your Understanding
Answer each question, then compare with the brief solutions below.
- Why can’t the rank of an observed statistic be assumed uniform when assignment paths have unequal probabilities?
- In a two-sided randomization test, an observed statistic has magnitude 3. Three paths have magnitudes 5, 3, and 1 with probabilities 0.15, 0.25, and 0.60. What is the inclusive-tail p-value?
- Under a candidate constant additive effect of 1.5 units, an observed treated outcome is 9 units. What control outcome is imputed?
- Why must the allocation probability for a later participant be recalculated along each simulated path?
- Does inverting a sharp constant-effect test automatically give a confidence interval for the average treatment effect when effects vary? Explain.
Solutions. (1) The assignment mechanism can give different probabilities to different ranks, so rank probabilities need not be equal; use the weighted tail. (2) Include magnitudes 5 and 3: \(0.15+0.25=0.40\). (3) The imputed control outcome is \(9-1.5=7.5\) units. (4) The adaptive rule may depend on earlier outcomes or assignments, which can differ across paths and change the later conditional probability. (5) No. The sharp constant-effect model imputes every missing outcome, while an average-effect null generally does not identify those individual missing outcomes.)