Why the Assignment Rule Belongs in the Reference Distribution
In a response-adaptive trial, each new assignment can depend on what has happened earlier. The probability of assigning the next participant to treatment may change as responses accumulate, and it may depend on which participants received which treatments. Consequently, the possible treatment-allocation sequences need not be equally likely. A randomization analysis that replaces the trial’s rule with a different allocation scheme answers a different design-based question.
The earlier material on potential outcomes and sharp null hypotheses supplies the key device for inference: under a sharp null, every participant’s outcome is known under every possible assignment. That lets us hold outcomes fixed while considering the assignments the trial could have generated. But the assignments must be generated according to the actual sequential rule, not a convenient substitute.
The history must represent what was available at the time, not information learned later. If outcomes are released only after a delay, for instance, they cannot enter the assignment probability for participants enrolled during that delay. The probabilities \(q_i\) are properties of the randomization procedure, not probabilities that a fitted outcome model assigns to the observed data.
Path Probabilities and Sharp-Null Inference
Write a complete assignment path as \(a=(a_1,\ldots,a_n)\). By the chain rule for conditional probabilities, its probability under the design is
The history in each factor is the history that would arise along that particular path. If an earlier assignment changes which arm’s response summary is updated, a different path can produce a different history and therefore different probabilities for later assignments.
Let \(Y_i(1)\) and \(Y_i(0)\) be participant \(i\)'s potential outcomes. Under the sharp null of no effect for any participant, \(Y_i(1)=Y_i(0)=y_i\). Thus \(y_i\) is known from the observed outcome regardless of the assignment path. For a prespecified statistic \(T(A,y)\), the randomization distribution assigns each possible path its design probability and evaluates the statistic using the fixed outcomes \(y=(y_1,\ldots,y_n)\).
This is an upper-tail randomization p-value for a statistic where larger values are more inconsistent with the null in the direction of interest. A two-sided test needs a prespecified two-sided extremeness rule. With a discrete distribution, including paths tied with the observed statistic generally makes the nonrandomized test conservative rather than continuously uniform under the null.
Worked Examples
Worked Example: A Response Changes the Next Assignment Probability
Suppose a hypothetical two-participant trial assigns participant 1 to treatment with probability \(0.5\). For participant 2, the rule assigns treatment with probability \(0.8\) if participant 1 was treated and responded, and probability \(0.2\) otherwise. Consider the observed history \(A_1=1\), with a response, followed by \(A_2=0\).
The first assignment has probability \(0.5\). Given the observed history, the probability of assigning participant 2 to control is \(1-0.8=0.2\). Therefore, the probability of this complete path is
The path \((A_1,A_2)=(1,1)\) instead has probability \(0.5(0.8)=0.40\). The two paths are not equiprobable, even though each starts with the same first assignment. Treating the four possible two-person paths as equally likely would not represent this rule.
Worked Example: A Sharp-Null Reference Distribution With Unequal Path Weights
Consider a hypothetical three-participant trial with binary responses and sharp-null outcomes \(y=(1,0,1)\). The assignment rule starts with \(P(A_1=1)=0.5\). For participant 2, \(q_2(1\mid H_1)=0.8\) if participant 1 was assigned to treatment and responded, and \(0.2\) otherwise. For participant 3, \(q_3(1\mid H_2)=0.75\) if participant 2 was assigned to treatment and did not respond, and \(0.25\) otherwise. These probabilities are stipulated as part of this hypothetical design.
Use the prespecified statistic \(T(A,y)=\sum_{i=1}^3 A_i y_i\), the number of observed responses among those assigned to treatment. Suppose the observed path is \(A_{\mathrm{obs}}=(1,0,1)\), for which \(T(A_{\mathrm{obs}},y)=1(1)+0(0)+1(1)=2\). The path probabilities follow by multiplying the conditional probabilities along each path:
| Path \(A\) | Path probability | \(T(A,y)\) |
|---|---|---|
| 000 | \(0.5(0.8)(0.75)=0.300\) | 0 |
| 001 | \(0.5(0.8)(0.25)=0.100\) | 1 |
| 010 | \(0.5(0.2)(0.25)=0.025\) | 0 |
| 011 | \(0.5(0.2)(0.75)=0.075\) | 1 |
| 100 | \(0.5(0.2)(0.75)=0.075\) | 1 |
| 101 | \(0.5(0.2)(0.25)=0.025\) | 2 |
| 110 | \(0.5(0.8)(0.25)=0.100\) | 1 |
| 111 | \(0.5(0.8)(0.75)=0.300\) | 2 |
The weights sum to \(1.000\), as probabilities over all paths must. For the upper-tail test, paths at least as extreme as the observed path have \(T\geq2\): paths 101 and 111. The p-value is therefore \(0.025+0.300=0.325\). It is not the probability of the observed path alone, \(0.025\), and it is not obtained by counting paths as equally likely.
The stipulated rule is unusual, but the calculation illustrates the relevant requirement: each path’s weight must reflect its conditional assignment probabilities along its own history. Under the sharp null, the response sequence \(y\) remains fixed while assignments are varied; the arm membership in the history can still change from path to path.
Worked Example: Comparing With a Fixed-Count Permutation Reference
Use the data and statistic from the previous example. A common alternative would be to condition on the observed total of two treatment assignments and treat the three ways to assign exactly two participants to treatment as equally likely: 011, 101, and 110. Under this fixed-count reference, their statistics are respectively 1, 2, and 1. The upper-tail probability is
That differs from the actual-rule p-value, \(0.325\). More importantly, the fixed-count calculation conditions on a total and assigns equal weight to the resulting paths; the response-adaptive design does not generally do either. A fixed-count reference is justified only when it corresponds to the actual randomization distribution or to a valid conditioning argument for the design. Similar-looking numerical answers do not make the reference distributions interchangeable.
Justification, Diagnostics, and Alternatives
The validity logic can be stated directly. Under the sharp null, the vector \(y\) does not change with assignment. If the randomization device follows the recorded mechanism, then the observed path is drawn from the path probabilities specified by that mechanism. Consequently, ranking \(T(A,y)\) over paths with those probabilities gives the null reference distribution. The upper-tail probability includes the observed statistic and all more extreme values; for a discrete distribution, the probability of landing in such an upper tail is at most the nominal level when the rejection threshold is chosen from that distribution. This argument depends on the sharp null and the assignment rule, not on an outcome-model approximation.
For a sharp null, use each participant’s observed outcome as that participant’s outcome under every assignment.
For each possible path, the next assignment probability must condition on the history that path would have produced, including only information available at that point.
Generate or enumerate paths under the actual rule, calculate the prespecified statistic for each, and compare their weighted distribution with the observed statistic.
Practical diagnostics start with the design record. Check that assignment probabilities are available at every decision point, that the recorded history contains the inputs used by the rule, and that the probabilities assigned to permitted options sum to one. Compare a replay of the observed trial with the recorded assignments and verify that response timing, eligibility changes, and any operational overrides are handled as specified. If a response or allocation needed by the rule is missing from the record, the resulting uncertainty cannot be repaired by silently substituting a uniform permutation.
| Approach | Reference distribution uses | When it is appropriate |
|---|---|---|
| Actual-rule randomization inference | The trial’s sequential conditional probabilities | When the mechanism is known and the null makes the required outcomes imputable |
| Fixed-allocation permutation | Equal-weight reallocations with a fixed treatment count | When that allocation scheme, or a justified conditional distribution, matches the design |
| Model-based inference | An assumed probability model for outcomes, with its stated design handling | When its outcome-model assumptions are defensible; it is not a substitute for an unknown randomization rule |
A central limitation is that a sharp null supplies outcomes under every assignment. A weak null about an average effect does not, by itself, specify every missing potential outcome, so the same exact sharp-null construction cannot simply be claimed for that weaker hypothesis. In addition, an unknown, misrecorded, or incompletely implemented assignment mechanism undermines the design-based reference distribution. In those cases, the analysis must state what is known, what assumptions replace the missing design information, and how conclusions depend on them.
Common Mistakes
- Wrong: “The trial randomized, so every allocation with the observed treatment count is equally likely.” Correction: Sequential adaptation can give paths different probabilities, including paths with different treatment counts.
- Wrong: “Under the sharp null, keep the observed assignment history fixed while generating the reference distribution.” Correction: Keep the null-implied outcomes fixed, but allow assignments and the histories they generate to vary.
- Wrong: “Use the probability of the observed path as the p-value.” Correction: A tail p-value sums the probabilities of all paths at least as extreme under the prespecified statistic.
- Wrong: “A plausible outcome model can fill in an undocumented allocation rule.” Correction: Outcome modeling and the randomization mechanism are different assumptions; the former does not establish the latter.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- A two-step design assigns the first participant with probability \(0.5\). For the second, \(P(A_2=1\mid A_1=1)=0.7\). What is the probability of path \((1,0)\)?
- Why can response-adaptive paths have unequal probabilities even if the first assignment is a fair randomization?
- Under a sharp null, which information is held fixed when assignments are varied: outcomes, assignments, or both?
- In the three-participant example, what is the upper-tail p-value for observed path 111 using \(T(A,y)=\sum_i A_i y_i\)?
- When does an equal-weight, fixed-count permutation reference match an adaptive trial’s randomization distribution?
Solutions. (1) \(P(1,0)=P(A_1=1)P(A_2=0\mid A_1=1)=0.5(1-0.7)=0.15\). (2) Later probabilities depend on the assignments and responses in the realized history, so different paths can have different products of conditional probabilities. (3) Hold the sharp-null outcomes fixed and vary assignments according to the design; the assignments are not held fixed. (4) For path 111, \(T=1+0+1=2\). The paths with \(T\geq2\) are 101 and 111, with probabilities \(0.025\) and \(0.300\), so the p-value is \(0.325\). (5) Only when the actual design, or a justified conditioning argument, gives that fixed-count set of paths equal probabilities; response adaptation alone does not establish this.