Why the Assignment Path Matters
A Fisher randomization test asks how unusual the observed test statistic would be under the assignments the design could have generated, if the sharp null hypothesis were true. With adaptive assignment, those possibilities generally do not have equal probabilities: the chance of assigning the next participant may depend on the assignments or responses already observed. The reference distribution must therefore use the probability of each complete path under the actual sequential rule.
Here, Fisher test means a randomization test based on the assignment mechanism, not the separate Fisher exact test for a contingency table. The earlier material on potential outcomes and sharp null hypotheses supplies the key premise: under a sharp null of no treatment effect for any participant, each participant’s outcome is fixed regardless of assignment. Thus, to evaluate a candidate assignment path, the observed outcomes can be held fixed while the assignments are varied.
For the test below, assume the enrolled participants and the assignment algorithm are fixed; the algorithm’s conditional probabilities are correctly specified and used at every step; and the test statistic is defined for every supported path. The sharp null is \(H_0:Y_i(1)=Y_i(0)\) for every participant \(i\), where \(Y_i(1)\) and \(Y_i(0)\) are the potential outcomes under treatment and control. Assume consistency and no interference, so observed outcomes correspond to the assigned treatment and do not depend on other participants’ assignments. Under \(H_0\), write the common, fixed outcome as \(Y_i\). If the design uses prior observed outcomes to adapt, those outcomes are determined along each candidate path under this null. Any additional randomization affecting assignments must also be represented in the stated mechanism or appropriately included in the reference distribution.
These conditions make the probability law over paths known under the null. They do not require assignments to be independent, or the probabilities to remain constant from one participant to the next. What matters is that the full path probability is calculated from the correct conditional probability at each step.
Weighted Randomization P-Values
Choose a test statistic \(S(a;Y)\) before inspecting the randomization results, with larger values defined as more extreme in the direction being tested. For a two-sided test, specify a two-sided extremeness rule in advance, for example by ordering paths by the absolute value of a prespecified contrast. Under the sharp null, the outcomes \(Y\) are held fixed, and only \(a\) varies. The upper-tail randomization p-value for the observed path \(A_{\mathrm{obs}}\) is
The sum includes ties. It is a probability under the design, not a count of paths divided by the total number of paths. If paths have unequal probabilities, treating each path as equally likely generally gives a different reference distribution.
Worked Example: Enumerating a Three-Participant Adaptive Design
Consider a hypothetical three-participant trial. The first participant has probability \(0.5\) of assignment to \(T\). For participant 2, the probability of \(T\) is \(0.75\) if participant 1 received \(T\), and \(0.25\) otherwise. For participant 3, the probability of \(T\) is \(0.6\) if at least one of the first two participants received \(T\), and \(0.4\) if neither did. These conditional probabilities define the hypothetical design; the probability of a path is their product along that path.
For example, along path CCT, the first two assignments are both C, so the probability of T for participant 3 is \(0.4\), not \(0.6\). Therefore
The full enumeration is:
| Path | Conditional factors | Path probability |
|---|---|---|
| TTT | \(0.5(0.75)(0.6)\) | 0.225 |
| TTC | \(0.5(0.75)(0.4)\) | 0.150 |
| TCT | \(0.5(0.25)(0.6)\) | 0.075 |
| TCC | \(0.5(0.25)(0.4)\) | 0.050 |
| CTT | \(0.5(0.25)(0.6)\) | 0.075 |
| CTC | \(0.5(0.25)(0.4)\) | 0.050 |
| CCT | \(0.5(0.75)(0.4)\) | 0.150 |
| CCC | \(0.5(0.75)(0.6)\) | 0.225 |
The probabilities sum to \(1.000\), as a check. Suppose the sharp null holds and the fixed outcomes, in participant order, are \(Y=(0,2,0)\) units. Use the statistic \(S(a;Y)=\sum_{i:a_i=1}Y_i\), the sum of outcomes among those assigned to treatment, measured in the same outcome units. The statistic is \(2\) exactly on paths assigning participant 2 to \(T\); it is \(0\) on all other paths. In particular, CTC has statistic \(2\), since participant 2 is assigned to treatment. If CTC is observed, the upper-tail p-value is
This calculation uses both the fixed outcomes under the sharp null and the unequal probabilities of the paths. The p-value is the probability, under this adaptive design and the null, of a statistic at least as large as the observed value.
Why the Test Controls Type-I Error
The justification is a finite probability argument. Under \(H_0\), the outcomes used to calculate the statistic are fixed, and the realized path is drawn from the distribution assigning probability \(\pi(a)\) to each supported path. Group paths with the same statistic value. Order the distinct statistic values from largest to smallest, and let \(m_j\) be the total probability of paths in group \(j\). For a path in group \(j\), the upper-tail p-value is the cumulative probability
The sequence \(c_j\) increases as the statistic becomes less extreme. Rejection at level \(\alpha\) includes precisely those initial groups whose cumulative probabilities are at most \(\alpha\). If there is at least one such group, the probability of rejection is the cumulative probability of the last included group, which is at most \(\alpha\). If there are no such groups, the probability of rejection is zero. Thus, in all cases,
This proves finite-sample type-I error control. The inequality may be strict because statistic values are discrete and the probability masses of tied groups may not add up to exactly \(\alpha\). “Exact” describes use of the finite-sample randomization distribution and its valid error control; it does not promise that every chosen level is attained exactly.
Worked Example: Type-I Error With Unequal Path Probabilities
Consider a hypothetical two-participant design. The first participant has probability \(0.5\) of assignment to \(T\). The second has probability \(0.8\) of \(T\) if the first received \(T\), and \(0.2\) if the first received \(C\). The path probabilities are
Suppose the sharp-null outcomes are fixed at \(Y_1=1\) and \(Y_2=2\) units, and use the treatment-outcome sum \(S\). The statistics are \(3\) on TT, \(1\) on TC, \(2\) on CT, and \(0\) on CC. Their upper-tail p-values, computed by adding the probabilities of paths with at least that statistic, are:
| Observed path | Statistic | Upper-tail p-value |
|---|---|---|
| TT | 3 | 0.4 |
| CT | 2 | 0.4 + 0.1 = 0.5 |
| TC | 1 | 0.4 + 0.1 + 0.1 = 0.6 |
| CC | 0 | 1.0 |
At \(\alpha=0.55\), rejection occurs on TT and CT, whose total probability is \(0.4+0.1=0.5\). The type-I error is therefore \(0.5\), no more than \(0.55\). At \(\alpha=0.6\), TC is also included, so the rejection probability is \(0.6\). The calculation shows both how discreteness can make a test conservative at one level and how, at another level, the cumulative probability can equal that level.
Ties, Diagnostics, and Limitations
Ties are handled by including every path with a statistic equal to the observed statistic. This gives a valid, potentially conservative p-value. A different convention can invalidate the guarantee: using only paths with a strictly larger statistic drops the probability mass of paths tied with the observation. Random tie-breaking is an optional way to attain exact rejection probability at a chosen level when the statistic is discrete. It adds an independent random draw and is not the usual nonrandomized Fisher p-value.
Worked Example: Randomized Tie-Breaking
Use the three-participant design and fixed outcomes from the first example, where \(S=2\) has probability \(0.5\) and \(S=0\) has probability \(0.5\). Let \(U\) be an independent random variable uniformly distributed between 0 and 1, and define
If the observed statistic is \(2\), then \(p_{\mathrm{rand}}=0+0.5U\). If it is \(0\), then \(p_{\mathrm{rand}}=0.5+0.5U\). At level \(\alpha=0.3\), an observed statistic of \(2\) leads to rejection when \(0.5U\leq0.3\), or \(U\leq0.6\). Since this statistic occurs with probability \(0.5\), its contribution to the rejection probability is \(0.5(0.6)=0.3\). An observed statistic of \(0\) cannot yield \(p_{\mathrm{rand}}\leq0.3\). The overall rejection probability is therefore \(0.3\). This optional randomization spreads each tied probability mass across its corresponding p-value interval; it does not change the ordinary conservative p-value convention.
Practical checks are essential before interpreting a result. Confirm that the conditional probabilities refer to the history available at each assignment, and recalculate them separately along each candidate path. Check that the supported path probabilities sum to one. Recompute the statistic for every path using the sharp-null outcomes, and verify that the chosen tail and tie rule match the test plan. If outcomes were used by the adaptive algorithm, confirm that their histories are reconstructed under each path as well. A path probability that uses a condition not met on that path, or a statistic that changes outcomes instead of assignments under the sharp null, changes the reference distribution and can destroy the guarantee.
The result is specific to the sharp null: it posits no effect for every participant, making all relevant outcomes imputable under every assignment. It does not by itself establish type-I error control for a weaker null that permits individual effects to vary while averaging to zero. Nor does it repair an incorrectly implemented or incompletely recorded assignment mechanism. A Fisher test can be computationally demanding when there are many supported paths; regardless of how the distribution is computed, the mathematical target is the same design-weighted distribution, not a uniform distribution over paths.
Common Mistake
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- In a sequential design, what determines the probability of a complete assignment path?
- Why can outcomes be held fixed across candidate paths when testing a sharp null?
- In the two-participant example, what is the p-value for observed path TC, and why?
- Why does including ties give type-I error at most \(\alpha\), even if it does not give exactly \(\alpha\)?
- Name two checks that can reveal an incorrectly constructed adaptive randomization distribution.
Solutions. (1) Multiply the conditional probability of each assignment along that particular path, using the history that would occur along it. (2) Under the sharp null, each participant’s outcome is the same under treatment and control, so the observed outcome is known under every candidate assignment. (3) TC has statistic \(1\); paths with statistic at least \(1\) are TT, CT, and TC, with total probability \(0.4+0.1+0.1=0.6\). (4) Tied paths form a probability mass that may not fit the level exactly; cumulative probability through the last rejected statistic group cannot exceed \(\alpha\). (5) Check that path probabilities sum to one and that each factor uses the correct conditional probability for the history on that path. Recomputing the statistic under the sharp null for every path is another important check.