Why the Reference Distribution Matters
A permutation test is only as appropriate as the rule used to generate its reference distribution. In a fixed-allocation trial, treatment labels are assigned according to a pre-specified randomization scheme, such as choosing uniformly among all allocations with specified arm counts. Permuting labels while preserving those counts can then reproduce the assignment law. In a response-adaptive trial, however, the probability of assigning the next participant may change with earlier observed responses. The final allocation counts do not, by themselves, describe that sequential law.
The issue is not that labels can never be permuted in an adaptive trial. The issue is whether the permutation distribution assigns the same probabilities to allocations as the trial’s actual randomization mechanism. If it does not, the resulting p-value need not have the advertised randomization-test error control.
Write \(Z_i=1\) when participant \(i\) is assigned to A and \(Z_i=0\) when assigned to B. Let \(H_{i-1}\) denote the observed history before participant \(i\) is assigned. The adaptive mechanism specifies \(\pi_i(z_i\mid H_{i-1})=P(Z_i=z_i\mid H_{i-1})\). The probability of a complete path \(z=(z_1,\ldots,z_n)\) is the product of its sequential assignment probabilities:
For a randomization test of a sharp null, each participant’s outcome is fixed across treatment assignments under the null. Thus, for any proposed assignment path, the outcomes that would be observed under the null can be specified, and the adaptive mechanism can be replayed using those outcomes. The relevant path probabilities are those produced by that mechanism—not automatically equal weights for all paths with the same final counts.
When Are Fixed-Count Permutations Correct?
Let \(C(Z)\) denote the vector of final treatment-arm counts. Under the adaptive mechanism, the conditional probability of a path given its counts is
The history in the denominator is the history that would arise along path \(u\). Fixed-count permutations are correct for this conditional distribution only when the adaptive path probabilities are equal for all paths with the conditioned counts. This is guaranteed by complete randomization conditional on those counts, but it is not guaranteed merely because the observed counts are fixed after the trial.
The condition can be checked directly. If two paths have the same counts but different products of sequential assignment probabilities, they cannot both receive the uniform probability assigned by a fixed-count permutation test. Conditioning on the counts removes differences in how likely different count totals were; it does not generally remove differences in the probabilities of paths within a count total.
Worked Example: Conditioning on Counts Does Not Make Paths Uniform
Consider three participants. The first is assigned to A or B with probability \(1/2\) each. Under the sharp null, suppose the fixed outcomes in participant order are \(y=(1,0,0)\). The second participant is assigned to A with probability \(0.8\) if the first response is 1, and the third is assigned to A with probability \(1/2\). Calculate the probabilities of paths with exactly one assignment to A.
Under the sharp null, the first response is 1 whichever arm the first participant receives. Therefore, the probability of assigning participant 2 to A is \(0.8\), and the probability of assigning that participant to B is \(0.2\). The three paths with exactly one A are ABB, BAB, and BBA. Multiplying the sequential probabilities gives
The probability of ending with exactly one A is \(0.05+0.20+0.05=0.30\). Conditional on that count, the path probabilities are \(0.05/0.30=1/6\), \(0.20/0.30=2/3\), and \(0.05/0.30=1/6\), respectively. A fixed-count permutation distribution instead assigns \(1/3\) to each path. The two conditional distributions are different, even though both use the same final counts.
How the Mismatch Changes a Test
To see the inferential consequence, use the difference in observed sample means as the statistic:
For the fixed outcomes \(y=(1,0,0)\) and exactly one A assignment, path ABB places the outcome 1 in A and gives \(T=1-0=1\). Paths BAB and BBA place an outcome 0 in A; the B group then has outcomes 1 and 0, so each gives \(T=0-(1+0)/2=-0.5\). These statistic values are unchanged across paths under the sharp null.
Worked Example: A Fixed-Count Permutation Gives a Different P-Value
Suppose the realized path is ABB, so \(T_{\mathrm{obs}}=1\), and consider the pre-specified upper-tail test. Calculate the exact conditional p-value under the adaptive mechanism and under uniform fixed-count permutations.
Under the adaptive mechanism, the conditional path probabilities from the previous calculation are \(1/6,2/3,1/6\). Only ABB has \(T\geq T_{\mathrm{obs}}=1\), so
Under uniform fixed-count permutations, each of the three paths has probability \(1/3\), and again only ABB is at least as extreme in the upper tail. Thus
These are different p-values because the reference distributions give different weights to the same statistic values. Under the sharp null and the stated assignment mechanism, the adaptive probability is the appropriate conditional randomization probability. The fixed-count value is not justified by the fact that the observed count is one.
This mismatch can go in either direction, depending on the observed path and the test’s tail. For example, if BAB is observed and the lower tail is specified, then \(T_{\mathrm{obs}}=-0.5\). Both BAB and BBA are in the lower tail. Their total conditional probability under the adaptive law is \(2/3+1/6=5/6\), whereas the uniform fixed-count value is \(2/3\). Neither a fixed-count permutation p-value nor an adaptive p-value is the probability that the null hypothesis is true.
Path Probabilities Depend on Order and Feedback
The product formula also makes clear why treating an adaptive trial as if it had one static allocation set can fail. A path’s probability reflects each assignment decision at the point it was made. Changing the order of labels can change which histories lead to which decisions, even when the final counts remain unchanged. The following example calculates that effect for a larger set of possible paths.
Worked Example: Comparing a Four-Participant Path With Uniform Permutations
There are four participants. The first assignment is fair. For each later participant, the probability of A is \(0.75\) if the immediately preceding response was 1 and \(0.25\) if it was 0. Under the sharp null, take the fixed outcome sequence to be \(y=(1,0,1,0)\). Compare the probability of path ABAB with the uniform probability among paths having two A assignments.
For path ABAB, the first assignment has probability \(0.5\). The second label is B after response 1, so its probability is \(0.25\). The third label is A after response 0, so its probability is \(0.25\). The fourth label is B after response 1, so its probability is \(0.25\). Therefore,
To obtain the conditional probability given two A assignments, sum the probabilities of all six possible paths with those counts. Using the same sequential rule gives the following values:
Their sum is \(0.40625\), so \(P(ABAB\mid C=2)=0.0078125/0.40625\approx0.01923\). A uniform permutation among the \(\binom{4}{2}=6\) paths gives \(1/6\approx0.1667\) to ABAB. The difference is substantial: the observed counts alone do not reproduce the sequential assignment probabilities.
Derivation and Practical Checks
The uniformity condition follows directly from conditional probability. For a fixed count vector \(c\), let \(w(z)=\prod_i\pi_i(z_i\mid H_{i-1})\) be the probability of path \(z\). Then
If \(w(z)\) has the same value for every path with counts \(c\), the denominator is that common value multiplied by the number of such paths. Each conditional probability is therefore the reciprocal of the number of paths, which is exactly the fixed-count permutation law. If two paths with counts \(c\) have different \(w(z)\), their conditional probabilities differ, so the uniform law is not correct. This establishes both the sufficient condition and the failure check.
A practical audit should reconstruct the assignment mechanism and compare path weights, rather than relying on the final arm totals. In particular:
- Check whether the probability of each assignment depended on responses already observed, and identify which responses were available at each decision.
- For paths with the observed counts, calculate or simulate their probabilities using the adaptive rule. Compare these with the equal weights imposed by label permutations.
- Verify that the null used to generate outcomes is sharp enough to supply the outcomes needed to replay the adaptive rule.
- Check that the recorded allocation algorithm, including any probability floors, caps, or forced assignments, matches the rule actually used.
If the adaptive path probabilities are known, the main alternative is to evaluate the statistic under paths generated or enumerated from that actual mechanism, using the appropriate path weights. Conditional randomization analysis restricts attention to paths with the observed counts but retains their correct conditional probabilities. Unconditional analysis uses the full assignment law, allowing counts to vary as the mechanism dictates. These approaches answer different conditional or unconditional questions; neither licenses replacing unequal adaptive weights with equal permutation weights.
Fixed-count permutation tests remain useful when the design truly assigns uniformly among allocations with fixed counts, or when the adaptive path probabilities happen to be constant within the conditioned count set. They can also be a useful sensitivity comparison, provided the difference from the actual design-based reference distribution is made explicit. They are not repaired simply by matching the observed counts, sorting labels, or using a large number of permutations: more draws from the wrong distribution still approximate the wrong distribution more precisely.
Common Pitfall and Limits
Wrong reasoning: “The trial ended with the observed number of participants in each arm, so every rearrangement of those labels was equally likely.” This treats the final count as if it had been fixed by the design before the trial. In an adaptive trial, the assignments may have been made sequentially with probabilities that changed as responses arrived.
Correction: Establish the probability of each assignment path from the recorded sequential rule. Uniform fixed-count permutations are justified only if those path probabilities are equal after conditioning on the counts. The examples above assume a known allocation rule, a fixed participant order, and a sharp null that makes outcomes available for replay. If the rule was not recorded accurately, or if the null does not determine the outcomes needed to replay it, the stated exact randomization calculation cannot simply be assumed to apply.
The point is about matching a reference distribution to a design, not about declaring every permutation procedure invalid. The actual assignment law can be checked against the uniform conditional law, and the comparison can reveal whether the distinction matters for a particular statistic and observed trial. But a close numerical match in one example does not establish the required probability equality in general.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions below.
- What condition makes a fixed-count permutation distribution equal to the adaptive assignment distribution conditional on the counts?
- In the three-participant example, what is the adaptive conditional probability of BAB given one A assignment?
- For the observed path ABB in that example, why is the upper-tail adaptive p-value \(1/6\), not \(1/3\)?
- In the four-participant example, how many paths have two A assignments, and what probability does the uniform permutation distribution give each?
- Does increasing the number of draws from a fixed-count permutation distribution correct unequal adaptive path probabilities? Explain briefly.
Solutions. 1. The sequential path probabilities must be equal for all paths having the conditioned counts; then conditioning makes each path equally likely. 2. \(P(BAB\mid C=1)=0.20/0.30=2/3\). 3. Only ABB has \(T\geq1\), and its adaptive conditional probability is \(0.05/0.30=1/6\); the value \(1/3\) incorrectly gives all three paths equal weight. 4. There are \(\binom{4}{2}=6\) paths, so uniform permutations assign each probability \(1/6\). 5. No. More draws reduce simulation error for the uniform permutation distribution, but do not change its weights to match the adaptive law.