Conditional And Unconditional Randomization Tests

Why Choose a Conditional Reference Distribution?

In an adaptive trial, the probability of an assignment path depends on the sequence of assignments and the information available at each step. A randomization test can use the distribution over all paths permitted by the design. Alternatively, it can restrict attention to paths sharing a specified feature, such as the observed allocation count, and use the assignment distribution conditional on that feature. These are different reference distributions, so they can give different p-values even when they test the same sharp null.

The key is that conditioning does not license replacing the design distribution with a convenient one. The conditional probabilities must come from the original path probabilities, and the conditioning event must leave a well-defined reference distribution. The derivation below uses a sharp null, which fixes every participant’s outcome under every assignment. This lets us evaluate the adaptive assignment mechanism along every candidate path.

Definition: Let $Z$ denote the complete assignment path and let $Y$ denote the outcomes. Write $q(z\mid Y)$ for the probability the design assigns path $z$, given the outcome values used by the adaptive rule. An unconditional randomization test compares the observed statistic with its distribution under $q(z\mid Y)$ over all possible paths. For an event $C$ defined on the paths, an event-conditional randomization test uses the distribution $$ P(Z=z\mid C,Y)=\frac{q(z\mid Y)}{\sum_{u\in C}q(u\mid Y)},\qquad z\in C, $$ provided $P(C\mid Y)>0$. Paths outside $C$ receive conditional probability zero.

The outcome notation in this definition is especially important for response-adaptive designs. Under the sharp null $H_0:Y_i(A)=Y_i(B)$ for every participant $i$, each participant’s outcome is fixed across candidate assignments. The rule can therefore be replayed along each path using the same potential outcomes, while allowing its assignment probabilities to vary with the path history. The path probability is the product of the relevant sequential assignment probabilities along that path.

Full-Path and Conditional Tests

Suppose $T(z,Y)$ is a statistic whose value can be calculated for every path under the sharp null. For an upper-tail test, the unconditional randomization p-value is $$ p_{\mathrm{uncond}}=\sum_z q(z\mid Y)\,I\{T(z,Y)\geq T(z_{\mathrm{obs}},Y)\}. $$ If we condition on $C$, the corresponding conditional p-value is $$ p_{\mathrm{cond}}= \frac{\sum_{z\in C}q(z\mid Y)\,I\{T(z,Y)\geq T(z_{\mathrm{obs}},Y)\}} {\sum_{z\in C}q(z\mid Y)}, $$ when the observed path belongs to $C$. Here $I\{\cdot\}$ equals one when its condition holds and zero otherwise. For a lower-tail test, reverse the inequality; for a two-sided test, specify the extremeness rule in advance and apply it consistently to every path. Including ties in the tail is the usual convention for a conservative exact p-value.

The unconditional test averages over both paths inside and outside $C$. The conditional test asks how unusual the observed statistic is among paths in $C$, given that $C$ occurred. Neither distribution is automatically preferable in every setting. Conditioning may be useful when the feature is part of the design question or when the conditional law has a clear interpretation. It can also discard information, reduce power, or produce a p-value that differs substantially from the unconditional one.

Worked Example: An Unconditional Test Over All Paths

Consider a hypothetical two-participant, two-arm adaptive design. The first participant is assigned to A with probability $0.5$. For the second participant, the probability of A is $0.8$ if the first assignment was A, and $0.4$ if it was B. Under the sharp null, suppose the fixed outcomes are $Y_1=2$ and $Y_2=5$. Use $T(z,Y)=\sum_{i=1}^2(2z_i-1)Y_i$, where $z_i=1$ denotes A and $z_i=0$ denotes B. For an observed path AA, calculate the upper-tail unconditional p-value.

The path probabilities are $q(AA)=0.5(0.8)=0.4$, $q(AB)=0.5(0.2)=0.1$, $q(BA)=0.5(0.4)=0.2$, and $q(BB)=0.5(0.6)=0.3$. They sum to $1$.

The statistic values are $T(AA)=2+5=7$, $T(AB)=2-5=-3$, $T(BA)=-2+5=3$, and $T(BB)=-2-5=-7$. Only AA has $T(z,Y)\geq T(AA,Y)=7$. Thus $$ p_{\mathrm{uncond}}=q(AA)=0.4. $$

For this hypothetical design, the upper-tail unconditional p-value is $0.400$. It is the probability, under the full adaptive assignment mechanism and the sharp null, of a statistic at least as large as the observed value.

Conditioning Requires Renormalizing the Design Law

To condition correctly, first identify the event $C$, then retain the original probability of every path in that event, and finally divide by the total probability of the event. This normalization matters: adaptive designs generally do not assign equal probability to paths merely because they share a feature. The conditional distribution is uniform only if the original path probabilities are equal within $C$.

Worked Example: Conditioning on the Allocation Count

Use the same hypothetical design and fixed outcomes as above. Suppose the observed path is AB and the test conditions on $C=\{\text{exactly one participant is assigned to A}\}$. Calculate the lower-tail conditional p-value for the observed statistic.

The event contains AB and BA, with original probabilities $0.1$ and $0.2$. Therefore $P(C)=0.1+0.2=0.3$, and the conditional path probabilities are $$ P(AB\mid C)=\frac{0.1}{0.3}=\frac{1}{3},\qquad P(BA\mid C)=\frac{0.2}{0.3}=\frac{2}{3}. $$

The observed statistic is $T(AB,Y)=-3$. Among the paths in $C$, only AB has a statistic less than or equal to $-3$. Hence $$ p_{\mathrm{cond}}=\frac{0.1}{0.1+0.2}=\frac{1}{3}\approx 0.333. $$

The conditional p-value is about $0.333$, not $0.5$: AB and BA have different conditional probabilities. The unconditional lower-tail probability for the same observed path would include BB as well, giving $0.1+0.3=0.4$. The two p-values answer different reference-distribution questions.

When Does the Conditional Test Remain Valid?

Under a sharp null, the full-path randomization p-value is exact or conservative because the observed path is drawn from the design distribution used to rank its statistic. The same argument applies conditionally: once $C$ has occurred, the observed path is distributed according to the original path law renormalized over $C$. Thus, if the design law is correctly specified, $C$ has positive probability, and the statistic and conditioning event are evaluated consistently under the null, the conditional p-value is super-uniform given $C$:

P(p_{\mathrm{cond}}\leq \alpha\mid C,H_0)\leq \alpha,\qquad 0\leq\alpha\leq1.

This inequality means that the conditional test rejects no more often than its nominal level among repetitions in which $C$ occurs. Discreteness and tied statistics can make the test conservative. For the ordinary finite-sample randomization guarantee, the null must be sharp enough to determine outcomes under all candidate assignments. The assignment algorithm—including any response-dependent transition probabilities—must also be known or reconstructed correctly.

Specify the null and the design.
Under a sharp null, fix each participant’s outcome across assignments. Record the actual sequential assignment probabilities, including their dependence on the trial history.

Define the conditioning event.
State precisely which paths satisfy $C$. The event must have positive probability under the null design, and the observed path must belong to it for the observed conditional test.

Reweight paths within the event.
Keep each path’s original probability and divide by $P(C)$. Do not substitute a uniform distribution unless equal probabilities within $C$ have been established.

Evaluate the statistic and tail.
Compute the statistic for every retained path under the null, then sum the conditional probabilities in the prespecified tail, including ties according to the chosen convention.

The justification is direct. Conditional on $C$, the probability of any retained path $z$ is $q(z\mid Y)/P(C\mid Y)$. Ranking the statistic using these probabilities therefore ranks the observed path according to its actual conditional sampling distribution. For a discrete distribution, the total conditional probability assigned to outcomes at least as extreme as an observed value cannot exceed the corresponding tail area; ties may add probability mass, making the resulting test conservative. This is the same finite-support argument used for an unconditional randomization test, applied to the restricted support.

Diagnostics, Alternatives, and Limitations

Before reporting a conditional result, check that the event is not impossible, that the retained path probabilities sum to one after normalization, and that the observed path is included. Recompute transition probabilities along candidate paths rather than assuming the adaptive algorithm produces equal weights. If the event is selected after inspecting the observed statistic, the selection rule must be accounted for; choosing a conditioning event or tail because it gives a more favorable result is not justified by the simple conditional argument.

A useful alternative is the unconditional test, which uses all paths and avoids restricting inference to a particular event. It may use more of the design distribution, but it answers a different question and need not yield a smaller p-value. A conditional test is not an automatic correction for adaptation: it is valid only when the conditional law follows from the actual design and the null assumptions. Nor does conditioning on a feature such as the final allocation count automatically create a uniform permutation distribution.

Finally, exactness under a sharp null should not be transferred without justification to a weaker null, such as a null average treatment effect with heterogeneous individual effects. Under a weak null, the observed outcomes need not be the outcomes that would have occurred along every reassigned path. In an adaptive design, this can also leave the response-dependent assignment probabilities under alternative paths unspecified. A conditional reference law formed by holding observed outcomes fixed may then fail to represent the sampling distribution for that weaker null.

Worked Example: Why a Sharp-Null Reference Law May Fail for a Weak Null

Suppose, hypothetically, there are two participants and the event being conditioned on is exactly one assignment to A. Participant 1 has potential outcomes $Y_1(A)=2$, $Y_1(B)=0$; participant 2 has $Y_2(A)=-2$, $Y_2(B)=0$. The average of the two individual treatment effects is $(2+(-2))/2=0$, but the sharp null is false. Compare the observed-outcome statistic $T=$ treated outcome minus control outcome under the two paths in the event.

For AB, participant 1 receives A and participant 2 receives B, so $T=2-0=2$. For BA, participant 2 receives A and participant 1 receives B, so $T=-2-0=-2$. The outcomes differ across paths even though the average treatment effect is zero. Holding the observed outcomes fixed while reallocating assignments would not reproduce these potential outcomes under the other path.

Thus the fixed-outcome randomization argument established for a sharp null does not, by itself, validate this conditional test for a weak average-effect null. A method with an appropriate weak-null justification is needed; the sharp-null conditional reference law cannot simply be reused.

Common Mistakes

Assuming conditioning makes paths equally likely. It does so only when the original design probabilities are equal within the conditioning event. Otherwise, renormalize the original probabilities.
Using only paths with the observed feature but forgetting the denominator. The retained probabilities must be divided by $P(C)$ to form a conditional distribution.
Conditioning on an event with probability zero. There is no conditional reference law for an event the design cannot produce.
Calling a conditional p-value the probability that the null is true. It is a tail probability calculated under the null design, conditional on the specified event.
Extending sharp-null validity to a weak null without proof. Under a weak null, potential outcomes and adaptive probabilities may not be determined for every candidate path.

Key takeaway: A conditional randomization test is valid when its reference law is the original assignment-path law restricted to a positive-probability event and properly renormalized, with the null and design assumptions sufficient to evaluate every retained path.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions.

If paths in an event $C$ have original probabilities $0.12$, $0.18$, and $0.30$, what is the conditional probability of the second path given $C$?
In the worked adaptive design, what is $P(AA\mid Z_1=A)$?
Why is conditioning on an event of probability zero invalid?
What assumption permits outcomes to be held fixed while evaluating alternative paths in an exact randomization test?
Does a valid conditional test necessarily give a smaller p-value than the unconditional test? Explain.

Solutions. (1) The event probability is $0.12+0.18+0.30=0.60$, so the second path has conditional probability $0.18/0.60=0.30$. (2) $P(AA\mid Z_1=A)=q(AA)/P(Z_1=A)=0.4/0.5=0.8$. (3) Conditional probabilities require division by $P(C)$; if it is zero, the event cannot occur and no conditional reference law is defined. (4) The sharp null fixes each participant’s outcome under every assignment. (5) No. Conditioning changes the reference distribution and may either increase or decrease the tail probability; the tests answer different questions.