Conditioning On Allocation Counts

Why conditioning on final counts requires care

A randomization test for an adaptive trial must respect how assignments were generated. If treatment probabilities changed in response to accumulating trial data, two assignment sequences with the same final treatment count need not have had the same probability. Conditioning on that count is legitimate only if the conditional reference distribution still represents the design’s probabilities among the paths that could have produced the count.

This distinction matters for a sharp-null test. Under a sharp null of no treatment effect for any unit, each unit’s outcome is fixed regardless of its assignment. The allocation algorithm can therefore be replayed across candidate assignment paths, using the outcomes that would be observed along each path. If the algorithm depends on past outcomes, those outcomes—and hence subsequent allocation probabilities—must be included in each path’s probability.

Definition: For $n$ sequentially assigned units, let $A_t\in\{0,1\}$ be the assignment of unit $t$, and write $A=(A_1,\ldots,A_n)$ for the complete assignment path. Let $H_{t-1}$ denote the design-relevant history before assigning unit $t$, and let $\pi_t(H_{t-1})=P(A_t=1\mid H_{t-1})$. The path probability is $$ P(A=a)=\prod_{t=1}^n \pi_t(H_{t-1}(a))^{a_t}\{1-\pi_t(H_{t-1}(a))\}^{1-a_t}. $$ Here $H_{t-1}(a)$ is the history induced by path $a$, including the responses observed along that path. The final treatment count is $N_1=\sum_{t=1}^n A_t$. For a count $m$ with $P(N_1=m)>0$, the conditional path probability is $$ P(A=a\mid N_1=m)= \frac{P(A=a)\,\mathbf{1}\{\sum_{t=1}^n a_t=m\}} {\sum_{b:\,\sum_{t=1}^n b_t=m}P(A=b)}. $$

These formulas presume that the allocation mechanism is known and that its conditional probabilities are correctly specified for every history used in the calculation. Paths with zero design probability are not part of the conditional support. The denominator is the design probability of the conditioning event; if it is zero, conditioning on that count is not defined.

A small adaptive design, worked path by path

Consider a hypothetical three-unit trial. The first unit is assigned to treatment with probability $0.5$. After the first response $Y_1$ is observed, unit 2 is assigned to treatment with probability $0.8$ if $Y_1=1$, and with probability $0.2$ if $Y_1=0$. Unit 3 is assigned with probability $0.5$, regardless of history. Assume independent randomization at each step conditional on the stated history. Under the sharp null, take the fixed responses to be $Y_1=1,Y_2=0,Y_3=0$. The response-dependent probability for unit 2 is therefore $0.8$ on every path considered here.

Worked Example: Conditional Weights for One Treated Unit

Find the conditional distribution of the assignment path given $N_1=1$. The three paths with exactly one treated unit are $100$, $010$, and $001$.

For path $100$, the first assignment has probability $0.5$, the second is control with probability $1-0.8=0.2$, and the third is control with probability $0.5$. Thus $$ P(A=100)=0.5(0.2)(0.5)=0.05. $$

For path $010$, the first assignment is control with probability $0.5$, the second is treatment with probability $0.8$, and the third is control with probability $0.5$: $$ P(A=010)=0.5(0.8)(0.5)=0.20. $$ For path $001$, the first two assignments are control, with probabilities $0.5$ and $0.2$, and the third is treatment with probability $0.5$: $$ P(A=001)=0.5(0.2)(0.5)=0.05. $$

The probability of the conditioning event is $P(N_1=1)=0.05+0.20+0.05=0.30$. Dividing each path probability by $0.30$ gives $$ P(100\mid N_1=1)=\frac{1}{6},\qquad P(010\mid N_1=1)=\frac{2}{3},\qquad P(001\mid N_1=1)=\frac{1}{6}. $$ The conditional distribution is not uniform: path $010$ is four times as likely as either of the other compatible paths.

The calculation uses the response-dependent part of the design explicitly. Under a sharp null, the outcomes are fixed across assignments, so the response that determines unit 2’s probability remains $Y_1=1$. Under alternatives where outcomes depend on treatment, this fixed-outcome replay is not justified in the same way; this conditional randomization argument is for testing a sharp null.

Validity of the conditional randomization test

Let $T(A,Y)$ be a test statistic, with larger values chosen to be more extreme for a prespecified upper-tail alternative. Under the sharp null, the outcomes $Y$ are fixed, so each candidate path $a$ determines a statistic $T(a,Y)$. For the observed path $A_{\text{obs}}$, the design-weighted conditional upper-tail p-value is $$ p_{\mathrm{cond}}= \sum_{a:\,N_1(a)=N_1(A_{\text{obs}})} P(A=a\mid N_1=N_1(A_{\text{obs}})) \mathbf{1}\{T(a,Y)\geq T(A_{\text{obs}},Y)\}. $$

The conditional validity argument is direct. Under the sharp null and the stated design, conditional on $N_1=m$, the observed assignment path has exactly the probabilities $P(A=a\mid N_1=m)$. The p-value is the conditional probability mass at outcomes at least as extreme as the observed statistic. For a discrete statistic, ties can make this p-value conservative, but cannot make its conditional rejection probability exceed the nominal level.

Theorem: Suppose the sharp null holds, the allocation probabilities and response histories are correctly represented, and $P(N_1=m)>0$. If $p_{\mathrm{cond}}$ is computed from the design-weighted conditional path distribution, then for every $0\leq\alpha\leq1$, $$ P(p_{\mathrm{cond}}\leq\alpha\mid N_1=m)\leq\alpha. $$ Consequently, $P(p_{\mathrm{cond}}\leq\alpha)\leq\alpha$ as well, since the unconditional probability is an average of the conditional rejection probabilities over possible counts.

To see the discrete-tail reasoning, list the distinct statistic values from most to least extreme and add their conditional probability masses in that order. The p-value assigned to any value is the cumulative mass through that value. A rejection at level $\alpha$ can include only values whose cumulative mass is at most $\alpha$; the total mass of those values is therefore at most $\alpha$. Ties are treated together by the tail definition, which can leave the test conservative.

Worked Example: A Conditional Upper-Tail Test

Use the previous design and fixed outcomes $Y=(1,0,0)$. Suppose the observed path is $100$, so the observed treatment count is one. Use the treatment-minus-control difference in observed means, $$ T(a,Y)=\bar{x}_{1}(a)-\bar{x}_{0}(a), $$ where $\bar{x}_{1}(a)$ and $\bar{x}_{0}(a)$ are the means among treated and control units under path $a$.

For path $100$, the treated mean is $1$ and the control mean is $(0+0)/2=0$, so $T=1$. For path $010$, the treated mean is $0$ and the control mean is $(1+0)/2=0.5$, so $T=-0.5$. For path $001$, the treated mean is also $0$, and the control mean is again $0.5$, so $T=-0.5$.

Only path $100$ has a statistic at least as large as the observed value $1$. Its conditional probability is $1/6$, so the design-weighted upper-tail p-value is $p_{\mathrm{cond}}=1/6$, or approximately $0.1667$. At significance level $0.05$, this test does not reject the sharp null. The p-value is not the probability that the null is true; it is the conditional design probability, under the null, of a statistic at least this extreme.

Uniform permutations are not automatically justified

A fixed-count permutation test gives equal weight to every assignment path with the observed count. That is the correct conditional randomization distribution only when the design’s conditional path probabilities are equal across those paths (or when another argument establishes validity for the particular test). Adaptive assignment commonly makes the conditional weights unequal, as in the three-unit example.

If equal weights do not match the design weights, the usual randomization-test justification does not apply. However, it would also be too strong to conclude that every uniform-weight p-value must be invalid: a particular statistic and tail can yield a valid test by coincidence. In that case validity requires separate calibration; it does not follow merely from conditioning on the count.

Worked Example: Why the Statistic Matters for Uniform Weights

Suppose three paths with the same count have true conditional design probabilities $0.10,0.45,0.45$. First, let the statistic be $0$ on the first path and $1$ on the other two, and use a lower-tail uniform-weight p-value. On the first path this p-value is $1/3$, while on either of the other paths it is $1$. Therefore $$ P(p_{\mathrm{uniform}}\leq 1/3\mid N_1=m)=0.10\leq 1/3. $$ This particular test is valid at level $1/3$, despite the mismatch between uniform and design weights.

Now instead let the three statistic values be distinct and ordered so the path with true probability $0.45$ has the smallest value. For a lower-tail uniform test, that path receives p-value $1/3$, and the other paths receive $2/3$ and $1$. Thus $$ P(p_{\mathrm{uniform}}\leq 1/3\mid N_1=m)=0.45>1/3. $$ This test is anti-conservative at level $1/3$. The contrast shows why unequal weights make uniform p-values not generally valid, but do not determine validity for every statistic.

The unconditional alternative uses the full design distribution over all paths rather than conditioning on $N_1$. It can avoid conditioning on a count, but still requires the adaptive path probabilities and sharp-null outcomes to be represented correctly. A conditional test answers a conditional calibration question: among repetitions with this final count, how unusual is the observed statistic? Conditioning may alter power and discards information carried by the count; it is not automatically more or less powerful than an unconditional test.

Diagnostics and limitations

Before interpreting a count-conditioned test, verify that the probability assigned to every compatible path is the product of the probabilities the actual algorithm would use along that path. In response-adaptive designs, this means recomputing the history-dependent probabilities after each candidate assignment and response. Reusing the probabilities from the observed path for all alternatives is generally wrong.

Check the support. Enumerate or simulate only paths the design can generate. A zero-probability path cannot be added to a conditional reference set.
Check the count event. Confirm that the observed count has positive design probability and that the conditional path weights sum to one, up to numerical error.
Check the response history. Under the sharp null, replay the algorithm with the fixed outcomes; do not assume allocation probabilities are constant if the rule uses prior responses.
Check the test target. This argument gives finite-sample conditional validity for the sharp null. A weak average-effect null is not a sharp null and does not, by itself, fix every unit’s outcomes under every path.
Check the tail and ties. Specify the extremeness ordering before computing the p-value. For a two-sided test, define the two-sided statistic or ordering explicitly and use it consistently.

Enumeration may become impractical when the number of units is large; Monte Carlo methods can approximate the design-weighted conditional distribution, but their simulation scheme must sample from that distribution rather than uniformly from compatible paths. Also, a count-conditioned result pertains to the count on which it is conditioned. It does not establish that this count is uninformative or that conditioning preserves power.

Common mistake

Wrong reasoning: “The observed trial had $m$ treated units, so every assignment with $m$ treated units is equally likely under the randomization test.”

Correction: Equal weighting follows from the conditional design only if the design gives equal probabilities to those paths. For an adaptive design, calculate each path probability from its sequential allocation probabilities, then normalize over paths with count $m$. If using a different weighting scheme, establish its validity separately for the statistic and test being used.

Key takeaway: Conditioning on final allocation counts preserves a valid sharp-null randomization distribution when the conditional reference distribution uses the actual design probabilities among compatible paths. The count alone does not make those paths equiprobable.

Check Your Understanding

Answer each question, then compare with the solutions below.

Three compatible paths have probabilities $0.2,0.3,0.5$. What are their conditional probabilities given that these are all paths with the observed count?
Why must response-dependent allocation probabilities be recomputed along each candidate path under a sharp null?
Suppose a count-conditioned design distribution assigns probabilities $0.6$ and $0.4$ to two paths. Is an equal-weight reference distribution automatically valid?
What does a conditional randomization p-value condition on, and what does it not mean?

Solutions. (1) The probabilities already sum to one, so the conditional weights are $0.2,0.3,0.5$. In general, divide each compatible path probability by the sum of all compatible path probabilities. (2) Candidate assignments can produce different histories, and the algorithm may choose different subsequent probabilities for those histories. Under the sharp null, fixed outcomes allow those histories and path probabilities to be calculated. (3) No. Equal weights differ from the design weights, so the ordinary conditional randomization justification does not apply. The uniform test might still be valid for a particular statistic, but that requires separate calibration. (4) It conditions on the observed final treatment count and evaluates extremeness under the sharp null using the design-weighted paths with that count. It is not the probability that the null is true.