Randomization Tests for Treatment Assignments

Why the Assignment Mechanism Matters

In a randomized experiment, treatment labels are assigned by a specified procedure: perhaps a fixed number of participants is selected for treatment, or treatment is assigned separately within blocks. A randomization test asks whether the observed treatment-effect statistic would be unusual if the experiment were repeated under that same procedure and a stated null hypothesis were true. The key is to replay the actual assignment mechanism, not to shuffle labels in a way the experiment could never have produced.

This approach uses the experiment’s random assignment as the source of randomness. It does not require the observed outcomes to follow a normal distribution. But an exact randomization test does require a clear null hypothesis that tells us what outcomes would have been under every allowable assignment.

Potential Outcomes and the Sharp Null

Definition: For unit $i$, let $Y_i(1)$ be the outcome that would be observed if the unit received treatment, and let $Y_i(0)$ be the outcome that would be observed if the unit received control. The observed outcome is $Y_i(1)$ for a treated unit and $Y_i(0)$ for a control unit. The sharp null hypothesis of no treatment effect states that $Y_i(1)=Y_i(0)$ for every unit $i$. It is called sharp because it specifies every unit’s outcome under either assignment.

For example, if the outcome is a symptom score measured after treatment, $Y_i(1)=Y_i(0)$ says that assigning treatment rather than control would not change that score for any participant. A randomization test of this sharp null holds the observed outcomes fixed and changes only which units are assigned to treatment.

For the test to reflect the experiment, the assignment rule and its probabilities must be known and reproduced. The treatment and outcome must also be well-defined, and one unit’s assignment must not change another unit’s outcome; otherwise the potential outcomes indexed only by that unit’s treatment may not describe the experiment. The sharp null is stronger than saying that the average treatment effect is zero: individual effects could differ, with some positive and some negative, while averaging to zero. The simple fixed-outcome reassignment test below is exact for the sharp null, not automatically for that weaker average-effect null.

Test statistic: A test statistic is a numerical summary chosen to measure the pattern relevant to the question. For a two-group experiment with a quantitative outcome, one common choice is the difference in sample means, treatment minus control: $$ T = \bar{x}_{\text{treatment}} - \bar{x}_{\text{control}}. $$ A positive value favors higher outcomes in the treatment group; a negative value favors lower outcomes. The direction and extremeness rule for the test should match the research question.

How to Construct the Test

Suppose $N$ units are assigned according to a known random mechanism. Under the sharp null, each unit’s observed outcome is also its outcome under every alternative allowable assignment. For each assignment the mechanism permits, calculate the chosen statistic from those fixed outcomes. The resulting collection of statistics is the randomization distribution under the null.

State the null and the statistic.
Specify the sharp null and choose a statistic that captures the effect and direction of interest.

Record the actual assignment rule.
Identify which assignments were possible and their probabilities, including any fixed group sizes or blocks.

Generate allowable assignments.
Enumerate them all for an exact test, or draw assignments according to the mechanism for a Monte Carlo test.

Recalculate the statistic.
For every generated assignment, use the fixed outcomes and the new treatment labels to calculate the statistic.

Compare with the observed result.
Count the assignments at least as extreme as the observed statistic under the chosen one- or two-sided rule, then interpret the resulting p-value in context.

If all allowable assignments are equally likely, an exact p-value is the fraction of them whose statistics are at least as extreme as the observed statistic. If assignments have different probabilities, the tail probability must instead weight each assignment by its actual probability. A p-value is a probability calculated under the null and the assignment mechanism; it is not the probability that the null hypothesis is true.

Worked Examples

Worked Example: Complete Randomization With Four Units

In a hypothetical experiment, four participants are randomly assigned, with exactly two receiving a treatment. The observed outcomes are 4, 6, 7, and 9 symptom-score points. The participants with outcomes 4 and 7 received treatment; the other two received control. Use the difference in means, treatment minus control, and test the sharp null against a two-sided alternative.

The observed statistic is $$ T_{\text{obs}} = \frac{4+7}{2} - \frac{6+9}{2} = 5.5 - 7.5 = -2 \text{ points}. $$ Under the sharp null, the four observed scores stay fixed. Complete randomization permits each of the $\binom{4}{2}=6$ pairs of participants to be the treated pair, each with equal probability. For any pair with treated-outcome sum $S_T$, the total score is 26, so $$ T = \frac{S_T}{2} - \frac{26-S_T}{2} = S_T - 13. $$

The six possible treated pairs and their statistics are: outcomes 4 and 6, $T=-3$; 4 and 7, $T=-2$; 4 and 9, $T=0$; 6 and 7, $T=0$; 6 and 9, $T=2$; and 7 and 9, $T=3$. The observed value is $-2$. For a two-sided test, statistics at least as extreme have $|T| \geq 2$. Four of the six assignments meet this condition, so $$ p = \frac{4}{6} \approx 0.667. $$ This test provides little evidence against the sharp null: a statistic at least this far from zero occurs in two-thirds of the allowable assignments under that null. This conclusion concerns evidence in this experiment, not the probability that the null is true.

Worked Example: Respecting Blocks in the Assignment

Suppose a hypothetical study pairs participants into two blocks based on a baseline characteristic. Within each block, a coin flip assigns one participant to treatment and the other to control. In block 1, the observed outcomes are 10 points for control and 13 for treatment; in block 2, they are 8 for control and 9 for treatment. Use the average of the two within-block differences and test the sharp null two-sided.

The observed within-block differences are $13-10=3$ and $9-8=1$ points. Thus the statistic is $$ T_{\text{obs}} = \frac{3+1}{2} = 2 \text{ points}. $$ There are four allowable assignments, because either participant can receive treatment in each of the two blocks. For the first block, the difference is either $+3$ or $-3$, depending on which participant receives treatment. For the second, it is either $+1$ or $-1$. The four possible average differences are therefore $$ \frac{3+1}{2}=2,\quad \frac{3-1}{2}=1,\quad \frac{-3+1}{2}=-1,\quad \frac{-3-1}{2}=-2. $$

All four assignments are equally likely under the stated mechanism. Two of the four have $|T| \geq |T_{\text{obs}}|=2$, so $p=2/4=0.5$. The observed difference is not especially unusual under this two-sided test. Reassigning treatment across all participants without preserving the blocks would not replay this experiment’s assignment procedure.

Worked Example: Monte Carlo Randomization Test

A hypothetical experiment assigns exactly 5 of 10 participants to treatment, with all such assignments equally likely. The chosen statistic is the difference in mean improvement, treatment minus control, and the observed difference is 7.2 improvement points. The research question is whether treatment increases improvement, so the alternative is one-sided: unusually large positive statistics count as evidence against the sharp null.

There are $\binom{10}{5}=252$ allowable assignments, so exact enumeration is possible, but a larger experiment might make it impractical. Suppose 4,999 assignments are instead sampled uniformly from the 5-of-10 assignments. For each draw, the 5 assigned participants’ outcomes are used to calculate a treatment-minus-control difference. Assume 11 simulated statistics are at least 7.2 points.

Using the usual plus-one estimate for a Monte Carlo randomization p-value, $$ \hat{p}_{\text{MC}} = \frac{b+1}{B+1} = \frac{11+1}{4999+1} = \frac{12}{5000} = 0.0024, $$ where $b=11$ is the number of simulated statistics at least as large as observed and $B=4999$ is the number of random assignments drawn. This small estimated tail probability indicates strong evidence against the sharp null in the direction of increased improvement, subject to Monte Carlo simulation error. It does not establish that every participant benefits or quantify the probability that the null is true.

Why the Test Is Valid Under the Sharp Null

The justification follows directly from the random assignment and the sharpness of the null. Once outcomes are observed, the sharp null says those same outcomes would have been observed under every allowable assignment. Therefore the only changing ingredient in a hypothetical repetition is the assignment itself. If the assignment mechanism gives each allowable assignment its specified probability, the observed assignment is one draw from that same distribution.

For equally likely assignments, let $\mathcal{A}$ be the set of allowable assignments and let $T(a)$ be the statistic for assignment $a$. The exact upper-tail p-value for a one-sided test is $$ p = \frac{\#\{a \in \mathcal{A}: T(a) \geq T_{\text{obs}}\}}{|\mathcal{A}|}. $$ For a two-sided test based on distance from zero, replace the comparison with $|T(a)| \geq |T_{\text{obs}}|$. Because the observed assignment is itself among the assignments counted when tied at the observed statistic, this tail fraction is calibrated by the assignment distribution under the sharp null. With ties, the test can be conservative rather than having exactly its nominal rejection probability.

This argument does not require random sampling from a larger population. It supports inference about the effect of the randomized treatment assignment for the units in the experiment under the stated null and assignment procedure. Generalizing to a wider population requires additional justification.

Common Mistakes and Practical Interpretation

Shuffling labels without reproducing the design. If the experiment fixed the number treated, preserve that number; if it randomized within blocks, reassign within those blocks. Otherwise the reference distribution does not represent the experiment.
Confusing a sharp null with a zero average effect. The sharp null says every unit has zero effect. A zero average could conceal different individual effects, so the fixed-outcome reassignment argument does not automatically test that weaker claim.
Choosing the tail after seeing the result. Decide whether evidence is one-sided or two-sided from the scientific question. A two-sided rule based on $|T|$ counts unusually large effects in either direction.
Overstating a small p-value. A small p-value means the observed statistic is unusual under the sharp null and the specified randomization. It does not give the probability that the null is true, prove causation beyond the experiment’s design, or show that all individuals respond alike.

Randomization tests are especially useful when the experimental assignment is known and a standard distributional model for outcomes is questionable. Their strength comes from respecting the design and stating exactly what null is being tested. A carefully chosen statistic and a faithful replay of the assignment mechanism make the resulting evidence interpretable.

Check Your Understanding

Answer these questions by referring to the assignment rule and the null hypothesis, not just to the observed outcomes.

What does the sharp null hypothesis specify about $Y_i(1)$ and $Y_i(0)$ for each unit?
An experiment assigns exactly 3 of 8 participants to treatment. Why would shuffling labels to create assignments with 2 or 4 treated participants fail to reproduce the design?
In a blocked experiment, how should treatment labels be reassigned when constructing the randomization distribution?
Explain why a randomization test of the sharp null does not automatically test the hypothesis that the average treatment effect is zero.
A two-sided exact test gives $p=0.04$. State what this says about the observed statistic under the sharp null and assignment mechanism, and one thing it does not say.