Fisher Randomization Tests for the Sharp Null

Why Use a Randomization Test?

In a randomized experiment, the treatment assignment is generated by a known design. A Fisher randomization test uses that design to ask: if the sharp null were true, how often would random assignment produce a test statistic at least as extreme as the one observed? Its reference distribution comes from the assignments the experiment could have produced, not from a normal approximation.

The key is that the sharp null specifies every unit’s treatment effect. Under that null, each unit would have the same outcome whether assigned to treatment or control. This makes all outcomes needed to evaluate alternative assignments available from the observed data. The resulting test is exact for the specified randomization design, subject to the chosen statistic and tail rule.

The Sharp Null and the Assignment Set

Definition: For unit $i$, let $Y_i(1)$ be the outcome under treatment and $Y_i(0)$ the outcome under control. The sharp null of no effect for every unit is $H_0: Y_i(1)=Y_i(0)$ for every $i=1,\ldots,N$. Equivalently, each unit’s treatment effect $Y_i(1)-Y_i(0)$ is zero. If $Z_i$ indicates assignment to treatment, the observed outcome is $Y_i=Z_iY_i(1)+(1-Z_i)Y_i(0)$.

Under this null, the observed outcome for each unit equals both of its potential outcomes. Write that common value as $Y_i$. If a different assignment were made, the same $Y_i$ would therefore be observed for that unit. We can reassign the fixed outcomes according to every assignment allowed by the design and calculate the statistic each assignment would produce.

For complete randomization of $N$ units with exactly $n_T$ assigned to treatment, the assignment set is

\Omega=\left\{\mathbf{z}\in\{0,1\}^N:\sum_{i=1}^N z_i=n_T\right\}, \qquad |\Omega|=\binom{N}{n_T}.

Each assignment in $\Omega$ has probability $1/|\Omega|$. A common statistic is the difference in observed group means,

T(\mathbf{Z},\mathbf{Y})= \frac{1}{n_T}\sum_{i:Z_i=1}Y_i - \frac{1}{N-n_T}\sum_{i:Z_i=0}Y_i.

A one-sided test might count assignments with $T$ at least as large as the observed value. A two-sided test might count assignments with $|T|$ at least as large as the observed absolute value. The statistic and the definition of “at least as extreme” should be specified before inspecting the randomization results. There is no single mandatory two-sided rule for every statistic.

The Exact Randomization Test

Definition: Given a statistic $T$ and an extremeness rule, the exact randomization p-value is the proportion of allowed assignments that produce a statistic at least as extreme as the observed statistic, assuming the sharp null. For a two-sided test based on absolute values, $$ p_{\mathrm{exact}}= \frac{1}{|\Omega|} \sum_{\mathbf{z}\in\Omega} \mathbf{1}\left\{ |T(\mathbf{z},\mathbf{Y})| \geq |T(\mathbf{Z}_{\mathrm{obs}},\mathbf{Y})| \right\}. $$

Here $\mathbf{1}\{\cdot\}$ equals 1 when the condition in braces holds and 0 otherwise. The observed assignment is included in $\Omega$, so it contributes at least one assignment to the tail count. If the test is conducted at level $\alpha$, reject $H_0$ when the p-value is no greater than $\alpha$. Because the assignment set is finite, attainable p-values are discrete and a test at a particular level may be conservative.

Worked Example: Enumerating All Assignments

Suppose four units are assigned to treatment by complete randomization, with exactly two treated. Their hypothetical observed outcomes are 1, 2, 4, and 8 units, respectively. The observed treated units are 1 and 3, so the observed difference in means is

T_{\mathrm{obs}}=\frac{1+4}{2}-\frac{2+8}{2} =2.5-5=-2.5\text{ outcome units}.

Under the sharp null, these four outcomes are fixed across all assignments. There are $\binom{4}{2}=6$ possible treated pairs. The total outcome is 15, so for a treated pair with outcome sum $s_T$, the difference in means is $s_T/2-(15-s_T)/2=s_T-7.5$. Enumerating the six pairs gives:

Treated pair	Treated outcome sum	Difference in means	Absolute difference
1 and 2	3	-4.5	4.5
1 and 3	5	-2.5	2.5
1 and 4	9	1.5	1.5
2 and 3	6	-1.5	1.5
2 and 4	10	2.5	2.5
3 and 4	12	4.5	4.5

For a two-sided test using absolute differences, four of the six assignments have an absolute difference at least $2.5$. Thus $p_{\mathrm{exact}}=4/6=0.667$. At a 0.05 significance level, this test does not reject the sharp null. The result says that, under the null and this assignment design, 4 of 6 possible assignments produce a difference at least as large in absolute value as the observed one. It does not say that the null has probability 0.667 of being true.

Why the Test Is Exact

Under the sharp null, the vector of outcomes $\mathbf{Y}$ is fixed, and the only randomness is the assignment. Under complete randomization, the actual assignment is uniformly distributed over $\Omega$. Therefore, the observed statistic has precisely the distribution obtained by calculating the statistic for every assignment in $\Omega$.

This also explains why the p-value controls the rejection probability. For any fixed outcomes under the null, rank the statistics over the assignments from most to least extreme according to the prespecified rule. The fraction of assignments at least as extreme as the observed assignment is its tail rank divided by $|\Omega|$. With no ties, a small tail fraction can occur only for assignments in a correspondingly small part of the assignment set. Ties include all assignments at the boundary, so they can make the test conservative rather than anti-conservative. No normality assumption is required for this design-based calibration.

Worked Example: Approximating the Distribution by Simulation

Consider five units with hypothetical outcomes 1, 3, 4, 7, and 10 outcome units. Exactly two are treated, and the observed treated units are 1 and 5. The observed difference is

T_{\mathrm{obs}}=\frac{1+10}{2}-\frac{3+4+7}{3} =5.5-\frac{14}{3} =\frac{5}{6}\approx0.8333\text{ outcome units}.

There are $\binom{5}{2}=10$ allowed assignments. If enumerating all ten is inconvenient, draw $B$ assignments independently and uniformly from those ten, with replacement, and calculate the statistic for each using the fixed outcomes. Suppose five illustrative draws are treated pairs 1 and 2; 2 and 4; 3 and 4; 2 and 5; and 1 and 4. Their differences in means are:

Pair 1 and 2: $2-(21/3)=-5$.
Pair 2 and 4: $5-(15/3)=0$.
Pair 3 and 4: $5.5-(14/3)=5/6\approx0.8333$.
Pair 2 and 5: $6.5-(12/3)=2.5$.
Pair 1 and 4: $4-(17/3)=-5/3\approx-1.667$.

Four of the five simulated statistics have absolute value at least $5/6$. The usual add-one Monte Carlo estimate is

\hat p_{\mathrm{MC}}= \frac{1+\#\{b:|T_b|\geq|T_{\mathrm{obs}}|\}}{B+1} =\frac{1+4}{5+1} =\frac{5}{6}\approx0.8333.

This is a Monte Carlo result based on five draws, not the exact p-value from all ten assignments. Its coarse resolution means that more draws would usually be needed for a stable estimate, especially if a decision is near the chosen significance level. The add-one calculation avoids reporting a zero p-value merely because no simulated draw entered the tail. Under the sharp null, the observed assignment and the independent simulated assignments are exchangeable, which gives this rank-based Monte Carlo test valid error control; ties remain conservative.

Choosing a Statistic

The randomization argument applies to any statistic that can be calculated for each assignment under the sharp null. A difference in means is often easy to interpret, but a studentized statistic can put the observed contrast on a scale that accounts for the groups’ variability. For exact calibration, the statistic must be recomputed for each assignment, including any assignment-dependent standard error. The reference distribution is the resulting randomization distribution, not a theoretical $t$ distribution.

Worked Example: A Studentized Statistic

Four units have hypothetical outcomes 1, 2, 5, and 10 outcome units. Exactly two are treated, and units 1 and 4 are observed in treatment. Use the one-sided statistic $t=D/SE$, where $D$ is the difference in means and

SE=\sqrt{\frac{s_T^2}{2}+\frac{s_C^2}{2}}.

For the observed pair, the treatment mean is $(1+10)/2=5.5$, the control mean is $(2+5)/2=3.5$, and $D=2$. The sample variances are $s_T^2=40.5$ and $s_C^2=4.5$. Therefore,

SE=\sqrt{\frac{40.5}{2}+\frac{4.5}{2}} =\sqrt{22.5}\approx4.743, \qquad t_{\mathrm{obs}}=\frac{2}{4.743}\approx0.4216.

The six possible treated pairs give the following studentized statistics. Each standard error uses the sample variances within the groups formed by that assignment.

Treated pair	Difference $D$	Standard error	Statistic $t$
1 and 2	-6	$\sqrt{6.5}\approx2.550$	-2.353
1 and 3	-3	$\sqrt{20}\approx4.472$	-0.6708
1 and 4	2	$\sqrt{22.5}\approx4.743$	0.4216
2 and 3	-2	$\sqrt{22.5}\approx4.743$	-0.4216
2 and 4	3	$\sqrt{20}\approx4.472$	0.6708
3 and 4	6	$\sqrt{6.5}\approx2.550$	2.353

For the upper-tailed test, three of the six statistics are at least $0.4216$, so $p_{\mathrm{exact}}=3/6=0.500$. The studentization changes the ordering and scale of the statistics; it does not change the logic of exact calibration under the sharp null.

What the Test Does—and Does Not—Establish

A small p-value is evidence against the sharp null of no effect on every unit, relative to the specified assignment mechanism and statistic. Rejecting that null implies that the data are inconsistent with the claim that all unit effects are zero at the chosen significance level. It does not by itself identify which units were affected, establish a particular effect size, or show that treatment caused an outcome difference in a nonrandomized setting.

The sharp null differs from a weak null such as “the average treatment effect is zero.” A weak null allows some units to benefit and others to be harmed, as long as effects average to zero. Under that null, the unobserved potential outcomes generally cannot be imputed for every unit. The simple exact procedure described here therefore tests the sharp null, not every possible null about an average effect.

A Fisher p-value is conditional on the outcomes and assignment design: it describes the probability, under the sharp null, of obtaining a statistic at least as extreme through the random assignment. It is not the probability that the null is true, nor a measure of the probability that treatment caused the observed result. Its validity also depends on using the actual assignment mechanism; enumerating assignments the experiment could not have produced gives the wrong reference distribution.

Common Pitfalls

Testing a different null. Treating a test of zero effect for every unit as a direct test of zero average effect is incorrect. The sharp null makes every missing potential outcome imputable; the weak average-effect null generally does not.
Using the wrong assignment set. If the design fixed the number treated, assignments with other treatment counts are not part of the reference distribution.
Changing the extremeness rule after seeing results. Choosing a one-sided tail or a two-sided statistic only after seeing the observed direction can distort the intended error rate. State the statistic and tail rule in advance.
Holding an estimated standard error fixed across assignments. For a studentized statistic, calculate the standard error from the groups formed by each assignment. Otherwise the simulated or enumerated statistics do not match the stated statistic.
Interpreting a p-value as a probability about the null. A p-value is a tail probability under the null and design, not the probability the null is true.

Key takeaway: Under the sharp null, observed outcomes can be held fixed while treatment labels are reassigned according to the experiment’s actual design. Comparing the observed statistic with this exact or simulated reference distribution yields a test calibrated to random assignment.

Check Your Understanding

Answer each question using the sharp-null and assignment-distribution logic.

In a completely randomized experiment with 12 units and 5 treated, how many assignments belong to the exact reference set?
Why can the sharp null make every unit’s outcome under an unobserved assignment known?
For a two-sided difference-in-means test, what does it mean for a reassigned statistic to be at least as extreme as the observed statistic?
Why is a test of zero effect for every unit not automatically a test of zero average treatment effect?
If a studentized statistic is used, which quantities must be recalculated for each assignment?

Treated pair	Difference \(D\)	Standard error	Statistic \(t\)
1 and 2	-6	\(\sqrt{6.5}\approx2.550\)	-2.353
1 and 3	-3	\(\sqrt{20}\approx4.472\)	-0.6708
1 and 4	2	\(\sqrt{22.5}\approx4.743\)	0.4216
2 and 3	-2	\(\sqrt{22.5}\approx4.743\)	-0.4216
2 and 4	3	\(\sqrt{20}\approx4.472\)	0.6708
3 and 4	6	\(\sqrt{6.5}\approx2.550\)	2.353