Randomization Tests with Cluster Assignment

Why the Assignment Mechanism Matters

When treatment is assigned to whole schools, clinics, villages, or firms, the randomized units are clusters, not the individuals within them. A valid randomization test must therefore compare the observed result with results from assignments the design could actually have produced. Permuting individual treatment labels instead would describe a different experiment and generally give the wrong reference distribution.

Randomization tests are especially useful when the number of clusters is small, because their validity under a sharp null can follow from the assignment design rather than a large-cluster approximation. But “exact” does not mean that the test has fine p-value resolution, that it tests every possible causal null, or that the original assignment was randomized. The design, null hypothesis, and statistic all matter.

Potential Outcomes and the Sharp Null

Suppose there are \(G\) clusters, indexed by \(g=1,\ldots,G\), and cluster \(g\) contains individuals \(i=1,\ldots,n_g\). Let \(Z_g\) indicate whether cluster \(g\) is assigned treatment. When assignment is at the cluster level, all individuals in that cluster share \(Z_g\). Write \(Y_{ig}(1)\) and \(Y_{ig}(0)\) for individual \(i\)'s potential outcomes if its cluster is assigned treatment or control. The observed outcome is \(Y_{ig}=Z_gY_{ig}(1)+(1-Z_g)Y_{ig}(0)\).

A randomization test needs a null hypothesis that lets us determine the outcomes that would have been observed under each permitted assignment. The sharp null of no treatment effect for any individual does this directly. A sharp null of a constant additive effect \(\tau_0\) does so as well: under that null, \(Y_{ig}(1)=Y_{ig}(0)+\tau_0\), so outcomes can be adjusted by the hypothesized effect before evaluating assignments.

This is different from a weak null, such as a zero average treatment effect, which allows individual effects to differ. Under a weak null, unobserved potential outcomes are not generally determined by observed outcomes, so the same direct enumeration is not automatically an exact test. The word “exact” below refers to validity under the stated sharp null and assignment mechanism.

Building the Randomization Distribution

Let \(\mathcal{A}\) be the set of assignments allowed by the design, and let \(\pi(z)\) be the design probability of assignment \(z\in\mathcal{A}\). For each assignment, calculate the statistic \(T(z)\) using the outcomes implied by the null. Under the no-effect sharp null, outcomes stay fixed while treatment labels are reassigned. The randomization p-value for a two-sided test based on absolute magnitude is

Here \(z_{\mathrm{obs}}\) is the observed assignment and \(\mathbf{1}\{\cdot\}\) equals one when the condition is true and zero otherwise. If all assignments in \(\mathcal{A}\) are equally likely, the p-value is the fraction at least as extreme as the observed statistic. If assignment probabilities differ, the assignments must be weighted by \(\pi(z)\); counting them equally is not generally valid. Including ties in the tail is the standard non-randomized convention.

The set \(\mathcal{A}\) is determined by the design. Complete randomization with a fixed number of treated clusters permits only assignments with that number treated. Bernoulli assignment permits assignments according to the specified cluster-level treatment probabilities. In a blocked or paired design, assignments must respect the restrictions within each block or pair. The test does not invent a new assignment scheme after seeing the outcomes.

Choosing and Recomputing a Statistic

The statistic should reflect the effect being studied and be specified without choosing it to make the observed result look extreme. One simple choice is the difference in mean cluster outcomes, giving each cluster equal weight. If the scientific target gives equal weight to individuals, an individual-weighted difference may instead be appropriate; with unequal cluster sizes, these statistics answer different descriptive questions. A regression coefficient or a studentized statistic can also be used, provided it is defined for every assignment being considered.

For every permitted reassignment, recompute the complete statistic, including any regression fit, residual-based standard error, or other assignment-dependent component. Holding the observed standard error fixed while permuting treatment labels does not produce the randomization distribution of a studentized statistic. Cluster-robust standard errors may be part of a chosen statistic, but the design-based reference distribution still comes from the cluster assignment scheme.

Blocked and Paired Cluster Assignment

The reference distribution must retain the restrictions of blocked assignment. For example, if clusters are paired before randomization and one cluster in each pair is assigned treatment, the possible assignments are the within-pair swaps. Permuting treatment across pairs would not represent that design. Blocking often aims to improve comparability; in the test, its direct role is to define which treatment reallocations are permitted and how likely they are.

A Derivation of Sharp-Null Validity

The reason the test is valid under the sharp null is the random assignment itself. If the null holds, the observed outcomes do not depend on which permitted assignment occurred. Thus, before the assignment is drawn, the statistic is a fixed function of the assignment, and its randomization distribution is precisely the distribution generated by the design. A tail probability calculated from that distribution is a valid p-value: under the null, the probability that it falls at or below a threshold \(\alpha\) is at most \(\alpha\), with discreteness or ties making the test potentially conservative.

For a finite equally likely assignment set of size \(M\), order the \(M\) statistic values from most to least extreme under the selected tail ordering. At most \(\lfloor\alpha M\rfloor\) assignments can have tail rank at most \(\alpha M\). Since each assignment has probability \(1/M\), the probability of a p-value no greater than \(\alpha\) is at most \(\lfloor\alpha M\rfloor/M\leq\alpha\). With unequal assignment probabilities, the same principle uses the design probabilities: the tail probability is the total probability mass of assignments at least as extreme as the observed one. Therefore, validity depends on using the actual assignment probabilities, not on a normal approximation to a statistic.

What Few Clusters Limit

The number of randomized clusters, rather than the number of people measured within them, determines how many distinct cluster assignments are available. With \(G\) clusters and exactly \(m\) treated, there are at most \(\binom{G}{m}\) assignments before accounting for any additional restrictions. A large number of individuals within a few clusters can make cluster outcomes more precisely measured, but it does not create more treatment allocations.

Consequently, exact p-values can be discrete and may never reach a chosen threshold. For example, if a design has only 20 equally likely assignments, its p-values change in increments of \(1/20=0.05\); a two-sided tail may have still fewer attainable values because assignments come in tied or symmetric pairs. A randomization test can be valid but have low power or limited ability to reject at a stringent level. Enumerating assignments does not remove that limitation.

When all assignments can be enumerated, use the full design distribution. If only a random sample of \(B\) assignments is drawn to approximate it, a common Monte Carlo estimate is \((1+K)/(B+1)\), where \(K\) is the number of sampled assignments at least as extreme as the observed statistic. The added one accounts for the observed assignment and avoids a zero estimate; this approximation is not the same as enumerating a small complete assignment set.

Common Pitfalls

• Permuting individuals. Reasoning that every person could have received either label ignores cluster-level assignment. Reassign clusters according to the actual design.
• Using unrestricted permutations in a blocked design. Assignments that violate the pairing or block restrictions were not possible. Restrict the reference set accordingly.
• Treating a sharp-null test as a test of any average effect null. A sharp null specifies every unit's effect and permits missing outcomes to be imputed under each assignment. A zero average effect alone generally does not.
• Counting unequal-probability assignments equally. This substitutes a uniform distribution for the design distribution. Weight each assignment by its actual probability.
• Interpreting “exact” as “highly informative.” Exactness concerns error control under the null and design; few clusters can still yield coarse p-values and limited power.
• Changing the statistic across assignments. If the statistic includes a fitted regression or standard error, recompute it for every allocation using the same rule.

A cluster randomization test is a design-based comparison, not a substitute for a randomized design when treatment was assigned by other means. Its strongest guarantee is precise but limited: under a stated sharp null, with a correctly represented assignment mechanism, the observed statistic can be judged against the outcomes the design could have generated.