Randomization Inference Under Interference

Why Randomization Inference Requires Care Under Interference

In an ordinary randomized trial without interference, a sharp null hypothesis says that every unit’s outcome would be the same under every treatment assignment. That makes randomization inference direct: the observed outcomes can be treated as fixed while the assignments are redrawn according to the trial’s design. With interference, a person’s outcome may depend both on that person’s treatment and on other people’s treatments. A null hypothesis about only one of those dimensions may therefore leave outcomes unknown under some reassigned exposure conditions.

The key is to match the reassignment scheme to the null being tested. If testing a direct effect while allowing spillovers, reassignments must preserve the peer-exposure conditions relevant to each unit. If testing a spillover effect while allowing direct effects, they must preserve each unit’s own treatment. A global sharp null, by contrast, permits reassignment across both dimensions because it specifies no effect of either.

Three null hypotheses illustrate the distinction. A sharp null of no direct effect states \(Y_i(1,g)=Y_i(0,g)\) for every unit \(i\) and relevant exposure \(g\), while allowing outcomes to vary across peer exposures. A sharp null of no spillover effect states \(Y_i(z,g)=Y_i(z,g')\) for every unit \(i\), own treatment \(z\), and relevant peer exposures \(g,g'\), while allowing outcomes to vary with own treatment. The global sharp null states \(Y_i(z,g)=Y_i(z',g')\) for all relevant combinations.

These are finite-population hypotheses about the potential outcomes of the units in the experiment. A randomization test evaluates how unusual the observed statistic would be if the specified null held, under the assignments the design could actually have produced. It does not require a sampling model for the outcomes.

Assignment Probabilities and Exact Tests

Let \(\Omega\) be the set of assignments allowed by the design, with assignment probability \(\Pr(A=a)\) for \(a\in\Omega\). For a statistic \(T(a)\), an upper-tail randomization p-value is the total design probability of assignments whose statistic is at least as large as the observed statistic. A lower-tail test reverses the inequality; a two-sided test needs a stated measure of extremeness, such as \(|T(a)|\) when the null value is zero.

For a conditional test, restrict \(\Omega\) to assignments satisfying the chosen conditioning event \(C\), and renormalize their original design probabilities. The statistic and its outcomes must be imputable under the null for every assignment in this restricted set. Using equal weights is correct only when the assignments in the set are equally likely under the original design.

The justification is a rank argument. Under the null, the statistic that would be observed for each allowed assignment is known. The realized assignment is distributed according to the design probabilities, so its statistic has the very distribution used to calculate the tail probability. The probability of landing in the most extreme fraction of that distribution is no greater than that fraction; ties can only make the inclusive-tail rejection set larger in steps, so a nonrandomized test may reject less often than its nominal level.

Two-Stage Randomized Saturation Designs

In a two-stage saturation design, clusters are first assigned to saturation levels, and then units within each cluster are assigned treatment according to the cluster’s level. For example, the first stage may assign some clusters to a low probability \(q_L\) and others to a high probability \(q_H\). The second stage might assign each unit independently with its cluster’s probability, or assign a fixed number of treated units. The test must reproduce whichever second-stage rule the experiment used.

For a design that selects a fixed number of clusters for each saturation level and then treats each unit independently with probability \(q_{S_c}\) in cluster \(c\), the probability of a complete assignment is the first-stage probability multiplied by the conditional second-stage probabilities. If the first-stage allocation is uniform over permitted allocations, its probability is the reciprocal of their number.

Here \(S\) is the vector of cluster saturation assignments, \(s_c\) is cluster \(c\)’s assigned level, \(n_c\) is its size, and \(z_i\) is unit \(i\)’s treatment. For fixed-count assignment within clusters, replace the Bernoulli factor by the probability of the permitted treatment subset—for example, \(1/{n_c\choose m_c}\) when exactly \(m_c\) units are selected uniformly.

An exposure mapping must also be specified. In the examples below, \(G_i\) is the saturation level assigned to unit \(i\)’s cluster. This is a design-defined measure of peer exposure, not necessarily the realized fraction of that unit’s peers treated. If the analysis instead defines peer exposure from realized peer assignments, it must calculate that exposure anew for every reassignment.

Testing a Direct-Effect Null by Conditioning on Peer Exposure

Suppose the null is no direct effect at a fixed peer exposure, but spillovers are not ruled out. Under an unrestricted reassignment, a unit might move from one peer-exposure condition to another. Its observed outcome would then not reveal its outcome under that new peer exposure. A conditional test avoids this problem by retaining only assignments with the same \(G_i\) for every unit. Within that restricted set, the direct-effect null lets us impute each unit’s outcome under either own-treatment value at its fixed exposure.

This example also shows why “conditioning” is more than a computational choice. It defines the assignments over which the direct-effect null supplies imputable outcomes. If fixing every unit’s peer exposure leaves no treatment variation, the conditional test cannot assess the direct effect for that design and exposure condition.

Testing a Spillover-Effect Null by Conditioning on Own Treatment

For a sharp null of no spillover effect, outcomes are known across peer-exposure changes only when own treatment is held fixed. A conditional spillover test therefore retains assignments with the observed \(Z_i\) for every unit and varies the assigned peer exposures according to the design. The design probabilities of those assignments may not be equal; they must be inherited from and normalized under the original two-stage assignment mechanism.

Using the Full Two-Stage Design Under a Global Sharp Null

When the null states that outcomes do not depend on own treatment or peer exposure, the observed outcome for each unit is known under every assignment. The test can then use the full two-stage assignment distribution without conditioning on \(G_i\) or \(Z_i\). The next example calculates an exact tail probability using the stage-one saturation lottery and stage-two Bernoulli assignments.

What Makes a Randomization Test Valid—and What Can Go Wrong

The essential validity conditions are concrete. The assignment mechanism must be known and represented correctly, including fixed totals, cluster restrictions, and unequal probabilities. The exposure mapping and potential-outcome null must be stated. Finally, every outcome needed to compute the statistic under every included assignment must be determined by the null and observed data. Under a component null, that last condition is often met only after conditioning on the other exposure dimension.

Common mistake: “The direct-effect null says treatment has no effect, so I can freely reassign treatment and keep all observed outcomes fixed.” That reasoning silently assumes outcomes are also invariant to peer exposure. Under interference, a unit whose peer exposure changes may have a different outcome even if its own-treatment effect is zero. The correction is to condition on the relevant peer exposures, or to test a broader sharp null that also rules out the exposure changes’ effects. For a spillover test that allows direct effects, the corresponding safeguard is to condition on own treatment.

Conditioning can reduce the number of assignments substantially. The resulting test can be exact but uninformative if few assignments remain, or impossible if only the observed assignment satisfies the condition. A small p-value is evidence against the particular sharp null and test statistic, not proof of a specific alternative mechanism. Likewise, failure to reject does not establish that direct or spillover effects are absent.