Why Few Treated Clusters Change the Inference Problem
In a difference-in-differences (DiD) study, treatment is often assigned at the level of a state, district, hospital, or other cluster. Outcomes may be recorded for thousands of people, but if only a handful of clusters receive treatment, the amount of independent information about treatment assignment remains limited. People within a cluster may share shocks, policies, and trends; treating their outcomes as independent can therefore make uncertainty look much smaller than it is.
Cluster-robust standard errors allow outcomes and regression errors to be dependent within a cluster while relying on independence across clusters. Their usual large-sample justification concerns the number of independent clusters, not simply the number of people or person-period observations. With few clusters, the estimated variance and the reference distribution for a test can both be poor approximations. The problem is particularly acute when only a few clusters are treated, even if there are many control clusters.
The cluster should generally reflect the level at which treatment is assigned and the level at which important dependence occurs. For example, if a policy is assigned to districts and outcomes within each district are serially correlated over time, clustering at the district level is usually central to inference. Earlier material on inference with serially correlated outcomes explains why ignoring this dependence can understate uncertainty. Few-cluster methods address a limitation of inference; they do not establish that a DiD comparison identifies a causal effect.
What Conventional Cluster-Robust Inference Relies On
For a linear regression \(y=X\beta+u\), let \(S_g=X_g'u_g\) be the regression-score contribution of cluster \(g\). Ignoring finite-sample correction factors, a cluster-robust variance estimator has the form
Here, \(G\) is the number of independent clusters. The estimator aggregates cluster-level score contributions, rather than treating every person-period as an independent contribution. Its standard large-sample rationale requires enough independent clusters for the aggregate score to be adequately approximated by a normal distribution and for its variance to be estimated reliably. When \(G\) is small, neither approximation is guaranteed. Unequal cluster sizes, high-leverage clusters, and a very small number of treated clusters can make the problem worse.
A \(t\)-reference distribution with a degrees-of-freedom adjustment can improve some finite-sample behavior, but it does not turn a small number of clusters into a large one. Nor does a large number of people per cluster necessarily solve the problem: if treatment varies only across a few clusters, the treatment contrast still rests on few independent assignment units.
Worked Example: A Large Individual Sample, but Few Assignment Units
Suppose a policy is assigned to 8 districts: 2 treated and 6 untreated. Each district contributes 500 people in each of two periods, for 8,000 person-period records. A DiD regression estimates a policy effect of 3.5 outcome points, and the district-clustered standard error is 1.088 points. What does the sample size imply about the inference?
The test statistic formed from the reported coefficient and standard error is
The arithmetic is \(3.5/1.088=3.2169\ldots\), which rounds to 3.217. This statistic does not by itself supply a reliable p-value: a reference distribution and a defensible approximation for it are also needed. There are 8 independent districts, not 8,000 independent treatment assignments, and only 2 treated districts. Using a standard normal reference distribution as if the large number of person-period records guaranteed its validity would ignore the small number of clusters.
The estimate and standard error are stated in outcome points. The estimate remains a numerical contrast, but its uncertainty should be assessed with a method appropriate to the cluster count and design. Neither a bootstrap nor a randomization test can repair a DiD design whose identifying assumptions are not credible.
Wild Cluster Bootstrap-t
The wild cluster bootstrap is a resampling approach designed to preserve within-cluster dependence while approximating the null distribution of a test statistic. In a commonly used null-restricted wild cluster bootstrap-\(t\) procedure, the researcher tests a stated null hypothesis, such as \(H_0:\beta_{\mathrm{treat}}=0\), and proceeds as follows:
Impose the null restriction when estimating the model, and obtain residuals under that restriction.
For bootstrap draw \(b\), multiply the residuals for every observation in cluster \(g\) by the same random weight \(w_g^{(b)}\). This retains the cluster as the unit whose residual contribution is perturbed.
Use the fitted null values and the multiplied residuals to construct bootstrap outcomes, then re-estimate the model and calculate its bootstrap-\(t\) statistic.
Use the share of bootstrap statistics at least as extreme as the observed statistic to approximate a p-value under the null. Confidence intervals can be obtained by inverting appropriate bootstrap tests.
The multipliers are commonly drawn from distributions with mean zero and variance one. Rademacher weights take values \(+1\) and \(-1\), each with probability one-half. With very few clusters, some implementations use a richer finite-support distribution, such as Webb six-point weights, because the set of possible Rademacher sign patterns is very limited. The choice of weights and implementation matters; no single variant is uniformly reliable in every design.
The procedure is not an exact replay of the treatment assignment mechanism. It approximates a null distribution by perturbing cluster-level residual contributions and remains dependent on suitable regularity and approximation conditions. In particular, it is not guaranteed to work well when there are extremely few treated clusters, substantial imbalance, or one or two clusters with high leverage. It should be reported as an inference method with limitations, not as an automatic correction for a small cluster count.
Worked Example: Reading a Wild Bootstrap p-Value
Consider the hypothetical estimate and clustered standard error from the previous example: 3.5 outcome points and 1.088 points. The observed statistic for testing \(H_0:\beta_{\mathrm{treat}}=0\) is \(t_{\mathrm{obs}}=3.217\). Suppose a null-restricted wild cluster bootstrap generates \(B=999\) bootstrap-\(t\) statistics, of which 37 have absolute value at least 3.217.
Using the common finite-Monte-Carlo adjustment, the estimated two-sided bootstrap p-value is
Under this specified bootstrap procedure, 3.8% of the adjusted bootstrap reference distribution is at least as extreme as the observed statistic. This is evidence against the tested null at the 5% level according to that procedure, but it is not the probability that the null is true. The result also does not show that the method's approximation is reliable with only 2 treated districts. The estimate, standard error, cluster count, bootstrap variant, and assignment context should all accompany the reported p-value.
Randomization-Based Inference
Randomization-based inference uses the treatment-assignment mechanism itself. If a known design assigned treatment to a specified number of clusters at random, a test can compare the observed statistic with statistics calculated under the other assignments permitted by that design. For a sharp null hypothesis of no treatment effect for any cluster, outcomes are fixed under the null, so the statistic under each permitted assignment can be calculated.
Exactness depends on the design. The assignment probabilities, any strata or restrictions, and the unit of assignment must be represented correctly when generating the reference distribution. In an observational DiD study, permuting treatment labels is not automatically justified: the researcher needs a credible basis for treating those assignments as exchangeable under the null, possibly conditional on known design features. A randomization test of the sharp null also does not, without additional assumptions, provide an exact test of a weaker null that only states that an average treatment effect is zero.
Worked Example: Exact Test With Two Treated Clusters
Suppose 2 of 6 districts were assigned treatment uniformly at random, with every pair of treated districts equally likely. Consider the following hypothetical district-level changes from before to after treatment, in outcome points: 7, 5, 4, 3, 2, and 1. The observed treated districts have changes 7 and 5. Use the difference between the treated mean change and the control mean change as the statistic, and test the sharp null of no effect in any district.
The observed statistic is
There are \(\binom{6}{2}=15\) assignments permitted by the design. The total of all six changes is 22. If a proposed treated pair has sum \(s\), its treated mean is \(s/2\) and the remaining four districts' mean is \((22-s)/4\). Thus its statistic is
A two-sided test counts assignments with \(|T(s)|\geq 3.5\). The condition is \(|3s-22|\geq 14\), which requires \(s\leq 8/3\) or \(s\geq 12\). The smallest possible pair sum is \(1+2=3\), so no pair meets the first condition. The only pair with sum at least 12 is the observed pair, 7 and 5. Therefore one of the 15 assignments is at least as extreme as the observed assignment, and the exact p-value is
For this assignment mechanism and sharp null, the observed statistic is not significant at the 5% level. This exact calculation does not rely on a large-cluster normal approximation. Its validity does rely on the stipulated random assignment of exactly 2 of the 6 districts and on the sharp null used to calculate outcomes under each assignment.
What Each Approach Can and Cannot Establish
| Feature | Wild cluster bootstrap-t | Randomization-based inference |
|---|---|---|
| Reference distribution | Generated by cluster-level residual multipliers under a fitted null model | Generated by assignments allowed under the actual assignment mechanism |
| Key justification | Bootstrap approximation is adequate for the cluster structure and sample size | Assignment mechanism is known or credibly justified; the tested sharp null specifies outcomes under assignments |
| Strength | Can be used in regression settings where treatment was not literally randomized, subject to approximation conditions | Can give an exact test for a sharp null under a correctly represented random assignment design |
| Important limitation | Not guaranteed to be reliable with extremely few or highly influential treated clusters | Not justified by arbitrary label permutations, and a sharp-null test is not automatically inference for an average effect |
The methods answer related but distinct questions. A bootstrap approximates a sampling distribution under modeling and regularity conditions. A randomization test uses a distribution over treatment assignments under a design and a specified null. Neither should be selected solely because it produces a preferred p-value. In either case, the DiD estimand and its identification assumptions remain separate from the uncertainty calculation.
Assignment Counts and Practical Limits
The assignment mechanism also determines how finely a randomization p-value can be resolved. With a small number of permitted assignments, an exact test has a discrete reference distribution; it cannot produce arbitrary p-values. This is a feature of the design, not a computational defect. Moreover, many possible assignments do not necessarily mean many treated clusters: the number of treated units remains important for the reliability of cluster-asymptotic approximations.
Worked Example: Resolution With Two Treated Clusters
Suppose 2 of 10 districts are treated, and assignment is uniform over all pairs. There are
permitted assignments. For a non-randomized statistic with no ties, an observed assignment is included among those counted as at least as extreme as itself. If it is the only such assignment, the smallest attainable one-sided p-value is \(1/45\approx 0.0222\). A two-sided p-value depends on how many assignments are at least as extreme in either direction; it is not necessarily twice this value. Even with 45 possible assignments, there are still only 2 treated districts, so this count alone does not establish that a cluster-asymptotic bootstrap is reliable.
Common Mistakes and a Design Checklist
A frequent mistake is to reason, “There are thousands of observations, so the cluster-robust standard error must be accurate.” This confuses the number of recorded outcomes with the number of independent assignment units. The correction is to report the number of clusters overall and by treatment status, then assess inference in light of that structure.
Another mistake is to call a permutation test exact merely because it enumerates every possible rearrangement of treatment labels. Enumeration is exact only relative to a defensible assignment mechanism and the stated null. If treatment was not randomly assigned, unrestricted shuffling may compare the observed design with assignments that were never plausible. Likewise, a wild cluster bootstrap p-value is not an exact design-based p-value.
Before reporting inference with few treated clusters, check the following:
- Assignment unit: Identify where treatment varies and cluster uncertainty at a level that reflects assignment and dependence.
- Cluster counts: Report both the total number of independent clusters and the numbers treated and untreated. Note substantial imbalance or influential clusters.
- Bootstrap specification: State the wild cluster bootstrap variant, null restriction, number of draws, and how the p-value or interval was calculated.
- Randomization justification: State the actual assignment mechanism or explain why the proposed assignments are credible under the design. Respect strata and other restrictions.
- Null and estimand: Distinguish tests of a sharp no-effect null from claims about an average treatment effect, and keep identification assumptions separate from uncertainty calculations.
Check Your Understanding
Answer each question using the distinction between the assignment mechanism and the uncertainty procedure.
- A study has 12,000 person-period observations but treatment was assigned to 6 districts, of which 2 were treated. Why is the individual observation count not a sufficient reason to trust a normal approximation for clustered inference?
- In a null-restricted wild cluster bootstrap-t, why are residuals multiplied by a common weight within each cluster rather than independently for every person?
- What assignment information is needed to justify a randomization-based test when 3 of 9 clusters were treated?
- Why does an exact test of the sharp null of no effect in any cluster not automatically provide an exact test of a null that only states an average effect is zero?
- If a bootstrap p-value is small, what can it say about the tested null, and what does it not establish about DiD identification?