Inference With Few Clusters in DiD

Why the Number of Clusters Matters

In a difference-in-differences analysis, outcomes from units in the same assignment cluster may share shocks. Cluster-robust standard errors allow this within-cluster dependence, but their usual justification relies on having many independent clusters. A data set can contain thousands of individuals and still provide little information for inference if treatment was assigned in only a small number of schools, hospitals, or regions.

The key sample size for cluster-robust inference is the number of independent clusters, not the number of individual observations. With few clusters, the estimated variance can be biased downward, and the usual normal reference distribution can make tests too likely to reject a true null hypothesis. These problems can be acute when only a few clusters receive treatment or when cluster sizes and leverage differ substantially.

For a two-period illustration, a common regression is

where \(Y_{ict}\) is the outcome for unit \(i\) in cluster \(c\) at time \(t\), \(\alpha_c\) and \(\lambda_t\) are cluster and time effects, \(D_c\) indicates a treated cluster, and \(Post_t\) indicates the post-intervention period. The coefficient \(\tau\) is the DiD interaction estimate. A causal interpretation still requires the relevant identification assumptions, including parallel trends; changing the standard error method does not establish them. The discussion here concerns uncertainty quantification for the estimated coefficient.

Write the regression in matrix form as \(Y=X\hat{\beta}+\hat{u}\), with \(X_c\) and \(\hat{u}_c\) denoting the regressors and residuals for cluster \(c\). The basic cluster sandwich covariance estimator has the form

Each cluster contributes one score-like quantity to the “meat” of this estimator. With many independent clusters, the aggregate can often be approximated by a normal distribution. With few clusters, that approximation and the estimated variance may both be poor. The usual large-sample standard error alone does not account for this uncertainty about the reference distribution.

Finite-Sample Adjustments and Degrees of Freedom

One common adjustment, often called CR1, multiplies the CR0 covariance estimate by a finite-sample factor. A frequently used version is

where \(N\) is the number of observations and \(K\) is the number of estimated regression coefficients. Implementations vary in the correction they use. The factor can increase the estimated variance, but it does not by itself ensure valid inference with few clusters. In particular, multiplying a covariance estimate by a correction factor and choosing an appropriate reference distribution are separate decisions.

A simple degrees-of-freedom choice for a cluster-robust test is a \(t\) distribution with \(G-1\) degrees of freedom rather than the standard normal distribution. This recognizes that the number of independent cluster contributions is limited. It is a useful basic adjustment, but \(G-1\) need not accurately reflect the information available for a particular coefficient when treatment is concentrated in a few clusters or cluster leverage is uneven.

Bias-Reduced Variance Estimation

A further option is a bias-reduced cluster covariance estimator, commonly called CR2. Its adjustment accounts for the leverage of observations or clusters in fitting the regression, in a manner analogous to small-sample corrections for heteroskedasticity-robust variance estimation. The aim is to reduce finite-sample bias in the estimated covariance; the adjustment is not a guarantee that the resulting test is exact.

CR2 is commonly paired with a Satterthwaite approximation for the degrees of freedom of the coefficient or contrast being tested. This approximation uses the estimated variability and influence pattern of that specific contrast. Consequently, its degrees of freedom can be fractional and can be much smaller than \(G-1\). The standard error adjustment and the degrees-of-freedom calculation are both relevant: reporting a CR2 standard error but silently using a normal reference can discard an important part of the small-sample procedure.

Wild Cluster Bootstrap Inference

A wild cluster bootstrap-t is another method for testing a DiD coefficient with few clusters. The procedure constructs a reference distribution by repeatedly generating bootstrap outcomes under the null hypothesis, while preserving dependence within each cluster. A typical implementation restricts the regression under \(H_0:\tau=\tau_0\), applies a common random weight to the residuals within each cluster, and refits the regression in each bootstrap sample. It then compares the observed studentized statistic with the bootstrap statistics.

For example, with Rademacher weights, each cluster independently receives a weight of \(+1\) or \(-1\), each with probability \(1/2\). Applying the same weight to all residuals in a cluster preserves their within-cluster pattern. Each bootstrap replication must use the same regression specification and cluster definition as the original analysis. The null restriction matters: for a test of zero effect, the bootstrap samples should be generated under the zero-effect null rather than treating the unrestricted fitted effect as true.

For \(B\) bootstrap replications, a commonly used two-sided bootstrap \(p\)-value with a finite-simulation adjustment is

where \(t_{obs}\) is the observed studentized statistic, \(t_b^*\) is the statistic in bootstrap replication \(b\), and \(I(\cdot)\) equals 1 when its condition is true and 0 otherwise. The added 1 prevents reporting a zero \(p\)-value merely because none of the simulated statistics exceeded the observed one.

What These Methods Can and Cannot Fix

Small-sample corrections and bootstrap procedures address the uncertainty calculation, not the causal design. They do not repair violations of parallel trends, treatment misclassification, or a cluster definition that fails to capture relevant dependence. They also cannot create independent treatment assignments that the study did not have.

In particular, few treated clusters are a serious limitation even if the total number of clusters seems moderate. If a policy is adopted by only two regions, for example, the evidence about treatment effects may be highly sensitive to those regions. A bootstrap with many replications does not turn those two treated clusters into many independent pieces of information. When feasible, researchers should report the numbers of treated and comparison clusters, cluster sizes, the variance correction, reference distribution and degrees of freedom, and the bootstrap design and replication count.

A common mistake is to reason: “There are thousands of people in the data, so the cluster-robust \(z\)-test must be reliable.” The correction is that people within a cluster may share shocks, so they are not thousands of independent contributions to the variance estimate. Inference must reflect the independent clusters and how treatment is distributed among them.

No small-cluster method is uniformly best for every design. A transparent analysis may report a well-justified bias-reduced method and a wild cluster bootstrap as a sensitivity check, while emphasizing the small number of independent clusters. Agreement between methods can be reassuring, but it is not proof that their assumptions hold.