Diagnosing Few-Cluster Inference

Why the Number of Clusters Matters

Cluster-robust inference allows observations within a cluster to have dependent errors while treating different clusters as independent. Its large-sample justification comes from having many independent clusters, not merely many observations. A data set can contain thousands of people yet provide weak support for a cluster-robust Wald test if those people are concentrated in only a handful of schools, hospitals, firms, or geographic areas.

Diagnosing few-cluster risk therefore involves more than counting clusters. Cluster sizes may be highly unequal, a small number of clusters may have unusually high regression leverage, or a treatment may be assigned to very few clusters. Each feature can concentrate the information in a small part of the data. These diagnostics do not produce a universal threshold at which inference becomes valid or invalid; they help explain how fragile a particular analysis may be.

The Conditions Behind Cluster-Robust Wald Inference

Consider the linear regression model \(y_g=X_g\beta+u_g\), where \(y_g\) and \(X_g\) contain the outcomes and regressors for cluster \(g\). The coefficient vector \(\beta\) is common across clusters. The key dependence assumption is that errors may be arbitrarily dependent within a cluster, while different clusters are independent: \(u_g\) is independent of \(u_h\) for \(g\ne h\). Regression specification and moment conditions are also needed for the target coefficient and its variance to be well defined.

The cluster-robust covariance estimator is built from cluster-level contributions to the regression score. In ordinary least squares, a contribution has the form \(X_g'\hat u_g\), where \(\hat u_g\) is the vector of fitted residuals in cluster \(g\). The sandwich estimator combines the outer products of these contributions with the inverse cross-product matrix of the regressors. Its central feature is that it sums across clusters, rather than treating each observation as an independent source of information.

This estimator is motivated by asymptotics in which \(G\) grows and no small set of clusters dominates the relevant score variation. Under suitable regularity conditions, a sum of many independent cluster contributions can be approximated by a normal distribution. A conventional cluster-robust Wald statistic typically uses that approximation, even though its covariance estimate is itself calculated from a finite number of cluster contributions. With few clusters, the normal approximation and the estimated variance can both be unreliable. The resulting test can be miscalibrated, often rejecting too often, but the direction and size of the error are not guaranteed in every design.

Small-sample adjustments such as CR2, CR3, and Satterthwaite degrees of freedom can address some finite-sample features, as covered in the earlier material on those methods. Their usefulness depends on the model and design. They do not create additional independent clusters, and a nominal reference distribution does not guarantee accurate inference when information is highly concentrated.

Four Diagnostics to Examine

A useful diagnosis combines the cluster count with the structure of the regression and treatment assignment. Record the observations below before interpreting a cluster-robust test.

• Cluster count. Report \(G\) and, where relevant, the number of clusters in each treatment or comparison group. A count such as \(G=12\) is more informative than reporting only the total number of observations.
• Size imbalance. Compare the distribution of \(n_g\), including the smallest and largest cluster and the share of observations in the largest clusters. Unequal sizes can mean that a few clusters contribute disproportionately, although sizes alone do not determine score contributions.
• Regression leverage. Assess whether the fitted regressor design gives one or a few clusters unusually strong influence over the coefficient estimates. Cluster-level leverage can be examined from the cluster blocks of the hat matrix.
• Treatment allocation. For cluster-level assignment, count treated and control clusters, not just treated and control observations. Also examine whether treatment variation is concentrated in a small number of clusters or coincides with high-leverage covariate patterns.

These are descriptive diagnostics, not universal pass/fail cutoffs. A balanced set of clusters can still have too small a \(G\) for a good approximation, and a large \(G\) does not automatically protect a poorly supported coefficient. The size-based calculation and examples below show what these checks can and cannot reveal.

Cluster Leverage and Concentrated Design Information

Leverage concerns the regressor design, rather than the size of a cluster by itself. For a regression with \(k\) columns in \(X\), the hat matrix is \(H=X(X'X)^{-1}X'\). Its diagonal blocks \(H_{gg}\), formed from the rows belonging to cluster \(g\), describe how strongly that cluster's regressor values participate in fitting the model. A simple cluster summary is \(\operatorname{tr}(H_{gg})\), the trace of the block. The traces sum to \(k\); unusually large values indicate that some clusters account for a disproportionate share of fitted-design leverage.

High leverage matters for cluster-robust inference because the variance estimator uses fitted residuals. A cluster whose observations strongly determine the fitted coefficients can have residuals that understate the uncertainty associated with its own contribution. Leverage adjustments such as CR2 and CR3 address this issue in specified ways, but a diagnostic should still report whether the design places much weight on a few clusters.

Treatment Allocation and the Number of Informative Clusters

When treatment is assigned to whole clusters, treatment status is constant within each cluster. Repeating the treatment indicator for every person in a clinic does not create new independent treatment assignments. For a treatment effect, the relevant allocation summary includes the number of treated clusters and the number of control clusters, as well as the overall \(G\).

This can be seen in a simple regression with an intercept and a binary treatment indicator at the cluster level. If there are \(G_T\) treated clusters and \(G_C\) control clusters, the diagonal leverage is \(1/G_T\) for each treated cluster and \(1/G_C\) for each control cluster. Thus, a small treatment arm creates high leverage for each cluster in that arm, even if each cluster contains many observations. Covariates and unequal cluster sizes can make the actual observation-level design more complicated, so the fitted design should be inspected as well.

Why a Conventional Wald Test Can Be Miscalibrated

For a scalar coefficient, a conventional cluster-robust Wald statistic is often written \(t=\hat\beta_j/SE(\hat\beta_j)\) under \(H_0:\beta_j=0\). Its large-sample reference is commonly standard normal, or a related reference distribution. The calculation is a ratio: its numerator is a coefficient estimate and its denominator is a covariance estimate. With few clusters, neither part is guaranteed to behave as the approximation requires.

The basic justification can be summarized directly. The regression estimate is driven by a sum of cluster-level score contributions. If many independent clusters contribute and no small set dominates, a central limit approximation may make the standardized sum close to normal. With only a few clusters, the sum has few terms. If cluster sizes, leverage, or treatment allocation are imbalanced, the terms may also be unequal in importance. The estimated covariance then has only a few cluster contributions from which to assess variation. A normal-reference Wald test ignores this extra finite-sample uncertainty, so its actual rejection probability can differ materially from the stated significance level.

A small-sample degrees-of-freedom method may be more appropriate than a normal reference in some settings, but it is not an automatic cure. The estimated degrees of freedom can be much lower than \(G-1\) when the tested contrast is supported by only a few cluster contributions. Even a carefully chosen reference distribution cannot compensate fully for a design with extremely sparse treatment allocation or nearly singular support for the coefficient.

A Practical Diagnostic and Reporting Sequence

Common Mistake: Counting People Instead of Independent Units

A common error is to argue that cluster-robust inference must be reliable because the regression includes thousands of observations. That reasoning counts rows while ignoring the assumed dependence structure. If errors may move together within a school, the independent information for the cluster-robust approximation comes principally from schools, not from treating every student as an independent draw. The correction is to report and assess \(G\), then examine whether size, leverage, or treatment allocation makes the information still more concentrated.

The diagnostics are strongest when interpreted together. Many similarly sized clusters with dispersed leverage and treatment assignment provide a more favorable setting than the same cluster count with one dominant cluster or only two treated clusters. Still, no single summary establishes validity. The purpose is to make the structure behind the standard error visible and to temper conclusions when a conventional Wald test depends on a weak many-cluster approximation.