Why Few Clusters Cause Inference Problems

Why the Number of Clusters Matters

Cluster-robust standard errors allow observations within a cluster to be dependent while treating different clusters as independent. This is useful when, for example, patients are grouped within clinics or students within schools. But the protection they offer depends on having enough independent clusters. A data set can contain thousands of observations and still provide little support for cluster-based inference if those observations come from only a few clusters.

The key is that the variance estimate is built from cluster-level contributions. With few clusters, those contributions may be highly variable, a small number of clusters may dominate the calculation, and the usual large-sample reference distribution may be a poor approximation. The problem is not simply that the total sample size is small: it is that the number of independent pieces of information is limited.

Model and Cluster-Robust Inference

Suppose there are \(G\) clusters and \(n_g\) observations in cluster \(g\), for \(g=1,\ldots,G\). Let \(N=\sum_{g=1}^{G} n_g\) be the total number of observations. Consider the linear regression model

y_{ig}=x_{ig}^{\mathsf T}\beta+u_{ig},

where \(y_{ig}\) is the outcome, \(x_{ig}\) is the vector of predictors (including an intercept if specified), \(\beta\) is the vector of regression coefficients, and \(u_{ig}\) is the error for observation \(i\) in cluster \(g\). The usual cluster-robust approach permits errors to be dependent within a cluster, including having different variances, but assumes errors are independent across clusters. It also requires the regression model to be suitably specified for the coefficient or estimating target under study, and requires enough independent clusters for the large-sample approximation to be credible.

Write \(X_g\) for the design matrix in cluster \(g\), and \(\hat{u}_g\) for the vector of fitted residuals there. The conventional cluster-robust sandwich variance estimator for ordinary least squares is

\widehat{\operatorname{Var}}_{\mathrm{CR}}(\hat{\beta}) = (X^{\mathsf T}X)^{-1} \left( \sum_{g=1}^{G} X_g^{\mathsf T}\hat{u}_g\hat{u}_g^{\mathsf T}X_g \right) (X^{\mathsf T}X)^{-1}.

The middle term adds one outer product per cluster. It allows arbitrary patterns of residual dependence within each cluster, but it has only \(G\) cluster-level contributions. That is why \(N\) alone cannot tell us whether this variance estimate is stable.

Key idea: Conventional cluster-robust inference is justified by having many independent clusters, not merely many observations. A large \(N\) does not compensate automatically for a small \(G\).

Why Few Clusters Destabilize Inference

The large-sample justification works at the cluster level. In broad terms, the coefficient estimate depends on adding contributions from independent clusters. When many clusters contribute and no one cluster dominates, the sum can be approximated by a normal distribution. The estimated variance can also stabilize as it aggregates information across clusters. With few clusters, neither approximation is guaranteed to work well.

A useful way to see the issue is to consider inference about a single contrast \(a^{\mathsf T}\beta\), where \(a\) is a fixed vector. The cluster contribution to that contrast’s estimated variance can be written as

q_g=a^{\mathsf T}(X^{\mathsf T}X)^{-1}X_g^{\mathsf T}\hat{u}_g, \qquad \widehat{\operatorname{Var}}_{\mathrm{CR}}(a^{\mathsf T}\hat{\beta}) =\sum_{g=1}^{G}q_g^2.

If many \(q_g\) values contribute comparably, the sum of squares is less dependent on any single cluster. If one or two values dominate, the variance estimate can change substantially when those clusters change. With only a few clusters, there is also limited information for estimating how much cluster contributions vary.

The corresponding test statistic is often compared with a standard normal distribution under a large-cluster approximation. Some software or analyses instead use a \(t\) reference distribution with a small number of degrees of freedom. Neither choice makes the conventional variance estimate automatically reliable when \(G\) is very small. A small-cluster reference distribution may change the critical values, but it cannot create independent clusters that were not observed.

Worked Example: The Same Standard Error, Different Reference Distributions

Suppose a hypothetical regression has \(G=6\) independent clinics and estimates a treatment coefficient of \(0.40\) outcome units, with a conventional cluster-robust standard error of \(0.20\) outcome units. Consider a two-sided test of \(H_0:\beta=0\). The estimated statistic is

t_{\mathrm{obs}}=\frac{0.40-0}{0.20}=2.00.

Under a standard normal reference distribution, the two-sided p-value is approximately \(2P(Z\geq 2.00)=0.0455\), where \(Z\sim N(0,1)\). Under a \(t\) reference distribution with \(G-1=5\) degrees of freedom, the two-sided p-value is approximately \(0.1019\).

These p-values differ because the \(t_5\) distribution has heavier tails than the standard normal distribution. This comparison does not establish that either calculation is exact for these six clinics. It shows why a conclusion based on a normal reference distribution can be sensitive to the reference choice when the cluster count is small. The coefficient and standard error alone do not remove that concern.

Diagnosing Dominant Cluster Contributions

A cluster count is a first diagnostic, not the whole diagnosis. The distribution of cluster contributions also matters. If one cluster supplies a large share of the variance estimate for a coefficient, inference may depend heavily on that cluster. This can happen because its residual pattern is unusual, because it has substantial leverage, or both.

Worked Example: One Cluster Dominates the Variance Contribution

Suppose a hypothetical analysis has five independent clinics. For a particular coefficient contrast, the calculated cluster contributions to the standard error are \(q_1=8\) and \(q_2=q_3=q_4=q_5=-2\), in compatible score units. These values sum to zero, as can occur for a regression score, but the variance uses their squares:

\sum_{g=1}^{5}q_g^2 =8^2+(-2)^2+(-2)^2+(-2)^2+(-2)^2 =64+16=80.

The first clinic contributes \(64/80=0.80\), or 80%, of the estimated variance. Each other clinic contributes \(4/80=0.05\), or 5%. Although the data include five clinics, the estimated uncertainty for this contrast is dominated by one of them. A modest change in that clinic’s outcomes or residuals could have a large effect on the estimated standard error. This pattern is a warning of fragile inference, not a proof that the coefficient estimate is biased.

Cluster Leverage and Limited Independent Variation

Cluster leverage describes how strongly a cluster’s predictor values contribute to the fitted regression design. For the ordinary least-squares projection matrix \(H=X(X^{\mathsf T}X)^{-1}X^{\mathsf T}\), let \(H_{gg}\) denote the block corresponding to observations in cluster \(g\). A useful summary is the cluster leverage

h_g=\operatorname{tr}(H_{gg}).

The cluster leverages sum to the number of fitted regression coefficients, \(p\): \(\sum_{g=1}^{G}h_g=p\). A cluster with a relatively large \(h_g\) carries a comparatively large share of the design’s leverage. This is a diagnostic summary, not a universal cutoff. Its interpretation depends on the model and on which coefficient is being studied.

Leverage matters because fitted residuals are computed after the model has been fit. A cluster that strongly influences the fit can have residuals that understate how much that cluster would differ from a fit based on the other clusters. In a small-cluster setting, this can make the cluster’s contribution to the sandwich estimate less informative about the variation across independent clusters.

Worked Example: A Predictor Varies Across Only Two Clusters

Imagine a hypothetical study with eight schools, each contributing the same number \(m\) of students. A binary school-level predictor equals 1 in two schools and 0 in the other six. The regression includes an intercept and this predictor, so \(p=2\). Because the predictor is constant within each school, the fitted values distinguish the two predictor groups using only two treated schools and six control schools.

For this balanced design, each student in a treated school has diagonal leverage \(1/(2m)\): there are \(2m\) observations in the treated group. Summing over the \(m\) students in one treated school gives cluster leverage \(h_g=m/(2m)=0.50\). For a control school, each student has leverage \(1/(6m)\), giving \(h_g=m/(6m)=0.1667\). The total is

2(0.50)+6(0.1667)\approx 2=p.

Each treated school accounts for \(0.50/2=25\%\) of the total leverage, and the treatment comparison has only two independent treated clusters. The study has many student observations, but the school-level predictor has only eight independent values, with very limited variation among treated schools. This combination of a small cluster count and concentrated leverage calls for caution about conventional cluster-robust inference.

A Short Justification: Why Many Clusters Support the Approximation

Let \(S_g\) denote a cluster’s contribution to the regression score. Under the across-cluster independence assumption, the total score is a sum of independent cluster contributions. If \(G\) grows, and no individual cluster contributes a substantial share of the total variability, a central limit theorem can make the appropriately scaled sum approximately normal. In the same setting, adding many cluster-level variance contributions can make their sum a more stable estimate of variability.

The “no dominant cluster” condition is important. If one cluster’s contribution remains large relative to the total as \(G\) grows, the sum need not behave like a normal sum of many comparable independent pieces. With a fixed, small \(G\), there is no general reason to expect the large-\(G\) approximation to be accurate. The usual cluster-robust formula can still be calculated, but its availability is not a guarantee of reliable inference.

What to Check in Practice

Begin by counting the independent clusters relevant to the assumed dependence structure. Then check whether those clusters contribute comparably to the coefficient being studied. A useful diagnostic review asks:

How many independent clusters are there, and are some clusters much larger than others?
Does the predictor of interest vary within clusters, or is its information primarily at the cluster level?
Are only a few clusters in one predictor or treatment category?
Do cluster score contributions or leverage summaries show that one or a few clusters dominate?
Would the substantive conclusion change noticeably under a plausible alternative reference distribution?

There is no universal cluster-count threshold that separates safe from unsafe inference. Risk depends on the number of clusters, their balance, the design, the predictor being tested, and the concentration of cluster contributions. Reporting \(N\) without \(G\), or reporting \(G\) without explaining where predictor variation occurs, can conceal the relevant limitation.

Common Mistake

Wrong reasoning: “There are 2,000 observations, so the cluster-robust standard error must be reliable.” This treats observations within the same cluster as if they supplied fully independent information about uncertainty. If those 2,000 observations come from six clinics, the independent cluster-level information is still limited.

Correction: Report and assess the number of independent clusters, then examine whether cluster contributions and leverage are concentrated. A large total sample can improve estimation within clusters, but it cannot by itself replace having more independent clusters. Do not interpret a computed standard error or a normal-reference p-value as proof that the few-cluster approximation is adequate.

Key takeaway: Few clusters can make both the estimated cluster-robust variance and its reference distribution unreliable. Cluster counts, the location of predictor variation, score contributions, and leverage together provide a more informative diagnosis than the total observation count alone.

Check Your Understanding

Use the ideas in this tutorial to answer the following questions.

A study has 900 patients from nine clinics. Why is 900 not the relevant count by itself when judging cluster-robust inference?
For one coefficient, one cluster contributes 60% of the sum of squared cluster contributions. What does this suggest, and what does it not prove?
Why can a normal-reference p-value differ from a p-value based on a \(t\) distribution with a small number of degrees of freedom?
In a regression with a school-level predictor, why should the number of schools in each predictor group be checked?
Why is a fixed rule such as “at least 30 clusters is always enough” not a complete diagnostic?