Sensitivity to Alternative Cluster Definitions

Why Compare Cluster Definitions?

Cluster-robust inference allows observations within a cluster to have dependent errors while treating different clusters as independent. But applied data may admit more than one defensible clustering level. Patients may share a clinic and a health system; students may share a school and a district; people may share a workplace and a neighborhood. Comparing results across reasonable definitions helps reveal how strongly inference depends on the assumed pattern of dependence.

This is a sensitivity analysis, not a contest to find the standard error that produces the preferred conclusion. The clustering choice should be motivated by the study design and plausible sources of shared variation. A comparison can show that conclusions are robust to some choices, or that they depend on an assumption that needs further attention. It cannot, by itself, identify the true dependence structure.

Definition: A cluster definition specifies which observations are grouped together for cluster-robust inference. With the data, regression model, and included observations held fixed, changing only that definition leaves the fitted regression coefficients unchanged, but can change their estimated standard errors, confidence intervals, and test results.

What the Comparison Changes

For ordinary least squares regression, let \(X\) be the design matrix, let \(\hat{\boldsymbol{\beta}}\) be the fitted coefficient vector, and let \(e_i\) be the residual for observation \(i\). Under a particular cluster definition with \(G\) clusters, write \(X_g\) for the rows of \(X\) in cluster \(g\), and \(e_g\) for the corresponding residual vector. The basic cluster-robust covariance estimator is

\widehat{\operatorname{Var}}_{\mathrm{cluster}}(\hat{\boldsymbol{\beta}}) = (X^\mathsf{T}X)^{-1} \left(\sum_{g=1}^{G} X_g^\mathsf{T}e_g e_g^\mathsf{T}X_g\right) (X^\mathsf{T}X)^{-1}.

The middle term aggregates the regression score contributions within each cluster before forming their outer products. Thus, the estimator allows errors to be dependent within a cluster; it does not treat every observation as an independent source of information. Different definitions regroup the same observations, changing this middle term. In practice, analysts may also apply a finite-sample correction; any comparison should use a stated, consistent inference procedure.

A useful way to see why merging clusters does not guarantee a larger estimated variance is to consider the score totals \(S_1\) and \(S_2\) for two groups. Keeping them separate contributes \(S_1S_1^\mathsf{T}+S_2S_2^\mathsf{T}\). Combining them contributes

(S_1+S_2)(S_1+S_2)^\mathsf{T} = S_1S_1^\mathsf{T} + S_2S_2^\mathsf{T} + S_1S_2^\mathsf{T} + S_2S_1^\mathsf{T}.

The added cross-products can increase or decrease particular entries of the estimated covariance matrix. The effect on a coefficient's standard error therefore depends on the observed score patterns, not simply on whether the new definition has fewer clusters. Moreover, fewer clusters can mean less reliable large-sample inference, even if the reported standard error is not larger.

Conditions for a Meaningful Sensitivity Analysis

Before comparing estimates, state the regression model and what is held fixed. For the usual cluster-robust interpretation, the mean model must be appropriate for the coefficient or contrast of interest, observations in different clusters must be independent (or close enough for the chosen approximation), and dependence within a cluster may be unrestricted. Inference also relies on having enough independent clusters for the approximation used; a small number of clusters can make conventional standard errors and confidence intervals unreliable.

The candidate definitions should correspond to plausible shared shocks or dependence mechanisms, or to relevant design features such as the level of assignment. They should be formed from the observational structure, not selected after looking for the smallest standard error or a desired significance result. Keep the outcome, predictors, observations, coefficient interpretation, and regression specification fixed. If these change too, the comparison no longer isolates sensitivity to the cluster definition.

For confidence intervals in the examples below, we use a stated procedure: the reported cluster-robust standard error and a \(t\) critical value with \(G-1\) degrees of freedom. Other procedures, including small-sample corrections, may use different degrees of freedom. The key is to report the procedure and apply it consistently when comparing alternatives.

Worked Example: Clinics or Health Systems?

Hypothetical data. A study records patient outcomes at 48 clinics belonging to 12 health systems. A fixed regression estimates the association between a clinic-level program indicator and an outcome measured in points. The coefficient is \(4.20\) points under either clustering definition. Suppose clinic-level clustering gives \(SE=0.90\), while health-system clustering gives \(SE=1.35\). Assume the reported standard errors use the same correction and that the stated confidence-interval procedure uses \(t_{G-1}\).

With clinics as clusters, \(G=48\), so \(df=47\). Using \(t_{0.975,47}=2.012\), the 95% interval is

4.20 \pm 2.012(0.90) = 4.20 \pm 1.811 = (2.389,\ 6.011)\text{ points}.

With health systems as clusters, \(G=12\), so \(df=11\). Using \(t_{0.975,11}=2.201\), the interval is

4.20 \pm 2.201(1.35) = 4.20 \pm 2.971 = (1.229,\ 7.171)\text{ points}.

Both intervals are above zero under the stated procedure, but the health-system interval is much wider. The estimate remains \(4.20\) points because the regression did not change; the estimated uncertainty changed. The comparison suggests that accounting for dependence at the health-system level matters for precision. It does not establish that health-system clustering is correct: that choice needs substantive and design justification, and 12 clusters may itself make the approximation fragile.

Reading Differences Without Overclaiming

A sensitivity result is most useful when it is described in terms of what changes and why that change matters. If standard errors and intervals are similar across defensible definitions, the reported inference may be relatively insensitive to those particular choices. If they differ substantially, the dependence assumption has practical consequences and deserves explicit discussion. In either case, the result is conditional on the model, sample, and candidate definitions considered.

Do not read the largest standard error as automatically correct or as a guaranteed upper bound. The score calculation above shows why merging groups need not increase every variance estimate. Nor does an unchanged coefficient show that the dependence assumptions are harmless: point estimates can be identical while uncertainty changes enough to alter the conclusions supported by an interval or test.

Worked Example: Schools or Neighborhoods?

Hypothetical data. A regression relates an educational outcome, measured in points, to hours of tutoring. The same 600 pupils attend 40 schools and live in 25 neighborhoods; schools and neighborhoods overlap, rather than one being nested entirely within the other. The fitted coefficient is \(2.60\) points per tutoring hour under both one-way cluster definitions. Suppose the school-clustered standard error is \(0.70\) and the neighborhood-clustered standard error is \(1.05\). Use \(t_{G-1}\) for the illustrative intervals.

For school clustering, \(G=40\), \(df=39\), and \(t_{0.975,39}=2.023\). Therefore,

2.60 \pm 2.023(0.70) = 2.60 \pm 1.416 = (1.184,\ 4.016)\text{ points per hour}.

For neighborhood clustering, \(G=25\), \(df=24\), and \(t_{0.975,24}=2.064\). Therefore,

2.60 \pm 2.064(1.05) = 2.60 \pm 2.167 = (0.433,\ 4.767)\text{ points per hour}.

The neighborhood-based interval is wider, although it still excludes zero under this procedure. Since each pupil shares both a school and a neighborhood, comparing these two one-way results is diagnostic rather than proof that either captures all dependence. If both sources plausibly induce dependence, a single clustering dimension may not represent the structure adequately; a suitable method for crossed dependence may be needed.

A Practical Comparison

Specify the fixed analysis.
Record the outcome, predictors, sample, fitted model, coefficient or contrast, and any correction used. These stay the same across comparisons.

Name defensible alternatives.
List cluster definitions motivated by assignment, sampling, or plausible shared influences. Explain why each could matter before examining which produces the widest interval.

Check the cluster counts and independence claim.
For each definition, report the number of clusters and consider whether different clusters plausibly act as independent units. Few clusters call for particular caution.

Compare uncertainty on a common basis.
Report the unchanged coefficient alongside each standard error and confidence interval. State the degrees-of-freedom rule and correction so the intervals can be interpreted fairly.

Explain the sensitivity in context.
Say whether substantive conclusions change, and describe which dependence assumptions account for the comparison. Do not treat the results as a data-driven vote for one clustering level.

Worked Example: How Much Does the Conclusion Depend on Clustering?

Hypothetical data. A fixed regression estimates the association between a workplace training measure and employee output, in output units per training unit. There are 100 schools and 20 districts, with schools nested within districts. The estimated coefficient is \(-1.80\) under either definition. Suppose the school-clustered standard error is \(0.48\) and the district-clustered standard error is \(0.73\), with the same correction and \(t_{G-1}\) interval procedure.

For schools, \(df=99\) and \(t_{0.975,99}=1.984\). The interval is

-1.80 \pm 1.984(0.48) = -1.80 \pm 0.952 = (-2.752,\ -0.848)\text{ output units per training unit}.

For districts, \(df=19\) and \(t_{0.975,19}=2.093\). The interval is

-1.80 \pm 2.093(0.73) = -1.80 \pm 1.528 = (-3.328,\ -0.272)\text{ output units per training unit}.

Both intervals are below zero using this procedure, but the district-clustered interval is wider. That difference indicates greater uncertainty under the district-level dependence assumption. Because there are only 20 districts, the district-based inference also deserves caution: a narrow or wide interval does not erase the limitations of having few independent clusters.

Common Mistake and Key Takeaway

Common mistake: “The clinic-clustered standard error is smaller, so clinics must be the correct clustering level.” This reasoning chooses a definition by its result. A small standard error is not evidence that observations are independent across the chosen clusters, and a larger one is not automatically valid. Instead, justify the candidate definitions from the study design and plausible dependence, then show how conclusions vary across them. If plausible dependence is nested or crossed in a way that one-way clustering cannot capture, acknowledge that limitation and consider a method suited to the structure.

Key takeaway: With the regression and data held fixed, alternative cluster definitions change the estimated uncertainty, not the fitted coefficient. Sensitivity comparisons make the consequences of dependence assumptions visible; they do not choose the correct assumption automatically.

Check Your Understanding

Use the definitions and examples above to answer the following questions.

With the data and regression specification fixed, what changes when the cluster definition changes, and what remains unchanged?
Why is it not generally true that combining two clusters must increase a coefficient's cluster-robust variance estimate?
A confidence interval becomes wider under a defensible alternative clustering level. What does this reveal, and what does it not prove?
Why should candidate cluster definitions be motivated before comparing their standard errors?
Suppose observations share both schools and neighborhoods, with those groupings crossed. Why might neither one-way clustering analysis alone be a sufficient description of dependence?