Satterthwaite Degrees of Freedom for Clustered Tests

Why Clustered Tests Need a Contrast-Specific Reference Distribution

With clustered errors, the independent units for inference are clusters, not individual observations. Even when a cluster-robust covariance estimator is adjusted for leverage, using a conventional normal or \(t\) reference distribution with a generic degrees of freedom can be poorly calibrated when the number of clusters is limited. The relevant information also depends on the particular coefficient or linear combination being tested: some contrasts may be supported broadly across clusters, while others depend heavily on only a few.

Satterthwaite degrees of freedom address this by approximating the variability of a scalar contrast’s estimated variance. The calculation uses how the cluster-level contributions to that variance are distributed and, importantly, how those contributions covary. The result is a contrast-specific degrees of freedom for an approximate \(t\) interval or test, not a universal count of clusters minus one.

Model and Cluster-Level Contributions

Let \(y_g\) be the outcome vector and \(X_g\) the design matrix for cluster \(g\), for \(g=1,\ldots,G\). Stack these across clusters to form \(y\) and \(X\), and consider the linear regression model \(y=X\beta+\varepsilon\), where \(X\) has full column rank. Assume the errors have mean zero and are independent across clusters; dependence and unequal variances within a cluster are allowed. For the moment calculation, use a specified working covariance \(\Phi\), block diagonal across clusters, and a Gaussian working model for \(\varepsilon\). These assumptions provide the reference distribution used to calculate the approximation; they do not assert that the working covariance is the true covariance.

For a scalar contrast \(\theta=c'\beta\), where \(c\) is a specified vector, the OLS estimate is \(\hat\theta=c'\hat\beta\), with \(\hat\beta=(X'X)^{-1}X'y\). Write \(\hat u_g=y_g-X_g\hat\beta\). With a CR2 adjustment matrix \(A_g\), the adjusted cluster score is \(X_g'A_g\hat u_g\). The corresponding scalar contribution to the contrast’s CR2 variance estimator is \(\psi_g=c'(X'X)^{-1}X_g'A_g\hat u_g\).

The \(\psi_g\) are linear functions of the regression errors under the working model. Although errors are independent across clusters, fitted residuals and adjusted contributions need not be independent across clusters: estimation of \(\beta\) links them. Define \(\psi=(\psi_1,\ldots,\psi_G)'\) and let \(\Omega=\operatorname{Cov}(\psi)\), calculated under the working model. Then \(\hat V=\psi'\psi\). Using the full covariance matrix \(\Omega\) accounts for dependence among the contributions.

Moment Matching and the Degrees-of-Freedom Formula

Under the Gaussian working model, \(\psi\) is a mean-zero multivariate normal vector with covariance \(\Omega\). A symmetric nonnegative covariance matrix has nonnegative eigenvalues \(\lambda_1,\ldots,\lambda_G\). Rotating \(\psi\) into the eigenvector directions shows that \(\psi'\psi\) has the same distribution as \(\sum_j\lambda_j Z_j^2\), where the \(Z_j\) are independent standard normal random variables. Thus the estimated variance is generally a weighted sum of chi-square variables, rather than one exact scaled chi-square variable.

To derive the second identity, diagonalize \(\Omega\). The mean of \(\sum_j\lambda_j Z_j^2\) is \(\sum_j\lambda_j\), since \(E(Z_j^2)=1\). Its variance is \(\sum_j\lambda_j^2\operatorname{Var}(Z_j^2)=2\sum_j\lambda_j^2\), since \(\operatorname{Var}(Z_j^2)=2\) and the terms are independent. These sums are \(\operatorname{tr}(\Omega)\) and \(2\operatorname{tr}(\Omega^2)\), respectively.

Approximate \(\hat V\) by \(a\chi^2_\nu\), where \(a>0\) is a scale and \(\nu\) is the degrees of freedom. A scaled chi-square has mean \(a\nu\) and variance \(2a^2\nu\). Equating these with the moments above gives \(a=\operatorname{tr}(\Omega)/\nu\), and then \[ \nu=\frac{\{\operatorname{tr}(\Omega)\}^2}{\operatorname{tr}(\Omega^2)}. \] This is the Satterthwaite moment match. In practice, \(\Omega\) is obtained from the design, CR2 adjustment, and working covariance model; the formula is not calculated from the observed \(\psi_g\) alone.

The formula also clarifies why the answer is not simply \(G-1\). If the contributions are uncorrelated and have equal variances, \(\Omega=\lambda I_G\), so \(\nu=(G\lambda)^2/(G\lambda^2)=G\). If their variances or covariances are uneven, the effective degrees of freedom can be smaller. The eigenvalue form, \(\nu=(\sum_j\lambda_j)^2/\sum_j\lambda_j^2\), makes this concentration visible: a few dominant eigenvalues yield a smaller value than many similarly sized eigenvalues.

Using the Approximation for Confidence Intervals

Let \(\widehat{SE}=\sqrt{\hat V}\). For a nominal \(100(1-\alpha)\%\) interval, use the \(t\) quantile with the contrast-specific Satterthwaite degrees of freedom: \(\hat\theta\pm t_{1-\alpha/2,\nu}\widehat{SE}\). The interval is an approximation based on the working model and moment match. Its confidence level describes the long-run coverage of the procedure under its assumptions and approximation, not a probability that a fixed parameter lies in the particular interval.

Using the Approximation for a Test

For a two-sided test of \(H_0:c'\beta=\theta_0\) against \(H_a:c'\beta\ne\theta_0\), calculate \(t=(\hat\theta-\theta_0)/\widehat{SE}\). Compare its absolute value with a \(t_\nu\) reference distribution, or compute the approximate two-sided p-value \(2P(T_\nu\ge |t|)\), where \(T_\nu\) has a \(t\) distribution with \(\nu\) degrees of freedom. The p-value is a tail probability under the null reference model; it is not the probability that the null hypothesis is true.

Practical Interpretation and Common Pitfalls

The degrees of freedom are contrast-specific: changing \(c\) can change the cluster contributions and their covariance, and hence \(\nu\), even with the same regression and number of clusters. They are also conditional on the chosen working covariance and CR2 adjustment. A fractional value is expected and should be used as such in the \(t\) quantile calculation. The method supplies an approximation for small-sample inference; it does not guarantee accurate coverage or testing in every design.

• Using the number of clusters as the degrees of freedom. This ignores the contrast-specific contribution structure. Calculate the Satterthwaite value for the contrast being reported.
• Dropping off-diagonal entries of \(\Omega\). The trace of \(\Omega^2\) includes squared off-diagonal entries as well as squared diagonal entries. Include both when contributions covary.
• Using a normal critical value because the standard error is cluster robust. Robust covariance estimation and reference-distribution choice are separate decisions. Use the stated \(t_\nu\) approximation when this procedure is being used.
• Calling the approximation exact. Moment matching aligns the first two moments of the estimated variance with a scaled chi-square distribution. It does not make the full variance-estimator distribution, or the resulting studentized statistic, exactly chi-square or \(t\).