Testing Contrasts With Clustered Data

Why Test a Contrast?

Regression questions often concern a combination of coefficients rather than any one coefficient by itself. A treatment effect might depend on a baseline measurement, for example, so the effect at a particular baseline value is a sum of a treatment coefficient and an interaction coefficient. Another question might ask whether two slopes are equal, or whether the fitted responses at two specified covariate profiles differ. Each question can be expressed as a linear contrast of regression coefficients.

When observations are clustered, the standard error for that contrast must reflect the dependence within clusters. A cluster-robust covariance matrix provides the estimated variances and covariances of the fitted coefficients. The contrast’s variance uses all of those entries, not just the individual coefficient variances. This matters especially for a difference between two fitted effects: their estimates may move together or in opposite directions across repeated samples.

Model, Contrast, and Conditions

Suppose the regression model has coefficient vector \(\beta\), and let \(\hat{\beta}\) be its fitted estimate. A contrast is a specified weighted sum of the coefficients, \(c^\mathsf{T}\beta\), where \(c\) is a fixed vector of weights. The contrast estimate is \(c^\mathsf{T}\hat{\beta}\). For example, \(c=(1,-1)^\mathsf{T}\) gives the difference between two coefficients, \(\beta_1-\beta_2\).

For clarity, consider ordinary least squares with observations grouped into \(G\) clusters. Write \(X_g\) for the predictor matrix and \(e_g\) for the residual vector within cluster \(g\); the full predictor matrix is \(X\). The uncorrected cluster-robust covariance estimate has the sandwich form below. A small-sample correction, when used, multiplies this estimate by the chosen correction factor. In a test, use the same selected cluster-robust covariance estimate that is used for the corresponding standard errors.

This estimate allows residuals within a cluster to be dependent and to have unequal variances. It relies on the regression mean model being suitable for the question, independent clusters, adequate variation to estimate the coefficients, and a sufficiently large number of independent clusters for the cluster-robust approximation to be credible. It does not make observations within a cluster independent. With few clusters, the resulting inference can be unreliable even when the number of observations is large.

Let \(\widehat{V}_{\mathrm{CR}}\) denote the estimated covariance matrix, including any selected small-sample correction. The estimated variance of a contrast follows from the covariance rule for a weighted sum:

For a null value \(\delta_0\), the hypotheses are \(H_0:c^\mathsf{T}\beta=\delta_0\) and \(H_a:c^\mathsf{T}\beta\ne\delta_0\) for a two-sided test. The test statistic is the estimated contrast’s distance from the null value, measured in its cluster-robust standard errors. A common reference approximation is a \(t\) distribution with \(G-1\) degrees of freedom. This is an approximation, not an exact result for arbitrary clustered data; the chosen degrees of freedom and any small-sample correction should be reported.

Why Covariances Enter the Calculation

For two estimated coefficients, \(\hat{\beta}_1\) and \(\hat{\beta}_2\), the variance of their weighted sum is

This identity follows by expanding the variance of the sum: the two individual variance terms remain, and the cross-product appears twice, once in each order. Thus, a covariance can increase or decrease the contrast variance. The sign of its contribution depends on both the covariance and the contrast weights.

Testing a Difference Between Coefficients

A difference between two coefficients is a contrast with weights \(1\) and \(-1\). The covariance term is then subtracted twice when the covariance is positive. This reflects the fact that two positively related estimates tend to move together, which can make their difference more stable than either estimate alone. Conversely, a negative covariance can make their difference less stable.

Differences Between Fitted Effects

A fitted effect at a specified covariate profile is itself a linear combination of regression coefficients. For two profiles, subtracting their fitted effects produces another linear contrast. More generally, if the fitted effect at profile A is \(r_A^\mathsf{T}\beta\) and at profile B is \(r_B^\mathsf{T}\beta\), their difference is \((r_A-r_B)^\mathsf{T}\beta\). Its variance is calculated using the same full covariance matrix as any other contrast. This is why a difference in fitted effects should not be calculated by combining two standard errors as if the fitted effects were independent.

Calculation and Interpretation

A reliable analysis separates the scientific question from the arithmetic. First state the quantity of interest in context; then write it as a coefficient contrast. After confirming that the contrast is estimable, obtain its variance from the cluster-robust covariance matrix. Finally, test the specified null value using the chosen cluster-based reference approximation. The resulting \(p\)-value measures how unusual a statistic at least as far from the null as the observed statistic would be under the null and the approximation. It is not the probability that the null hypothesis is true.

The same contrast variance also supports a confidence interval: the estimate plus or minus an appropriate critical value times its cluster-robust standard error. The interval procedure and the test should use compatible degrees of freedom and covariance corrections. If a two-sided test rejects at level \(\alpha\), the corresponding \(1-\alpha\) interval excludes the null value, subject to using the same approximation.

Common Mistake: Dropping the Covariance

A frequent error is to calculate the variance of a difference as only the sum of the two coefficient variances, or to treat two fitted effects as independent. For example, in the slope comparison above, omitting covariance would give \(0.0400+0.0400=0.0800\), rather than the correct \(0.0500\). That would overstate the standard error and alter the test. With a negative covariance and same-sign contrast weights, the covariance contribution is negative and reduces the variance; the direction must be determined from the product of the weights and covariance, not guessed from the word “covariance.” Always use the full matrix calculation.