Robust Covariance for Linear Contrasts

Why Linear Combinations Need the Full Covariance Matrix

A regression question often concerns more than one coefficient at a time. An analyst may want to compare two slopes, add two components of a fitted effect, or calculate a specified combination of several coefficients. Each quantity is a linear contrast of the coefficient vector. Its uncertainty depends not only on the separate variances of the coefficient estimates, but also on how those estimates vary together.

The heteroskedasticity-robust covariance matrix records both kinds of information. This tutorial uses that full matrix to obtain a contrast’s standard error and an asymptotic confidence interval. The central calculation is simple to state, but omitting its off-diagonal entries can change the answer substantially.

Definition: Let \(\hat{\beta}\) be a \(k\)-dimensional regression coefficient estimator and let \(c\) be a specified \(k\)-dimensional vector of constants. The linear contrast is \(c'\beta\), and its estimator is \(c'\hat{\beta}\). If \(\widehat{V}\) estimates the covariance matrix of \(\hat{\beta}\), the estimated variance and standard error of the contrast estimator are \(c'\widehat{V}c\) and \(\sqrt{c'\widehat{V}c}\), respectively.

The Robust Covariance Matrix and Contrast Variance

For clarity, consider the linear regression \(y_i=x_i'\beta+u_i\), where \(x_i\) is a \(k\)-dimensional column vector, \(i=1,\ldots,n\), and the design matrix has full column rank. OLS estimates \(\beta\) by \(\hat{\beta}\), and \(\hat{u}_i=y_i-x_i'\hat{\beta}\) is the residual for observation \(i\). The HC0 heteroskedasticity-robust covariance estimator is

\widehat{V}_{HC0} = (X'X)^{-1} \left(\sum_{i=1}^n x_i x_i'\hat{u}_i^2\right) (X'X)^{-1}.

This estimator allows the conditional variance of \(u_i\) to differ across observations. In particular, it does not replace each \(\hat{u}_i^2\) with a common variance estimate. Its consistency relies on regularity conditions for OLS and the robust covariance estimator, including independent sampling (or suitable dependence conditions), finite moments, and a nonsingular limiting regressor second-moment matrix. For this tutorial, assume those conditions hold. The conditional error variance need not be constant.

The matrix \(\widehat{V}_{HC0}\) has one row and one column for every coefficient. The diagonal entries estimate the individual coefficient variances; an off-diagonal entry estimates covariance between a pair of coefficient estimates. For \(c'\hat{\beta}\), every entry can matter. The full variance expansion is

c'\widehat{V}_{HC0}c = \sum_{j=1}^k\sum_{\ell=1}^k c_jc_\ell\widehat{V}_{HC0,j\ell}.

A positive covariance contributes positively when the two contrast weights have the same sign, and negatively when their signs differ. A negative covariance reverses those effects. Thus the covariance terms can make a contrast more or less precise than a calculation based only on the separate coefficient variances would suggest.

Why the Contrast Formula Works

The variance rule follows directly from the covariance matrix definition. For any random vector \(Z\) with covariance matrix \(V\), the scalar \(c'Z\) has variance \(c'Vc\). Applying this rule to the sampling variation of \(\hat{\beta}\) gives the contrast variance; replacing the unknown covariance matrix by its robust estimate gives the estimated variance.

The regression representation also shows how the robust estimator enters. With \(X\) containing the rows \(x_i'\), OLS satisfies \(\hat{\beta}-\beta=(X'X)^{-1}\sum_i x_i u_i\). Write \(Q_n=X'X/n\). Under the stated regularity conditions, the normalized sum of \(x_i u_i\) has an asymptotic covariance determined by the variation in \(x_i u_i\). In a random-design formulation, its population “meat” is \(\Omega=E[u_i^2x_ix_i']\), while \(Q=E[x_ix_i']\). Consequently, the asymptotic covariance of \(\sqrt{n}(\hat{\beta}-\beta)\) is \(Q^{-1}\Omega Q^{-1}\). The HC0 estimator replaces these population components with their sample counterparts, producing the sandwich matrix above. For a contrast, premultiplying by \(c'\) and postmultiplying by \(c\) gives the corresponding scalar covariance.

Result: If the robust covariance estimator consistently estimates the asymptotic covariance of \(\hat{\beta}\), then \(c'\widehat{V}_{HC0}c\) estimates the sampling variance of \(c'\hat{\beta}\). Under asymptotic normality, an approximate \(100(1-\alpha)\%\) confidence interval for \(c'\beta\) is \(c'\hat{\beta}\ \pm\ z_{1-\alpha/2}\sqrt{c'\widehat{V}_{HC0}c}\), where \(z_{1-\alpha/2}\) is the corresponding standard normal critical value.

Worked Contrasts

Worked Example: Difference Between Two Coefficients

Suppose a hypothetical regression reports two coefficient estimates, \(\hat{\beta}_1=1.40\) and \(\hat{\beta}_2=0.700\), measured in outcome points per unit of their respective regressors. The relevant block of the HC0 covariance estimate is

\widehat{V} = \begin{pmatrix} 0.160 & 0.072\\ 0.072 & 0.090 \end{pmatrix}.

The target is \(\beta_1-\beta_2\), so \(c=(1,-1)'\) and the point estimate is \(1.40-0.700=0.700\) points per unit. Applying the contrast formula gives

\widehat{\operatorname{Var}}(\hat{\beta}_1-\hat{\beta}_2) = 0.160+0.090-2(0.072) = 0.106, \qquad SE=\sqrt{0.106}=0.3256.

The negative sign on the covariance term comes from multiplying the contrast weights \(1\) and \(-1\). Using the 1.96 standard-normal critical value, the 95% asymptotic interval, calculated with unrounded values, is

0.700\pm1.96\sqrt{0.106} = 0.700\pm0.6381 = [0.0619,\ 1.3381]\text{ points per unit}.

Under the model and asymptotic approximation, this interval gives a range of plausible values for the difference in the two coefficients. Its endpoints are just above zero, so the interval indicates a small positive estimated difference at this confidence level. It does not mean there is a 95% probability that this particular fixed coefficient difference lies inside the calculated interval.

Worked Example: Sum of Two Coefficients

In a second hypothetical regression, two coefficients measured on the same outcome scale have estimates \(\hat{\beta}_1=0.400\) and \(\hat{\beta}_2=0.300\). Suppose their robust covariance block is

\widehat{V} = \begin{pmatrix} 0.040 & 0.030\\ 0.030 & 0.090 \end{pmatrix}.

For the sum \(\beta_1+\beta_2\), use \(c=(1,1)'\). The estimate is \(0.400+0.300=0.700\) outcome units. This time the covariance contribution is positive:

\widehat{\operatorname{Var}}(\hat{\beta}_1+\hat{\beta}_2) = 0.040+0.090+2(0.030) = 0.190, \qquad SE=\sqrt{0.190}=0.4359.

The approximate 95% confidence interval is

0.700\pm1.96\sqrt{0.190} = 0.700\pm0.8543 = [-0.1543,\ 1.5543]\text{ outcome units}.

The positive covariance increases the variance of this sum relative to the variance \(0.040+0.090=0.130\) that would result from incorrectly treating the estimates as uncorrelated. The interval describes uncertainty in the combined coefficient, not two separate intervals for the individual coefficients.

Worked Example: A Contrast of Three Coefficients

Consider a hypothetical model with coefficient estimates \(\hat{\beta}=(2.00,\ 0.500,\ -0.200)'\). Suppose the full robust covariance estimate for these coefficients is

\widehat{V} = \begin{pmatrix} 0.040 & 0.010 & 0\\ 0.010 & 0.090 & -0.015\\ 0 & -0.015 & 0.160 \end{pmatrix}.

Suppose the target is \(\beta_1-2\beta_2+\beta_3\), with all three coefficients on compatible scales. Here \(c=(1,-2,1)'\), and the estimated contrast is \(2.00-2(0.500)-0.200=0.800\) outcome units. The matrix-vector product is

\widehat{V}c = \begin{pmatrix} 0.020\\ -0.185\\ 0.190 \end{pmatrix}, \qquad c'\widehat{V}c = 0.020+(-2)(-0.185)+0.190 = 0.580.

Therefore \(SE=\sqrt{0.580}=0.7616\) outcome units. Using unrounded values, the approximate 95% interval is

0.800\pm1.96\sqrt{0.580} = 0.800\pm1.4927 = [-0.6927,\ 2.2927]\text{ outcome units}.

This example illustrates why the contrast vector is useful: it specifies the target, while the same full covariance matrix supplies its uncertainty. The matrix multiplication includes the variances and all pairwise covariance contributions without requiring a separate rule for each possible combination.

Common Mistake: Dropping the Covariance Terms

A frequent error is to calculate a contrast variance by adding or subtracting coefficient variances alone. Variances do not subtract when taking a difference, and the coefficient estimates are not generally uncorrelated. For a difference, the correct expression is \(\widehat{V}_{11}+\widehat{V}_{22}-2\widehat{V}_{12}\); for a sum, it is \(\widehat{V}_{11}+\widehat{V}_{22}+2\widehat{V}_{12}\). For a longer contrast, use the full matrix product \(c'\widehat{V}c\).

In the difference example, ignoring the covariance would give an estimated variance of \(0.160+0.090=0.250\), rather than \(0.106\). That would produce an SE of \(0.500\), rather than \(0.3256\), and therefore a different interval. The mistake is not fixed by making a more careful rounding choice: it is a different, incorrect variance calculation.

Key takeaway: First write the target as \(c'\beta\). Then calculate its estimate as \(c'\hat{\beta}\), its robust variance as \(c'\widehat{V}c\), and its standard error as the square root of that variance. Retain the off-diagonal covariance entries throughout.

Check Your Understanding

Use the full covariance matrix whenever the target combines multiple coefficients.

For a contrast \(\beta_1-\beta_2\), what contrast vector should be used, and why does the covariance term enter with a minus sign?
If two coefficient estimates have variances \(0.08\) and \(0.12\), and covariance \(0.02\), calculate the estimated variance of their sum.
In what situation would the variance of a difference be larger than the sum of the two individual variances?
Explain why the diagonal entries of a robust covariance matrix are not sufficient to calculate the standard error of a general linear contrast.
What does a 95% asymptotic confidence interval for a coefficient contrast mean, and what does it not mean?