Deriving OLS Covariance Under Unequal Variance

Why the OLS Covariance Formula Changes

Least squares chooses coefficients to fit the observed outcomes, but the fitted coefficients also vary across hypothetical repetitions of the data. Their covariance describes that variation, conditional on the predictor values. When error variances differ from observation to observation, the uncertainty in the coefficients depends not just on the overall design but on which observations are noisy and how each observation contributes to the fit.

The key distinction is between the actual conditional covariance under a specified error-variance pattern and the simpler formula used when all errors have the same variance. The unequal-variance expression follows directly from the OLS estimator; it does not require a new fitting method.

Model and Assumptions

Definition: Let \(Y\) be the \(n \times 1\) vector of outcomes, \(X\) the \(n \times p\) design matrix, and \(\beta\) the \(p \times 1\) coefficient vector. The linear model is \(Y=X\beta+\varepsilon\). For the conditional covariance result below, assume \(X\) has full column rank, \(E(\varepsilon\mid X)=0\), and \(\operatorname{Cov}(\varepsilon\mid X)=\Omega=\operatorname{diag}(\sigma_1^2,\ldots,\sigma_n^2)\). Thus the error variance for observation \(i\), conditional on \(X\), is \(\sigma_i^2\), which may differ across observations; the diagonal covariance matrix also specifies that distinct errors have zero conditional covariance.

The design matrix is treated as fixed, or equivalently the calculations are conditional on its observed values. It contains a column of ones if the model includes an intercept. The full-rank condition makes \(X^\mathsf{T}X\) invertible, so the ordinary least-squares (OLS) estimator is

\hat{\beta}=(X^\mathsf{T}X)^{-1}X^\mathsf{T}Y.

No normality assumption is needed to derive this covariance. The conditional mean-zero assumption gives conditional unbiasedness, while the specified conditional covariance of the errors determines the covariance of \(\hat{\beta}\). If errors have nonzero conditional covariances, the same algebra uses their full conditional covariance matrix rather than a diagonal matrix; here we focus on unequal individual variances with zero covariance between distinct errors.

Deriving the Conditional Covariance

Substitute \(Y=X\beta+\varepsilon\) into the OLS expression. This separates the fixed coefficient vector from the part of the estimate that varies with the errors.

\hat{\beta} =(X^\mathsf{T}X)^{-1}X^\mathsf{T}(X\beta+\varepsilon) =\beta+(X^\mathsf{T}X)^{-1}X^\mathsf{T}\varepsilon.

Write \(A=(X^\mathsf{T}X)^{-1}X^\mathsf{T}\). Then \(\hat{\beta}=\beta+A\varepsilon\). Because \(E(\varepsilon\mid X)=0\), \(E(\hat{\beta}\mid X)=\beta\). Since \(\beta\) is fixed conditional on \(X\), subtracting the conditional mean leaves \(A\varepsilon\). Applying the definition of covariance gives the result:

\begin{aligned} \operatorname{Cov}(\hat{\beta}\mid X) &=E\!\left[(\hat{\beta}-\beta)(\hat{\beta}-\beta)^\mathsf{T}\mid X\right]\\ &=E\!\left[A\varepsilon\varepsilon^\mathsf{T}A^\mathsf{T}\mid X\right]\\ &=A\operatorname{Cov}(\varepsilon\mid X)A^\mathsf{T}\\ &=(X^\mathsf{T}X)^{-1}X^\mathsf{T}\Omega X(X^\mathsf{T}X)^{-1}. \end{aligned}

In the last line, \(A^\mathsf{T}=X(X^\mathsf{T}X)^{-1}\), because \(X^\mathsf{T}X\) and its inverse are symmetric. The expression has size \(p \times p\), as required for the covariance matrix of a \(p\)-coefficient vector. Its diagonal entries are the conditional variances of the estimated coefficients; its off-diagonal entries are their conditional covariances.

Theorem: Under the stated model and assumptions, \(\operatorname{Cov}(\hat{\beta}\mid X)=(X^\mathsf{T}X)^{-1}X^\mathsf{T}\Omega X(X^\mathsf{T}X)^{-1}\). If every conditional error variance is the same, so that \(\Omega=\sigma^2 I\), this reduces to the homoskedastic formula \(\operatorname{Cov}(\hat{\beta}\mid X)=\sigma^2(X^\mathsf{T}X)^{-1}\).

For the reduction, substitute \(\Omega=\sigma^2 I\) into the unequal-variance expression. The middle product becomes \(X^\mathsf{T}(\sigma^2I)X=\sigma^2X^\mathsf{T}X\); multiplying by the two outer inverse matrices leaves \(\sigma^2(X^\mathsf{T}X)^{-1}\). The familiar formula therefore depends on a common conditional variance, not merely on the use of OLS.

Worked Example: A Three-Point Simple Regression

Suppose a hypothetical simple regression has an intercept and one predictor, with predictor values \(x=0,1,2\). Assume the three conditional error variances, in squared outcome units, are \(1,4,1\), respectively, and the errors have zero conditional means and zero covariances with one another. Find the conditional covariance matrix of the OLS intercept and slope.

The design matrix and diagonal error covariance matrix are

X= \begin{pmatrix} 1&0\\ 1&1\\ 1&2 \end{pmatrix}, \qquad \Omega= \begin{pmatrix} 1&0&0\\ 0&4&0\\ 0&0&1 \end{pmatrix}.

First, \(X^\mathsf{T}X=\begin{pmatrix}3&3\\3&5\end{pmatrix}\), whose inverse is \(\frac{1}{6}\begin{pmatrix}5&-3\\-3&3\end{pmatrix}\). Therefore the matrix \(A=(X^\mathsf{T}X)^{-1}X^\mathsf{T}\), which gives the weights on the three outcomes for each coefficient, is

A= \begin{pmatrix} 5/6&1/3&-1/6\\ -1/2&0&1/2 \end{pmatrix}.

Each coefficient is a weighted sum of the outcomes. With uncorrelated errors, the covariance matrix is the sum of each observation's variance times the outer product of its weight column. In this case,

\begin{aligned} \operatorname{Cov}(\hat{\beta}\mid X) &=A\Omega A^\mathsf{T}\\ &=1 \begin{pmatrix}5/6\\-1/2\end{pmatrix} \begin{pmatrix}5/6&-1/2\end{pmatrix} +4 \begin{pmatrix}1/3\\0\end{pmatrix} \begin{pmatrix}1/3&0\end{pmatrix} +1 \begin{pmatrix}-1/6\\1/2\end{pmatrix} \begin{pmatrix}-1/6&1/2\end{pmatrix}\\ &= \begin{pmatrix} 7/6&-1/2\\ -1/2&1/2 \end{pmatrix}. \end{aligned}

For example, the intercept variance is \(25/36+4(1/9)+1/36=42/36=7/6\). The covariance is \((5/6)(-1/2)+4(1/3)(0)+(-1/6)(1/2)=-1/2\), and the slope variance is \(1/4+4(0)+1/4=1/2\). These are conditional quantities: they describe coefficient variation for this design and variance pattern.

If instead all three error variances were a common \(\sigma^2\), the formula would be \(\frac{\sigma^2}{6}\begin{pmatrix}5&-3\\-3&3\end{pmatrix}\). That is not the covariance matrix obtained from the unequal variances above.

How the Design and Variances Work Together

The matrix expression can also be read observation by observation. Let \(a_i\) be column \(i\) of \(A=(X^\mathsf{T}X)^{-1}X^\mathsf{T}\). Then \(\hat{\beta}=\sum_{i=1}^n a_iY_i\). Under the diagonal error-covariance assumption,

\operatorname{Cov}(\hat{\beta}\mid X) =\sum_{i=1}^n \sigma_i^2 a_i a_i^\mathsf{T}.

Thus an observation matters through both its conditional variance and its coefficient weights. A large variance contributes more, but its effect on a particular coefficient also depends on that observation's position in the design. Covariances between estimated coefficients can be positive, negative, or zero because the components of the weight vectors may have different signs.

Worked Example: Unequal Variances Can Create Coefficient Covariance

Consider a hypothetical regression with an intercept and predictor values \(-1,-1,1,1\). The conditional error variances are \(1,4,9,16\), in that order. Assume conditional mean-zero errors with zero covariances across observations. Calculate the covariance matrix.

The columns of \(X\) are orthogonal: their sums are \(4\) and \(0\), and the sum of squared predictor values is \(4\). Hence \(X^\mathsf{T}X=\begin{pmatrix}4&0\\0&4\end{pmatrix}\). The coefficient weight vectors are \((1/4,1/4,1/4,1/4)\) for the intercept and \((-1/4,-1/4,1/4,1/4)\) for the slope.

The intercept variance is the sum of the four error variances times \(1/16\): \( (1+4+9+16)/16=30/16=15/8\). The slope variance has the same value, since each slope weight also has squared value \(1/16\). Their covariance uses the products of the intercept and slope weights: \( (-1-4+9+16)/16=20/16=5/4\). Therefore,

\operatorname{Cov}(\hat{\beta}\mid X) = \begin{pmatrix} 15/8&5/4\\ 5/4&15/8 \end{pmatrix}.

Although the design columns are orthogonal, the coefficient estimates are not conditionally uncorrelated under this variance pattern. The observations at \(x=1\), which have the larger variances, contribute with positive weights to both estimates, producing positive covariance. With a common error variance, the homoskedastic formula would instead give zero off-diagonal covariance because \(X^\mathsf{T}X\) is diagonal.

Worked Example: The Simple-Regression Slope Variance

A hypothetical simple regression records predictor values \(1,2,4\) hours. Suppose the conditional error variances are \(1,9,4\) squared outcome units, with zero conditional error means and zero covariances between distinct errors. Find the conditional variance of the estimated slope.

For simple regression with an intercept, define \(\bar{x}\) as the predictor mean and \(S_{xx}=\sum_i(x_i-\bar{x})^2\). The OLS slope can be written as \(\hat{\beta}_1=\sum_i (x_i-\bar{x})Y_i/S_{xx}\). Because the errors are uncorrelated, its conditional variance is the sum of the squared weights times the individual error variances:

\operatorname{Var}(\hat{\beta}_1\mid X) =\frac{\sum_i (x_i-\bar{x})^2\sigma_i^2}{S_{xx}^2}.

Here \(\bar{x}=(1+2+4)/3=7/3\), and \(S_{xx}=(-4/3)^2+(-1/3)^2+(5/3)^2=42/9=14/3\). The three slope weights are \((-4/3)/(14/3)=-2/7\), \((-1/3)/(14/3)=-1/14\), and \((5/3)/(14/3)=5/14\). Substitution gives

\begin{aligned} \operatorname{Var}(\hat{\beta}_1\mid X) &=\left(-\frac{2}{7}\right)^2(1) +\left(-\frac{1}{14}\right)^2(9) +\left(\frac{5}{14}\right)^2(4)\\ &=\frac{16+9+100}{196} =\frac{125}{196} \approx 0.6378. \end{aligned}

The variance is in squared outcome-units-per-hour. The observations do not contribute equally: the point at \(x=4\) has a comparatively large slope weight and a conditional variance of \(4\), while the point at \(x=2\) has a small slope weight even though its conditional variance is largest.

What the Formula Does—and Does Not—Say

The unequal-variance formula is a statement about the sampling covariance of OLS under the specified model. It shows that OLS can remain conditionally unbiased when variances differ, while its covariance need not equal the homoskedastic expression. It does not, by itself, provide a practical estimate of the unknown \(\sigma_i^2\). The calculations above treat those variances as specified hypothetical quantities in order to show what determines the covariance.

The result also does not change the mean model \(E(Y\mid X)=X\beta\), guarantee that this model is appropriate, or establish that observed differences are caused by predictors. A covariance calculation describes uncertainty conditional on the design and the assumptions; it cannot repair a misspecified mean relationship.

Common Mistake

Pitfall: “OLS was used, so the coefficient covariance must be \(\sigma^2(X^\mathsf{T}X)^{-1}\).” That conclusion silently assumes every observation has the same conditional error variance. Under unequal variances, the appropriate theoretical expression is \((X^\mathsf{T}X)^{-1}X^\mathsf{T}\Omega X(X^\mathsf{T}X)^{-1}\). The common-variance formula follows only when \(\Omega=\sigma^2I\).

A related oversight is to look only at the diagonal entries of the covariance matrix. Those entries give individual coefficient variances, but the off-diagonal entries describe how coefficient estimates vary together. Both parts are needed to understand the full joint uncertainty in the coefficient vector.

Check Your Understanding

Use the model and assumptions in this tutorial to answer the following questions.

Which assumption allows the conditional covariance of the coefficient vector to be written using a diagonal matrix \(\Omega\)?
Starting from \(\hat{\beta}=\beta+A\varepsilon\), explain why its conditional covariance is \(A\Omega A^\mathsf{T}\).
What condition on the individual error variances reduces the unequal-variance expression to \(\sigma^2(X^\mathsf{T}X)^{-1}\)?
In the four-point example, why can orthogonal design columns coexist with nonzero covariance between the estimated intercept and slope?
For simple regression, what two features determine each observation's contribution to the slope variance?