The GLS Model With Structured Error Covariance

Why Model the Error Covariance?

In linear regression, the pattern of error variation matters as well as its average. Errors may have unequal variances, or observations may have correlated errors. When a covariance structure is specified, generalized least squares (GLS) uses it to form a coefficient estimate that accounts for that structure. This is different from retaining the ordinary least-squares estimate and changing only its reported standard errors.

The key object is the conditional covariance matrix of the errors. Its diagonal entries describe error variances, while its off-diagonal entries describe covariances between observations. The inverse covariance matrix determines how the GLS objective weights residuals and their joint patterns.

Definition: The linear model with structured error covariance is $$ y=X\beta+u,\qquad E[u\mid X]=0,\qquad \operatorname{Var}(u\mid X)=\Omega. $$ Here $y$ is an $n$-vector of outcomes, $X$ is an $n\times k$ design matrix, $\beta$ is a $k$-vector of coefficients, and $u$ is an $n$-vector of errors. The matrix $\Omega$ is symmetric positive definite and represents the specified conditional covariance structure.

Conditions and the GLS Estimator

The conditional mean restriction $E[u\mid X]=0$ is the exogeneity condition used here. It says that, conditional on the design matrix, the errors have mean zero; it does not say that the errors are independent or have equal variances. The covariance restriction describes their second moments and may include unequal variances or dependence across observations. A covariance matrix alone does not specify the full probability distribution of the errors, and normality is not needed to define the GLS coefficient estimator.

For the formula below, assume that $\Omega$ is known and positive definite, and that $X$ has full column rank $k$. Positive definiteness means $a'\Omega a>0$ for every nonzero vector $a$. It ensures that $\Omega^{-1}$ exists and is also positive definite. Full column rank ensures that $X'\Omega^{-1}X$ is invertible. The covariance structure is called structured when its entries follow a specified pattern, such as unequal diagonal variances, dependence within groups, or a common covariance pattern across observations.

Definition: Given these conditions, the GLS estimator minimizes the covariance-weighted sum of squared residuals, $$ Q(b)=(y-Xb)'\Omega^{-1}(y-Xb), $$ over coefficient vectors $b$. Its unique minimizer is $$ \hat{\beta}_{GLS}=(X'\Omega^{-1}X)^{-1}X'\Omega^{-1}y. $$

This expression shows how the assumed covariance structure enters the estimator: it is the inverse covariance matrix, not the covariance matrix itself, that weights the residuals. If $\Omega$ is multiplied by a positive scalar, that scalar cancels from the coefficient formula. Thus only the relative covariance pattern determines the GLS coefficients, though the scale still matters for their uncertainty.

Why the Formula Minimizes the GLS Criterion

The minimization can be verified directly. Since $\Omega^{-1}$ is symmetric, expanding the criterion gives $Q(b)=y'\Omega^{-1}y-2b'X'\Omega^{-1}y+b'X'\Omega^{-1}Xb$. Differentiating with respect to $b$ and setting the gradient to zero gives the GLS normal equations. Full column rank and positive definiteness make their coefficient matrix positive definite, so the solution is the unique minimum.

\begin{aligned} \nabla_b Q(b)&=-2X'\Omega^{-1}y+2X'\Omega^{-1}Xb,\\ X'\Omega^{-1}X\hat{\beta}_{GLS}&=X'\Omega^{-1}y,\\ \hat{\beta}_{GLS}&=(X'\Omega^{-1}X)^{-1}X'\Omega^{-1}y. \end{aligned}

The same model conditions also give a useful description of the estimator's sampling behavior, conditional on $X$ and on the specified $\Omega$. Substituting $y=X\beta+u$ into the estimator shows that its conditional expectation is $\beta$. Its conditional covariance is obtained by applying the covariance rule for a linear transformation of $u$.

\begin{aligned} E[\hat{\beta}_{GLS}\mid X] &=(X'\Omega^{-1}X)^{-1}X'\Omega^{-1}X\beta=\beta,\\ \operatorname{Var}(\hat{\beta}_{GLS}\mid X) &=(X'\Omega^{-1}X)^{-1}X'\Omega^{-1} \Omega\Omega^{-1}X(X'\Omega^{-1}X)^{-1}\\ &=(X'\Omega^{-1}X)^{-1}. \end{aligned}

These statements concern the model with the stated covariance matrix. They do not establish that GLS is more efficient than other linear unbiased estimators; that comparison requires a separate result. In applications, the covariance matrix may also involve unknown parameters, but the examples here treat it as known to make the role of its structure explicit.

Worked Example: Diagonal Covariance and a Two-Coefficient Model

Suppose a hypothetical dataset has three observations, an intercept and one regressor, and a known diagonal error covariance. The values are $x=(0,5,1)'$, $y=(0,20/3,16/9)'$, and $\Omega=\operatorname{diag}(30/11,20,4/3)$. The design matrix has rows $(1,x_i)$. Assume $E[u\mid X]=0$. The diagonal entries are positive, so $\Omega$ is positive definite; the two design columns are linearly independent, so $X$ has full column rank.

The inverse covariance and the required products are

\Omega^{-1}=\operatorname{diag}(11/30,1/20,3/4),\qquad X'\Omega^{-1}X= \begin{pmatrix}7/6&1\\1&2\end{pmatrix}.

For the cross-product with the outcome, first multiply by $\Omega^{-1}$: $\Omega^{-1}y=(0,1/3,4/3)'$. Therefore

X'\Omega^{-1}y= \begin{pmatrix}5/3\\3\end{pmatrix},\qquad (X'\Omega^{-1}X)^{-1}= \begin{pmatrix}3/2&-3/4\\-3/4&7/8\end{pmatrix}.

Multiplying gives $\hat{\beta}_{GLS}=(1/4,11/8)'$: the estimated intercept is $0.25$ outcome units, and the estimated change in the conditional mean is $1.375$ outcome units per regressor unit. For example, the intercept component is $(3/2)(5/3)-(3/4)(3)=5/2-9/4=1/4$; the slope component is $-(3/4)(5/3)+(7/8)(3)=-5/4+21/8=11/8$.

The diagonal structure means that the errors are uncorrelated across these observations under the specified model, while their variances differ. The reciprocal variances appear on the diagonal of $\Omega^{-1}$, so observations with smaller error variances receive larger diagonal weights in the GLS criterion. This interpretation relies on the stated covariance model, not merely on the observed outcome values.

Worked Example: Exchangeable Covariance for an Intercept

Consider three hypothetical outcomes $y=(10,13,16)'$ and an intercept-only design $X=\mathbf{1}$, a column of three ones. Suppose the conditional error covariance is $\Omega=\begin{pmatrix}4&2&2\\2&4&2\\2&2&4\end{pmatrix}=2I+2\mathbf{1}\mathbf{1}'$. Its eigenvalues are $8,2,2$, all positive, so it is positive definite. The intercept-only design has full column rank, and assume conditional mean zero.

The inverse is $\Omega^{-1}=\frac12 I-\frac18\mathbf{1}\mathbf{1}'$. Each row of this inverse sums to $1/8$, so $\mathbf{1}'\Omega^{-1}\mathbf{1}=3/8$. Also, the entries of $\Omega^{-1}y$ sum to $39/8$. Hence

\hat{\beta}_{GLS} =(\mathbf{1}'\Omega^{-1}\mathbf{1})^{-1} \mathbf{1}'\Omega^{-1}y =\frac{39/8}{3/8}=13.

In this balanced exchangeable case, GLS estimates the common mean by the arithmetic average, $ (10+13+16)/3=13$. That does not mean the covariance structure is irrelevant: the conditional variance of this estimate is $(\mathbf{1}'\Omega^{-1}\mathbf{1})^{-1}=8/3$ outcome-units squared. Treating the observations as independent with variance $4$ would instead give variance $4/3$ for the average, which does not reflect the specified positive covariance between observations.

Worked Example: Dependence Within Two Groups

Suppose four hypothetical observations belong to two groups, with outcomes $y=(2,4,5,7)'$. The design includes an intercept and an indicator for group B, so its rows are $(1,0),(1,0),(1,1),(1,1)$. Assume conditional mean zero and the block covariance

\Omega= \begin{pmatrix} 2&1&0&0\\ 1&2&0&0\\ 0&0&2&1\\ 0&0&1&2 \end{pmatrix}.

Each two-by-two block has eigenvalues $3$ and $1$; consequently the full matrix is positive definite. Both groups are observed, so the intercept and group-indicator columns are linearly independent. The inverse of each block is $\frac13\begin{pmatrix}2&-1\\-1&2\end{pmatrix}$. Applying this inverse to the design and outcome gives

X'\Omega^{-1}X= \begin{pmatrix}4/3&2/3\\2/3&2/3\end{pmatrix}, \qquad X'\Omega^{-1}y= \begin{pmatrix}6\\4\end{pmatrix}.

Solving the normal equations yields $\hat{\beta}_{GLS}=(3,3)'$. The fitted mean for group A is $3$, and the fitted mean for group B is $3+3=6$. The block structure represents positive covariance between observations in the same group and zero covariance between groups. Here the two observations within each group have the same covariance pattern, and the estimate of each group mean equals its arithmetic average: $3$ for A and $6$ for B. The example illustrates how a structured covariance can encode dependence without changing these particular group-mean estimates.

Reading the Covariance Structure

The examples illustrate several general points. A diagonal $\Omega$ allows observation-specific variances. Off-diagonal entries represent conditional covariance, and blocks can express dependence limited to particular groups. The inverse matrix translates that pattern into the GLS criterion: a residual is not always assessed independently of the other residuals. When errors are correlated, their joint residual pattern matters.

GLS therefore depends on a model for the errors as well as a model for the conditional mean. Exogeneity concerns the conditional mean of the errors; positive definiteness concerns whether the covariance structure defines valid, nondegenerate variation. Neither condition alone implies the other. A design matrix also needs sufficient rank: even a valid covariance matrix cannot identify coefficients that the columns of $X$ do not distinguish.

Common Pitfall

A common mistake is to reason, “Ordinary least squares has already produced the coefficients, so GLS only changes the standard errors.” That is generally wrong. GLS minimizes a different criterion, and its coefficient estimate can differ from ordinary least squares whenever the covariance weighting changes the fit. In contrast, a cluster-robust or HAC covariance estimator typically changes the estimated uncertainty while leaving the original coefficient estimate unchanged. These are distinct approaches: covariance-robust inference does not by itself apply GLS weights.

Another pitfall is to treat a plausible covariance pattern as known just because it has a convenient form. The formulas above describe GLS conditional on the specified $\Omega$. If that matrix is only an approximation, the interpretation of the weights and the sampling properties need to be assessed in relation to the actual error process. Specifying a covariance structure is a modeling choice, not a consequence of observing correlated residuals.

Key takeaway: In the GLS model, conditional exogeneity identifies the target conditional mean, while a symmetric positive-definite $\Omega$ describes error variation and dependence. Its inverse determines the covariance-weighted coefficient estimate, provided the design matrix has full column rank.

Check Your Understanding

Use the definitions and calculations above to answer these questions.

What does $E[u\mid X]=0$ assert, and why does it not imply that the errors are independent?
Why must $\Omega$ be positive definite for the GLS formula as stated?
In the diagonal-covariance example, calculate the GLS slope contribution from the displayed inverse matrix and $X'\Omega^{-1}y$.
In the exchangeable-covariance example, what is the conditional variance of the estimated intercept, and what covariance feature makes it differ from the independence calculation?
Explain why changing from ordinary least squares to GLS can change coefficient estimates, whereas changing only to a cluster-robust covariance estimate does not change the fitted coefficients.