Why the Covariance Assumption Matters
Generalized least squares (GLS) uses a specified covariance structure for the regression errors to construct an estimator of the regression coefficients. When that structure is the true conditional covariance, GLS is not merely a way to obtain a different estimate: it has a precise efficiency guarantee. Among all estimators that are linear in the observed response and unbiased for every coefficient vector, GLS has the smallest covariance matrix in the positive-semidefinite ordering.
The qualification about the covariance structure is essential. The result is a finite-sample statement under a correctly specified error covariance, not a claim that any weighting scheme improves on ordinary least squares (OLS). If the covariance used to construct GLS is wrong, the estimator may remain unbiased under an appropriate mean assumption, but the Gauss–Markov efficiency guarantee no longer applies.
Model and Assumptions
Full column rank ensures that the coefficients are identified from the design: \(X'X\) and \(X'\Omega^{-1}X\) are invertible. Positive definiteness of \(\Omega\) ensures that the inverse covariance \(\Omega^{-1}\) exists. The conditional mean assumption makes the GLS estimator unbiased, because its coefficient on \(y\) satisfies \((X'\Omega^{-1}X)^{-1}X'\Omega^{-1}X=I_k\).
Here “linear estimator” means an estimator of the form \(Ay\), where \(A\) is a nonrandom \(k\times n\) matrix conditional on \(X\). Such an estimator is unbiased for every \(\beta\) under the stated mean model exactly when \(AX=I_k\). The comparison is among these linear unbiased estimators; it is not a comparison with every conceivable estimator, including biased or nonlinear ones.
The Gauss–Markov Efficiency Result for GLS
The positive-semidefinite comparison is the precise meaning of “best” here. It implies that each individual GLS coefficient has variance no greater than the corresponding coefficient from another linear unbiased estimator. It also compares every linear combination of coefficients, including contrasts. It does not require the two covariance matrices to have a component-by-component ordering in every off-diagonal entry.
Why the Covariance Difference Is Nonnegative
Write the GLS estimator as \(A_{GLS}y\), where \(A_{GLS}=(X'\Omega^{-1}X)^{-1}X'\Omega^{-1}\). Let \(Ay\) be any other linear unbiased estimator, and define \(D=A-A_{GLS}\). Both estimators are unbiased, so \(AX=I_k\) and \(A_{GLS}X=I_k\). Subtracting gives \(DX=0\).
The covariance of any linear estimator \(By\) is \(B\Omega B'\), conditional on \(X\). Expanding the covariance for \(A=A_{GLS}+D\) gives $$ A\Omega A' =A_{GLS}\Omega A_{GLS}' +A_{GLS}\Omega D' +D\Omega A_{GLS}' +D\Omega D'. $$ The two cross terms are zero. In fact, $$ A_{GLS}\Omega D' =(X'\Omega^{-1}X)^{-1}X'D' =(X'\Omega^{-1}X)^{-1}(DX)' =0, $$ and the other cross term is its transpose. Therefore, $$ \operatorname{Var}(Ay\mid X)-\operatorname{Var}(\hat{\beta}_{GLS}\mid X) =D\Omega D'. $$ For any vector \(c\), \(c'D\Omega D'c=(D'c)'\Omega(D'c)\geq 0\), because \(\Omega\) is positive definite. Thus \(D\Omega D'\) is positive semidefinite, proving the theorem.
This argument also identifies when another estimator ties GLS in a particular direction: its variance difference for \(c'\beta\) is zero exactly when \(D'c=0\). A different linear unbiased estimator can therefore have the same variance as GLS for some contrasts while having a larger variance for others.
Worked Example: GLS and OLS for a Heteroskedastic Intercept
Suppose two hypothetical observations follow \(y_i=\mu+u_i\), with independent errors satisfying \(E(u_i)=0\), \(\operatorname{Var}(u_1)=1\), and \(\operatorname{Var}(u_2)=4\). The covariance matrix is known to be \(\Omega=\begin{pmatrix}1&0\\0&4\end{pmatrix}\), which is positive definite, and the design matrix is a column of ones with full rank. For observed values \(y_1=2\) and \(y_2=6\), compare GLS with OLS.
In this intercept-only model, GLS is the inverse-variance weighted mean: $$ \hat{\mu}_{GLS} =\frac{y_1/1+y_2/4}{1/1+1/4} =\frac{2+6/4}{1+1/4} =\frac{3.5}{1.25} =2.800. $$ The GLS variance is the reciprocal of the total weight: $$ \operatorname{Var}(\hat{\mu}_{GLS})=\frac{1}{1+1/4}=0.8000. $$
OLS gives the arithmetic mean, \(\hat{\mu}_{OLS}=(2+6)/2=4.000\). Its variance, using independence and the stated variances, is $$ \operatorname{Var}(\hat{\mu}_{OLS})=\frac{1^2+4}{2^2}=\frac{5}{4}=1.250. $$ Both estimators are unbiased for \(\mu\), but GLS has lower variance here because its weights use the correct covariance. The observed estimates differ; that difference is not itself evidence of bias or inefficiency.
Worked Example: Comparing Covariance Matrices in a Regression
Consider a hypothetical regression with three observations and a constant and a slope regressor. Let $$ X=\begin{pmatrix}1&-1\\1&0\\1&1\end{pmatrix}, \qquad \Omega=\begin{pmatrix}1&0&0\\0&4&0\\0&0&1\end{pmatrix}, \qquad y=\begin{pmatrix}1\\2\\5\end{pmatrix}. $$ Assume the errors have conditional mean zero and the stated known covariance. The two columns of \(X\) are linearly independent, and all diagonal entries of \(\Omega\) are positive, so the rank and positive-definiteness conditions hold.
The GLS cross-product matrix is $$ X'\Omega^{-1}X =\begin{pmatrix}9/4&0\\0&2\end{pmatrix}, $$ so the conditional covariance of GLS is $$ \operatorname{Var}(\hat{\beta}_{GLS}\mid X) =(X'\Omega^{-1}X)^{-1} =\begin{pmatrix}4/9&0\\0&1/2\end{pmatrix}. $$ The GLS coefficient estimate is $$ \hat{\beta}_{GLS} =\begin{pmatrix}9/4&0\\0&2\end{pmatrix}^{-1} X'\Omega^{-1}y =\begin{pmatrix}26/9\\2\end{pmatrix} \approx \begin{pmatrix}2.889\\2.000\end{pmatrix}. $$
For comparison, OLS has \(X'X=\begin{pmatrix}3&0\\0&2\end{pmatrix}\), and its conditional covariance under this error structure is $$ \operatorname{Var}(\hat{\beta}_{OLS}\mid X) =(X'X)^{-1}X'\Omega X(X'X)^{-1} =\begin{pmatrix}2/3&0\\0&1/2\end{pmatrix}. $$ The difference is $$ \operatorname{Var}(\hat{\beta}_{OLS}\mid X) -\operatorname{Var}(\hat{\beta}_{GLS}\mid X) =\begin{pmatrix}2/9&0\\0&0\end{pmatrix}, $$ which is positive semidefinite. GLS has a smaller variance for the intercept, while the slope variances are equal in this design. The theorem permits equality for some coefficients or contrasts; it does not require a strict improvement everywhere.
Why Correct Specification Is Essential
The proof depends on the same \(\Omega\) appearing both in the GLS weights and in the true covariance of the errors. If the matrix used for weighting is wrong, the identity that eliminates the cross terms no longer establishes a positive-semidefinite covariance difference relative to other unbiased estimators. The estimator formed with the working matrix can still be unbiased when the conditional mean is correctly specified, but its variance must be evaluated using the true covariance.
Worked Example: Misspecified Weights Can Be Less Efficient
Suppose the true model is again an intercept-only model with two independent hypothetical errors, both having mean zero and variance 1. Thus the true covariance is \(I_2\), and the assumptions for comparing unbiased estimators hold. Imagine, however, that a working GLS procedure incorrectly uses \(\Omega_W=\operatorname{diag}(1,4)\).
The resulting estimator is \(\hat{\mu}_W=(4/5)y_1+(1/5)y_2\). It is unbiased because its weights sum to one. Under the true covariance \(I_2\), its variance is $$ \operatorname{Var}(\hat{\mu}_W) =(4/5)^2+(1/5)^2 =\frac{17}{25} =0.6800. $$ OLS is \(\hat{\mu}_{OLS}=(y_1+y_2)/2\), with variance $$ \operatorname{Var}(\hat{\mu}_{OLS})=(1/2)^2+(1/2)^2=0.5000. $$ Here the misspecified weighted estimator is unbiased but less efficient than OLS. Under the true covariance \(I_2\), OLS is the GLS estimator, so the comparison agrees with the theorem. This example shows why using a covariance matrix in a formula does not by itself guarantee an efficiency gain: it must be the true covariance for the stated result to apply.
Scope of the Efficiency Claim
The Gauss–Markov result is conditional on the design matrix and requires a correct conditional mean and covariance model. It does not require normally distributed errors. Normality is therefore not needed for this covariance comparison, though it may be relevant to other distributional calculations. Nor does the theorem say that GLS always produces a numerically closer estimate in a particular data set; it compares repeated-sampling variances under the model.
When the true covariance is \(\sigma^2 I\), GLS and OLS coincide, so the efficiency comparison yields no improvement. When the true covariance is not proportional to the identity, GLS can improve precision by accounting for the way errors vary or move together. If the covariance is estimated rather than known, the exact known-\(\Omega\) finite-sample result does not directly establish the performance of the resulting feasible GLS estimator.
Common Mistake
- Wrong reasoning: “I used GLS weights, so my estimator must be more efficient than OLS.” The formula alone does not support that claim.
- Correction: The BLUE guarantee requires the weighting covariance to equal the true conditional error covariance, as well as the stated mean, rank, and positive-definiteness conditions. With a misspecified covariance, calculate variance using the true covariance; there is no general GLS-over-OLS efficiency ordering.
Check Your Understanding
Work through these questions using the assumptions and comparisons developed above.
- What condition must a linear estimator \(Ay\) satisfy to be unbiased for every \(\beta\) in the model \(y=X\beta+u\)?
- In the GLS efficiency proof, why does \(DX=0\) make the cross-covariance term vanish?
- What does positive semidefinite ordering imply about the variance of a specified linear combination of coefficients?
- Why does the GLS efficiency theorem not require normally distributed errors?
- If GLS uses an incorrect covariance matrix but the conditional mean is still zero, can the estimator remain unbiased? Does the BLUE guarantee still follow?