The Gauss–Markov Theorem for OLS

Why a Variance Comparison Matters

Finite-sample unbiasedness says that, under strict exogeneity, OLS has the correct conditional mean for the coefficient vector. Unbiasedness alone does not distinguish OLS from other estimators that are also centered on the target. The Gauss–Markov theorem adds a precision result: under homoskedastic, uncorrelated errors, OLS has no greater conditional variance than any other linear unbiased estimator.

The comparison is conditional on the observed regressors. It concerns a specified class of estimators—those linear in the outcome vector and unbiased for every coefficient vector—not every conceivable way to estimate the coefficients. The theorem does not require normally distributed errors.

Model and Assumptions

Let $y$ be an $n\times 1$ outcome vector, $X$ an $n\times k$ regressor matrix, $\beta$ a $k\times 1$ coefficient vector, and $u$ an $n\times 1$ error vector. Treat $X$ as fixed, or condition throughout on its observed value. The linear model is

y=X\beta+u.

The theorem uses the following assumptions:

Full column rank: $X$ has rank $k$, so $X'X$ is invertible.
Conditional mean zero: $E[u\mid X]=0$.
Homoskedasticity and no cross-observation correlation: $\operatorname{Var}(u\mid X)=\sigma^2 I_n$, where $\sigma^2>0$ and $I_n$ is the $n\times n$ identity matrix.

The last condition means that every error has conditional variance $\sigma^2$, and distinct errors have conditional covariance zero. It is a statement about the conditional covariance matrix; it does not say that the errors are independent or normally distributed. Under these assumptions, the OLS estimator is

\hat{\beta}_{OLS}=(X'X)^{-1}X'y.

Definition: A linear estimator of $\beta$, conditional on $X$, has the form $\tilde{\beta}=Ay$, where $A$ is a $k\times n$ matrix that may depend on $X$, but not on $y$. It is conditionally unbiased for every $\beta$ if $E[\tilde{\beta}\mid X]=\beta$ for every coefficient vector in the model.

Since $E[y\mid X]=X\beta$, a linear estimator $Ay$ is conditionally unbiased for every $\beta$ exactly when $AX=I_k$. This condition will identify the set of estimators against which OLS is compared.

The Gauss–Markov Theorem

Theorem: Under full column rank, $E[u\mid X]=0$, and $\operatorname{Var}(u\mid X)=\sigma^2 I_n$, OLS is the best linear unbiased estimator (BLUE) of $\beta$. Specifically, for every linear conditionally unbiased estimator $\tilde{\beta}=Ay$, $$ \operatorname{Var}(\tilde{\beta}\mid X)-\operatorname{Var}(\hat{\beta}_{OLS}\mid X) $$ is positive semidefinite. Equivalently, every fixed linear contrast $c'\beta$ has no greater conditional variance when estimated by $c'\hat{\beta}_{OLS}$ than by $c'\tilde{\beta}$.

“Positive semidefinite” means that a symmetric matrix $M$ satisfies $c'Mc\geq 0$ for every vector $c$ of the appropriate dimension. Thus the theorem compares the variance of every linear combination of the coefficient estimates. It does not claim that each off-diagonal covariance entry is numerically smaller.

Why OLS Has the Smallest Covariance Matrix

Write $B=(X'X)^{-1}X'$, so that $\hat{\beta}_{OLS}=By$, and consider any other linear unbiased estimator $Ay$. Because $AX=I_k$ and $BX=I_k$, their difference $C=A-B$ satisfies $CX=0$. The OLS covariance follows directly from the model assumptions:

\operatorname{Var}(\hat{\beta}_{OLS}\mid X) =B\operatorname{Var}(u\mid X)B' =\sigma^2 BB' =\sigma^2(X'X)^{-1}.

For the competing estimator, $A=B+C$. Expanding its conditional variance gives

\begin{aligned} \operatorname{Var}(Ay\mid X) &=\sigma^2 AA'\\ &=\sigma^2(B+C)(B+C)'\\ &=\sigma^2\left(BB'+BC'+CB'+CC'\right). \end{aligned}

The two cross-products vanish. In fact, $CX=0$ implies $X'C'=0$, and hence $BC'=(X'X)^{-1}X'C'=0$; taking the transpose gives $CB'=0$. Subtracting the OLS covariance therefore leaves

\operatorname{Var}(Ay\mid X)-\operatorname{Var}(\hat{\beta}_{OLS}\mid X) =\sigma^2 CC'.

For any vector $c$, the quadratic form of the difference is

c'\sigma^2 CC'c=\sigma^2(C'c)'(C'c)\geq 0.

This proves the theorem. It also explains the source of the variance penalty: a different unbiased linear estimator adds a component $Cy$ that does not change its conditional mean, because $CX=0$, but can add conditional variance.

Worked Example: Computing OLS for a Small Design

Suppose a hypothetical design has an intercept and one regressor with values $0,1,2$. The observed outcomes, measured in points, are $2,3,5$. Assume the conditional model has mean zero errors and covariance $\sigma^2 I_3$, with $\sigma^2=4$ points squared, and that the stated design is fixed. The design matrix has full column rank because the regressor is not constant.

The two columns of $X$ are $(1,1,1)'$ and $(0,1,2)'$. The OLS slope is the centered cross-product divided by the centered sum of squares:

\hat{\beta}_1 =\frac{(0-1)2+(1-1)3+(2-1)5} {(0-1)^2+(1-1)^2+(2-1)^2} =\frac{-2+0+5}{1+0+1} =1.5\text{ points per regressor unit}.

The intercept is the sample outcome mean minus the slope times the regressor mean:

\hat{\beta}_0 =\frac{2+3+5}{3}-1.5(1) =\frac{11}{6} \approx 1.833\text{ points}.

The OLS covariance matrix is $\sigma^2(X'X)^{-1}$. For this design, its entries are

\operatorname{Var}(\hat{\beta}_{OLS}\mid X) =4 \begin{pmatrix} 5/6 & -1/2\\ -1/2 & 1/2 \end{pmatrix} = \begin{pmatrix} 3.333 & -2\\ -2 & 2 \end{pmatrix}\text{ points}^2.

The diagonal entries give conditional variances of the intercept and slope estimates. The negative off-diagonal entry is their conditional covariance; it does not indicate that either variance is negative.

Comparing OLS With Another Unbiased Estimator

Worked Example: An Unbiased Slope With Greater Variance

Use the same hypothetical fixed design $x=(0,1,2)'$ and error assumptions. Instead of the OLS slope, consider $\tilde{\beta}_1=y_2-y_1$, using the observations at regressor values 1 and 0. This is a linear estimator with weights $(-1,1,0)$. Its weights sum to zero, so the intercept cancels; their weighted regressor values sum to one. Therefore its conditional expectation is $\beta_1$, and it is unbiased for the slope.

Under OLS, the slope weights are $(-1/2,0,1/2)$. Since the errors are uncorrelated and each has variance $4$ points squared, the OLS slope variance is

\operatorname{Var}(\hat{\beta}_1\mid X) =4\left[\left(-\frac12\right)^2+0^2+\left(\frac12\right)^2\right] =4\left(\frac12\right) =2\text{ points}^2.

The alternative slope uses two error terms with weights $-1$ and $1$, so

\operatorname{Var}(\tilde{\beta}_1\mid X) =4\left[(-1)^2+1^2+0^2\right] =8\text{ points}^2.

Both estimators are conditionally unbiased, but the alternative has four times the conditional variance in this design. This comparison illustrates that unbiasedness does not by itself ensure precision, and that the theorem’s minimum-variance claim applies within the class of linear unbiased estimators.

A Weighted Least-Squares Alternative

Weighted least squares (WLS) provides a useful example of a different linear estimator. For a fixed positive-definite weight matrix $W$, its estimator is

\hat{\beta}_{WLS}=(X'WX)^{-1}X'Wy.

When $X$ has full column rank, this estimator is linear in $y$. Its coefficient matrix multiplied by $X$ is $I_k$, so it is conditionally unbiased under $E[u\mid X]=0$. The Gauss–Markov theorem does not say WLS can never be useful: it says that under the particular covariance assumption $\sigma^2 I_n$, no linear unbiased estimator has lower conditional variance than OLS.

Worked Example: Comparing OLS With Unequal Fixed Weights

Return to the design $x=(0,1,2)'$, and use $W=\operatorname{diag}(1,2,3)$. Keep the errors conditionally uncorrelated with equal variance $\sigma^2$. The WLS regressor mean is $(1\cdot0+2\cdot1+3\cdot2)/6=4/3$, and its weighted centered sum of squares is

\sum_{i=1}^{3}w_i(x_i-\bar{x}_w)^2 =1(0-4/3)^2+2(1-4/3)^2+3(2-4/3)^2 =\frac{10}{3}.

The WLS slope is a weighted sum of the outcomes. Its three coefficients are $w_i(x_i-\bar{x}_w)/(10/3)$, giving

\hat{\beta}_{1,WLS} =-0.4y_1-0.2y_2+0.6y_3.

These coefficients sum to zero, and their weighted sum against the regressor values is $0(-0.4)+1(-0.2)+2(0.6)=1$. Thus the estimator cancels the intercept and has conditional expectation $\beta_1$. Its conditional variance is

\operatorname{Var}(\hat{\beta}_{1,WLS}\mid X) =\sigma^2\left[(-0.4)^2+(-0.2)^2+(0.6)^2\right] =0.56\sigma^2.

For OLS the slope variance is $0.5\sigma^2$. In particular, if $\sigma^2=4$ points squared, these variances are $2.24$ and $2$ points squared, respectively. The unequal fixed weights have not improved precision under homoskedastic errors in this design. The comparison is consistent with the theorem; it does not establish that WLS is less precise for every design and every choice of weights.

Scope and a Common Pitfall

The word “best” in BLUE means minimum conditional covariance, in the positive-semidefinite sense, among estimators that are both linear in $y$ and conditionally unbiased for every $\beta$. It does not mean OLS has the smallest variance among all possible estimators, or the smallest mean squared error among estimators that may be biased or nonlinear. Nor does the theorem, on its own, justify a probability distribution for the coefficient estimates or a particular inference procedure. Normality is not needed for the variance comparison.

Pitfall: “The Gauss–Markov theorem says OLS is always the most precise estimator, so it must be best even if the errors have unequal variances.”
Correction: The theorem’s covariance conclusion uses $\operatorname{Var}(u\mid X)=\sigma^2 I_n$. If that assumption fails, the proof’s covariance calculation changes, and the theorem does not establish OLS’s minimum-variance property for that setting.
Pitfall: “Because OLS is BLUE, every entry of its covariance matrix must be less than the corresponding entry for any competitor.”
Correction: The comparison is a matrix ordering: every linear contrast has no greater variance under OLS. It is not an entry-by-entry comparison of covariances.

Key takeaway: Under full column rank, conditional mean zero, and homoskedastic uncorrelated errors, every linear unbiased estimator differs from OLS by a component that adds a positive-semidefinite covariance term. That is the precise sense in which OLS is BLUE.

Check Your Understanding

Use the assumptions and variance comparison developed above to answer these questions.

What matrix condition makes a linear estimator $Ay$ conditionally unbiased for every $\beta$?
Why does $CX=0$, where $C=A-(X'X)^{-1}X'$, make the cross-products in the covariance comparison vanish?
In the three-point design $x=(0,1,2)'$, why is $y_2-y_1$ an unbiased estimator of the slope?
What does positive-semidefinite covariance dominance imply about a fixed linear contrast of $\beta$?
Does the Gauss–Markov theorem require normally distributed errors? State the error covariance assumption it does require.