The Gauss-Markov Result for Simple Regression

Why “Best” Needs a Precise Meaning

Ordinary least squares (OLS) is often described as the “best” way to estimate a regression coefficient. That statement is meaningful only after specifying the class of methods being compared and the criterion used to judge them. The Gauss-Markov result supplies that precision: under stated assumptions, OLS has the smallest variance among linear unbiased estimators of each regression coefficient. It does not say that OLS is best among every possible estimator.

For this result, treat the predictor values as specified numbers. The response for observation \(i\) follows a simple linear regression model. The predictor values must contain variation; otherwise a slope cannot be estimated. Importantly, the result does not require normally distributed errors. Normality is useful for particular inference procedures, but it is not a condition for the variance comparison proved here.

Model and assumptions: For \(i=1,\ldots,n\), let \(Y_i=\beta_0+\beta_1x_i+\varepsilon_i\), where the \(x_i\) are specified values, at least two of which differ. Assume \(E[\varepsilon_i]=0\), \(\operatorname{Var}(\varepsilon_i)=\sigma^2\) for every \(i\), and \(\operatorname{Cov}(\varepsilon_i,\varepsilon_j)=0\) for \(i\ne j\). The common variance satisfies \(\sigma^2\ge 0\). These assumptions imply \(E[Y_i]=\beta_0+\beta_1x_i\) and that the errors are uncorrelated with equal variance. Independence is sufficient for the error covariance condition, but is stronger than required.

Write \(\bar{x}=n^{-1}\sum_{i=1}^n x_i\) and \(S_{xx}=\sum_{i=1}^n(x_i-\bar{x})^2\). The requirement that the predictor values are not all equal is exactly the condition \(S_{xx}>0\). Under the model, the least-squares estimates are

\widehat{\beta}_1 =\frac{\sum_{i=1}^n(x_i-\bar{x})Y_i}{S_{xx}}, \qquad \widehat{\beta}_0=\bar{Y}-\widehat{\beta}_1\bar{x}.

A linear estimator of a coefficient means an estimator formed as a weighted sum of the responses, with weights determined by the predictor values. Unbiasedness means its expected value equals the coefficient for every possible pair \((\beta_0,\beta_1)\) in the model. The phrase “best linear unbiased” refers to having the smallest variance within this particular class.

Theorem: Under the stated simple regression assumptions, \(\widehat{\beta}_0\) and \(\widehat{\beta}_1\) are unbiased and have minimum variance among all estimators that are linear in \(Y_1,\ldots,Y_n\) and unbiased for the corresponding coefficient. If \(\sigma^2>0\), the minimizing weights for each coefficient are unique. If \(\sigma^2=0\), all admissible linear unbiased estimators have variance zero, so the minimum is still attained by OLS but need not be unique.

Why OLS Has the Smallest Variance

The key is to characterize every linear unbiased estimator of the slope. Write such an estimator as \(T=\sum_{i=1}^n a_iY_i\). Its expectation is \(\beta_0\sum_i a_i+\beta_1\sum_i a_ix_i\). For this to equal \(\beta_1\) for every value of \(\beta_0\) and \(\beta_1\), the weights must satisfy two constraints: \(\sum_i a_i=0\) and \(\sum_i a_ix_i=1\).

The OLS slope has weights \(c_i=(x_i-\bar{x})/S_{xx}\). They satisfy the constraints because the centered predictor values sum to zero, and \(\sum_i c_ix_i=\sum_i(x_i-\bar{x})x_i/S_{xx}=1\). Any other admissible slope weights can therefore be written \(a_i=c_i+d_i\), where \(\sum_i d_i=0\) and \(\sum_i d_ix_i=0\). The variance comparison follows directly.

\begin{aligned} \operatorname{Var}\left(\sum_i a_iY_i\right) &=\sigma^2\sum_i(c_i+d_i)^2\\ &=\sigma^2\sum_i c_i^2 +2\sigma^2\sum_i c_id_i +\sigma^2\sum_i d_i^2. \end{aligned}

The cross-product is zero: \(\sum_i c_id_i=\{\sum_i x_id_i-\bar{x}\sum_i d_i\}/S_{xx}=0\), by the two constraints on \(d_i\). Thus the variance of the alternative estimator is the OLS variance plus \(\sigma^2\sum_i d_i^2\), a nonnegative quantity. Also, \(\sum_i c_i^2=1/S_{xx}\), so \(\operatorname{Var}(\widehat{\beta}_1)=\sigma^2/S_{xx}\).

\operatorname{Var}(T)-\operatorname{Var}(\widehat{\beta}_1) =\sigma^2\sum_{i=1}^n d_i^2\ge 0.

This derivation also explains the equality case. When \(\sigma^2>0\), equality requires \(\sum_i d_i^2=0\), which requires every \(d_i=0\); hence the OLS weights are the unique minimizing weights. When \(\sigma^2=0\), the variance difference is zero for every admissible set of weights. OLS remains a minimizer, but uniqueness does not follow.

The same argument applies to the intercept. Its OLS weights are \(b_i=1/n-\bar{x}(x_i-\bar{x})/S_{xx}\), and they satisfy \(\sum_i b_i=1\) and \(\sum_i b_ix_i=0\), the constraints for unbiasedness for \(\beta_0\). The OLS intercept variance is \(\sigma^2(1/n+\bar{x}^2/S_{xx})\). Any other unbiased linear intercept estimator differs by weights \(d_i\) with \(\sum_i d_i=0\) and \(\sum_i d_ix_i=0\); its variance exceeds the OLS variance by \(\sigma^2\sum_i d_i^2\).

Worked Comparisons

Worked Example: Comparing Two Linear Slope Estimators

Suppose a hypothetical portfolio study uses predictor values \(x=(0,1,2)\) in the model \(Y_i=\beta_0+\beta_1x_i+\varepsilon_i\), with zero-mean, uncorrelated errors, each having variance \(\sigma^2=4\). Compare the OLS slope with the estimator using weights \(a=(1/2,-2,3/2)\). Identify the method and the mistake this comparison avoids.

Method. First check that the proposed weights meet the unbiasedness constraints for a slope. Their sum is \(1/2-2+3/2=0\), and their weighted predictor sum is \(0(1/2)+1(-2)+2(3/2)=1\). Thus the proposed estimator is linear and unbiased for \(\beta_1\).

Here \(\bar{x}=1\) and \(S_{xx}=(-1)^2+0^2+1^2=2\). The OLS weights are \((-1/2,0,1/2)\), so its variance is \(4\{(1/2)^2+(1/2)^2\}=2\), agreeing with \(\sigma^2/S_{xx}=4/2=2\).

The alternative estimator has variance \(4\{(1/2)^2+(-2)^2+(3/2)^2\}=4(1/4+4+9/4)=26\). Its variance exceeds the OLS variance by \(26-2=24\). The difference in weights is \(d=(1,-2,1)\), giving \(\sigma^2\sum_i d_i^2=4(1+4+1)=24\), which checks the result.

Common mistake avoided: Comparing arbitrary formulas without checking unbiasedness. Here the weight constraints are satisfied, so the Gauss-Markov comparison applies; it shows that OLS has lower variance in this comparison.

Worked Example: Comparing Linear Intercept Estimators

Use predictor values \(x=(1,2,3)\) and the same equal-variance, uncorrelated error assumptions, now with \(\sigma^2=3\). Compare the OLS intercept with the estimator using weights \((7/3,-5/3,1/3)\).

Method. For an estimator \(\sum_i b_iY_i\) to be unbiased for \(\beta_0\), its weights must satisfy \(\sum_i b_i=1\) and \(\sum_i b_ix_i=0\). The proposed weights sum to \((7-5+1)/3=1\); their predictor-weighted sum is \(7/3-10/3+3/3=0\). It is therefore a linear unbiased intercept estimator.

For these predictor values, \(\bar{x}=2\) and \(S_{xx}=1+0+1=2\). The OLS intercept weights are \(1/3-2(-1)/2=4/3\), \(1/3-2(0)/2=1/3\), and \(1/3-2(1)/2=-2/3\). Its variance is \(3\{(4/3)^2+(1/3)^2+(-2/3)^2\}=3(21/9)=7\).

The alternative estimator’s variance is \(3\{(7/3)^2+(-5/3)^2+(1/3)^2\}=3(75/9)=25\). The weight difference is \((1,-2,1)\), so the variance excess is \(3(1+4+1)=18\); indeed, \(25-7=18\). OLS has the smaller variance.

Common mistake avoided: Checking only that the weights sum to one. That condition alone does not establish unbiasedness for the intercept; the predictor-weighted sum must also be zero.

Worked Example: Applying the Result Without Normal Errors

Consider a hypothetical study with predictor values \(x=(0,1,2)\), model \(Y_i=5+2x_i+\varepsilon_i\), and independent errors with the same distribution: \(\varepsilon_i=-1\) with probability \(2/3\), and \(\varepsilon_i=2\) with probability \(1/3\). Determine whether the Gauss-Markov result applies and calculate the OLS slope and intercept variances.

Method. The errors have mean \((-1)(2/3)+(2)(1/3)=0\). Their variance is \(2/3+4/3=2\), since the mean is zero. They are independent, and therefore uncorrelated, with common variance \(\sigma^2=2\). The predictor values vary, with \(\bar{x}=1\) and \(S_{xx}=2\). All the Gauss-Markov assumptions hold.

The slope variance is \(\sigma^2/S_{xx}=2/2=1\). The intercept variance is \(2\{1/3+1^2/2\}=2(5/6)=5/3\). The error distribution is not normal: it takes only two values and has unequal probabilities. That does not prevent the variance-minimization result from applying.

Common mistake avoided: Requiring normal errors for the Gauss-Markov theorem. Normality is not needed for this comparison; zero means, equal variances, zero covariances, and a nonconstant predictor are the relevant conditions.

What the Result Does Not Say

“Best” in BLUE means best linear unbiased estimator: minimum variance among unbiased estimators that are linear in the observed responses. It does not establish that OLS has minimum variance among all unbiased estimators, including nonlinear ones, and it does not compare biased estimators by mean squared error. A separate argument would be needed to make either broader claim.

The conditions matter as well. If the error variances differ or errors are correlated, the variance calculation used in the proof no longer reduces to \(\sigma^2\sum_i a_i^2\); the stated Gauss-Markov comparison is not justified. This does not, by itself, prove OLS is inferior in every such case. It means that this theorem cannot be invoked without its assumptions.

Key takeaway: Check the model assumptions and the estimator’s unbiasedness constraints before applying the result. Under the simple regression assumptions, OLS minimizes variance within the class of linear unbiased estimators. It does not claim superiority over every estimator, and normal errors are not required.

Check Your Understanding

For each problem, check both the assumptions and what class of estimators is being compared.

In a simple regression with \(x=(0,2,4)\) and uncorrelated equal-variance errors of variance \(\sigma^2\), find \(S_{xx}\) and the variance of the OLS slope. (Answer: \(S_{xx}=8\); \(\operatorname{Var}(\widehat{\beta}_1)=\sigma^2/8\).)
For \(x=(1,2,3)\), verify that OLS intercept weights are \((4/3,1/3,-2/3)\), then give their variance when \(\sigma^2=2\). (Answer: The weights sum to 1 and their predictor-weighted sum is 0; the variance is \(14/3\).)
With \(x=(0,1,3)\), consider slope weights \((-1/3,0,1/3)\). Verify unbiasedness, calculate their variance, and compare with OLS when \(\sigma^2=1\). (Answer: They are unbiased; their variance is \(2/9\), compared with OLS variance \(3/14\), so OLS has the smaller variance by \(1/126\).)
For \(x=(0,1,2)\), suppose the errors are uncorrelated but have variances \((1,4,1)\). Does the equal-variance Gauss-Markov result apply? (Answer: No; the error variances are unequal, so the stated theorem’s assumptions are not met.)
Suppose errors are independent, have mean zero and common positive variance, but follow a nonnormal distribution. Does nonnormality alone prevent OLS from being BLUE under the simple regression model with varying predictor values? (Answer: No; normality is not required for the Gauss-Markov result.)

What You'll Learn