What Heteroskedasticity Changes in Least Squares

Why Unequal Variance Matters

A least-squares regression line is calculated from the observed outcomes and predictors. If the errors have different variances for different observations, that calculation does not automatically change. Yet the uncertainty attached to the fitted coefficients can change substantially. This distinction matters because a regression analysis uses coefficients to describe estimated relationships, but uses standard errors to quantify how much those estimates would vary across repeated samples.

The key condition for unbiased OLS coefficients is exogeneity: conditional on the observed predictors, the errors have mean zero. Equal error variances are not required for that result. By contrast, the familiar conventional standard-error formula relies on a common error variance, and unequal variances can also make OLS less precise than another linear unbiased estimator.

Definition: In the linear regression model \(Y=X\beta+u\), \(Y\) is the vector of outcomes, \(X\) is the predictor matrix, \(\beta\) is the vector of regression coefficients, and \(u\) is the vector of errors. Assume \(X\) has full column rank. Exogeneity means \(E(u\mid X)=0\). Heteroskedasticity means that the conditional error variances are not all equal: \(\operatorname{Var}(u_i\mid X)=\sigma_i^2\), with at least two \(\sigma_i^2\) differing. For the covariance calculations below, errors are also assumed conditionally uncorrelated.

The coefficient estimate is \(\hat{\beta}=(X'X)^{-1}X'Y\). The full-rank condition ensures that this expression is defined. Heteroskedasticity concerns the spread of the errors, not their conditional means, so it does not by itself violate exogeneity.

Why OLS Coefficients Can Remain Unbiased

An estimator is unbiased conditional on \(X\) if its expected value over repeated outcomes, holding the predictor values fixed, equals the target coefficient vector. Substituting the model into the OLS formula shows exactly where this property comes from.

\begin{aligned} \hat{\beta} &=(X'X)^{-1}X'(X\beta+u)\\ &=\beta+(X'X)^{-1}X'u,\\ E(\hat{\beta}\mid X) &=\beta+(X'X)^{-1}X'E(u\mid X)\\ &=\beta. \end{aligned}

The final equality uses exogeneity, \(E(u\mid X)=0\). No equal-variance condition appears in this argument. Thus unequal error variances alone do not bias OLS coefficients. This does not mean that every estimated coefficient equals the true coefficient in a particular sample; it means that the estimates average to the true coefficients over repeated samples with the same \(X\).

Worked Example: An Unbiased Slope With Different Error Variances

Suppose a hypothetical simple regression has fixed predictor values \(x=0,1,2\), and its conditional mean is \(E(Y_i\mid X)=2+3x_i\). The errors are conditionally independent, have mean zero, and have variances \(1,4,9\), respectively. Find the expression for the fitted slope error and its expected value.

For these three predictor values, \(\bar{x}=1\) and \(S_{xx}=\sum (x_i-\bar{x})^2=1+0+1=2\). The slope error is

\hat{\beta}_1-3 =\frac{\sum (x_i-\bar{x})u_i}{S_{xx}} =\frac{-u_1+u_3}{2}.

Since \(E(u_1\mid X)=E(u_3\mid X)=0\),

E(\hat{\beta}_1-3\mid X) =\frac{-E(u_1\mid X)+E(u_3\mid X)}{2}=0.

So the slope estimate is conditionally unbiased despite the different error variances. Its variance, however, depends on those variances:

\operatorname{Var}(\hat{\beta}_1\mid X) =\frac{\operatorname{Var}(u_1\mid X)+\operatorname{Var}(u_3\mid X)}{4} =\frac{1+9}{4}=2.5.

The middle observation has zero weight in the slope formula for this design. This example shows why a coefficient's expected value and its sampling variability are separate questions.

What Changes for Standard Errors

A standard error estimates the sampling standard deviation of a coefficient estimator. Under the classical equal-variance model, the errors are conditionally uncorrelated and share a common variance \(\sigma^2\). The coefficient covariance matrix then has the familiar form \(\sigma^2(X'X)^{-1}\). In practice, \(\sigma^2\) is replaced by the residual mean square \(s^2=\mathrm{SSE}/(n-p)\), where \(p\) is the number of fitted coefficients, giving the conventional estimated covariance matrix \(s^2(X'X)^{-1}\).

With unequal variances, let \(\Omega=\operatorname{Var}(u\mid X)\). If errors are conditionally uncorrelated, \(\Omega\) is diagonal with entries \(\sigma_i^2\). Because OLS is a linear transformation of \(Y\), its conditional covariance is

\operatorname{Var}(\hat{\beta}\mid X) =(X'X)^{-1}X'\Omega X(X'X)^{-1}.

This follows from \(\hat{\beta}-\beta=Au\), where \(A=(X'X)^{-1}X'\), and the covariance rule \(\operatorname{Var}(Au\mid X)=A\operatorname{Var}(u\mid X)A'\). If \(\Omega=\sigma^2 I\), the expression reduces to \(\sigma^2(X'X)^{-1}\); if the variances differ, that simplification generally does not hold. Consequently, the conventional standard-error formula does not generally estimate the correct coefficient uncertainty under heteroskedasticity.

Worked Example: Conventional Slope Uncertainty Can Be Too Small

Consider a hypothetical simple regression with fixed predictor values \(-2,-1,1,2\). The errors are conditionally independent, have mean zero, and have variances \(9,1,1,9\), in that order. Compare the true slope variance with the expected conventional estimate of that variance.

Here \(\bar{x}=0\), \(S_{xx}=4+1+1+4=10\), and \(n-p=4-2=2\). The slope estimator has error \(\sum x_i u_i/S_{xx}\), so its true conditional variance is

\operatorname{Var}(\hat{\beta}_1\mid X) =\frac{\sum x_i^2\sigma_i^2}{S_{xx}^2} =\frac{4(9)+1(1)+1(1)+4(9)}{10^2} =\frac{74}{100}=0.74.

The conventional estimated slope variance is \(s^2/S_{xx}\). To find its expected value, use the regression residual-maker matrix \(M=I-H\), where \(H\) is the projection matrix onto the fitted intercept and predictor. For this design, the diagonal entries of \(H\) are \(1/4+x_i^2/10\): they are \(0.65\) for the two outer observations and \(0.35\) for the two inner observations. Therefore the diagonal entries of \(M\) are \(0.35\) and \(0.65\), respectively. Since \(E(\mathrm{SSE}\mid X)=\sum (1-h_{ii})\sigma_i^2\) for conditionally uncorrelated errors,

\begin{aligned} E(s^2\mid X) &=\frac{2(0.35)(9)+2(0.65)(1)}{2}\\ &=\frac{6.3+1.3}{2}=3.8,\\ E\left(\frac{s^2}{S_{xx}}\middle|X\right) &=\frac{3.8}{10}=0.38. \end{aligned}

The true slope variance is \(0.74\), whereas the expected conventional variance estimate is \(0.38\). Thus, in this design, the conventional formula underestimates slope variance on average. This is a comparison of variances, not of the expected standard error itself. Unequal variances do not always cause underestimation; the direction and size of the discrepancy depend on how the variances align with the predictor values and regression design.

If conventional standard errors are inaccurate, tests and confidence intervals built from them can also be unreliable. A reported p-value may not have its intended calibration, and a confidence interval may not achieve its stated repeated-sampling coverage. The coefficient estimate itself can remain unbiased while these inferential summaries fail to reflect its actual sampling variability. Normal errors do not repair the mismatch in the variance formula.

Efficiency Is a Separate Issue

Among linear unbiased estimators, an estimator is more efficient when it has smaller variance. Under the Gauss–Markov conditions, including equal error variances and no error correlation, OLS is the best linear unbiased estimator (BLUE): it has the smallest variance in that class. With heteroskedasticity, OLS may still be linear and unbiased, but the BLUE guarantee no longer follows. Another linear unbiased estimator can use the differing variances more effectively and have smaller variance.

Worked Example: A Less Variable Unbiased Estimate

Suppose a hypothetical model contains only an intercept: \(Y_i=\beta_0+u_i\). There are two independent errors with conditional means zero and variances \(1\) and \(9\). Compare OLS with an estimator that gives more weight to the lower-variance observation.

OLS estimates the intercept by the ordinary average, \(\hat{\beta}_0=(Y_1+Y_2)/2\). Its expectation is \(\beta_0\), and independence gives

\operatorname{Var}(\hat{\beta}_0\mid X) =\frac{1^2(1)+1^2(9)}{(1+1)^2} =\frac{10}{4}=2.5.

Now consider \(\tilde{\beta}_0=0.9Y_1+0.1Y_2\). The weights sum to one, so \(E(\tilde{\beta}_0\mid X)=0.9\beta_0+0.1\beta_0=\beta_0\): this estimator is also unbiased. Its variance is

\operatorname{Var}(\tilde{\beta}_0\mid X) =0.9^2(1)+0.1^2(9) =0.81+0.09=0.90.

In this example the second estimator has substantially lower variance than OLS, while both are unbiased. The example demonstrates a possibility, not a universal numerical advantage of a particular alternative: efficiency depends on the design and the variance pattern.

What to Conclude From Heteroskedasticity

Unequal error variances change neither the definition of the OLS coefficients nor their conditional unbiasedness when the model is correctly specified and exogeneity holds. They do change the covariance calculation, so conventional standard errors can be wrong. They also remove the equal-variance condition needed for the Gauss–Markov efficiency guarantee. These are three distinct claims and should be reported separately.

A common mistake is to reason, “The estimated regression coefficients are unchanged, so the standard errors and tests must be fine.” The correction is that coefficients and their uncertainty come from different calculations. Exogeneity controls the conditional expectation of OLS; the error-variance pattern controls its covariance and affects the validity of the conventional uncertainty formula.

Another mistake is to say that heteroskedasticity necessarily biases OLS or necessarily makes conventional standard errors too small. Neither conclusion follows from unequal variances alone. Bias depends on whether the conditional mean-zero assumption holds; the direction of a standard-error discrepancy depends on the variance pattern and the design.

Key takeaway: Under full rank and \(E(u\mid X)=0\), OLS remains conditionally unbiased even with unequal error variances. Heteroskedasticity generally invalidates the conventional equal-variance covariance formula and can make OLS less efficient, so coefficient estimates and uncertainty estimates must not be conflated.

Check Your Understanding

Use the distinction between coefficient expectation, coefficient variance, and efficiency to answer these questions.

Which condition gives conditional unbiasedness of OLS: equal error variances or conditional mean-zero errors? Explain briefly.
Why does the conventional covariance formula \(s^2(X'X)^{-1}\) generally fail under heteroskedasticity?
Can OLS coefficients be unbiased even if conventional standard errors are inaccurate? State why.
In the slope example, why did the middle observation contribute nothing to the slope error expression?
Does heteroskedasticity prove that conventional standard errors are too small? Explain what determines the direction of any discrepancy.