Collinearity Does Not Automatically Bias Least Squares

Collinearity and the Meaning of Bias

When predictors are strongly related, it is tempting to conclude that the fitted regression coefficients must be systematically wrong. That conclusion confuses two different properties of an estimator. Bias concerns the estimator's average over repeated outcomes under the same predictor values; precision concerns how much its estimates vary from one outcome sample to another. Collinearity can make coefficients imprecise, but it does not by itself make ordinary least-squares (OLS) coefficients biased.

The qualification matters: unbiasedness is a result under a model assumption about the errors, not a guarantee supplied by OLS in every setting. We will state that assumption, derive the result, and distinguish the effect of near collinearity from a violation of the model.

The Model and the Unbiasedness Result

For \(n\) observations and \(p\) regression coefficients, write the multiple linear regression model as

\mathbf{Y}=\mathbf{X}\boldsymbol{\beta}+\boldsymbol{\varepsilon}.

Here \(\mathbf{Y}\) is the vector of outcomes, \(\mathbf{X}\) is the known design matrix containing the predictor values (and, if included, a column of ones for the intercept), \(\boldsymbol{\beta}\) is the vector of population coefficients, and \(\boldsymbol{\varepsilon}\) is the vector of errors. We condition on the observed \(\mathbf{X}\): the predictor values are held fixed while considering repeated possible outcomes.

Definition: An estimator \(\hat{\boldsymbol{\beta}}\) is unbiased conditional on \(\mathbf{X}\) if \(E(\hat{\boldsymbol{\beta}}\mid\mathbf{X})=\boldsymbol{\beta}\). Its conditional bias is \(E(\hat{\boldsymbol{\beta}}\mid\mathbf{X})-\boldsymbol{\beta}\). Unbiasedness means that the estimator averages to the target over repeated outcomes at these predictor values; it does not mean every fitted estimate equals the target.

If \(\mathbf{X}\) has full column rank, then \(\mathbf{X}^{\mathsf T}\mathbf{X}\) is invertible and the OLS estimator is

\hat{\boldsymbol{\beta}} =(\mathbf{X}^{\mathsf T}\mathbf{X})^{-1}\mathbf{X}^{\mathsf T}\mathbf{Y}.

The key assumption for unbiasedness is \(E(\boldsymbol{\varepsilon}\mid\mathbf{X})=\mathbf{0}\): for each fixed set of predictor values, the errors have mean zero. Full column rank is also required for this unique OLS coefficient formula. Neither assumption requires the predictors to be uncorrelated. They may be strongly, but not exactly, linearly related.

Theorem: If \(\mathbf{X}\) has full column rank and \(E(\boldsymbol{\varepsilon}\mid\mathbf{X})=\mathbf{0}\), then OLS is conditionally unbiased: \(E(\hat{\boldsymbol{\beta}}\mid\mathbf{X})=\boldsymbol{\beta}\). The result holds whether the predictors are weakly or strongly correlated.

Why the Result Holds

Substitute the model equation for \(\mathbf{Y}\) into the OLS formula. Then take the conditional expectation, treating \(\mathbf{X}\) as fixed:

\begin{aligned} E(\hat{\boldsymbol{\beta}}\mid\mathbf{X}) &=E\left((\mathbf{X}^{\mathsf T}\mathbf{X})^{-1} \mathbf{X}^{\mathsf T}\mathbf{Y}\mid\mathbf{X}\right)\\ &=(\mathbf{X}^{\mathsf T}\mathbf{X})^{-1}\mathbf{X}^{\mathsf T} E(\mathbf{Y}\mid\mathbf{X})\\ &=(\mathbf{X}^{\mathsf T}\mathbf{X})^{-1}\mathbf{X}^{\mathsf T} \mathbf{X}\boldsymbol{\beta}\\ &=\boldsymbol{\beta}. \end{aligned}

The third line follows because \(E(\mathbf{Y}\mid\mathbf{X})=\mathbf{X}\boldsymbol{\beta}\), a consequence of the zero conditional mean of the errors. In the last line, the matrix product reduces to the identity matrix. Predictor correlation affects the entries of \((\mathbf{X}^{\mathsf T}\mathbf{X})^{-1}\), but it does not alter this cancellation as long as the matrix is invertible.

Worked Example: Checking the Conditional Expectation

Consider a hypothetical regression with two centered predictors. Suppose their cross-products give \(\mathbf{X}^{\mathsf T}\mathbf{X}= \begin{pmatrix}100&97.5\\97.5&100\end{pmatrix}\). The strong positive relationship between the predictors is represented by the large off-diagonal entries. Assume the true coefficient vector is \(\boldsymbol{\beta}=(2,1)^{\mathsf T}\), the design has full rank, and \(E(\boldsymbol{\varepsilon}\mid\mathbf{X})=\mathbf{0}\). What is the conditional expected OLS estimate?

The determinant is \(100(100)-(97.5)^2=10{,}000-9{,}506.25=493.75\), which is positive. Thus the matrix is invertible, with inverse \[ (\mathbf{X}^{\mathsf T}\mathbf{X})^{-1} =\frac{1}{493.75} \begin{pmatrix}100&-97.5\\-97.5&100\end{pmatrix}. \]

Since \(E(\mathbf{Y}\mid\mathbf{X})=\mathbf{X}\boldsymbol{\beta}\), the expected estimate is \[ E(\hat{\boldsymbol{\beta}}\mid\mathbf{X}) =(\mathbf{X}^{\mathsf T}\mathbf{X})^{-1} \mathbf{X}^{\mathsf T}\mathbf{X} \begin{pmatrix}2\\1\end{pmatrix} = \begin{pmatrix}2\\1\end{pmatrix}. \] The predictors' strong relationship does not shift the conditional expectation of either coefficient.

Unbiased Does Not Mean Precise

The same design can produce substantial variation in estimated coefficients. To quantify that variation, add the usual constant-variance condition: \(\operatorname{Var}(\boldsymbol{\varepsilon}\mid\mathbf{X})=\sigma^2\mathbf{I}\), where \(\sigma^2\) is the error variance and \(\mathbf{I}\) is the identity matrix. Under this condition,

\operatorname{Var}(\hat{\boldsymbol{\beta}}\mid\mathbf{X}) =\sigma^2(\mathbf{X}^{\mathsf T}\mathbf{X})^{-1}.

This variance result explains the distinction: collinearity leaves the expectation calculation intact but can make entries on the diagonal of the inverse matrix large. Those diagonal entries are the coefficient variances; their square roots are the standard errors. The earlier material on variance inflation factors and standard errors develops this precision consequence. Here the essential point is that high variance is not the same as nonzero bias.

Worked Example: Strong Collinearity and a Large Standard Error

Continue with two hypothetical centered predictors whose sums of squares are both 100 and whose cross-product is 97.5. Their correlation is \(r=97.5/\sqrt{100(100)}=0.975\). Suppose the error variance is \(\sigma^2=4\) outcome-units squared. Compare the variance and standard error of the first coefficient with what they would be if the cross-product were zero while the sums of squares and error variance stayed the same.

For the correlated design, the relevant inverse-matrix diagonal entry is \[ \frac{100}{100(100)-(97.5)^2} =\frac{100}{493.75}\approx0.2025. \] Therefore, \[ \operatorname{Var}(\hat{\beta}_1\mid\mathbf{X}) =4\left(\frac{100}{493.75}\right) \approx0.8101, \qquad SE(\hat{\beta}_1\mid\mathbf{X}) =\sqrt{0.8101}\approx0.9001 \] outcome-units per unit of predictor.

With zero cross-product, the inverse-matrix diagonal entry would be \(1/100=0.01\), giving variance \(4(0.01)=0.0400\) and standard error \(\sqrt{0.0400}=0.2000\) outcome-units per unit of predictor. Thus, for this fixed design comparison, collinearity makes the first coefficient's variance about \(0.8101/0.0400=20.25\) times as large and its standard error about \(0.9001/0.2000=4.500\) times as large. This is a loss of precision, not evidence of bias.

Worked Example: One Estimate Can Miss Without Being Biased

Suppose the true coefficients in the preceding correlated design are \(\beta_1=2\) and \(\beta_2=1\). In one hypothetical realization, let the error vector have predictor cross-products \(\mathbf{X}^{\mathsf T}\boldsymbol{\varepsilon}=(0.5,-0.5)^{\mathsf T}\). Find the difference between the fitted coefficients and their true values.

From the OLS formula and the model, \[ \hat{\boldsymbol{\beta}}-\boldsymbol{\beta} =(\mathbf{X}^{\mathsf T}\mathbf{X})^{-1} \mathbf{X}^{\mathsf T}\boldsymbol{\varepsilon}. \] Using the inverse above, \[ \hat{\boldsymbol{\beta}}-\boldsymbol{\beta} =\frac{1}{493.75} \begin{pmatrix}100&-97.5\\-97.5&100\end{pmatrix} \begin{pmatrix}0.5\\-0.5\end{pmatrix} =\frac{1}{493.75} \begin{pmatrix}98.75\\-98.75\end{pmatrix} = \begin{pmatrix}0.2\\-0.2\end{pmatrix}. \] So this realization gives \(\hat{\beta}_1=2.2\) and \(\hat{\beta}_2=0.8\). These values differ from the truth, but a single outcome realization cannot establish bias. Under the zero conditional mean assumption, the deviations average to zero over repeated outcomes at the same predictor values.

When the Unbiasedness Claim Does Not Apply

The theorem does not say that OLS is unbiased regardless of how the data arise. If \(E(\boldsymbol{\varepsilon}\mid\mathbf{X})\ne\mathbf{0}\), then the same calculation gives

E(\hat{\boldsymbol{\beta}}\mid\mathbf{X}) =\boldsymbol{\beta} +(\mathbf{X}^{\mathsf T}\mathbf{X})^{-1}\mathbf{X}^{\mathsf T} E(\boldsymbol{\varepsilon}\mid\mathbf{X}).

The additional term need not be zero. For example, if the conditional mean of the errors is \(\mathbf{X}\boldsymbol{\gamma}\), then the expected estimate is \(\boldsymbol{\beta}+\boldsymbol{\gamma}\). The shift comes from the nonzero conditional error mean, not from collinearity itself. Collinearity can magnify the consequences of error in a particular coefficient estimate, but it does not create this violation.

Worked Example: A Nonzero Error Mean Can Shift the Estimate

Suppose a hypothetical model has two predictors, the same full-rank correlated design as above, and \(E(\boldsymbol{\varepsilon}\mid\mathbf{X})=\mathbf{X}(0.1,0)^{\mathsf T}\). Find the conditional expectation of the OLS coefficient vector if the true coefficients are \((2,1)^{\mathsf T}\).

Substitution into the expectation formula gives \[ E(\hat{\boldsymbol{\beta}}\mid\mathbf{X}) =\boldsymbol{\beta} +(\mathbf{X}^{\mathsf T}\mathbf{X})^{-1} \mathbf{X}^{\mathsf T}\mathbf{X} \begin{pmatrix}0.1\\0\end{pmatrix} = \begin{pmatrix}2\\1\end{pmatrix} + \begin{pmatrix}0.1\\0\end{pmatrix} = \begin{pmatrix}2.1\\1.0\end{pmatrix}. \] Here the first coefficient has conditional bias \(0.1\), because the zero conditional mean assumption fails. The calculation identifies the failure as a feature of the error mean, not a consequence of the predictors' correlation.

Exact Collinearity and Practical Interpretation

Near collinearity means that predictor columns are close to a linear combination of one another, while the design matrix still has full column rank. The OLS coefficient vector remains uniquely defined, and the unbiasedness argument still works, although the estimates may have large standard errors. With exact collinearity, the columns are linearly dependent and \(\mathbf{X}^{\mathsf T}\mathbf{X}\) is not invertible. The unique coefficient formula and the theorem in that form then do not apply: distinct coefficient vectors can produce the same fitted values. This is a failure of unique identification, not a case in which a uniquely defined OLS coefficient is biased.

A coefficient's unbiasedness also does not guarantee that its estimate is close to the target in a particular data set, nor that its usual confidence interval is narrow. When interpreting an analysis, keep the questions separate: does the model support an unbiasedness claim, and how much uncertainty remains in the coefficient estimate? Collinearity principally raises the second question.

Mistake: “The predictors are highly correlated, so OLS must systematically overestimate one coefficient.”
Correction: Under full rank and \(E(\boldsymbol{\varepsilon}\mid\mathbf{X})=\mathbf{0}\), the conditional expectation equals the true coefficient vector. Correlation can increase variance and make individual estimates unstable, but it does not alone cause systematic overestimation.
Mistake: “My fitted coefficient differs from the population coefficient, so the estimator is biased.”
Correction: A realized estimate can differ from its target even when the estimator is unbiased. Bias is about the repeated-outcome average, not the error in one fitted sample.

Key takeaway: With a full-rank design and zero conditional mean errors, OLS coefficients are conditionally unbiased regardless of predictor correlation. Collinearity can still inflate their variance and standard errors; unbiasedness is not a promise of precision.

Check Your Understanding

Answer each question using the assumptions and distinctions developed above.

In precise terms, what does it mean for an OLS coefficient estimator to be unbiased conditional on \(\mathbf{X}\)?
Which assumption about the errors is central to the conditional unbiasedness result? Does that result require predictors to be uncorrelated?
Explain why a coefficient estimate that differs from its true value in one sample does not, by itself, demonstrate bias.
What is the difference between the effect of near collinearity and exact collinearity on the OLS coefficient vector?
If \(E(\boldsymbol{\varepsilon}\mid\mathbf{X})\ne\mathbf{0}\), which part of the conditional expectation calculation can produce a nonzero shift from \(\boldsymbol{\beta}\)?