Misspecified Linear Regression and the Projection Target

Why a Linear Model Has a Target Even When It Is Wrong

A linear regression model is often used even when the conditional mean of the response is not exactly linear in the predictors. In that setting, there may be no coefficient vector that describes the conditional mean perfectly. There is still a well-defined best linear predictor under squared-error risk: the population least-squares projection. It depends on the distribution of the predictors and response, and it is the target that empirical least squares estimates when the model is held fixed.

The projection is not automatically a causal effect, nor does its existence imply that the linear model is correctly specified. Its meaning is narrower: among the predictors in the chosen linear span, it minimizes expected squared prediction error. We derive that target, examine the precise meaning of its residual equations, and show why empirical least squares converges to it under standard moment and nonsingularity conditions.

Population Projection and Its Normal Equations

Let \((Y,Z)\) be a random response and feature vector, where \(Y\) is real-valued and \(Z\in\mathbb{R}^p\). A coefficient vector \(b\in\mathbb{R}^p\) predicts \(Y\) by \(b^\mathsf{T}Z\). Assume \(E[Y^2]<\infty\) and \(E[\|Z\|^2]<\infty\), so the squared-error risk is finite for every fixed \(b\).

Define the feature second-moment matrix \(Q=E[ZZ^\mathsf{T}]\) and the feature-response moment vector \(g=E[ZY]\). The stated moment assumptions make all entries finite. If \(Q\) is positive definite, it is invertible, and the projection is unique.

Proof. Expand the risk:

Because \(Q\) is positive definite, \(R(b)\) is a strictly convex quadratic function. Completing the square around \(Q^{-1}g\) gives

The final term is nonnegative and equals zero only at \(b=Q^{-1}g\). Thus the unique minimizer is \(b^*=Q^{-1}g\), and subtracting the minimum risk gives the stated excess-risk identity. Finally,

This proves the theorem. \(\square\)

The equation \(E[Ze]=0\) means that the residual is orthogonal to each included feature in the \(L^2\) inner product \(E[UV]\). Without an intercept, this does not in general say that the residual has mean zero or that it is uncorrelated with each feature. If \(Z\) includes a constant coordinate equal to 1, the corresponding equation gives \(E[e]=0\). In that case, for every other feature \(Z_j\), \(\operatorname{Cov}(Z_j,e)=E[Z_je]-E[Z_j]E[e]=0\).

Residual Orthogonality Is Not Always Zero Covariance

The distinction between an inner-product equation and a covariance equation matters especially when a regression has no intercept. The normal equations impose \(E[Z_je]=0\) for each included feature, not necessarily \(\operatorname{Cov}(Z_j,e)=0\). The next calculation shows why those statements are different.

Empirical Least Squares Converges to the Projection

Now let \((Y_i,Z_i)\), \(i=1,\ldots,n\), be independent and identically distributed observations from the same distribution as \((Y,Z)\). Empirical least squares minimizes \(P_n(Y-b^\mathsf{T}Z)^2\), where \(P_n\) denotes the average over the \(n\) observations. Define \(Q_n=n^{-1}\sum_{i=1}^n Z_iZ_i^\mathsf{T}\) and \(g_n=n^{-1}\sum_{i=1}^n Z_iY_i\). When \(Q_n\) is invertible, the empirical minimizer is \(\hat b_n=Q_n^{-1}g_n\).

Proof. Each entry of \(Z_iZ_i^\mathsf{T}\) is integrable: for coordinates \(j,k\), Cauchy–Schwarz gives \(E[|Z_jZ_k|]\leq\{E[Z_j^2]E[Z_k^2]\}^{1/2}<\infty\). Similarly, \(E[|Z_jY|]\leq\{E[Z_j^2]E[Y^2]\}^{1/2}<\infty\). The strong law of large numbers applied to each of the finitely many entries therefore yields \(Q_n\xrightarrow{a.s.}Q\) and \(g_n\xrightarrow{a.s.}g\).

The determinant is a continuous function of the matrix entries, so \(\det(Q_n)\xrightarrow{a.s.}\det(Q)\). Positive definiteness of \(Q\) implies \(\det(Q)>0\); therefore, with probability one, \(\det(Q_n)\neq 0\) for all sufficiently large \(n\). For those \(n\), the empirical minimizer is unique and equals \(Q_n^{-1}g_n\). Matrix inversion is continuous at every invertible matrix, so \(Q_n^{-1}\xrightarrow{a.s.}Q^{-1}\). Combining these convergences gives

This proves the theorem. \(\square\)

The finite second-moment assumptions serve two roles: they ensure the population squared-error risk is finite and make the entries used in the strong-law argument integrable. Positive definiteness is an identifiability condition for the chosen features: no nonzero linear combination of them is zero almost surely. If it fails, the coefficient vector need not be unique, and the displayed inverse formula is unavailable.

Interpretation and a Common Pitfall

Under misspecification, the projection target is distribution-dependent. If the predictor distribution changes, the best linear approximation can change even when the conditional mean function stays the same. The projection is consequently a precise prediction target for a specified population and feature set, not a claim that the linear model is the true conditional mean.

• Pitfall: saying normal equations make the residual uncorrelated with every feature. That statement is too strong without an intercept. The equations give \(E[Z_je]=0\), an \(L^2\) orthogonality condition. To conclude zero covariance, one also needs \(E[e]=0\); an included constant feature supplies that equation.
• Pitfall: treating nonsingularity as a technical detail. If features are linearly dependent in the population, different coefficient vectors can give the same prediction almost surely. The projection prediction may remain meaningful, but a unique coefficient target and the inverse formula require nonsingularity.
• Pitfall: calling the empirical coefficient the population target in a finite sample. The sample coefficients generally differ from \(b^*\). The consistency theorem is an asymptotic statement under independent, identically distributed sampling and the stated moment and nonsingularity assumptions.