The Projection Theorem and Normal Equations

Why Projection Residuals Must Be Orthogonal

A population linear projection chooses coefficients to make the squared prediction error as small as possible on average. The central result is that, at the best choice of coefficients, the remaining error has zero expected product with every regressor included in the prediction. These zero-moment conditions are the normal equations.

The result is useful because it turns an optimization problem into a system of equations. It also identifies exactly when the best linear predictor has a unique coefficient vector: the regressors must have a nonsingular second-moment matrix. This is a statement about best prediction within a specified linear class; it does not require the conditional mean of the outcome to be linear.

Population Projection and Its Assumptions

Let $Y$ be a scalar outcome and let $X$ be a $k$-dimensional random vector of included regressors. The vector may contain a constant regressor. Assume $E[Y^2]$ and every component of $E[XX']$ are finite, so the mean squared prediction error is well defined. For any coefficient vector $b$, the linear predictor is $X'b$.

Definition: The population linear projection coefficient $\beta$ is a coefficient vector that minimizes $E[(Y-X'b)^2]$ over all $b\in\mathbb{R}^k$. Its projection residual is $e=Y-X'\beta$. The normal equations are $E[Xe]=0$, equivalently $E[XX']\beta=E[XY]$.

This definition concerns the best predictor among linear combinations of the included regressors. If the conditional mean $E[Y\mid X]$ is not linear, the projection is still defined. Nor does calling $e$ a residual imply it is independent of $X$: the result established here is about expected products.

The Projection Theorem

Theorem: Suppose $E[Y^2]<\infty$, the components of $X$ have finite second moments, and $Q=E[XX']$ is nonsingular. Then there is a unique coefficient vector minimizing $E[(Y-X'b)^2]$, given by $$ \beta=Q^{-1}E[XY]. $$ The corresponding residual $e=Y-X'\beta$ satisfies $E[Xe]=0$. Thus each included regressor has zero expected product with the projection residual.

The equations express one condition for each regressor. If $X=(1,X_1,\ldots,X_{k-1})'$ includes an intercept, its first normal equation is $E[e]=0$, so the projection residual has mean zero. The other equations state $E[X_j e]=0$ for each nonconstant regressor $X_j$. Without an intercept, a zero residual mean is not generally implied.

Why the Normal Equations Hold

Write $Q=E[XX']$, $q=E[XY]$, and $m=E[Y^2]$. The objective function, or population prediction risk, is

$$ R(b)=E[(Y-X'b)^2]=m-2b'q+b'Qb. $$

The equality follows by expanding the square and taking expectations. The matrix $Q$ is symmetric and positive semidefinite because, for every vector $a$,

a'Qa=E[(X'a)^2]\geq 0.

As a result, if $Q$ is also nonsingular, it is positive definite: $a'Qa>0$ for every nonzero $a$. The minimizer can now be derived by completing the square. Set $\beta=Q^{-1}q$, so $q=Q\beta$. Substituting this relation into the risk gives

\begin{aligned} R(b) &=m-2b'Q\beta+b'Qb\\ &=m-\beta'Q\beta+(b-\beta)'Q(b-\beta)\\ &=R(\beta)+(b-\beta)'Q(b-\beta). \end{aligned}

Since $Q$ is positive definite, the final added term is nonnegative and equals zero only when $b=\beta$. Therefore $\beta$ uniquely minimizes the risk. This identity also shows how much worse another coefficient vector performs: its excess mean squared error is $(b-\beta)'Q(b-\beta)$.

Finally, the normal equations follow directly from the definition of $\beta$:

E[Xe]=E[X(Y-X'\beta)]=E[XY]-E[XX']\beta=q-Q\beta=0.

This is the projection theorem. The equations are not an additional modeling assumption about how $Y$ is generated. They are necessary properties of the best linear predictor, and under the stated nonsingularity condition they also characterize its unique coefficients.

Worked Example: A Projection With One Regressor and an Intercept

Suppose a hypothetical population has $E[X]=2$, $\operatorname{Var}(X)=4$, $E[Y]=10$, and $\operatorname{Cov}(X,Y)=6$. Find the best linear predictor $a+bX$ and check its residual moments.

Check the conditions. Assume $X$ and $Y$ have finite second moments. With $X$'s mean and variance as given, $E[X^2]=4+2^2=8$. The second-moment matrix for the regressors $(1,X)'$ is

Q= \begin{pmatrix} 1&2\\ 2&8 \end{pmatrix}, \qquad \det(Q)=8-4=4>0.

Thus $Q$ is nonsingular. Also $E[XY]=\operatorname{Cov}(X,Y)+E[X]E[Y]=6+(2)(10)=26$. The normal equations are

\begin{pmatrix} 1&2\\ 2&8 \end{pmatrix} \begin{pmatrix} a\\b \end{pmatrix} = \begin{pmatrix} 10\\26 \end{pmatrix}.

The first equation gives $a+2b=10$, and the second gives $2a+8b=26$. Subtracting twice the first equation from the second gives $4b=6$, so $b=1.5$ and $a=10-(2)(1.5)=7$. The best linear predictor is therefore $7+1.5X$.

For $e=Y-7-1.5X$, the residual mean is $E[e]=10-7-(1.5)(2)=0$. The regressor-residual moment is $E[Xe]=26-(7)(2)-(1.5)(8)=0$. Both normal equations hold, with the intercept accounting for the zero residual mean.

Worked Example: Solving Two Regressor Normal Equations

Consider a hypothetical population with $X=(1,X_1,X_2)'$, where $E[X_1]=E[X_2]=0$. Suppose the second moments are

E[XX']= \begin{pmatrix} 1&0&0\\ 0&4&1\\ 0&1&3 \end{pmatrix}, \qquad E[XY]= \begin{pmatrix} 5\\7\\-1 \end{pmatrix}.

Assume finite second moments. The lower-right block has determinant $(4)(3)-(1)(1)=11$, so the full second-moment matrix is nonsingular. The normal equations are

\begin{pmatrix} 1&0&0\\ 0&4&1\\ 0&1&3 \end{pmatrix} \begin{pmatrix} \beta_0\\\beta_1\\\beta_2 \end{pmatrix} = \begin{pmatrix} 5\\7\\-1 \end{pmatrix}.

The first equation gives $\beta_0=5$. The remaining equations are $4\beta_1+\beta_2=7$ and $\beta_1+3\beta_2=-1$. From the first, $\beta_2=7-4\beta_1$. Substitution into the second yields $\beta_1+3(7-4\beta_1)=-1$, hence $-11\beta_1=-22$, $\beta_1=2$, and $\beta_2=-1$. The unique best linear predictor is $5+2X_1-X_2$.

The specified moments are compatible, for example, with centered jointly normal regressors having covariance matrix $\begin{pmatrix}4&1\\1&3\end{pmatrix}$, and an independent mean-zero error $U$ with variance $4$, when $Y=5+2X_1-X_2+U$. In that construction the projection residual is $U$, and the prediction risk at the projection is $E[U^2]=4$. If the intercept coefficient alone were changed from $5$ to $4$, the prediction error would be $U+1$, with mean squared error $E[(U+1)^2]=4+2E[U]+1=5$. The one-unit increase agrees with the excess-risk identity.

Why Nonsingularity Matters

The second-moment matrix is nonsingular precisely when no nonzero coefficient direction produces a regressor combination that is zero with probability one. If it is singular, different coefficient vectors can describe the same predictor, so the coefficient vector need not be unique. This is why the theorem's uniqueness conclusion requires nonsingularity.

Worked Example: Perfect Collinearity and Nonunique Coefficients

Let a hypothetical population have $X_1$ equally likely to be $-1$ or $1$, let $X_2=2X_1$, and let $U$ be independent of $X_1$, equally likely to be $-1$ or $1$. Set $Y=3+4X_1+U$, and include an intercept along with $X_1$ and $X_2$.

All variables have finite second moments. But the regressors are perfectly collinear because $X_2-2X_1=0$ with probability one. Thus the second-moment matrix is singular. For any coefficients satisfying $\beta_0=3$ and $\beta_1+2\beta_2=4$, the predictor is

\beta_0+\beta_1X_1+\beta_2X_2 =3+(\beta_1+2\beta_2)X_1 =3+4X_1.

There are many such coefficient pairs, including $(\beta_1,\beta_2)=(4,0)$ and $(0,2)$, but they give the same predictor. Its residual is $U$; independence and the zero means give $E[U]=E[X_1U]=E[X_2U]=0$. Thus orthogonality can hold even though the coefficients are not unique. The nonsingular-matrix condition ensures uniqueness of the coefficients, not the existence of orthogonality as an idea.

What Orthogonality Does—and Does Not—Say

The normal equations describe average products: $E[X_j e]=0$. They do not generally imply $E[e\mid X]=0$, which is a stronger condition, and they do not imply that $e$ is independent of $X$. A zero product moment can hold even when the residual's distribution depends on the regressors. The projection theorem is therefore about the best linear prediction criterion, not a claim that the regressors explain every systematic feature of the outcome.

A common mistake is to see $E[Xe]=0$ and conclude that the residual is unrelated to the regressors in every possible sense. The correction is to state exactly what the equations establish: each included regressor has zero expected product with the projection residual. Stronger claims require additional assumptions. Likewise, orthogonality is not a causal interpretation of a coefficient; it follows from the definition of a best linear predictor.

In practice, the theorem gives a compact characterization of the population projection. It lets us determine its coefficients from the moments of the outcome and regressors, verify a proposed coefficient vector by checking residual moments, and diagnose why perfectly redundant regressors block unique coefficient identification.

Key takeaway: With finite second moments and a nonsingular $E[XX']$, the best linear predictor has unique coefficients solving $E[XX']\beta=E[XY]$. Its residual is orthogonal in the expected-product sense to every included regressor.

Check Your Understanding

Try these questions using the definitions and results above.

State the normal equations in terms of the projection residual $e$, and explain why they are equivalent to $E[XX']\beta=E[XY]$.
If a constant is included among the regressors, what does its normal equation imply about the mean of the projection residual?
Why does nonsingularity of $E[XX']$ guarantee that the coefficient vector is unique?
Does $E[Xe]=0$ by itself imply $E[e\mid X]=0$? Explain the distinction.
If two regressors satisfy $X_2=2X_1$ with probability one, what goes wrong with the unique-coefficient conclusion, and what can remain the same across coefficient choices?