Projection Matrices and Regression Geometry

Regression as Geometry in Observation Space

In a linear regression with \(n\) observations, the response vector \(y\) is a point in \(n\)-dimensional observation space. The columns of the design matrix \(X\) span the set of vectors the model can fit. The fitted vector is the projection of \(y\) onto that column space, and the residual vector is the part of \(y\) perpendicular to it.

The projection theorem and normal equations establish that, when \(X\) has full column rank, the fitted vector lies in the column space of \(X\) and the residual is orthogonal to every column of \(X\). Here we express those geometric facts as matrices and derive the properties that follow. These results concern the geometry of the sample vectors; they do not require a probability model for \(y\).

Setup: Let \(X\) be an \(n \times k\) design matrix with full column rank, and let \(y\) be an \(n \times 1\) response vector. Then \(X'X\) is invertible. The coefficient vector satisfying the normal equations is \(\hat{\beta}=(X'X)^{-1}X'y\), so the fitted vector is \(\hat{y}=X\hat{\beta}\), and the residual vector is \(e=y-\hat{y}\).

Substituting the coefficient vector into the fitted values gives a matrix that maps \(y\) directly to \(\hat{y}\). Subtracting that fitted vector from \(y\) gives a second matrix that maps \(y\) to \(e\).

Definition: The hat matrix and the residual-maker matrix are

H=X(X'X)^{-1}X', \qquad M=I_n-H.

Thus \(\hat{y}=Hy\) and \(e=My\). The hat matrix is so named because it puts a “hat” on \(y\): it transforms the observed response vector into its fitted counterpart. The matrix \(M\) removes the component of \(y\) that lies in the column space of \(X\).

Why These Matrices Are Projections

A projection onto a subspace leaves vectors in that subspace unchanged. Projecting twice therefore has the same result as projecting once. For \(H\), this property is called idempotence. Because \(X'X\) is symmetric and invertible, its inverse is symmetric, which also makes \(H\) symmetric. Symmetry and idempotence together characterize an orthogonal projection matrix.

H'=H, \qquad H^2=H.

The residual-maker matrix has the same two properties. It is symmetric because it is the difference of the symmetric matrices \(I_n\) and \(H\). Its idempotence follows by expanding the product and using \(H^2=H\):

M'=M, \qquad M^2=(I_n-H)^2=I_n-2H+H^2=I_n-H=M.

The two projections act on perpendicular subspaces. The fitted vector lies in the column space of \(X\), while the residual is perpendicular to that space. In matrix form, their interaction is zero:

HM=H(I_n-H)=H-H^2=0, \qquad MH=0.

In particular, \(X'e=0\). This states that the residual vector is orthogonal to each column of \(X\). It is also the normal-equations condition. The fitted vector and residual vector are perpendicular to one another because \(\hat{y}'e=y'HMe=0\).

Worked Example: Projecting a Response Onto a Regression Line

Consider three hypothetical observations with an intercept and a single predictor taking values \(0,1,2\). The design matrix and response vector are

X= \begin{bmatrix} 1&0\\ 1&1\\ 1&2 \end{bmatrix}, \qquad y= \begin{bmatrix} 1\\2\\4 \end{bmatrix}.

The columns of \(X\) are linearly independent, so \(X\) has full column rank. Computing \(X(X'X)^{-1}X'\) gives

H=\frac{1}{6} \begin{bmatrix} 5&2&-1\\ 2&2&2\\ -1&2&5 \end{bmatrix}.

Apply the hat matrix to \(y\):

\hat{y}=Hy =\frac{1}{6} \begin{bmatrix} 5(1)+2(2)-1(4)\\ 2(1)+2(2)+2(4)\\ -1(1)+2(2)+5(4) \end{bmatrix} = \begin{bmatrix} 5/6\\7/3\\23/6 \end{bmatrix}.

The residual vector is

e=y-\hat{y} = \begin{bmatrix} 1/6\\-1/3\\1/6 \end{bmatrix}.

Check orthogonality against both columns of \(X\): the residuals sum to \(1/6-1/3+1/6=0\), and their predictor-weighted sum is \(0(1/6)+1(-1/3)+2(1/6)=0\). Thus \(X'e=0\), as required. The residual-maker matrix for this design is \(M=I_3-H\); applying it to \(y\) produces the same residual vector.

Ranks, Dimensions, and Leverage

The column space of \(X\) has dimension \(k\), so its projection matrix \(H\) has rank \(k\). The residuals lie in the orthogonal complement of that space, which has dimension \(n-k\); therefore \(M\) has rank \(n-k\). These ranks also follow from the fact that \(H\) and \(M\) project onto complementary subspaces.

The diagonal entry \(h_{ii}\) of \(H\) is called the leverage of observation \(i\). It describes how much the fitted value at that observation depends on its own response, holding the design fixed: from \(\hat{y}=Hy\), \(\hat{y}_i=\sum_{j=1}^n h_{ij}y_j\), so \(h_{ii}\) is the coefficient multiplying \(y_i\). For an orthogonal projection, \(0\leq h_{ii}\leq1\), and the leverages sum to the rank:

\sum_{i=1}^n h_{ii}=\operatorname{tr}(H)=k.

The trace identity follows directly from the matrix definition and the cyclic property of the trace: \(\operatorname{tr}(H)=\operatorname{tr}((X'X)^{-1}X'X)=\operatorname{tr}(I_k)=k\). Since \(M=I_n-H\), its trace is \(n-k\). These are algebraic facts about the design matrix; a large leverage value alone does not establish that an observation is erroneous or influential.

Worked Example: Leverage and Residuals With a Sparse Design

Suppose four hypothetical responses are recorded, and the model has one regressor whose values are \(1,0,1,0\). Thus \(X=(1,0,1,0)'\), which has rank \(1\). Its hat matrix is

H=X(X'X)^{-1}X' =\frac{1}{2} \begin{bmatrix} 1&0&1&0\\ 0&0&0&0\\ 1&0&1&0\\ 0&0&0&0 \end{bmatrix}.

For \(y=(2,1,4,3)'\), the fitted values and residuals are

\hat{y}=Hy= \begin{bmatrix}3\\0\\3\\0\end{bmatrix}, \qquad e=y-\hat{y}= \begin{bmatrix}-1\\1\\1\\3\end{bmatrix}.

The residual is orthogonal to the sole design column: \(X'e=(-1)+1=0\). The diagonal of \(H\) is \((1/2,0,1/2,0)\), so its leverages sum to \(1\), the rank of \(X\). The residual-maker matrix has rank \(4-1=3\). This example also shows that a zero regressor value does not force a fitted value to equal the observed response: the model can fit only vectors in the span of \(X\).

Orthogonal Decomposition of Variation

Since \(y=\hat{y}+e\) and \(\hat{y}'e=0\), the squared length of the response vector splits into the squared length of its fitted component and that of its residual component. This is the Pythagorean theorem applied in observation space:

y'y=(\hat{y}+e)'(\hat{y}+e) =\hat{y}'\hat{y}+2\hat{y}'e+e'e =\hat{y}'\hat{y}+e'e.

The identity holds for a projection onto any linear column space, including one that does not contain a constant vector. It decomposes squared length measured from the origin. It should not automatically be interpreted as a decomposition of variation around the sample mean: that centered decomposition requires the model to include an intercept.

Worked Example: Checking the Orthogonal Decomposition

Return to the three-observation design with \(y=(1,2,4)'\). The response vector has squared length

$$ y'y=1^2+2^2+4^2=21. $$

Using the fitted values from the first example,

\hat{y}'\hat{y} =\left(\frac{5}{6}\right)^2 +\left(\frac{7}{3}\right)^2 +\left(\frac{23}{6}\right)^2 =\frac{25+196+529}{36} =\frac{125}{6}.

The residual squared length is

e'e=\left(\frac{1}{6}\right)^2 +\left(-\frac{1}{3}\right)^2 +\left(\frac{1}{6}\right)^2 =\frac{6}{36} =\frac{1}{6}.

Therefore \(\hat{y}'\hat{y}+e'e=125/6+1/6=21=y'y\). The equality holds because the fitted and residual vectors are perpendicular, not because each term has been centered around the sample mean.

A Short Derivation From the Normal Equations

The matrix formulas can also be checked directly against the defining projection conditions. The fitted vector \(Hy\) belongs to the column space of \(X\), since \(Hy=X[(X'X)^{-1}X'y]\). The residual is orthogonal to that space because

X'(y-Hy)=X'y-X'X(X'X)^{-1}X'y=0.

A vector in the column space of \(X\) whose difference from \(y\) is orthogonal to that space is exactly the orthogonal projection of \(y\) onto the space. This verifies that \(H\) gives the fitted vector and \(M\) gives the residual vector. It also explains why full column rank matters for the displayed inverse formula: if the columns are linearly dependent, \((X'X)^{-1}\) does not exist, and that formula cannot be used as written.

Common Mistakes and Why the Geometry Matters

Confusing \(H\) with \(M\). \(H\) produces fitted values, \(Hy\); \(M=I_n-H\) produces residuals, \(My\). They project onto perpendicular, complementary spaces.
Assuming every residual is zero. The projection makes residuals orthogonal to the design columns, not generally zero. Residuals vanish for every \(y\) only if the column space fills observation space.
Using the centered variation decomposition without checking the intercept. The identity \(y'y=\hat{y}'\hat{y}+e'e\) is about lengths from the origin. The familiar decomposition around the sample mean requires an intercept.
Treating leverage as a response diagnostic by itself. Leverage is determined by the design matrix. It indicates a position in predictor space, not whether the response is unusual or whether the observation changes a particular estimate substantially.

Key takeaway: With a full-column-rank design, \(H\) and \(M=I_n-H\) are symmetric, idempotent projections onto the model space and its orthogonal complement. Their orthogonality gives \(X'e=0\) and the decomposition \(y'y=\hat{y}'\hat{y}+e'e\).

Check Your Understanding

Try these questions without looking back.

For a full-column-rank \(n \times k\) design, write the definitions of \(H\) and \(M\), and state which vector each matrix produces when applied to \(y\).
Use \(H^2=H\) to show that \(HM=0\), where \(M=I_n-H\).
Why is the rank of \(M\) equal to \(n-k\) under the stated full-rank condition?
What does the identity \(X'e=0\) say geometrically?
Does \(y'y=\hat{y}'\hat{y}+e'e\) by itself establish a decomposition of variation around the sample mean? Explain.