Fitted Values and Residuals as Orthogonal Projections

Introduction

In multivariate regression, each case contributes a vector of responses, and the same design matrix is used to fit all response variables. The fitted response matrix and the residual matrix are therefore not separate calculations for each outcome: both arise by projecting the observed response matrix onto complementary subspaces of the case space.

This geometric view gives more than a compact formula. It explains why fitted values and residuals are orthogonal, and why the response cross-product splits into fitted and residual contributions without a cross-term. The centered version of that split has an additional condition: the model must include an intercept.

The Hat Matrix and Residual-Maker Matrix

Let $Y$ be an $n\times p$ response matrix and $X$ an $n\times q$ design matrix of full column rank $q$. The multivariate linear model has mean $XB$, where $B$ is a $q\times p$ coefficient matrix. The ordinary least-squares coefficient estimate is $\hat B=(X'X)^{-1}X'Y$. For the projection results below, no distributional assumption on the errors is required.

Definition: The hat matrix and the residual-maker matrix are $$ H=X(X'X)^{-1}X', \qquad M=I_n-H. $$ The fitted response matrix is $\hat Y=X\hat B=HY$, and the residual matrix is $E=Y-\hat Y=MY$. The columns of $H$ lie in the column space of $X$; $M$ projects onto its orthogonal complement.

The name “hat matrix” reflects that $H$ puts a hat on $Y$, producing $\hat Y$. The matrices $H$ and $M$ act on the rows of $Y$, that is, on the case dimension. They do not mix the $p$ response variables with one another.

Because $X$ has full column rank, $X'X$ is invertible and symmetric. Direct calculation gives $H'=H$ and $H^2=H$. Thus $H$ is a symmetric idempotent matrix, the defining matrix properties of an orthogonal projection. It follows that $M'=M$, $M^2=M$, and $HM=MH=0$.

Worked Example: An Intercept-Only Model

Suppose $n=3$, there are two responses, and the model includes only an intercept. Then $X=\mathbf{1}=(1,1,1)'$, so $$ H=\frac{1}{3}\mathbf{1}\mathbf{1}' =\frac{1}{3} \begin{pmatrix} 1&1&1\\ 1&1&1\\ 1&1&1 \end{pmatrix}. $$ For $$ Y= \begin{pmatrix} 1&2\\ 2&0\\ 3&1 \end{pmatrix}, $$ the column means are $2$ and $1$. Multiplying by $H$ repeats each mean in every row: $$ \hat Y=HY= \begin{pmatrix} 2&1\\ 2&1\\ 2&1 \end{pmatrix}, \qquad E=Y-\hat Y= \begin{pmatrix} -1&1\\ 0&-1\\ 1&0 \end{pmatrix}. $$ Each residual column sums to zero, so $\hat Y'E=0$: each fitted column is constant across cases, and is orthogonal to a residual column whose entries sum to zero. This is the projection geometry in its simplest form.

Orthogonality of Fitted Values and Residuals

For matrices with $n$ rows, orthogonality means that every column of one matrix is orthogonal to every column of the other. Equivalently, the matrix of pairwise inner products is zero. Here that matrix is $\hat Y'E$, a $p\times p$ matrix.

Theorem: Under the full-column-rank assumption on $X$, the fitted response matrix and residual matrix are orthogonal: $$ \hat Y'E=0_{p\times p}. $$ Consequently, the uncentered response cross-product decomposes as $$ Y'Y=\hat Y'\hat Y+E'E. $$

Proof. Since $\hat Y=HY$ and $E=MY$, their cross-product is $$ \hat Y'E=(HY)'(MY)=Y'H'MY. $$ The hat matrix is symmetric, so $H'=H$. Also $HM=H(I_n-H)=H-H^2=0$. Therefore $$ \hat Y'E=Y'HM Y=Y'0Y=0_{p\times p}. $$ Now write $Y=\hat Y+E$, multiply out, and use both $\hat Y'E=0$ and its transpose $E'\hat Y=0$: $$ Y'Y=(\hat Y+E)'(\hat Y+E) =\hat Y'\hat Y+\hat Y'E+E'\hat Y+E'E =\hat Y'\hat Y+E'E. $$ This proves both claims. $\square$

The entries of these cross-product matrices summarize sums of products across cases. For example, the $(j,k)$ entry of $\hat Y'E$ is the inner product of the fitted values for response $j$ and the residuals for response $k$. The zero matrix result includes both the same-response orthogonality and the cross-response orthogonality.

Worked Example: A Two-Response Regression With an Intercept

Take $x=(-1,0,1)'$, $X=[\mathbf{1},x]$, and $$ Y= \begin{pmatrix} 1&2\\ 2&1\\ 4&3 \end{pmatrix}. $$ For a regression on an intercept and $x$, each fitted column is its sample mean plus its estimated slope times $x$. The means are $7/3$ and $2$. The slopes are $$ \frac{(-1)(1)+0(2)+1(4)}{(-1)^2+0^2+1^2}=\frac{3}{2}, \qquad \frac{(-1)(2)+0(1)+1(3)}{2}=\frac{1}{2}. $$ Thus $$ \hat Y= \begin{pmatrix} 5/6&3/2\\ 7/3&2\\ 23/6&5/2 \end{pmatrix}, \qquad E= \begin{pmatrix} 1/6&1/2\\ -1/3&-1\\ 1/6&1/2 \end{pmatrix}. $$ Each residual column sums to zero, and its inner product with $x$ is also zero. For example, the first residual column has inner product $(-1)(1/6)+0(-1/3)+1(1/6)=0$. The fitted columns are combinations of $\mathbf{1}$ and $x$, so they are orthogonal to both residual columns.

The centered fitted values are the fitted slopes times $x$. Since $x'x=2$, their cross-product is $$ (\hat Y-\mathbf{1}\bar y')'(\hat Y-\mathbf{1}\bar y') = \begin{pmatrix} (3/2)^2(2)&(3/2)(1/2)(2)\\ (3/2)(1/2)(2)&(1/2)^2(2) \end{pmatrix} = \begin{pmatrix} 9/2&3/2\\ 3/2&1/2 \end{pmatrix}. $$ The residual cross-product is $$ E'E= \begin{pmatrix} 1/6&1/2\\ 1/2&3/2 \end{pmatrix}. $$ Their sum is $$ \begin{pmatrix} 14/3&2\\ 2&2 \end{pmatrix}, $$ which is the centered response cross-product $(Y-\mathbf{1}\bar y')'(Y-\mathbf{1}\bar y')$. Here $\bar y'=(7/3,2)$. Because the model includes an intercept, the fitted columns contain the constant component, and the residual columns have zero sums; this yields the centered decomposition.

When the Centered Decomposition Holds

The theorem gives an uncentered identity for any full-column-rank design. Many regression summaries instead describe variation around the response means. Define $C=I_n-\mathbf{1}\mathbf{1}'/n$, the centering matrix. The centered response cross-product is $Y'CY$.

If the model includes an intercept, then $\mathbf{1}$ belongs to the column space of $X$, and $H\mathbf{1}=\mathbf{1}$. Consequently, $\mathbf{1}'E=\mathbf{1}'MY=0$. The fitted values and residuals are orthogonal, and the residuals have zero column means. Writing $\bar y'= \mathbf{1}'Y/n$, we have $\hat Y$'s column means equal to $\bar y'$, so $$ Y-\mathbf{1}\bar y'=(\hat Y-\mathbf{1}\bar y')+E. $$ The two terms on the right are orthogonal: the centered fitted matrix has columns in the column space of $X$, while $E$'s columns are orthogonal to that space. Therefore $$ Y'CY=(\hat Y-\mathbf{1}\bar y')'(\hat Y-\mathbf{1}\bar y')+E'E. $$ This is the centered variation decomposition when an intercept is fitted.

Worked Example: Why an Intercept Matters for Centered Variation

Consider a one-column design $X=(1,1,0)'$, which does not include an intercept, and $$ Y= \begin{pmatrix} 2&1\\ 4&3\\ 1&5 \end{pmatrix}. $$ Since $X'X=2$, the projection is $$ H=\frac{1}{2} \begin{pmatrix} 1&1&0\\ 1&1&0\\ 0&0&0 \end{pmatrix}. $$ The fitted values average the first two rows and set the third row to zero: $$ \hat Y= \begin{pmatrix} 3&2\\ 3&2\\ 0&0 \end{pmatrix}, \qquad E= \begin{pmatrix} -1&-1\\ 1&1\\ 1&5 \end{pmatrix}. $$ The ordinary projection identity holds: $$ \hat Y'\hat Y= \begin{pmatrix} 18&12\\ 12&8 \end{pmatrix}, \qquad E'E= \begin{pmatrix} 3&7\\ 7&27 \end{pmatrix}, \qquad Y'Y= \begin{pmatrix} 21&19\\ 19&35 \end{pmatrix}. $$ Indeed, the first two matrices sum to the third. But the residual column means are $1/3$ and $5/3$, not zero. The centered fitted and residual matrices need not be orthogonal: for example, the centered first fitted column is $(1,1,-2)'$, and the centered first residual column is $(-4/3,2/3,2/3)'$. Their inner product is $-2$, not zero. Thus the centered cross-product does not split into centered fitted and residual cross-products without an additional cross-term. An intercept ensures the zero-residual-means property needed for that centered split.

Common Mistakes and Why the Geometry Matters

Assuming orthogonality without checking the fit. The result follows because $H$ is a symmetric projection and $HM=0$, not merely because fitted values and residuals have different names.
Confusing uncentered and centered cross-products. The identity $Y'Y=\hat Y'\hat Y+E'E$ follows for the full-rank design considered here. A decomposition around the response means additionally requires an intercept.
Treating the response variables as the projected dimension. The matrices $H$ and $M$ multiply on the left. They project case-wise patterns while retaining all response columns.

These identities matter because they separate the variation represented by the fitted model from the variation left in the residuals, without a cross-product between the two parts. The same projection structure applies to every response column, while the cross-product matrices preserve relationships among responses. The next step in using this geometry is to study the residual cross-product and its degrees of freedom.

Key takeaway: For a full-column-rank design, $H$ and $I_n-H$ are orthogonal projections. Hence $\hat Y=HY$ and $E=(I_n-H)Y$ satisfy $\hat Y'E=0$ and $Y'Y=\hat Y'\hat Y+E'E$. Centered variation has the analogous split when the model includes an intercept.

Check Your Understanding

Use the projection definitions and conditions above to answer these questions.

Why is $H$ symmetric and idempotent when $X$ has full column rank?
Show how $HM=0$ implies $\hat Y'E=0$.
What matrix identity decomposes the uncentered response cross-product?
What additional condition on the design is needed for the centered decomposition, and what property of the residuals follows?
In a multivariate model, what does the $(j,k)$ entry of $\hat Y'E$ represent?