Fitted and Residual Sums of Squares Under Normality

Why Separate Fitted Variation from Residual Variation?

In a fixed linear model, least squares splits a response vector into a component lying in the model space and an orthogonal remainder. The corresponding squared lengths describe fitted and residual variation. Under normality, this geometric split has a distributional consequence: the two projected vectors, and therefore their sums of squares, are independent when the errors have spherical covariance.

The word fitted needs care. Here the fitted sum of squares means the squared length of the projection of the response onto the model space, measured from the origin. It is not automatically a corrected sum of squares centered around the sample mean. That distinction matters for interpreting degrees of freedom and whether a sum is central.

Definition: Let $\mathcal S$ be a fixed linear subspace of $\mathbb R^n$, with dimension $p$. Let $P$ be the orthogonal projection matrix onto $\mathcal S$, and set $M=I_n-P$, the projection onto its orthogonal complement. For a response vector $\mathbf Y$, the fitted vector is $P\mathbf Y$, the residual vector is $M\mathbf Y$, the fitted sum of squares is $\mathbf Y^\mathsf{T}P\mathbf Y=\|P\mathbf Y\|^2$, and the residual sum of squares is $\mathbf Y^\mathsf{T}M\mathbf Y=\|M\mathbf Y\|^2$.

An orthogonal projection matrix is symmetric and idempotent: $P^\mathsf{T}=P$ and $P^2=P$. Since $M=I_n-P$, it too is symmetric and idempotent, and $PM=MP=0$. If $\mathcal S$ is the column space of a design matrix $X$ of rank $p$, then $P\mathbf Y$ is the least-squares fitted vector. This definition works even if a design matrix has redundant columns, because the projection is determined by the space rather than by a unique coefficient vector.

The Geometric Decomposition

The fitted and residual vectors are orthogonal: $(P\mathbf Y)^\mathsf{T}(M\mathbf Y)=\mathbf Y^\mathsf{T}PM\mathbf Y=0$. Also $\mathbf Y=P\mathbf Y+M\mathbf Y$. The Pythagorean theorem therefore gives the following identity for every response vector, whether or not it is random.

Theorem: For every $\mathbf y\in\mathbb R^n$, $$ \|\mathbf y\|^2=\|P\mathbf y\|^2+\|M\mathbf y\|^2. $$

To verify the identity directly, expand the squared norm:

\begin{aligned} \|\mathbf y\|^2 &=(P\mathbf y+M\mathbf y)^\mathsf{T}(P\mathbf y+M\mathbf y)\\ &=\mathbf y^\mathsf{T}P\mathbf y +2\mathbf y^\mathsf{T}PM\mathbf y +\mathbf y^\mathsf{T}M\mathbf y\\ &=\mathbf y^\mathsf{T}P\mathbf y+\mathbf y^\mathsf{T}M\mathbf y, \end{aligned}

because $PM=0$. This is a geometric identity, not an independence result. Independence requires distributional assumptions.

Worked Example: Projecting a Response onto a Line-and-Intercept Model

Take three design points $x=(-1,0,1)$, and let the model space consist of vectors $(a-b,a,a+b)^\mathsf{T}$. For the observed response $\mathbf y=(1,2,4)^\mathsf{T}$, calculate the fitted and residual sums of squares.

The design matrix has columns $(1,1,1)^\mathsf{T}$ and $(-1,0,1)^\mathsf{T}$, which are orthogonal and have squared lengths $3$ and $2$. The least-squares intercept and slope are therefore $a=(1+2+4)/3=7/3$ and $b=(-1+4)/2=3/2$. Thus $P\mathbf y=(5/6,7/3,23/6)^\mathsf{T}$, and $M\mathbf y=(1/6,-1/3,1/6)^\mathsf{T}$.

The fitted sum of squares is $25/36+49/9+529/36=(25+196+529)/36=750/36=125/6$. The residual sum of squares is $1/36+1/9+1/36=6/36=1/6$. The response’s squared length is $1^2+2^2+4^2=21$, and $125/6+1/6=21$, confirming the decomposition.

Exact Distributions and Independence Under Normality

Suppose the response has a multivariate normal distribution with spherical covariance: $\mathbf Y\sim N_n(\boldsymbol\mu,\sigma^2 I_n)$, where $\sigma^2>0$, and where the model space and its projection $P$ are fixed. A random design is not covered by this statement unless one conditions on the design and the stated Gaussian model holds conditionally.

Theorem: Let $P$ be a fixed orthogonal projection of rank $p$, and suppose $\mathbf Y\sim N_n(\boldsymbol\mu,\sigma^2I_n)$ with $\sigma^2>0$. Then $P\mathbf Y$ and $M\mathbf Y$ are independent. Moreover, $$ \frac{\mathbf Y^\mathsf{T}P\mathbf Y}{\sigma^2} \sim \chi^2_p\!\left(\frac{\|P\boldsymbol\mu\|^2}{\sigma^2}\right), \qquad \frac{\mathbf Y^\mathsf{T}M\mathbf Y}{\sigma^2} \sim \chi^2_{n-p}\!\left(\frac{\|M\boldsymbol\mu\|^2}{\sigma^2}\right), $$ and these two random variables are independent. Here $\chi^2_r(\lambda)$ denotes a noncentral chi-square distribution with $r$ degrees of freedom and noncentrality parameter $\lambda$; $\chi^2_r=\chi^2_r(0)$ is central chi-square.

Proof. Choose an orthonormal basis for $\mathcal S$ and one for its orthogonal complement. Put the basis vectors into the columns of an orthogonal matrix $Q=[U\ V]$, where $U$ has $p$ columns and $V$ has $n-p$ columns. In these coordinates, $Q^\mathsf{T}\mathbf Y=(U^\mathsf{T}\mathbf Y,V^\mathsf{T}\mathbf Y)^\mathsf{T}$ is normal with mean $Q^\mathsf{T}\boldsymbol\mu$ and covariance $Q^\mathsf{T}(\sigma^2I_n)Q=\sigma^2I_n$. Its density factors into a product of $p$ univariate normal densities for the first $p$ coordinates and $n-p$ for the remaining coordinates. Thus $U^\mathsf{T}\mathbf Y$ and $V^\mathsf{T}\mathbf Y$ are independent.

The projected vectors are $P\mathbf Y=UU^\mathsf{T}\mathbf Y$ and $M\mathbf Y=VV^\mathsf{T}\mathbf Y$. Each is a deterministic function of one of those independent coordinate vectors, so they are independent. Also, $\|P\mathbf Y\|^2=\|U^\mathsf{T}\mathbf Y\|^2$ and $\|M\mathbf Y\|^2=\|V^\mathsf{T}\mathbf Y\|^2$. The coordinates of $U^\mathsf{T}\mathbf Y/\sigma$ are independent unit-variance normal variables with means given by $U^\mathsf{T}\boldsymbol\mu/\sigma$. Their squared sum is noncentral chi-square with $p$ degrees of freedom and noncentrality $\|U^\mathsf{T}\boldsymbol\mu\|^2/\sigma^2=\|P\boldsymbol\mu\|^2/\sigma^2$. The same reasoning for $V$ gives the residual distribution and its noncentrality. Independence of the sums follows from independence of the coordinate blocks. $\square$

The theorem allows the mean to lie anywhere in $\mathbb R^n$. It does not require the mean to belong to the model space for independence. The mean’s location instead determines whether either chi-square distribution is central.

Key consequence: The residual sum of squares is central precisely when $M\boldsymbol\mu=0$, equivalently when $\boldsymbol\mu\in\mathcal S$. The fitted sum of squares is central precisely when $P\boldsymbol\mu=0$, equivalently when $\boldsymbol\mu\in\mathcal S^\perp$. Both are central only when $\boldsymbol\mu=0$.

Worked Example: A Correctly Specified Mean Gives a Central Residual Sum

Let $u=(1,1,1)^\mathsf{T}/\sqrt{3}$, and take the model space $\mathcal S=\operatorname{span}(u)$, so $p=1$. Suppose $\mathbf Y\sim N_3(2u,4I_3)$. Determine the exact distributions and independence of the fitted and residual sums of squares.

The mean $2u$ lies in $\mathcal S$, so $P\boldsymbol\mu=2u$ and $M\boldsymbol\mu=0$. Its squared length is $\|2u\|^2=4$. The theorem gives $\mathbf Y^\mathsf{T}P\mathbf Y/4\sim\chi^2_1(1)$, since the fitted noncentrality is $4/4=1$. It also gives $\mathbf Y^\mathsf{T}M\mathbf Y/4\sim\chi^2_2$, central, because the residual noncentrality is zero. The two sums are independent.

As a check on the scale, the expected fitted sum is $4(1+1)=8$, and the expected residual sum is $4(2)=8$. The first expectation includes both the variance contribution and the nonzero mean; the second has only its two residual degrees of freedom.

Worked Example: A Mean Outside the Model Makes the Residual Noncentral

In $\mathbb R^2$, let $P$ project onto the first coordinate axis and suppose $\mathbf Y\sim N_2((1,2)^\mathsf{T},I_2)$. Find the distributions of the two sums of squares.

Here $P\mathbf Y=(Y_1,0)^\mathsf{T}$ and $M\mathbf Y=(0,Y_2)^\mathsf{T}$. The fitted sum is $Y_1^2$; since $Y_1\sim N(1,1)$, $Y_1^2\sim\chi^2_1(1)$. The residual sum is $Y_2^2$; since $Y_2\sim N(2,1)$, $Y_2^2\sim\chi^2_1(4)$. The two coordinates are independent under the specified multivariate normal distribution, so the sums are independent. The residual is not central because the mean has a component of length $2$ in the orthogonal complement, giving noncentrality $2^2/1=4$.

When Is This Result Useful?

The exact laws identify degrees of freedom directly from dimensions: the fitted projection has $p$ coordinates, and the residual projection has $n-p$. They also make clear which model assumptions support familiar normal-theory calculations. If the mean belongs to the model space, residual variation has a central chi-square distribution, while the fitted variation may retain a nonzero mean contribution. Independence holds regardless of whether the mean belongs to the space, provided the response is Gaussian with covariance $\sigma^2I_n$ and the projection is fixed.

This result concerns the uncentered fitted sum $\|P\mathbf Y\|^2$. A corrected model sum of squares involves a different projection when the model contains an intercept; it should not be substituted for $\|P\mathbf Y\|^2$ without checking the relevant subspaces and degrees of freedom.

Common Mistake

Wrong reasoning: “The mean is in the model space, so both fitted and residual sums of squares are central chi-square variables.” The mean condition only makes the residual mean zero after projection: $M\boldsymbol\mu=0$. It usually leaves $P\boldsymbol\mu=\boldsymbol\mu\ne0$, so the fitted sum is noncentral. A second error is to infer independence merely from the Pythagorean identity. Orthogonality of the projections gives the identity for every vector, but independence here follows from the Gaussian distribution and spherical covariance.

Check Your Understanding

Use the projection definitions and normal-theory result to answer the following.

If the model space has dimension $p$, what are the degrees of freedom of the fitted and residual sums of squares?
Under what condition on $\boldsymbol\mu$ is the residual sum of squares central chi-square after division by $\sigma^2$?
Does independence of fitted and residual sums require $\boldsymbol\mu$ to lie in the model space? Explain.
For $\mathbf Y\sim N_n(\boldsymbol\mu,\sigma^2I_n)$, write the noncentrality parameter of the fitted sum.
Why does the Pythagorean decomposition alone not establish that the two sums are independent?