Residual Sums of Squares in Normal Samples

Why Centering Changes the Distribution

For a normal sample, the centered residual sum of squares measures variation around the sample mean rather than around a fixed, known center. That distinction matters: estimating the center from the same observations imposes one linear constraint on the residuals, so the resulting sum of squares has one fewer degree of freedom than an uncentered sum of squared standard normal variables.

The exact distribution follows from geometry. The vector of observations has a multivariate normal distribution, and subtracting the sample mean projects that vector away from the direction in which all coordinates are equal. An orthogonal change of coordinates isolates this direction and leaves \(n-1\) independent standard normal coordinates to contribute to the residual sum of squares.

Definition: Let \(X_1,\ldots,X_n\) be independent random variables with common distribution \(N(\mu,\sigma^2)\), where \(n\geq 2\) and \(\sigma^2>0\). Define the sample mean and centered residual sum of squares by

\bar X=\frac{1}{n}\sum_{i=1}^n X_i, \qquad R=\sum_{i=1}^n (X_i-\bar X)^2.

Here \(R\) is the unscaled residual sum of squares. The quantity \(R/\sigma^2\) is its version scaled by the population variance.

A chi-square random variable with \(k\) degrees of freedom, written \(\chi^2_k\), is the sum of squares of \(k\) independent \(N(0,1)\) random variables. In particular, \(k\) counts the independent standard normal coordinates being squared. The central result for \(R\) is as follows.

Theorem: Under the assumptions above,

\frac{R}{\sigma^2}\sim\chi^2_{n-1}.

Thus the centered residual sum of squares has \(n-1\) degrees of freedom. Its distribution does not depend on \(\mu\).

Derivation by Orthogonal Coordinates

Write the sample as the vector \(X=(X_1,\ldots,X_n)^{\mathsf T}\), and let \(\mathbf 1=(1,\ldots,1)^{\mathsf T}\). The vector \(u=\mathbf 1/\sqrt n\) has length one. The matrix \(P=I-uu^{\mathsf T}\) is the orthogonal projection onto the subspace perpendicular to \(u\). Since \(PX=X-\bar X\mathbf 1\), the residual sum of squares is the squared length of that projection.

R=\lVert PX\rVert^2 =\sum_{i=1}^n(X_i-\bar X)^2.

To see exactly which random coordinates contribute, standardize the sample vector by setting \(Z=(X-\mu\mathbf 1)/\sigma\). Its coordinates are independent \(N(0,1)\) variables. Choose an orthogonal \(n\) by \(n\) matrix \(Q\) whose first column is \(u\); such a matrix exists by extending \(u\) to an orthonormal basis. Define \(Y=Q^{\mathsf T}Z\).

Proof of the theorem: The joint density of \(Z\) at \(z\in\mathbb R^n\) is \((2\pi)^{-n/2}\exp(-z^{\mathsf T}z/2)\). The change of coordinates \(y=Q^{\mathsf T}z\) has Jacobian determinant of absolute value \(1\), and \(z^{\mathsf T}z=y^{\mathsf T}y\), because \(Q\) is orthogonal. Therefore the joint density of \(Y\) is

(2\pi)^{-n/2}\exp\left(-\frac12\sum_{j=1}^n y_j^2\right) =\prod_{j=1}^n\frac{1}{\sqrt{2\pi}}\exp\left(-\frac{y_j^2}{2}\right).

This factorization shows that \(Y_1,\ldots,Y_n\) are independent \(N(0,1)\) variables. Because the first column of \(Q\) is \(u\), the projection \(P\) removes the first coordinate in this orthonormal basis and retains the other \(n-1\). Since \(X=\mu\mathbf 1+\sigma Z\) and \(P\mathbf 1=0\), it follows that

\frac{R}{\sigma^2} =Z^{\mathsf T}PZ =\sum_{j=2}^n Y_j^2.

The right-hand side is, by definition, chi-square with \(n-1\) degrees of freedom. This proves the stated distribution and shows why \(\mu\) does not appear in it. \(\square\)

The same decomposition also explains the degrees of freedom without relying only on a matrix calculation: the residuals satisfy \(\sum_{i=1}^n(X_i-\bar X)=0\). Once any \(n-1\) residuals are specified, the final residual is determined. The orthogonal-coordinate argument makes this constraint precise for the distribution: exactly \(n-1\) independent standard normal coordinates remain after centering.

Density and Moments

For \(n\geq2\), the theorem gives the density of \(R\) by scaling the chi-square density. If \(k=n-1\), the density of a \(\chi^2_k\) variable \(W\) is \(w^{k/2-1}e^{-w/2}/(2^{k/2}\Gamma(k/2))\) for \(w>0\), where \(\Gamma\) is the gamma function. Applying the change of variable \(w=r/\sigma^2\) gives:

f_R(r)= \frac{r^{(n-1)/2-1}\exp\left(-r/(2\sigma^2)\right)} {2^{(n-1)/2}\Gamma\left((n-1)/2\right)(\sigma^2)^{(n-1)/2}}, \qquad r>0.

The expectation and variance follow from the sum-of-squares representation. If \(Z_0\sim N(0,1)\), then \(E[Z_0^2]=1\) and \(\operatorname{Var}(Z_0^2)=E[Z_0^4]-E[Z_0^2]^2=3-1=2\). The \(n-1\) squared coordinates in the representation are independent, so their expectations and variances add. Consequently,

E[R]=(n-1)\sigma^2, \qquad \operatorname{Var}(R)=2(n-1)\sigma^4.

These moments provide useful checks on calculations. In particular, the expected residual sum of squares is smaller than the expected uncentered sum of squared deviations from the true mean, which is \(n\sigma^2\). Centering uses the data to locate the center and removes one direction of variation from the sum.

Worked Applications

Worked Example: A Tail Probability with Three Observations

Suppose \(X_1,X_2,X_3\) are independent \(N(\mu,4)\) observations, where \(\mu\) is unspecified. Find \(P(R>8)\), with \(R=\sum_{i=1}^3(X_i-\bar X)^2\).

Here \(n=3\) and \(\sigma^2=4\), so the theorem gives \(R/4\sim\chi^2_2\). Therefore

P(R>8)=P\left(\frac{R}{4}>2\right)=P(\chi^2_2>2).

A \(\chi^2_2\) variable has density \(\frac12 e^{-w/2}\) for \(w>0\), so its survival probability at \(2\) is

P(\chi^2_2>2)=\int_2^\infty \frac12e^{-w/2}\,dw =\left[-e^{-w/2}\right]_{2}^{\infty} =e^{-1}\approx 0.3679.

Thus \(P(R>8)=e^{-1}\). The answer does not require knowing \(\mu\).

Worked Example: Checking an Observed Residual Sum of Squares

Suppose five observed values are \(2,4,6,8,10\). Calculate their residual sum of squares. Then, under a model in which the five observations are independent \(N(\mu,4)\) variables, state the distribution of the corresponding random residual sum of squares.

The observed mean is

\bar x=\frac{2+4+6+8+10}{5}=6.

The observed residual sum of squares is

$$ r=(2-6)^2+(4-6)^2+(6-6)^2+(8-6)^2+(10-6)^2 =16+4+0+4+16=40. $$

Under the stated model, \(n=5\) and \(\sigma^2=4\), so the random sum satisfies \(R/4\sim\chi^2_4\). The observed scaled value is \(r/4=10\). This is one realized value of a \(\chi^2_4\) statistic; it does not change the sampling distribution, which is determined by the model and sample size.

Worked Example: Expressing Residual Variation Through Contrasts

For four independent \(N(\mu,\sigma^2)\) observations, use orthogonal contrasts to verify the residual sum of squares for the data \((3,1,2,6)\), and identify the sampling distribution of the scaled result.

Three mutually orthogonal unit vectors perpendicular to \((1,1,1,1)^{\mathsf T}\) are

v_1=\frac{(1,-1,0,0)^{\mathsf T}}{\sqrt2},\qquad v_2=\frac{(1,1,-2,0)^{\mathsf T}}{\sqrt6},\qquad v_3=\frac{(1,1,1,-3)^{\mathsf T}}{\sqrt{12}}.

They form an orthonormal basis for the residual subspace. For \(x=(3,1,2,6)^{\mathsf T}\), the three contrast coordinates are

v_1^{\mathsf T}x=\frac{2}{\sqrt2}=\sqrt2,\qquad v_2^{\mathsf T}x=\frac{3+1-4}{\sqrt6}=0,\qquad v_3^{\mathsf T}x=\frac{3+1+2-18}{\sqrt{12}}=-2\sqrt3.

Their squared sum is \(2+0+12=14\). Directly, \(\bar x=3\), so the residual sum of squares is \((3-3)^2+(1-3)^2+(2-3)^2+(6-3)^2=0+4+1+9=14\), agreeing with the contrast calculation. Under the normal model, each contrast applied to \(X\) has mean zero and variance \(\sigma^2\); after division by \(\sigma\), the three coordinates are independent standard normals. Hence \(R/\sigma^2\sim\chi^2_3\).

Common Mistake: Counting the Wrong Directions

A tempting but incorrect argument is to standardize each observation around the population mean and conclude that the centered residual sum of squares has \(n\) degrees of freedom. Indeed, the uncentered sum \(\sum_{i=1}^n(X_i-\mu)^2/\sigma^2\) is \(\chi^2_n\), because it is the sum of squares of \(n\) independent standard normal variables. But that is not the same sum as \(R/\sigma^2\): \(R\) centers at the random sample mean, not at \(\mu\).

The exact relationship is

\sum_{i=1}^n(X_i-\mu)^2 = \sum_{i=1}^n(X_i-\bar X)^2+n(\bar X-\mu)^2.

To verify it, write \(X_i-\mu=(X_i-\bar X)+(\bar X-\mu)\), expand the squares, and use \(\sum_{i=1}^n(X_i-\bar X)=0\) to see that the cross term vanishes. The uncentered sum includes the component in the constant-vector direction, whereas \(R\) does not. Therefore its \(n\) degrees of freedom cannot be assigned to \(R\); the centered residual sum has \(n-1\).

Key takeaway: For independent observations from \(N(\mu,\sigma^2)\), centering at the sample mean projects away one direction. The exact result is \(R/\sigma^2\sim\chi^2_{n-1}\), with the degrees of freedom equal to the dimension of the residual subspace.

Check Your Understanding

Use the normal-sample assumptions and the orthogonal-coordinate argument to answer these questions.

For a sample of size \(n=8\) from \(N(\mu,\sigma^2)\), what is the distribution of \(R/\sigma^2\)?
Why does centering at \(\bar X\) remove one degree of freedom?
If \(n=3\) and \(\sigma^2=9\), express \(P(R\leq 9)\) as a probability involving a chi-square variable.
Explain why \(\sum_{i=1}^n(X_i-\mu)^2/\sigma^2\) and \(R/\sigma^2\) have different degrees of freedom.
For \(n=5\), what are \(E[R]\) and \(\operatorname{Var}(R)\) in terms of \(\sigma^2\)?