Nested Projections and Incremental Sums of Squares

Why Nested Projections Produce Increments

Suppose a sequence of models or subspaces becomes progressively larger. The change in a projected sum of squares from one stage to the next is an incremental sum of squares. For Gaussian vectors with spherical covariance, each such change corresponds to a component in a direction orthogonal to the preceding subspace. This geometry determines both the degrees of freedom and the distribution of the increment.

The scale matters: when the covariance is \(\sigma^2 I_p\), an unscaled sum of squares is generally \(\sigma^2\) times a chi-square variable, not a chi-square variable itself. The increment divided by \(\sigma^2\) has a chi-square law when its projected mean is zero, and a noncentral chi-square law otherwise.

Nested Subspaces and Their Projection Differences

Definition: Let \(S_0\subseteq S_1\subseteq\cdots\subseteq S_m\) be nested linear subspaces of \(\mathbb R^p\). Let \(P_j\) be the orthogonal projection matrix onto \(S_j\). For \(j=1,\ldots,m\), define the \(j\)th increment matrix by \(Q_j=P_j-P_{j-1}\). The corresponding incremental sum of squares for a vector \(x\) is \(x^{\mathsf T}Q_jx\).

The difference \(Q_j\) projects onto the part of \(S_j\) that is orthogonal to \(S_{j-1}\). Its rank is the number of new directions added at stage \(j\). If \(S_j=S_{j-1}\), the increment is zero and its rank is zero.

Lemma: Each \(Q_j\) is a symmetric idempotent matrix, so it is an orthogonal projection. Its rank is \(\dim(S_j)-\dim(S_{j-1})\). For distinct indices \(i\) and \(j\), \(Q_iQ_j=0\).

Proof. Orthogonal projection matrices are symmetric, so \(Q_j\) is symmetric. Since \(S_{j-1}\subseteq S_j\), the projections satisfy \(P_jP_{j-1}=P_{j-1}P_j=P_{j-1}\). Thus

Q_j^2=(P_j-P_{j-1})^2 =P_j-P_{j-1}-P_{j-1}+P_{j-1} =P_j-P_{j-1}=Q_j.

Therefore \(Q_j\) is a symmetric idempotent matrix. Also, \(S_j\) is the orthogonal direct sum of \(S_{j-1}\) and \(S_j\cap S_{j-1}^{\perp}\). The projection onto the latter space is \(P_j-P_{j-1}\), so its dimension, and hence the rank of \(Q_j\), is \(\dim(S_j)-\dim(S_{j-1})\).

For \(i<j\), nesting gives \(P_iP_j=P_i\). Expanding the product of the differences, or noting that the orthogonal component added at stage \(i\) lies in \(S_{j-1}\) while the component added at stage \(j\) is orthogonal to \(S_{j-1}\), gives \(Q_iQ_j=0\). Symmetry also gives \(Q_jQ_i=0\). Thus the increment subspaces are mutually orthogonal. \(\square\)

Gaussian Laws for the Increments

Theorem: Let \(X\sim N_p(\mu,\sigma^2 I_p)\), where \(\sigma^2>0\), and let \(S_0\subseteq S_1\subseteq\cdots\subseteq S_m\) be fixed subspaces with projection differences \(Q_j=P_j-P_{j-1}\). Write \(r_j=\dim(S_j)-\dim(S_{j-1})\). Then

\frac{X^{\mathsf T}Q_jX}{\sigma^2} \sim \chi^2_{r_j}(\lambda_j), \qquad \lambda_j=\frac{\mu^{\mathsf T}Q_j\mu}{\sigma^2}.

The increments \(X^{\mathsf T}Q_1X,\ldots,X^{\mathsf T}Q_mX\) are mutually independent. If \(r_j=0\), the \(j\)th increment is identically zero; this is the degenerate zero-dimensional case.

Here \(\chi^2_r(\lambda)\) denotes a noncentral chi-square distribution with \(r\) degrees of freedom and noncentrality parameter \(\lambda\). For \(r>0\), it is the distribution of \(\sum_{k=1}^r(Z_k+a_k)^2\), where the \(Z_k\) are independent \(N(0,1)\) variables and \(\sum_{k=1}^r a_k^2=\lambda\). When \(\lambda=0\), it is the central chi-square distribution \(\chi^2_r\).

Proof. For each increment subspace, choose an orthonormal basis and place its basis vectors in the columns of a matrix \(U_j\). Then \(Q_j=U_jU_j^{\mathsf T}\), and

X^{\mathsf T}Q_jX =(U_j^{\mathsf T}X)^{\mathsf T}(U_j^{\mathsf T}X).

The vector \(U_j^{\mathsf T}X\) is multivariate normal with mean \(U_j^{\mathsf T}\mu\) and covariance \(\sigma^2 I_{r_j}\), because the columns of \(U_j\) are orthonormal. Dividing by \(\sigma\) expresses the sum above, after division by \(\sigma^2\), as the sum of squares of \(r_j\) independent unit-variance normal variables with mean vector \(U_j^{\mathsf T}\mu/\sigma\). Its noncentrality is

\frac{\|U_j^{\mathsf T}\mu\|^2}{\sigma^2} =\frac{\mu^{\mathsf T}U_jU_j^{\mathsf T}\mu}{\sigma^2} =\frac{\mu^{\mathsf T}Q_j\mu}{\sigma^2}.

To establish mutual independence, concatenate orthonormal bases for all the increment subspaces. The resulting columns are mutually orthogonal by the lemma. Their coordinates of \(X\) are jointly Gaussian and have covariance zero between distinct coordinates, since \(\operatorname{Cov}(X)=\sigma^2I_p\). Jointly Gaussian coordinates with zero cross-covariances are independent. Each increment is a function only of the coordinates in its own subspace, so the increments are mutually independent. \(\square\)

The same argument can include the initial subspace \(S_0\) and the orthogonal complement \(S_m^\perp\): their projection matrices are \(P_0\) and \(I_p-P_m\). Together with the \(Q_j\), they partition the whole space into mutually orthogonal components. In particular, the projected sum of squares added between the first and last subspaces satisfies

X^{\mathsf T}(P_m-P_0)X =\sum_{j=1}^m X^{\mathsf T}Q_jX.

Worked Examples

Worked Example: Three One-Dimensional Increments

Let \(S_0=\{0\}\), \(S_1=\operatorname{span}(e_1)\), \(S_2=\operatorname{span}(e_1,e_2)\), and \(S_3=\mathbb R^3\), where \(e_1,e_2,e_3\) are the coordinate unit vectors. Suppose \(X\sim N_3((2,3,-1),9I_3)\). Find the increment laws and check the scale.

The projection matrices are \(P_0=0\), \(P_1=\operatorname{diag}(1,0,0)\), \(P_2=\operatorname{diag}(1,1,0)\), and \(P_3=I_3\). Therefore the three difference matrices are

Q_1=\operatorname{diag}(1,0,0),\quad Q_2=\operatorname{diag}(0,1,0),\quad Q_3=\operatorname{diag}(0,0,1).

Each rank is \(1\). The noncentrality parameters are the squared corresponding mean coordinates divided by \(9\): \(\lambda_1=4/9\), \(\lambda_2=1\), and \(\lambda_3=1/9\). Hence

X^{\mathsf T}Q_1X\sim 9\chi^2_1(4/9),\quad X^{\mathsf T}Q_2X\sim 9\chi^2_1(1),\quad X^{\mathsf T}Q_3X\sim 9\chi^2_1(1/9).

The increments are independent. For instance, the first increment is \(X_1^2\), and \(X_1/3\sim N(2/3,1)\), so \(X_1^2=9(X_1/3)^2\) has the stated law. The factor \(9\) is essential: the unscaled increment is not itself distributed as \(\chi^2_1(4/9)\).

Worked Example: A Mean Entirely in the Smaller Subspace

In \(\mathbb R^4\), define the orthonormal vectors \(v_1=(1,1,0,0)^{\mathsf T}/\sqrt2\) and \(v_2=(1,-1,0,0)^{\mathsf T}/\sqrt2\), and let \(v_3=e_3\), \(v_4=e_4\). Take \(S_0=\operatorname{span}(v_1)\), \(S_1=\operatorname{span}(v_1,v_2,v_3)\), and \(S_2=\mathbb R^4\). Let \(X\sim N_4(5v_1,4I_4)\).

The first increment space is spanned by \(v_2,v_3\), so \(Q_1=v_2v_2^{\mathsf T}+v_3v_3^{\mathsf T}\) and \(r_1=2\). The second is spanned by \(v_4\), so \(Q_2=v_4v_4^{\mathsf T}\) and \(r_2=1\). Since the mean \(5v_1\) belongs to \(S_0\), its projections onto both increment spaces are zero: \(\lambda_1=\lambda_2=0\). In coordinates,

X^{\mathsf T}Q_1X=\frac{(X_1-X_2)^2}{2}+X_3^2, \qquad X^{\mathsf T}Q_2X=X_4^2.

The theorem therefore gives \(X^{\mathsf T}Q_1X/4\sim\chi^2_2\) and \(X^{\mathsf T}Q_2X/4\sim\chi^2_1\), independently. Equivalently, the unscaled sums are \(4\chi^2_2\) and \(4\chi^2_1\). The ranks \(2\) and \(1\) match the number of new orthogonal directions at the two stages.

Worked Example: Noncentral Increments of Different Ranks

Let \(S_0=\operatorname{span}(e_1)\), \(S_1=\operatorname{span}(e_1,e_2,e_3)\), and \(S_2=\mathbb R^4\). Suppose \(X\sim N_4((2,1,-2,3),4I_4)\). The first increment adds \(e_2,e_3\), so \(Q_1=\operatorname{diag}(0,1,1,0)\) and \(r_1=2\). The second adds \(e_4\), so \(Q_2=\operatorname{diag}(0,0,0,1)\) and \(r_2=1\).

The sums are \(X^{\mathsf T}Q_1X=X_2^2+X_3^2\) and \(X^{\mathsf T}Q_2X=X_4^2\). Their noncentralities are

\lambda_1=\frac{1^2+(-2)^2}{4}=\frac54, \qquad \lambda_2=\frac{3^2}{4}=\frac94.

Thus \(X^{\mathsf T}Q_1X/4\sim\chi^2_2(5/4)\), while \(X^{\mathsf T}Q_2X/4\sim\chi^2_1(9/4)\). The two increments are independent. The component of the mean along \(e_1\) contributes to neither increment because \(e_1\) lies in the baseline subspace \(S_0\).

Interpretation and a Common Pitfall

The rank difference \(\dim(S_j)-\dim(S_{j-1})\) is the degrees of freedom contributed by the new directions. The noncentrality records how much of the Gaussian mean lies in those directions, measured in units of the variance. An increment is central precisely when \(Q_j\mu=0\), since \(\mu^{\mathsf T}Q_j\mu=\|Q_j\mu\|^2\).

A common mistake is to call \(X^{\mathsf T}Q_jX\) chi-square without accounting for the covariance scale. For \(X\sim N_p(\mu,\sigma^2I_p)\), it is the scaled increment \(X^{\mathsf T}Q_jX/\sigma^2\) that has the chi-square or noncentral chi-square law. For example, if the increment has rank one, the mean component is zero, and \(\sigma^2=9\), then the unscaled sum has law \(9\chi^2_1\), not \(\chi^2_1\). The correction is to divide by \(\sigma^2\) before identifying the chi-square distribution.

This decomposition is useful whenever sums of squares are compared across nested subspaces: it identifies which directions contribute at each stage, supplies the correct degrees of freedom, and shows when separate increments are independent. Both the spherical covariance assumption and Gaussianity matter for the independence conclusion established here.

Check Your Understanding

Use the projection-difference definition and theorem to answer the following.

If \(\dim(S_{j-1})=3\) and \(\dim(S_j)=7\), what is the rank of \(Q_j\)?
For \(X\sim N_p(\mu,5I_p)\), what quantity must be divided by \(5\) to obtain the stated noncentral chi-square law?
When is the \(j\)th increment central, even if \(\mu\ne 0\)?
If \(S_j=S_{j-1}\), what are \(Q_j\), its rank, and the corresponding increment?
Why does the theorem give independence between distinct increments under spherical Gaussian covariance?