Incremental Sums of Squares for Orthogonal Model Extensions

Why Track Each Model Extension?

When a model is enlarged in stages, it is often useful to record how much of the response vector is fitted at each stage. The change in fitted sum of squares is not determined simply by counting the columns added: some columns may be redundant, and the effective contribution depends on the part of the new model space not already represented. Orthogonal projection gives a precise way to define and calculate that contribution.

This tutorial considers a fixed response vector \(y\in\mathbb{R}^n\) and a sequence of nested, finite-dimensional model spaces. The resulting decomposition is geometric: it holds for any \(y\), without a probability model. Under a normal model, additional distributional conclusions may be available, but additivity alone does not establish independence or a particular distribution.

Model-Space Increments

Definition: Let \(V_0\subseteq V_1\subseteq\cdots\subseteq V_m\) be finite-dimensional subspaces of \(\mathbb{R}^n\). The \(j\)th model-space increment is \(W_j=V_j\cap V_{j-1}^{\perp}\), for \(j=1,\ldots,m\). It is the portion of \(V_j\) orthogonal to the preceding space \(V_{j-1}\). Write \(P_j\) for the orthogonal projection onto \(V_j\), and \(P_{W_j}\) for the projection onto \(W_j\). The sum of squares for the \(j\)th increment is \(SS_j=\|P_{W_j}y\|^2\).

Because \(V_{j-1}\subseteq V_j\), each vector in \(V_j\) can be split into its projection onto \(V_{j-1}\) and a remainder orthogonal to \(V_{j-1}\). Thus \(V_j=V_{j-1}\oplus W_j\), where \(\oplus\) denotes an orthogonal direct sum. In particular, the increment is a subspace, not necessarily the raw span of the columns newly listed in a design matrix.

Theorem: For every \(j=1,\ldots,m\), \(P_{W_j}=P_j-P_{j-1}\). Consequently, \(SS_j=y^\mathsf{T}(P_j-P_{j-1})y\), and \(\operatorname{rank}(P_j-P_{j-1})=\dim(V_j)-\dim(V_{j-1})\).

Proof. The orthogonal decomposition \(V_j=V_{j-1}\oplus W_j\) implies that the projection onto \(V_j\) is the sum of the projections onto its two orthogonal components: \(P_j=P_{j-1}+P_{W_j}\). Rearranging gives \(P_{W_j}=P_j-P_{j-1}\). An orthogonal projection is symmetric and idempotent, so \(P_{W_j}\) is symmetric and idempotent as well; its rank equals the dimension of its range, which is \(\dim(W_j)\). Since the sum is direct, \(\dim(V_j)=\dim(V_{j-1})+\dim(W_j)\). Therefore its rank is \(\dim(V_j)-\dim(V_{j-1})\). Finally, \(\|P_{W_j}y\|^2=y^\mathsf{T}P_{W_j}^\mathsf{T}P_{W_j}y=y^\mathsf{T}P_{W_j}y\), using symmetry and idempotence. Substitution gives the stated sum-of-squares formula. \(\square\)

The rank is the increment’s degrees of freedom: it counts the number of independent directions added to the model space, not the number of columns written in an added block. If \(V_j\) is obtained by adjoining columns of a matrix \(Z_j\) to a design matrix for \(V_{j-1}\), only the components of those columns orthogonal to \(V_{j-1}\) can increase the rank.

Worked Example: Coordinate Directions Added One at a Time

Let \(V_0=\operatorname{span}(e_1)\), \(V_1=\operatorname{span}(e_1,e_2)\), and \(V_2=\operatorname{span}(e_1,e_2,e_3)\) in \(\mathbb{R}^4\), where \(e_i\) are the coordinate unit vectors. For \(y=(2,-1,3,4)^\mathsf{T}\), calculate the baseline, both increments, and the final residual sum of squares.

The baseline projection keeps the first coordinate, so its fitted sum of squares is \(2^2=4\), with rank \(\dim(V_0)=1\). The first increment is \(W_1=\operatorname{span}(e_2)\). Its rank is \(\dim(V_1)-\dim(V_0)=2-1=1\), and \(SS_1=(-1)^2=1\).

The second increment is \(W_2=\operatorname{span}(e_3)\). Its rank is \(\dim(V_2)-\dim(V_1)=3-2=1\), and \(SS_2=3^2=9\). The final fitted sum of squares is \(4+1+9=14\). The residual is the component along \(e_4\), so its sum of squares is \(4^2=16\). As a check, \(\|y\|^2=2^2+(-1)^2+3^2+4^2=30=14+16\).

Adding Design Columns: Residualize Before Counting

Suppose the current model space is the column space of a matrix \(X\), and a block of columns \(Z\) is proposed for addition. Let \(P_X\) be the projection onto \(\mathcal{C}(X)\), the column space of \(X\). The new space is \(\mathcal{C}([X\ Z])\). The effective new directions are generated by \((I-P_X)Z\): each column has had its already-fitted part removed. Therefore, \(\dim(\mathcal{C}([X\ Z]))-\dim(\mathcal{C}(X))=\operatorname{rank}((I-P_X)Z)\).

To see this rank identity, write any vector in the enlarged space as \(Xa+Zb\). Its component orthogonal to \(\mathcal{C}(X)\) is \((I-P_X)Zb\). The orthogonal complement of \(\mathcal{C}(X)\) within the enlarged space is therefore exactly the column space of \((I-P_X)Z\). Its dimension is the rank of that matrix, and it is also the dimension increase from the old to the new space.

Worked Example: Orthogonal Columns With Unequal Lengths

Let the baseline space be generated by \(x_0=(1,1,0,0)^\mathsf{T}\). Extend it first by \(x_1=(1,-1,0,0)^\mathsf{T}\), then by \(x_2=(0,0,2,0)^\mathsf{T}\). For \(y=(3,1,2,5)^\mathsf{T}\), compute each increment and its rank.

The vectors are mutually orthogonal: \(x_0^\mathsf{T}x_1=1-1=0\), and both have zero inner product with \(x_2\). Their squared lengths are \(x_0^\mathsf{T}x_0=2\), \(x_1^\mathsf{T}x_1=2\), and \(x_2^\mathsf{T}x_2=4\). For a nonzero vector \(x\), the projection of \(y\) onto its span has squared length \((y^\mathsf{T}x)^2/(x^\mathsf{T}x)\).

The baseline contribution is \((y^\mathsf{T}x_0)^2/(x_0^\mathsf{T}x_0)=(3+1)^2/2=8\), with rank 1. The first increment contributes \((y^\mathsf{T}x_1)^2/(x_1^\mathsf{T}x_1)=(3-1)^2/2=2\), also with rank 1. The second increment contributes \((y^\mathsf{T}x_2)^2/(x_2^\mathsf{T}x_2)=(4)^2/4=4\), with rank 1. The final fitted sum of squares is \(8+2+4=14\). Since \(\|y\|^2=9+1+4+25=39\), the residual sum of squares is \(39-14=25\), corresponding to the remaining coordinate direction \(e_4\).

Additivity Across a Sequence

Theorem: Let \(V_0\subseteq\cdots\subseteq V_m\) be nested subspaces of \(\mathbb{R}^n\), and let \(W_j=V_j\cap V_{j-1}^{\perp}\). For every \(y\in\mathbb{R}^n\), \(\|P_my\|^2=\|P_0y\|^2+\sum_{j=1}^{m}y^\mathsf{T}(P_j-P_{j-1})y\). Moreover, with \(R_m=I-P_m\), \(\|y\|^2=\|P_0y\|^2+\sum_{j=1}^{m}y^\mathsf{T}(P_j-P_{j-1})y+\|R_my\|^2\). The rank of the final residual projection is \(n-\dim(V_m)\).

Proof. From the preceding result, \(P_j-P_{j-1}=P_{W_j}\). The spaces \(W_i\) and \(W_j\) are orthogonal whenever \(i<j\): \(W_i\subseteq V_i\subseteq V_{j-1}\), while \(W_j\subseteq V_{j-1}^{\perp}\). Also, \(V_0\) is orthogonal to every \(W_j\). Repeatedly using \(V_j=V_{j-1}\oplus W_j\) gives \(V_m=V_0\oplus W_1\oplus\cdots\oplus W_m\). The projection of \(y\) onto this orthogonal sum has squared length equal to the sum of the squared lengths of its component projections. Each increment’s squared length is \(y^\mathsf{T}(P_j-P_{j-1})y\), proving the first equality. The residual \(R_my=y-P_my\) is orthogonal to \(V_m\), hence to \(P_my\). Applying the Pythagorean identity to \(y=P_my+R_my\) proves the second equality. Finally, \(R_m\) projects onto \(V_m^\perp\), whose dimension is \(n-\dim(V_m)\); this is its rank. \(\square\)

The formula separates the fitted sum of squares into a baseline and successive additions, and then separates the total squared length into fitted and residual parts. The ranks partition in the same way: \(\dim(V_m)=\dim(V_0)+\sum_{j=1}^m[\dim(V_j)-\dim(V_{j-1})]\). Thus the degrees of freedom in the fitted components and residual add to \(n\).

Worked Example: A Redundant Column in an Added Block

In \(\mathbb{R}^3\), take \(X=e_1\), then add the block \(Z=[2e_1\ \ e_2\ \ e_3]\). For \(y=(1,2,-1)^\mathsf{T}\), determine the increase in model rank, the incremental sum of squares, and the residual sum of squares.

The original space has dimension 1. Since \(P_X\) projects onto \(e_1\), residualizing the added columns gives \((I-P_X)Z=[0\ \ e_2\ \ e_3]\). This matrix has rank 2, so the model rank increases by 2, not by the three columns listed in \(Z\). The new space is all of \(\mathbb{R}^3\).

The baseline fitted sum of squares is \((y^\mathsf{T}e_1)^2=1\). The increment projects onto \(\operatorname{span}(e_2,e_3)\), so its sum of squares is \(2^2+(-1)^2=5\) and its rank is 2. The final fitted sum of squares is \(1+5=6=\|y\|^2\); the residual is zero, with rank \(3-3=0\). The column \(2e_1\) adds no direction and contributes no increment.

Common Mistake: Counting Columns Instead of Directions

A tempting but incorrect calculation is to set the rank of an increment equal to the number of newly supplied predictors. That works only when all the new columns contribute linearly independent directions outside the preceding model space. A duplicate column, a multiple of an existing column, or a linear combination already represented in the model adds no dimension. Compute \(\operatorname{rank}((I-P_X)Z)\), or equivalently the difference between the dimensions of the new and old spaces.

Another important distinction is between geometric additivity and statistical independence. Orthogonal increments yield an additive identity for sums of squares for every fixed response vector. Distributional claims require additional assumptions and are not consequences of this identity alone. The order of model extensions also determines which directions are assigned to which increment, even though the total fitted space at the end may be unchanged.

Key takeaway: For a nested sequence, the \(j\)th increment is the part of \(V_j\) orthogonal to \(V_{j-1}\). Its sum of squares is \(y^\mathsf{T}(P_j-P_{j-1})y\), and its rank is \(\dim(V_j)-\dim(V_{j-1})\). Add these contributions to recover the final fitted sum of squares.

Check Your Understanding

Work through these questions using projections and dimensions, not column counts alone.

If \(\dim(V_{j-1})=4\) and \(\dim(V_j)=7\), what is the rank of the \(j\)th increment?
Why is \(P_j-P_{j-1}\) the projection onto the increment rather than merely a difference of fitted vectors?
A model has one existing column and is extended by two copies of that same column. What is the rank increase, and why?
State the decomposition of \(\|y\|^2\) into baseline fitted, incremental fitted, and final residual sums of squares.
Does the additive projection identity by itself imply that the increments are statistically independent? Explain.