When Sequential Sums of Squares Are Independent

Two Different Questions About Sequential Sums of Squares

A sequential sum of squares measures the reduction in residual variation when a model is enlarged by a term. There are two questions that are easy to confuse. For a fixed order of nested models, are the successive sums of squares independent? And if the terms are entered in a different order, do they receive the same sums of squares? Projection geometry answers these questions separately: increments in a fixed nested sequence are orthogonal, while invariance to reordering requires orthogonality between the terms after accounting for the baseline model.

Throughout, treat the design and its model spaces as fixed. Suppose the response vector is $Y=\mu+\varepsilon$, where $\varepsilon\sim N_n(0,\sigma^2 I_n)$ and $\sigma^2>0$. This is the spherical Gaussian error assumption: errors have a common variance and zero covariance between distinct coordinates. The mean vector $\mu$ need not belong to any of the model spaces.

Definition: For a linear subspace $M\subseteq\mathbb R^n$, let $P_M$ be the orthogonal projection matrix onto $M$. If $M_0\subseteq M_1\subseteq\cdots\subseteq M_k$ is a nested sequence of model spaces, the $j$th sequential increment matrix and sum of squares are $$ Q_j=P_{M_j}-P_{M_{j-1}}, \qquad SS_j=Y^\mathsf{T}Q_jY. $$ The degrees of freedom of this increment are $\operatorname{rank}(Q_j)=\dim(M_j)-\dim(M_{j-1})$.

What Nesting Guarantees

If $M_{j-1}\subseteq M_j$, the part of $M_j$ added at step $j$ is the subspace $A_j=M_j\cap M_{j-1}^{\perp}$. It is perpendicular to the old model space, and $M_j$ is the orthogonal sum of $M_{j-1}$ and $A_j$. Consequently, $Q_j=P_{A_j}$: the increment matrix is itself an orthogonal projection.

For any fixed nested sequence, the added subspaces $A_j$ are mutually orthogonal. In fact, if $i<j$, then $A_i\subseteq M_i\subseteq M_{j-1}$, whereas $A_j\subseteq M_{j-1}^{\perp}$. Thus the sequential increment matrices satisfy $Q_iQ_j=0$. This geometric fact holds for every nesting order; it does not say that changing the order leaves the increments unchanged.

Theorem: Suppose $Y\sim N_n(\mu,\sigma^2 I_n)$, with $\sigma^2>0$. For any finite collection of pairwise orthogonal subspaces $A_1,\ldots,A_k$, the sums $Y^\mathsf{T}P_{A_j}Y$ are mutually independent. Each sum divided by $\sigma^2$ has a noncentral chi-square distribution with degrees of freedom $\dim(A_j)$ and noncentrality parameter $\|P_{A_j}\mu\|^2/\sigma^2$. In particular, it is central when $P_{A_j}\mu=0$.

Proof. The projected vectors $P_{A_j}Y$ are jointly Gaussian because they are linear transformations of a Gaussian vector. For $i\ne j$, orthogonality gives $P_{A_i}P_{A_j}=0$, so $$ \operatorname{Cov}(P_{A_i}Y,P_{A_j}Y) =P_{A_i}(\sigma^2 I_n)P_{A_j} =\sigma^2 P_{A_i}P_{A_j} =0. $$ Jointly Gaussian vectors with zero cross-covariance are independent. Therefore the vectors $P_{A_j}Y$ are mutually independent, and so are their squared lengths $Y^\mathsf{T}P_{A_j}Y=\|P_{A_j}Y\|^2$, which are measurable functions of those vectors. In an orthonormal basis of $A_j$, the coordinates of $P_{A_j}Y/\sigma$ are independent unit-variance normal variables with mean vector given by the coordinates of $P_{A_j}\mu/\sigma$. Their squared sum has the stated noncentral chi-square distribution. $\square$

Applying the theorem to $A_j=M_j\cap M_{j-1}^{\perp}$ shows that sequential sums of squares in any one fixed nesting order are mutually independent under spherical Gaussian errors. Their distributions can be noncentral if the mean has a component in the increment subspace. Independence itself does not require the mean to lie in the smallest model.

Worked Example: Orthogonal Terms Give Order-Invariant Sums

Take no baseline term and let the two term spaces in $\mathbb R^3$ be spanned by $x_1=(1,0,0)^\mathsf{T}$ and $x_2=(0,1,0)^\mathsf{T}$. These vectors are perpendicular and each has squared length $1$. For the observed response $y=(2,1,3)^\mathsf{T}$, calculate the two sequential sums of squares.

Entering $x_1$ first gives $P_{x_1}=x_1x_1^\mathsf{T}$, so $y^\mathsf{T}P_{x_1}y=(x_1^\mathsf{T}y)^2=2^2=4$. The full model projection is onto the first two coordinate axes. Its sum of squares is $2^2+1^2=5$, so adding $x_2$ contributes $5-4=1$.

Reversing the order gives $1^2=1$ for $x_2$ first and $5-1=4$ for $x_1$ second. Each term receives the same sum of squares in either order. If the response has spherical Gaussian errors, the two sums are also independent, because their projection subspaces are perpendicular.

When Does Changing the Order Leave Each Term Unchanged?

Now let $M_0$ be a baseline space and let $T_1,\ldots,T_k$ be the subspaces contributed by terms. Define the baseline-adjusted term spaces $$ W_i=(I_n-P_{M_0})T_i =\{(I_n-P_{M_0})t:t\in T_i\}. $$ Each $W_i$ lies in $M_0^\perp$. The model after adding a set of terms $S$ is the orthogonal sum of $M_0$ and the span of the corresponding $W_i$'s. Thus orthogonality must be checked between the parts of terms that remain after projection away from the baseline, not necessarily between the original design columns.

Theorem: For a baseline space $M_0$ and term spaces $T_1,\ldots,T_k$, each term's sequential sum of squares is unchanged for every ordering and every response vector $y$ if and only if the baseline-adjusted spaces $W_i=(I_n-P_{M_0})T_i$ are pairwise orthogonal. A zero adjusted space contributes zero in every order.

Proof. First consider two terms with adjusted spaces $W_i,W_j$, and write $P_i=P_{W_i}$, $P_j=P_{W_j}$, and $P_U=P_{W_i+W_j}$. If $i$ is entered first, its increment matrix is $P_i$. If it is entered second, after $j$, its increment matrix is $P_U-P_j$. Having the same sum of squares for every response vector means these matrices are equal: two symmetric matrices $B,C$ with $y^\mathsf{T}By=y^\mathsf{T}Cy$ for every $y$ must be equal, by polarization of their quadratic forms. Hence order invariance for this pair implies $$ P_U=P_i+P_j. $$ The right side is an orthogonal projection only if $W_i\perp W_j$. To see the needed implication, square both sides and use $P_U^2=P_U$, $P_i^2=P_i$, and $P_j^2=P_j$. This gives $P_iP_j+P_jP_i=0$. Taking traces yields $2\operatorname{tr}(P_iP_j)=0$. But $\operatorname{tr}(P_iP_j)=\operatorname{tr}(P_jP_iP_j)=\|P_iP_j\|_F^2\geq0$, so $P_iP_j=0$, which is equivalent to $W_i\perp W_j$. If every ordering gives every term the same contribution, this two-term swap argument applies to every pair.

Conversely, suppose the $W_i$'s are pairwise orthogonal. After any collection of terms has been entered, adding term $i$ adds precisely $W_i$, perpendicular to the baseline-adjusted spaces already present. Its increment matrix is therefore $P_{W_i}$, regardless of the terms entered earlier. This proves order invariance and completes both directions. $\square$

Worked Example: Nonorthogonal Terms Are Independent in Order but Not Invariant

In $\mathbb R^2$, take no baseline and use $x_1=(1,0)^\mathsf{T}$, $x_2=(1,1)^\mathsf{T}$, and $y=(2,1)^\mathsf{T}$. The vectors are not orthogonal because $x_1^\mathsf{T}x_2=1$.

Entering $x_1$ first gives $y^\mathsf{T}P_{x_1}y=(x_1^\mathsf{T}y)^2/(x_1^\mathsf{T}x_1)=2^2/1=4$. The two vectors span $\mathbb R^2$, so the full-model sum of squares is $y^\mathsf{T}y=2^2+1^2=5$. The increment for $x_2$ second is $5-4=1$.

In the reverse order, the first contribution is $(x_2^\mathsf{T}y)^2/(x_2^\mathsf{T}x_2)=3^2/2=9/2$. The increment for $x_1$ second is $5-9/2=1/2$. Thus term-specific sums change with order. Nevertheless, in either fixed order the two increments are independent under spherical Gaussian errors: they project onto the mutually perpendicular added subspaces for that order. Independence in each order does not make the allocations order-invariant.

Worked Example: Checking Orthogonality After an Intercept

In $\mathbb R^4$, let the baseline be the intercept space spanned by $\mathbf 1=(1,1,1,1)^\mathsf{T}$. Let two one-dimensional terms be spanned by $a=(1,-1,1,-1)^\mathsf{T}$ and $b=(1,1,-1,-1)^\mathsf{T}$, and take $y=(3,1,2,0)^\mathsf{T}$. Both terms are perpendicular to the intercept since $\mathbf 1^\mathsf{T}a=\mathbf 1^\mathsf{T}b=0$. They are also perpendicular to each other, since $a^\mathsf{T}b=1-1-1+1=0$.

Each term vector has squared length $4$. The intercept projection of $y$ has coefficient $\mathbf 1^\mathsf{T}y/4=6/4=1.5$, so its fitted vector is $(1.5,1.5,1.5,1.5)^\mathsf{T}$, with sum of squares $4(1.5)^2=9$. For $a$, the projection coefficient is $a^\mathsf{T}y/4=(3-1+2-0)/4=1$, giving a sequential sum of squares $1^2(4)=4$. For $b$, the coefficient is $b^\mathsf{T}y/4=(3+1-2-0)/4=0.5$, giving $0.5^2(4)=1$.

The total response sum of squares is $3^2+1^2+2^2+0^2=14=9+4+1$, so these components account for all of $y$. The two term contributions remain $4$ and $1$ in either order. Under spherical Gaussian errors their sums of squares are independent. This example also shows why orthogonality should be assessed relative to the baseline: in this case the terms already happen to be perpendicular to it.

Common Mistake and Practical Use

Incorrect reasoning: “Sequential sums of squares are independent, so their values cannot depend on term order.” The first part is true under spherical Gaussian errors for increments in a fixed nested sequence. The conclusion does not follow: changing the order can change the increment subspaces, and therefore change which term receives which sum of squares. The nonorthogonal two-term example demonstrates this directly.

Before interpreting sequential sums, specify the order and identify the relevant model spaces. For order-invariant contributions, remove the baseline projection from each term and check pairwise orthogonality of the resulting spaces. When this condition fails, a particular ordering still yields orthogonal increments and independent sums under the stated Gaussian assumption, but the term-specific allocations describe that ordering rather than an order-free decomposition. If a term is completely explained by the baseline or by terms already entered, its increment has rank zero and contributes no new sum of squares.

Check Your Understanding

Use projection geometry to answer each question.

Why are the added subspaces in a fixed nested sequence mutually orthogonal?
Which assumptions allow the sums of squares from orthogonal projections to be independent?
For two terms added to a baseline model, which spaces must be checked for orthogonality to ensure order-invariant contributions?
Can sequential sums of squares be independent in each of two different term orders while the contribution assigned to a term changes? Explain.
If a term's baseline-adjusted space is zero, what is its sequential contribution in any ordering?

When Sequential Sums of Squares Are Independent

What You'll Learn