Sequential and Partial F Tests in Nonorthogonal Designs

Why the Order of Predictors Matters

In a regression model, a sum of squares can describe either the improvement from entering predictors in a particular order or the improvement from adding a predictor while adjusting for all the others. These are different comparisons when predictor subspaces are nonorthogonal. Consequently, the corresponding F statistics can differ even when they use the same full model and the same residual mean square.

The central question is not simply which predictors are in the model. It is which reduced model is being compared with the full model. Sequential sums of squares use models built up in a chosen order. Partial sums of squares compare the full model with a model that retains every predictor except the block being tested.

Definition: Let \(V_0\subseteq V_1\subseteq\cdots\subseteq V_p\) be the predictor subspaces in a specified order, with \(P_j\) the orthogonal projector onto \(V_j\). The sequential sum of squares for step \(j\) is \(Y^\mathsf{T}(P_j-P_{j-1})Y\), equivalently the reduction in residual sum of squares from \(V_{j-1}\) to \(V_j\). For a block of predictors, its partial sum of squares is the reduction in residual sum of squares when that block is added to a model containing all the other predictors. The partial sum of squares is therefore a comparison of the full model with the model omitting the tested block.

Here \(Y\) is the response vector. In the usual regression setting, every subspace includes the intercept, and \(V_p\) is the full-model column space. If \(P_p\) is the projector onto the full model, the full-model residual sum of squares is \(Y^\mathsf{T}(I-P_p)Y\), and its residual degrees of freedom are \(n-\operatorname{rank}(V_p)\).

A Nonorthogonal Design: Calculate Both Comparisons

Worked Example: Sequential Sums of Squares Change with Order

Consider four observations, an intercept, and two centered predictor columns: \(x_1=(-1,-1,1,1)^\mathsf{T}\) and \(x_2=(-1,0,0,1)^\mathsf{T}\). Let \(Y=(-2.5,-1.5,-0.5,4.5)^\mathsf{T}\). The predictors are not orthogonal because \(x_1^\mathsf{T}x_2=2\), not zero.

To make the geometry clear, write \(z=x_2-\tfrac12x_1=(-0.5,0.5,-0.5,0.5)^\mathsf{T}\). Then \(x_1^\mathsf{T}z=0\), \(x_1^\mathsf{T}x_1=4\), and \(z^\mathsf{T}z=1\). With \(r=(1,-1,-1,1)^\mathsf{T}\), we have \(Y=2x_1+3z+r\); \(r\) is orthogonal to the intercept, \(x_1\), and \(z\), and \(r^\mathsf{T}r=4\).

Order \(x_1\), then \(x_2\). Adding \(x_1\) after the intercept explains \((Y^\mathsf{T}x_1)^2/(x_1^\mathsf{T}x_1)=8^2/4=16\). After \(x_1\) is fitted, the part of \(x_2\) that adds a new direction is \(z\). Its sequential sum of squares is \((Y^\mathsf{T}z)^2/(z^\mathsf{T}z)=3^2/1=9\).

Order \(x_2\), then \(x_1\). Since \(x_2=\tfrac12x_1+z\), its squared length is \(2\), and \(Y^\mathsf{T}x_2=7\). Entering \(x_2\) first explains \(7^2/2=24.5\). Adding \(x_1\) second explains the remaining full-model improvement, \(25-24.5=0.5\), because the full model explains \(2^2(4)+3^2(1)=25\).

The full-model residual sum of squares is \(r^\mathsf{T}r=4\). Thus the sequential sums of squares are \(16,9\) in the first order and \(24.5,0.5\) in the reverse order. Their totals are both \(25\), but the amount assigned to each step depends on the order.

The partial sums of squares answer a different question. For \(x_1\) adjusted for \(x_2\), the reduced model contains the intercept and \(x_2\). Its explained sum of squares is \(24.5\), so the partial sum of squares for \(x_1\) is \(25-24.5=0.5\). For \(x_2\) adjusted for \(x_1\), the reduced model explains \(16\), giving partial sum of squares \(25-16=9\). These match the corresponding last-entry sequential sums of squares, not necessarily the first-entry values.

Worked Example: Orthogonal Predictors Make the Sums of Squares Agree

Let \(x_1=(-1,-1,1,1)^\mathsf{T}\), \(x_2=(-1,1,-1,1)^\mathsf{T}\), \(r=(1,-1,-1,1)^\mathsf{T}\), and \(Y=x_1+2x_2+r\). Each vector is centered; \(x_1\) and \(x_2\) are orthogonal, each has squared length \(4\), and \(r\) is orthogonal to both.

Adding \(x_1\) explains \((Y^\mathsf{T}x_1)^2/4=4^2/4=4\). Adding \(x_2\) explains \((Y^\mathsf{T}x_2)^2/4=8^2/4=16\), whether it is entered first or second, because the predictors are orthogonal. The full-model residual sum of squares is \(r^\mathsf{T}r=4\). Thus both orders give sequential sums of squares \(4\) for \(x_1\) and \(16\) for \(x_2\); the partial sums of squares have those same values.

Orthogonality makes each predictor’s fitted direction perpendicular to the other predictor directions, so fitting one does not change the component of \(Y\) along the other. In nonorthogonal designs, that separation generally fails.

Partial F Tests and Their Exact Calibration

A partial F test is a nested-model test. Suppose \(V_R\subseteq V_F\) are the reduced and full model subspaces, \(q=\dim(V_F)-\dim(V_R)\), and the full model has residual degrees of freedom \(\nu=n-\dim(V_F)>0\). Define \(\operatorname{SSE}_R=Y^\mathsf{T}(I-P_R)Y\) and \(\operatorname{SSE}_F=Y^\mathsf{T}(I-P_F)Y\). The partial sum of squares is \(\operatorname{SSE}_R-\operatorname{SSE}_F\).

Theorem: Suppose \(Y\sim N(\mu,\sigma^2 I)\), \(V_R\subseteq V_F\), and the null hypothesis is \(\mu\in V_R\). If \(q>0\) and \(\nu>0\), then \(\bigl[(\operatorname{SSE}_R-\operatorname{SSE}_F)/q\bigr]/(\operatorname{SSE}_F/\nu)\sim F_{q,\nu}\).

Proof. The matrix \(P_F-P_R\) is a symmetric idempotent matrix: using \(P_FP_R=P_RP_F=P_R\), its square is \(P_F-P_R\). Its rank is \(q\), and it projects onto the part of \(V_F\) perpendicular to \(V_R\). Also, \(\operatorname{SSE}_R-\operatorname{SSE}_F=Y^\mathsf{T}(P_F-P_R)Y\). Under the null, \(\mu\in V_R\), so \((P_F-P_R)\mu=0\). The normal quadratic-form result therefore gives \((\operatorname{SSE}_R-\operatorname{SSE}_F)/\sigma^2\sim\chi^2_q\).

The full-model residual is \((I-P_F)Y\). The matrix \(I-P_F\) is also symmetric and idempotent, has rank \(\nu\), and annihilates \(\mu\), since \(\mu\in V_R\subseteq V_F\). Hence \(\operatorname{SSE}_F/\sigma^2\sim\chi^2_\nu\). The two quadratic forms are independent: their projection matrices have product \((P_F-P_R)(I-P_F)=0\), and orthogonal projections of a multivariate normal vector are independent. Dividing each independent chi-square variable by its degrees of freedom gives the stated \(F\) distribution. \(\square\)

In the nonorthogonal example, testing \(x_1\) conditional on \(x_2\) uses \(q=1\), \(\operatorname{SSE}_R=4.5\), and \(\operatorname{SSE}_F=4\). Thus the partial sum of squares is \(0.5\), and the statistic is \(0.5/4=0.125\), with \(1\) and \(1\) degrees of freedom. Testing \(x_2\) conditional on \(x_1\) gives \(9/4=2.25\), again with \(1\) and \(1\) degrees of freedom. These are exact central F tests when their respective reduced-model null hypotheses hold.

What a Sequential F Test Tests

For an ordered sequence \(V_0\subseteq V_1\subseteq\cdots\subseteq V_p\), the sequential sum of squares at step \(j\) compares \(V_{j-1}\) with \(V_j\). It is possible to divide this quantity by the full-model mean squared error, \(\operatorname{SSE}_F/\nu\), to form a sequential F statistic. Its numerator degrees of freedom are \(q_j=\dim(V_j)-\dim(V_{j-1})\).

The key qualification is that the exact central F null for this comparison is \(\mu\in V_{j-1}\). If later predictors have nonzero effects, the mean need not lie in \(V_{j-1}\). The sequential numerator can then reflect those later effects as well as the effect of the current block. In contrast, a partial test of the current block retains the later predictors in both models and tests whether the mean lies in the full model with that block removed.

Worked Example: A Sequential Null Is Not Always a Conditional Null

Return to the nonorthogonal predictors, with full-model mean \(\mu=0.5x_1+3x_2=2x_1+3z\), and suppose \(\sigma^2=1\). The coefficient of \(x_1\) conditional on \(x_2\) is \(0.5\). Testing that coefficient uses the reduced subspace spanned by the intercept and \(x_2\). Since \(x_1-x_2\) is perpendicular to \(x_2\) and has squared length \(2\), the mean’s component in the added direction is \(0.5(x_1-x_2)\). Its squared length is \(0.5^2(2)=0.5\). The partial test therefore has noncentrality parameter \(0.5\) under this alternative.

Now consider entering \(x_1\) first and testing it sequentially against the intercept-only model. The mean’s projection onto \(x_1\) is \(2x_1\), whose squared length is \(2^2(4)=16\). Because the later predictor has a nonzero effect, this mean is not in the intercept-only subspace. The sequential comparison therefore has noncentrality parameter \(16\), not \(0.5\). It is not testing the same conditional coefficient as the partial test.

This distinction also explains why order can matter for sequential tests without making partial tests order-dependent: changing the order changes which earlier subspace is used as the sequential baseline. A partial test’s baseline is instead specified by the hypothesis—keep all other predictors, remove just the block being tested.

Common Mistake

Wrong reasoning: “The sequential sum of squares for a predictor is its partial sum of squares, so it tests that predictor’s coefficient after adjustment for every other predictor.”
Correction: A sequential sum of squares adjusts only for terms entered earlier. It equals the partial sum of squares when the tested term is entered last, or under conditions such as orthogonality that make the relevant projections agree. To test a block conditional on all other predictors, compare the full model with the model omitting only that block.

Key takeaway: Sequential sums of squares allocate improvement according to a chosen entry order. Partial sums of squares measure the improvement from adding a block to a model that already contains all other predictors. State the two models being compared before interpreting either F statistic.

Check Your Understanding

Answer each question by identifying the model comparison involved.

In what sense does a sequential sum of squares depend on predictor order?
For a partial test of one predictor, which predictors remain in the reduced model?
What null hypothesis gives the exact central F calibration for a comparison of nested model subspaces \(V_R\subseteq V_F\)?
In the first nonorthogonal example, why does the sequential sum of squares for \(x_1\) differ when it is entered first rather than last?
Give a condition under which sequential and partial sums of squares for a predictor agree, and explain why.