Why Partition a Multivariate Regression?
A regression model may contain predictor blocks with different roles: for example, a baseline set of covariates and a set of additional predictors. Partitioning the design and coefficient matrices makes it possible to ask exactly what the additional block contributes after accounting for the first. Because the response has several columns, the calculation applies to all outcomes together.
Let \(Y\) be an \(n\times q\) response matrix. Partition the \(n\times(p_1+p_2)\) design matrix into \(X_1\) and \(X_2\), and partition the \((p_1+p_2)\times q\) coefficient matrix conformably. The model for the mean response is \(X_1B_1+X_2B_2\). Throughout, assume \(X_1\) has full column rank and that the part of \(X_2\) not explained by \(X_1\) has full column rank. These conditions make the block estimates unique.
The residualized design \(Z_2\) is central: it describes what the second block contributes beyond the first. The least-squares objective is the sum of squared residuals across all cases and responses, written \(\|Y-X_1B_1-X_2B_2\|_F^2\), where \(\|A\|_F^2=\operatorname{tr}(A'A)\) is the squared Frobenius norm.
Deriving the Blockwise Estimates
For a fixed \(B_2\), least squares over \(B_1\) is an ordinary regression of \(Y-X_2B_2\) on \(X_1\). Its normal equations give \(B_1=(X_1'X_1)^{-1}X_1'(Y-X_2B_2)\). Substitution removes the part of the residual that \(X_1\) can fit, leaving the objective \(\|M_1Y-M_1X_2B_2\|_F^2=\|M_1Y-Z_2B_2\|_F^2\). Thus the second block is estimated from both the response and its own design after each has been residualized against \(X_1\).
To verify the formula for \(\hat B_2\), write \(W=M_1Y\). The reduced objective is \(\|W-Z_2B_2\|_F^2\). Its normal equations are \(Z_2'Z_2B_2=Z_2'W\). Since \(Z_2\) has full column rank, \(Z_2'Z_2\) is invertible, so the unique solution is \( (Z_2'Z_2)^{-1}Z_2'W\). Also, \(Z_2'=X_2'M_1\) because \(M_1\) is symmetric and idempotent. Therefore \(Z_2'Z_2=X_2'M_1X_2\), giving the stated expression. Substituting this estimate into the fixed-\(B_2\) formula for \(B_1\) completes the derivation. \(\square\)
Worked Example: An Intercept and One Added Predictor
Take three observations, two responses, an intercept block \(X_1=\mathbf{1}=(1,1,1)'\), and a predictor \(x=(-1,0,1)'\). Let the response matrix have columns \(y_1=(0,1,4)'\) and \(y_2=(2,0,3)'\). Calculate the estimates with and without \(x\).
The response means are \(5/3\) and \(5/3\), and \(x'x=2\). Since \(x'\mathbf{1}=0\), residualizing \(x\) against the intercept leaves \(x\) unchanged. The slopes are therefore \(x'Y/(x'x)=(4,1)/2=(2,1/2)\). Because \(x\) is centered, the intercept estimates remain the response means. Thus \( \hat B_1=(5/3,5/3)\) and \(\hat B_2=(2,1/2)\).
The fitted rows are \((-1/3,7/6)\) for the first observation, \((5/3,5/3)\) for the second, and \((11/3,13/6)\) for the third. The first response’s fitted values are \((-1/3,5/3,11/3)\), so its residuals are \((1/3,-2/3,1/3)\). The second-response fit is \((7/6,5/3,13/6)\), with residuals \((5/6,-5/3,5/6)\).
With the intercept alone, the first response’s residuals are \((-5/3,-2/3,7/3)\). Their squared length is \(25/9+4/9+49/9=78/9=26/3\). After adding \(x\), the squared residual length is \(1/9+4/9+1/9=6/9=2/3\). The reduction is \(26/3-2/3=8\), which also equals the squared fitted contribution \( (x' y_1)^2/(x'x)=4^2/2=8\).
What Changes When the Block Is Added?
The block estimates also give a concise description of the fitted response. Let \(P_{Z_2}=Z_2(Z_2'Z_2)^{-1}Z_2'\). This is the projection onto the column space of \(Z_2\). The fit from \(X_1\) alone is \(P_1Y\), while the fit from both blocks is \(X_1\hat B_1+X_2\hat B_2=P_1Y+Z_2\hat B_2=P_1Y+P_{Z_2}Y\). The added fitted component is therefore orthogonal to the column space of \(X_1\). This describes the change in fitted responses, not a claim that the numerical estimates of \(B_1\) must stay fixed.
For the proof, \(Z_2=M_1X_2\) lies in the orthogonal complement of the column space of \(X_1\). The combined design space is the orthogonal direct sum of the space spanned by \(X_1\) and the space spanned by \(Z_2\): indeed, each column of \(X_2\) is the sum of its projection onto \(X_1\) and its residualized column in \(Z_2\). The projection onto the combined space is consequently \(P_1+P_{Z_2}\), so its residual-maker is \(I-P_1-P_{Z_2}=M_1-P_{Z_2}\). Subtraction gives the displayed SSCP identity. Finally, for any \(q\)-vector \(a\), \(a'Y'P_{Z_2}Ya=(Ya)'P_{Z_2}(Ya)=\|P_{Z_2}Ya\|^2\geq 0\), since an orthogonal projection is symmetric and idempotent. This proves positive semidefiniteness. \(\square\)
Taking traces shows that adding a block cannot increase the total residual sum of squares. The SSCP identity is more informative than a single total: it describes the reduction jointly across responses, including their cross-products. It does not say that every individual entry of the SSCP matrix must decrease.
Worked Example: A Nonorthogonal Added Predictor
Use the same responses as above, but let the added predictor be \(v=(1,2,4)'\), while retaining an intercept as \(X_1\). Its mean is \(7/3\), so \(Z_2=M_1v=(-4/3,-1/3,5/3)'\). Its squared length is \(v'M_1v=(16+1+25)/9=14/3\).
The centered first response is \((-5/3,-2/3,7/3)'\). Its inner product with \(Z_2\) is \((20+2+35)/9=57/9=19/3\). The first slope is therefore \((19/3)/(14/3)=19/14\). For the second response, the centered values are \((1/3,-5/3,4/3)'\); the inner product is \((-4+5+20)/9=21/9=7/3\), giving slope \((7/3)/(14/3)=1/2\).
The intercept block must adjust to the new slopes: its estimates are the response means minus \(7/3\) times each slope. Hence \(\hat B_1=(-3/2,1/2)\), and \(\hat B_2=(19/14,1/2)\). For the first response, the fitted values are \(-1/7,\ 17/14,\ 55/14\), leaving residuals \(1/7,\ -3/14,\ 1/14\). Their squared length is \(1/49+9/196+1/196=1/14\). This agrees with the initial intercept-only squared residual length \(26/3\) minus the reduction \((19/3)^2/(14/3)=361/42\), since \(26/3-361/42=1/14\).
Worked Example: Orthogonal Blocks and the Fitted-Response Change
Let \(X_1\) contain an intercept and \(x=(-1,0,1)'\), and add \(z=(1,-2,1)'\) as the second block. The vector \(z\) is orthogonal to both the intercept and \(x\), since \(1-2+1=0\) and \(-1+0+1=0\). Thus \(Z_2=z\) and \(z'z=6\). Use the response matrix from the first example.
The new block’s coefficient row is \(z'Y/(z'z)\). The inner products are \(z'y_1=0-2+4=2\) and \(z'y_2=2-0+3=5\), so \(\hat B_2=(1/3,5/6)\). The fitted increment \(z\hat B_2\) has rows \((1/3,5/6)\), \((-2/3, -5/3)\), and \((1/3,5/6)\). It is orthogonal to the earlier fitted space by construction.
Since the blocks are orthogonal, adding \(z\) does not change the intercept or \(x\) coefficients from the first example. Adding the displayed increment to that earlier fit gives the response rows \((0,2)\), \((1,0)\), and \((4,3)\), exactly \(Y\). The full model is saturated here and has zero residuals. In a nonorthogonal design, the added fitted component remains orthogonal to the earlier design space, but the estimates for the earlier coefficient block can change, as the preceding example showed.
Common Pitfall and Practical Interpretation
A common mistake is to estimate the second block using \((X_2'X_2)^{-1}X_2'Y\) while ignoring \(X_1\). That is the coefficient estimate for a different regression, one that omits the first block. The correct estimate uses \(X_2'M_1X_2\) and \(X_2'M_1Y\): both the new design and the response must be adjusted for the first block. Another mistake is to assume that adding predictors can only leave old coefficients unchanged. That is true in special orthogonal designs, not in general. The invariant geometric statement is that the added part of the fitted response lies orthogonal to the earlier design space.
For more than two blocks, the same calculation applies when a new block is added to an existing combined design: residualize the new block against that entire existing design, then estimate its contribution from the corresponding residualized response. This blockwise view is useful when predictors are entered in groups, when their coefficient matrices need separate interpretation, and when comparing the fitted responses from nested models.
Check Your Understanding
Use the formulas and geometry in this tutorial to answer the following.
- What rank condition on \(X_2\) makes the block estimate unique once \(X_1\) is included?
- Why must the response as well as the new design block be residualized against \(X_1\)?
- In what sense is the fitted-response change orthogonal to the first block?
- What does positive semidefiniteness of \(Y'P_{Z_2}Y\) imply about the total residual sum of squares?
- When can adding a predictor block leave the earlier coefficient estimates unchanged?