Why Construct H and E From Residuals?
In a multivariate linear model, a hypothesis compares two fits to the same response matrix: the unrestricted least-squares fit and the fit subject to the coefficient restrictions. The matrices \(H\) and \(E\) record the contributions of those fits to the total residual variation. Constructing them from residuals clarifies what each matrix measures and makes their relationship directly checkable.
The unrestricted model is \(Y=XB+\text{error}\), where \(Y\) is an \(n\times q\) response matrix, \(X\) is an \(n\times p\) design matrix, and \(B\) is a \(p\times q\) coefficient matrix. Assume throughout that \(X\) has full column rank. Write \(P_X=X(X'X)^{-1}X'\) for the projection onto the column space of \(X\), and \(Q_X=I-P_X\) for the residual projection. These are orthogonal projections: \(P_X'=P_X\), \(Q_X'=Q_X\), and \(P_XQ_X=0\).
Here \(C\) is an \(r\times p\) matrix, \(M\) is a \(q\times s\) matrix, and \(\Theta\) is an \(r\times s\) specified matrix. For the explicit constrained-estimator formula below, assume \(C\) has full row rank and \(M\) has full column rank. No normality assumption is needed for these least-squares constructions. \(E\) describes residual variation remaining under the unrestricted fit; \(H\) describes the additional residual cross-product incurred by imposing the restrictions.
The Residual Decomposition
Both fitted response matrices lie in the column space of \(X\). Their difference therefore lies in that same space. Define \(D=\hat Y_U-\hat Y_R\). Since \(Y=\hat Y_U+R_U\), the restricted residual is \(R_R=R_U+D\). The unrestricted residual is orthogonal to every column of \(X\), so it is also orthogonal to every column of \(D\).
Proof. By the least-squares projection equations, \(X'R_U=X'Q_XY=0\). Since \(D=X(\hat B-\hat B_R)\), \[ R_U'D=R_U'X(\hat B-\hat B_R)=0. \] Expanding the restricted residual cross-product gives \[ R_R'R_R=(R_U+D)'(R_U+D) =R_U'R_U+R_U'D+D'R_U+D'D =E+D'D. \] The definition \(H=R_R'R_R-R_U'R_U\) then gives \(H=D'D\). For any vector \(a\in\mathbb R^q\), \(a'Ea=\|R_Ua\|^2\geq 0\) and \(a'Ha=\|Da\|^2\geq 0\); both matrices are therefore positive semidefinite. \(\square\)
This proof is the central residual check: compute the two residual matrices, take their cross-products, and confirm that their difference is also the cross-product of the fitted-response difference. The result holds for any feasible linear restrictions, including restrictions that combine coefficients across multiple responses.
Computing the Restricted Fit
The definition of \(H\) is useful even when a constrained fit is obtained numerically. Under the full-rank assumptions above, it can also be calculated directly from the unrestricted estimate and the restriction discrepancy. Set \[ G=(X'X)^{-1},\qquad A=CGC',\qquad S=M'M,\qquad K=C\hat B M-\Theta. \] The matrix \(K\) measures how far the unrestricted estimate is from satisfying the restriction. Full row rank of \(C\) makes \(A\) invertible, and full column rank of \(M\) makes \(S\) invertible.
Derivation. For any coefficient matrix \(B\), the orthogonal decomposition into the unrestricted residual and a fitted component gives \[ \|Y-XB\|_F^2=\|Q_XY\|_F^2+ \operatorname{tr}\!\left\{(B-\hat B)'X'X(B-\hat B)\right\}. \] Thus the constrained least-squares estimate is the feasible matrix closest to \(\hat B\) in the metric defined by \(X'X\). Consider the proposed correction \(\Delta=\hat B_R-\hat B=-GC'A^{-1}KS^{-1}M'\). It satisfies the constraint because \[ C\hat B_RM=C\hat BM-CGC'A^{-1}KS^{-1}M'M =C\hat BM-K=\Theta. \] Here \(CGC'=A\), \(M'M=S\), and \(K=C\hat BM-\Theta\).
To verify that this feasible point minimizes the objective, let \(V\) be any change from \(\hat B_R\) that preserves the restriction, so \(CVM=0\). The weighted inner product of \(\Delta\) with \(V\) is \[ \operatorname{tr}(\Delta'X'XV) =-\operatorname{tr}\!\left(MS^{-1}K'A^{-1}CV\right) =-\operatorname{tr}\!\left(S^{-1}K'A^{-1}CVM\right)=0. \] Therefore \[ \operatorname{tr}\{(\Delta+V)'X'X(\Delta+V)\} =\operatorname{tr}(\Delta'X'X\Delta)+ \operatorname{tr}(V'X'XV), \] and the last term is nonnegative. This proves that the proposed feasible point minimizes the constrained least-squares criterion. Finally, \(D=X(\hat B-\hat B_R)=-X\Delta\). Substitution, using \(X'X=G^{-1}\), gives \[ D'D=\Delta'X'X\Delta =M S^{-1}K'A^{-1}KS^{-1}M'. \] By the residual decomposition, this is \(H\). \(\square\)
This formula constructs \(H\) without first writing out every restricted fitted value. It also shows how the restriction discrepancy is adjusted for the design geometry through \(A^{-1}\), and for the selected response contrasts through \(S^{-1}\). When restrictions concern all response columns, \(M\) is the identity and the response-contrast adjustment simplifies.
Worked Constructions
Worked Example: Restricting a Predictor Effect Across Two Responses
Let \(x=(-1,0,1)'\), let \(X=[\mathbf 1,x]\), and take the response matrix \[ Y=\begin{pmatrix}-1&5\\-1&-2\\5&3\end{pmatrix}. \] Since \(X'X=\operatorname{diag}(3,2)\), direct least squares gives \(\hat B=\begin{pmatrix}1&2\\3&-1\end{pmatrix}\). The unrestricted residual columns are \(u_1=(1,-2,1)'\) and \(u_2=(2,-4,2)'\); each is orthogonal to both columns of \(X\).
Test the restriction that the two predictor slopes are both zero: take \(C=(0,1)\), \(M=I_2\), and \(\Theta=0\). The unrestricted slope row is \((3,-1)\), so the restricted fit sets that row to zero and retains the intercept row. The fitted-response difference is \(D=x(3,-1)\). Hence \[ H=D'D=(x'x)\begin{pmatrix}3\\-1\end{pmatrix}(3,-1) =2\begin{pmatrix}9&-3\\-3&1\end{pmatrix} =\begin{pmatrix}18&-6\\-6&2\end{pmatrix}. \] The unrestricted error matrix is \[ E=R_U'R_U =\begin{pmatrix}u_1'u_1&u_1'u_2\\u_2'u_1&u_2'u_2\end{pmatrix} =\begin{pmatrix}6&12\\12&24\end{pmatrix}. \] Since \(R_R=R_U+D\) and \(R_U'D=0\), the restricted residual cross-product is \[ R_R'R_R=E+H =\begin{pmatrix}24&6\\6&26\end{pmatrix}. \] The off-diagonal entries are cross-products between the two response residual columns; they need not be zero.
Worked Example: A Restriction on a Response Contrast
Use the same design and response matrix, for which \(\hat B\), \(E\), and the unrestricted slope row \((3,-1)\) were calculated above. Now test whether the difference between the two slopes is zero. Set \(C=(0,1)\), \(M=(1,-1)'\), and \(\Theta=0\). Here \(G=\operatorname{diag}(1/3,1/2)\), \(A=CGC'=1/2\), \(S=M'M=2\), and \(K=C\hat BM=3-(-1)=4\).
The restricted-estimator formula gives the slope-row correction \[ -GC'A^{-1}KS^{-1}M' =-\begin{pmatrix}0\\1/2\end{pmatrix}(2)(4)(1/2)(1,-1) =(0,-2,2), \] where the displayed product is a \(2\times2\) coefficient correction; its first row is \((0,0)\) and its second row is \((-2,2)\). Thus the restricted slope row is \((1,1)\), which satisfies the required zero difference. The fitted difference is \(D=x(2,-2)\). Therefore \[ H=D'D=2\begin{pmatrix}2\\-2\end{pmatrix}(2,-2) =\begin{pmatrix}8&-8\\-8&8\end{pmatrix}. \] The same answer follows directly from \(H=MS^{-1}K'A^{-1}KS^{-1}M'\): its scalar multiplier on \(MM'\) is \((1/2)(16)(2)(1/2)=8\). The negative off-diagonal entries reflect the direction of the contrast \((1,-1)'\), not a negative amount of total variation.
Worked Example: A Nonzero Affine Restriction
Again use the same design and response matrix, with \(\hat B=\begin{pmatrix}1&2\\3&-1\end{pmatrix}\). This time restrict the sum of the two intercepts to zero: \(C=(1,0)\), \(M=(1,1)'\), and \(\Theta=0\). Then \(A=1/3\), \(S=2\), and \(K=C\hat BM=1+2=3\). The estimator formula changes the intercept row by \[ -GC'A^{-1}KS^{-1}M' =-\begin{pmatrix}1/3\\0\end{pmatrix}(3)(3)(1/2)(1,1) =\begin{pmatrix}-1.5&-1.5\\0&0\end{pmatrix}. \] The restricted intercepts are \((-0.5,0.5)\), whose sum is zero. The fitted difference is constant across observations, with each row equal to \((1.5,1.5)\). Thus \[ H=D'D=3\begin{pmatrix}1.5\\1.5\end{pmatrix}(1.5,1.5) =\begin{pmatrix}6.75&6.75\\6.75&6.75\end{pmatrix}. \] For a formula check, \(MS^{-1}K'A^{-1}KS^{-1}M'\) has scalar multiplier \((1/2)(9)(3)(1/2)=6.75\) on \(MM'\), giving the same matrix. The unrestricted residuals are unchanged, so \(E=\begin{pmatrix}6&12\\12&24\end{pmatrix}\), and the restricted residual cross-product is \(E+H=\begin{pmatrix}12.75&18.75\\18.75&30.75\end{pmatrix}\).
Interpretation and a Common Pitfall
The decomposition \(R_R'R_R=H+E\) is an exact least-squares identity. It does not require a probability model. In applications, the design and restrictions determine the fitted-response difference \(D\); \(H=D'D\) measures the change in fit caused by those restrictions, while \(E=R_U'R_U\) retains the unrestricted model's residual variation. Both matrices are expressed in response coordinates, which is why their off-diagonal entries preserve cross-response information.
- Pitfall: treating \(H\) as the unrestricted fitted cross-product. In general, \(Y'P_XY\) includes fitted variation whether or not the restrictions matter. The relevant matrix is \(D'D\), or equivalently \(R_R'R_R-R_U'R_U\).
- Pitfall: using the restricted residual cross-product as \(E\). That quantity is \(E+H\), not \(E\). \(E\) is formed from unrestricted residuals.
- Pitfall: assuming every restriction is an observation-space projection. For restrictions that combine responses, such as \(CBM=\Theta\), there need not be a single \(n\times n\) projection representing the restricted fit for all response columns. Construct the fit under the coefficient constraint, then use \(D'D\) and \(R_U'R_U\).
Check Your Understanding
Use the residual construction to answer each question.
- Why does \(R_U'D=0\) hold when both fitted response matrices come from the same design matrix \(X\)?
- What is the difference between \(R_R'R_R\) and \(E\), and how is that difference related to \(H\)?
- For a restriction \(CBM=\Theta\), what roles do \(A=CGC'\) and \(S=M'M\) play in the direct formula for \(H\)?
- In the response-contrast example, explain why \(H\) has negative off-diagonal entries while remaining positive semidefinite.
- Which residual matrix defines \(E\): the restricted residuals or the unrestricted residuals? State the corresponding cross-product.