Worked Construction of H and E Under General Restrictions

From a General Restriction to H and E

In a multivariate linear model, a restriction can combine predictor coefficients and response directions at the same time. The notation \(C B M=\Theta\) expresses this compactly: \(C\) selects or combines rows of the coefficient matrix, while \(M\) selects or combines response directions. To construct the hypothesis and error sum-of-squares matrices correctly, keep track of those dimensions, check estimability, and distinguish the unrestricted residual variation from the variation attributable to imposing the restriction.

We use the model \(Y=XB+U\), where \(Y\) is an \(n\times p\) response matrix, \(X\) is an \(n\times q\) design matrix, and \(B\) is a \(q\times p\) coefficient matrix. Assume \(X\) has full column rank, and write \(S=X'X\). The unrestricted least-squares estimate is \(\hat B=S^{-1}X'Y\). The restriction matrices have dimensions \(C:r\times q\), \(M:p\times s\), and \(\Theta:r\times s\).

The full-rank conditions make the matrices \(C S^{-1}C'\) and \(M'M\) invertible. They also ensure that any \(r\times s\) target \(\Theta\) can be attained by some coefficient matrix \(B\). Let \(D=C\hat B M-\Theta\) be the discrepancy between the unrestricted estimate and the target. The restricted estimate is obtained by correcting \(\hat B\) just enough, in least-squares distance, to remove that discrepancy.

Estimability: A Design-Based Check

When \(X\) is not full column rank, not every coefficient contrast is identifiable. Let \(X^+\) denote the Moore–Penrose inverse. A row contrast \(cB\) is estimable exactly when \(c=cX^+X\), meaning that \(c\) lies in the row space of \(X\). For a nonzero, full-column-rank \(M\), the restriction \(CBM\) is estimable exactly when every row of \(C\) is estimable. In the full-rank setting used for the main construction, \(X^+X=I_q\), so this condition holds for any \(C\).

Restricted Least Squares and the Hypothesis Matrix

Write \(S=X'X\), \(A=(C S^{-1}C')^{-1}\), and \(G=(M'M)^{-1}\). The correction to the unrestricted estimate is \(\Delta=S^{-1}C'A D G M'\). It has dimensions \(q\times p\), just like \(B\). The restricted estimate is \(\tilde B=\hat B-\Delta\). The next result derives this formula and the resulting hypothesis matrix.

Proof. Any candidate coefficient matrix can be written \(\hat B-K\). Since \(X'\!(Y-X\hat B)=0\), its residual sum of squares is \[ (Y-X(\hat B-K))'(Y-X(\hat B-K)) =E+K'SK, \] where \(E=(Y-X\hat B)'(Y-X\hat B)\). Thus minimizing residual sum of squares subject to the restriction is equivalent to minimizing \(\operatorname{tr}(K'SK)\) subject to \(C K M=D\).

The proposed correction \(\Delta=S^{-1}C'A D G M'\) satisfies the constraint: \[ C\Delta M =(C S^{-1}C')A D G(M'M) =D. \] It is therefore feasible. For any other feasible correction \(K\), set \(L=K-\Delta\); then \(C L M=0\). The cross term in the objective vanishes: \[ \operatorname{tr}(L'S\Delta) =\operatorname{tr}\!\left(L'C'A D G M'\right) =\operatorname{tr}\!\left(G M'L'C'A D\right)=0, \] because \(M'L'C'=(C L M)'=0\). Hence \[ \operatorname{tr}(K'SK) =\operatorname{tr}(\Delta'S\Delta)+\operatorname{tr}(L'SL) \geq \operatorname{tr}(\Delta'S\Delta), \] with equality only when \(L=0\), since \(S\) is positive definite. This proves the restricted estimate formula and its least-squares property.

To obtain \(H\), substitute \(\Delta\) into \(\Delta'S\Delta\), retaining the order of the matrix products: \[ \Delta'S\Delta =M G D'A C S^{-1}S S^{-1}C'A D G M' =M G D'A(C S^{-1}C')A D G M' =M G D'A D G M'. \] The cancellation uses \(C S^{-1}C'\) between the two factors \(A=(C S^{-1}C')^{-1}\). The response-side factors remain \(M\) and \(M'\); no identity involving \(M\) is required. This proves the displayed formula for \(H\). \(\square\)

The unrestricted error matrix is \(E=(Y-X\hat B)'(Y-X\hat B)\). It summarizes residual variation after fitting the unrestricted model. The matrix \(H\) is the additional residual cross-product matrix induced by the restriction; it is not the entire restricted residual matrix.

Worked Construction With Numerical Data

Take \(n=4\), \(q=p=2\), and let the predictor matrix contain an intercept and a centered predictor \(x=(-3,-1,1,3)'\). Use the following design and response matrices:

We test the single general restriction \(C B M=\Theta\), with \(C=(0,1)\), \(M=(1,1)'\), and \(\Theta=(1)\). Thus the slope coefficients for the two responses must sum to one. The dimensions are \(C:1\times2\), \(B:2\times2\), \(M:2\times1\), and \(\Theta:1\times1\). Both restriction matrices have the required rank. The design has full column rank because its columns are nonzero and orthogonal, so \(C\) is estimable.

First calculate the design cross-product and unrestricted estimate:

The unrestricted slope sum is \(C\hat B M=(1,2)(1,1)'=3\), so \(D=3-1=2\). Also, \(C S^{-1}C'=1/20\), \(A=20\), and \(M'M=2\), so \(G=1/2\). The coefficient correction is

Consequently, \(\tilde B=\hat B-\Delta= \begin{pmatrix}2&-1\\0&1\end{pmatrix}\). Checking the restriction gives \(C\tilde B M=(0,1)(1,1)'=1=\Theta\), as required.

The hypothesis matrix is

For the unrestricted error matrix, the fitted values are \(X\hat B\). Subtracting these from the two response columns of \(Y\) gives residual vectors \((1,-1,-1,1)'\) and \((2,-4,2,0)'\). Their squared lengths are \(4\) and \(24\), and their inner product is \(4\). Therefore

What the Matrices Reveal—and a Common Pitfall

The example illustrates why \(H\) and \(E\) are separate matrices. \(E\) records the residual cross-products under the unrestricted fit, whereas \(H\) records the extra residual cross-products from the restriction. Both are \(p\times p\), symmetric, and positive semidefinite. For \(H\), this follows directly from \(H=\Delta'S\Delta\) and positive definiteness of \(S\); for \(E\), it follows because \(E\) is a residual matrix cross-product. In the example, \(H\) has identical rows and columns because the correction acts in the response direction \(M=(1,1)'\).

Common pitfall: It is tempting to simplify the formula for \(H\) by claiming that a response-side product involving \(M\) and \((M'M)^{-1}\) equals an identity matrix. This is generally invalid: the dimensions need not even permit that multiplication. Instead, substitute the full expression for \(\Delta\) into \(\Delta'S\Delta\), keep the factors in their valid order, and cancel only the middle predictor-space factor \(C S^{-1}C'\) against its inverse. The response-side factors \(M(M'M)^{-1}\) and \((M'M)^{-1}M'\) remain in the final formula.

A reliable construction therefore follows a fixed sequence: verify dimensions and estimability; compute \(\hat B\) and the discrepancy \(D\); form \(\Delta\) and check that \(C\tilde B M=\Theta\); then compute \(H=\Delta'S\Delta\) and \(E=(Y-X\hat B)'(Y-X\hat B)\). As a final numerical check, the restricted residual cross-product matrix must equal \(E+H\).