Hypothesis and Error Sum-of-Squares Matrices

Why Two Matrices Describe a Multivariate Comparison

When a linear model has several responses, comparing a restricted model with a larger nested model involves more than one residual sum of squares. Each response has its own sum of squares, and pairs of responses also have cross-products that describe how their fitted or residual variation occurs together. The comparison is therefore summarized by two matrices: the hypothesis sum-of-squares-and-cross-products matrix and the error sum-of-squares-and-cross-products matrix.

The hypothesis matrix records the change in fitted responses when moving from the reduced model to the full model. The error matrix records what remains unexplained by the full model. Their entries are descriptive quantities calculated from fitted values and residuals; defining them does not require a probability distribution. In particular, these matrices are not themselves a multivariate test. They provide the variation summaries used in subsequent comparisons.

Definitions for Nested Models

Let \(Y\) be an \(n\times p\) observed response matrix, with one row per case and one column per response. Consider two nested linear models for the same data. The reduced model allows a set of fitted response matrices, and the full model allows a larger set containing every fit permitted by the reduced model. Fit both models by least squares, and write their fitted response matrices as \(\widehat Y_R\) and \(\widehat Y_F\), respectively.

The full-model residual matrix is \(R_F=Y-\widehat Y_F\). The change in fitted responses between the two fits is \(D=\widehat Y_F-\widehat Y_R\). Both \(R_F\) and \(D\) are \(n\times p\) matrices, so their cross-product matrices are \(p\times p\).

Definition: The hypothesis SSCP matrix and error SSCP matrix for the nested comparison are

H=D'D=(\widehat Y_F-\widehat Y_R)'(\widehat Y_F-\widehat Y_R), \qquad E=R_F'R_F=(Y-\widehat Y_F)'(Y-\widehat Y_F).

The diagonal entries of \(H\) and \(E\) are sums of squares for individual responses. Their off-diagonal entries are cross-products between pairs of responses.

For example, \(H_{jj}\) is the sum of squared changes in fitted values for response \(j\), while \(H_{jk}\) is the sum of products of fitted-value changes for responses \(j\) and \(k\). Likewise, \(E_{jj}\) is the full-model residual sum of squares for response \(j\), and \(E_{jk}\) is the cross-product of the two full-model residual columns. The matrices are symmetric because \(D'D\) and \(R_F'R_F\) are symmetric.

The definition uses raw cross-products, with no division by sample size or degrees of freedom. A later inferential procedure may combine these matrices or use a scaled version, but that is distinct from defining \(H\) and \(E\).

Worked Example: Group Fits, Hypothesis Variation, and Error Variation

Four observations have two responses, and the reduced model has an intercept only. The full model has a separate mean for each of two groups. The observed response matrix, with the first two rows in group A and the last two in group B, is

Y= \begin{pmatrix} 2&1\\ 4&3\\ 6&5\\ 8&7 \end{pmatrix}.

The reduced-model mean is \((5,4)\), so its fitted matrix has four identical rows. The full-model group means are \((3,2)\) and \((7,6)\). Thus

\widehat Y_R= \begin{pmatrix} 5&4\\5&4\\5&4\\5&4 \end{pmatrix}, \qquad \widehat Y_F= \begin{pmatrix} 3&2\\3&2\\7&6\\7&6 \end{pmatrix}.

Subtracting gives

D= \begin{pmatrix} -2&-2\\ -2&-2\\ 2&2\\ 2&2 \end{pmatrix}, \qquad R_F=Y-\widehat Y_F= \begin{pmatrix} -1&-1\\ 1&1\\ -1&-1\\ 1&1 \end{pmatrix}.

Each row of \(D\) contributes an outer product with all entries equal to \(4\). There are four rows, so

H=D'D= \begin{pmatrix}16&16\\16&16\end{pmatrix}.

Each row of \(R_F\) contributes an outer product with all entries equal to \(1\). Therefore

E=R_F'R_F= \begin{pmatrix}4&4\\4&4\end{pmatrix}.

For either response, the full model accounts for \(16\) units of fitted change relative to the reduced fit, and leaves \(4\) units of residual sum of squares. The off-diagonal entries show that the changes and residuals for the two responses move together in this example.

The Decomposition and Its Proof

The full model fits at least as flexibly as the reduced model, but the matrices give a more informative accounting than a statement about total residual size. The reduced-model residual matrix splits into the fitted change \(D\) and the full-model residual \(R_F\). Least-squares fitting makes these two components orthogonal, response by response.

Theorem: For nested linear models fitted by least squares to the same response matrix,

(Y-\widehat Y_R)'(Y-\widehat Y_R)=H+E.

Thus the reduced-model residual SSCP is the sum of the hypothesis SSCP and the full-model error SSCP.

Proof. By the definitions of \(D\) and \(R_F\),

Y-\widehat Y_R =(Y-\widehat Y_F)+(\widehat Y_F-\widehat Y_R) =R_F+D.

Expanding the cross-product gives

(Y-\widehat Y_R)'(Y-\widehat Y_R) =R_F'R_F+D'D+R_F'D+D'R_F.

For least squares in the full model, each column of \(R_F\) is orthogonal to every vector in the full model’s fitted-value space. Each column of \(D\) belongs to that space: it is the difference of a full-model fitted column and a reduced-model fitted column, and nesting places both in the full-model space. Hence \(R_F'D=0\), and also \(D'R_F=0\). The remaining terms are \(E\) and \(H\), which proves the identity. \(\square\)

Both matrices also have a useful positivity property. For any vector \(a\in\mathbb{R}^p\),

a'Ha=a'D'Da=(Da)'(Da)=\lVert Da\rVert^2\geq 0, \qquad a'Ea=a'R_F'R_Fa=\lVert R_Fa\rVert^2\geq 0.

Therefore \(H\) and \(E\) are positive semidefinite. This does not mean that every entry is nonnegative: cross-products off the diagonal can be negative. The guaranteed nonnegativity applies to the quadratic forms above, not to each individual matrix entry.

Worked Example: A Hypothesis Matrix With Different Response Patterns

Suppose the reduced model fits the same response mean \((3,1)\) for four observations. The full-model fitted values differ from it by the rows of

D= \begin{pmatrix} 1&0\\ 0&2\\ -1&0\\ 0&-2 \end{pmatrix}.

These changes sum to zero in each response, as required for added fitted effects beyond an intercept-only reduced model. The full-model fitted matrix is therefore

\widehat Y_F= \begin{pmatrix} 4&1\\ 3&3\\ 2&1\\ 3&-1 \end{pmatrix}.

Take the full-model residual matrix to be

R_F= \begin{pmatrix} 1&2\\ -1&-2\\ 1&2\\ -1&-2 \end{pmatrix}.

For instance, these are valid least-squares fits when the full fitted-value space contains the intercept and the two columns of \(D\): each residual column is orthogonal to the intercept and to both columns of \(D\). The observed matrix is \(Y=\widehat Y_F+R_F\), namely

Y= \begin{pmatrix} 5&3\\ 2&1\\ 3&3\\ 2&-3 \end{pmatrix}.

The hypothesis matrix follows by taking column cross-products. The first column of \(D\) has squared length \(1^2+(-1)^2=2\), the second has squared length \(2^2+(-2)^2=8\), and their cross-product is zero. Hence

H= \begin{pmatrix}2&0\\0&8\end{pmatrix}.

The residual columns are \((1,-1,1,-1)'\) and twice that vector. Their squared lengths are \(4\) and \(16\), and their cross-product is \(8\). Thus

E= \begin{pmatrix}4&8\\8&16\end{pmatrix}.

Here the added fit changes the two responses in separate patterns, so \(H\) has zero cross-product. The residuals, in contrast, are proportional across responses, giving a positive off-diagonal entry in \(E\).

Changing the Units or Coordinates of the Responses

The SSCP matrices depend on how responses are represented. Suppose the responses are recoded by an invertible \(p\times p\) matrix \(A\), so that the transformed response matrix is \(YA\). Its fitted values and residuals are correspondingly \(\widehat Y_FA\), \(\widehat Y_RA\), and \(R_FA\). The fitted difference is \(DA\), so direct multiplication gives

H_{\text{new}}=(DA)'(DA)=A'HA, \qquad E_{\text{new}}=(R_FA)'(R_FA)=A'EA.

Thus the entries change with the response coordinates, while the matrices continue to encode sums of squares and cross-products in those coordinates. This is important when responses are measured in different units or when a useful linear combination of responses is considered.

Worked Example: Re-expressing Two Responses

Use the first worked example, where \(H=16J\) and \(E=4J\), with \(J=\begin{pmatrix}1&1\\1&1\end{pmatrix}\). Define new responses by

A=\begin{pmatrix}1&1\\1&-1\end{pmatrix}.

The first new response is the sum of the original responses, and the second is their difference. Since

A'\begin{pmatrix}1\\1\end{pmatrix} =\begin{pmatrix}2\\0\end{pmatrix},

we have \(A'JA=\begin{pmatrix}4&0\\0&0\end{pmatrix}\). Consequently,

H_{\text{new}}=A'HA= \begin{pmatrix}64&0\\0&0\end{pmatrix}, \qquad E_{\text{new}}=A'EA= \begin{pmatrix}16&0\\0&0\end{pmatrix}.

The difference response has no fitted change or residual variation in this particular data set: the original response columns are identical. The zero second diagonal entries are therefore a feature of the data, not a general rule about response transformations.

Interpretation and Common Pitfalls

The decomposition matters because it separates variation associated with enlarging the model from variation left after fitting the full model. Each matrix retains response-specific sums of squares on its diagonal and joint response information off the diagonal. This makes the pair a natural starting point for multivariate comparisons, while keeping clear which variation comes from the model change and which comes from residuals.

Confusing \(H\) with full-model residual variation. \(H\) is formed from \(\widehat Y_F-\widehat Y_R\), not from \(Y-\widehat Y_F\). The latter produces \(E\).
Dropping cross-products. Replacing either matrix by only its diagonal discards information about how response columns vary together.
Assuming all entries must be nonnegative. The matrices are positive semidefinite, but an off-diagonal cross-product may be negative.
Calling \(E\) a covariance estimate without scaling. \(E\) is a residual cross-product matrix. Any covariance estimate requires an appropriate scaling and its own assumptions.

Key takeaway: For nested least-squares fits, \(H\) is the cross-product matrix of the change in fitted responses, and \(E\) is the cross-product matrix of full-model residuals. Their sum is the reduced-model residual SSCP.

Check Your Understanding

Use the definitions and decomposition to answer the following questions.

In a two-response model comparison, what data matrix is cross-multiplied to form \(H\), and what matrix is cross-multiplied to form \(E\)?
What do the diagonal and off-diagonal entries of an SSCP matrix represent?
Why does least-squares fitting in the full model make the cross-terms in the decomposition vanish?
Can a positive semidefinite SSCP matrix have a negative off-diagonal entry? Explain.
If responses are transformed by a matrix \(A\), how do \(H\) and \(E\) change?