Residual Cross-Products Across Multiple Responses
In a multivariate linear model, each case contributes a vector of responses. Fitting the model leaves a residual vector for each case, and those residuals can vary together across responses. The matrix that collects their sums of squares and cross-products is the residual sum-of-squares-and-cross-products matrix, abbreviated residual SSCP matrix. Its dimensions, entries, and degrees of freedom describe different features, so it is important not to treat them as interchangeable.
Let \(Y\) be an \(n\times q\) response matrix and \(X\) an \(n\times p\) design matrix of full column rank \(p\). In the multivariate linear model, write \(Y=XB+U\), where \(B\) is a \(p\times q\) coefficient matrix and \(U\) is the error matrix. The projection onto the column space of \(X\) is \(H=X(X'X)^{-1}X'\). The residual projection is \(I-H\); multiplying \(Y\) by it gives the \(n\times q\) matrix of residuals.
The diagonal entry \(E_{jj}\) is the residual sum of squares for response \(j\). An off-diagonal entry \(E_{jk}\) is a residual cross-product: it records whether residuals for responses \(j\) and \(k\) tend to have the same or opposite signs. Because \(E=R'R\), the matrix is positive semidefinite: for every \(q\)-vector \(a\), \(a'Ea=(Ra)'(Ra)\geq 0\). Its entries are measured in products of response units. For example, if one response is measured in milligrams and another in milliliters, their cross-product has units of milligram-milliliters.
Why the Residual Degrees of Freedom Are \(n-p\)
The residual degrees of freedom count the dimension of the part of the \(n\)-dimensional case space left after fitting the design. Since \(X\) has rank \(p\), its column space has dimension \(p\). The orthogonal residual space therefore has dimension \(n-p\). This count does not depend on how many response columns are in \(Y\).
The matrix \(I-H\) is symmetric and idempotent: \((I-H)'=I-H\) and \((I-H)^2=I-H\). Its range is the residual space, so its rank is \(n-p\). For a symmetric idempotent matrix, the trace equals the rank: its eigenvalues are only zero or one, so both quantities count the eigenvalues equal to one. Consequently, \(\operatorname{tr}(I-H)=n-p\).
To see the distinction, \(E=R'R\) gives \(\operatorname{rank}(E)=\operatorname{rank}(R)\). The residual matrix \(R\) has \(q\) columns but its columns all lie in a space of dimension \(\nu\). Therefore,
So residual degrees of freedom are determined by the design, while the rank of \(E\) is also limited by the number of response variables and may reflect the particular residual data. In particular, an SSCP matrix can be singular even though its residual degrees of freedom are positive.
Worked Example: Dimensions and Degrees of Freedom
Suppose \(n=12\) cases have \(q=3\) responses, and the design matrix has \(p=4\) linearly independent columns. Find the dimensions and residual degrees of freedom of \(E\), and give the largest possible value of its rank.
The residual matrix \(R=(I-H)Y\) has the same dimensions as \(Y\), namely \(12\times 3\). Hence \(E=R'R\) has dimensions \(3\times 3\). The residual subspace has dimension $$ \nu=n-p=12-4=8. $$ The rank bound is $$ \operatorname{rank}(E)\leq \min(q,\nu)=\min(3,8)=3. $$ Thus the residual degrees of freedom are 8, while the SSCP matrix can have rank at most 3. It is not correct to report 3 as the residual degrees of freedom just because \(E\) is a \(3\times 3\) matrix.
A Derived Result: Expected Residual SSCP
The residual SSCP is not, by itself, a covariance matrix estimate. Under specified error assumptions, however, its expectation shows why scaling it by the residual degrees of freedom gives a useful covariance estimate. Assume the design \(X\) is fixed and full column rank, and the error rows \(u_i'\) of \(U\) are independent, have mean zero, and share the same \(q\times q\) covariance matrix \(\Sigma\). No normality assumption is required for the expectation calculation.
Because \(HXB=XB\), the residual projection removes the fitted mean: \((I-H)Y=(I-H)U\). Let \(M=I-H\), with entries \(m_{ij}\). Then \(E=U'MU\). Its \((a,b)\) entry is \(\sum_{i=1}^n\sum_{j=1}^n m_{ij}U_{ia}U_{jb}\). Independence and zero means make \(E[U_{ia}U_{jb}]=0\) when \(i\ne j\), while the common row covariance gives \(E[U_{ia}U_{ib}]=\Sigma_{ab}\) when \(i=j\). Therefore,
This holds for every pair \(a,b\), and hence for the whole matrix:
The scaling matters: \(E\) is a sum of residual cross-products, while \(E/(n-p)\) has the role of a covariance estimate. The denominator comes from the dimension of the residual subspace, not from the number \(n\) of cases alone. If errors are dependent across cases or have different covariance matrices, the expectation need not equal \((n-p)\Sigma\); the unbiasedness claim must not be carried over without checking the assumptions.
Worked Example: Computing a Residual SSCP and Scaling It
Consider an intercept-only model with \(n=3\) cases and \(q=2\) responses. Suppose the residual rows are \((1,0)\), \((0,1)\), and \((-1,-1)\). They sum to zero, as required for residuals from an intercept-only model. Compute \(E\), its degrees of freedom, and the covariance estimate obtained by dividing by those degrees of freedom.
The residual matrix is $$ R= \begin{pmatrix} 1&0\\ 0&1\\ -1&-1 \end{pmatrix}. $$ The residual SSCP is $$ E=R'R = \begin{pmatrix} 1&0&-1\\ 0&1&-1 \end{pmatrix} \begin{pmatrix} 1&0\\ 0&1\\ -1&-1 \end{pmatrix} = \begin{pmatrix} 2&1\\ 1&2 \end{pmatrix}. $$ The design has one column, the intercept, so \(p=1\) and \(\nu=3-1=2\). Thus $$ \frac{E}{\nu}= \begin{pmatrix} 1&0.5\\ 0.5&1 \end{pmatrix}. $$ The diagonal entries of \(E\) are the residual sums of squares, and its off-diagonal entries are the residual cross-products. The scaled matrix is the unbiased covariance estimate under the independent, common-covariance error assumptions; \(E\) itself is the unscaled SSCP.
Worked Examples: What the Rank Does and Does Not Count
Worked Example: Positive Degrees of Freedom but a Singular SSCP
Let \(n=3\), \(p=2\), and \(q=3\). Take \(X\) to have columns \((1,0,0)'\) and \((0,1,0)'\), so \(H=\operatorname{diag}(1,1,0)\) and \(I-H=\operatorname{diag}(0,0,1)\). Suppose the third row of \(Y\) is \((1,2,0)\); the first two rows do not affect the residuals.
Multiplication by \(I-H\) leaves only the third row, giving residual rows \((0,0,0)\), \((0,0,0)\), and \((1,2,0)\). Therefore $$ E=R'R= \begin{pmatrix} 1&2&0\\ 2&4&0\\ 0&0&0 \end{pmatrix}. $$ There are \(\nu=n-p=1\) residual degrees of freedom. The matrix has rank 1, not 3, because all residual variation lies along one response direction. Its singularity does not mean the residual degrees of freedom are zero; the design still leaves a one-dimensional residual space. In this example, dividing by \(\nu=1\) leaves \(E\) unchanged, but that does not make \(E\) a full-rank estimate of a three-dimensional covariance matrix.
Worked Example: A Cross-Product Has Two Response Units
Suppose a fitted model produces two residual columns, one measured in milligrams and one in milliliters. For three cases, let the residual pairs be \((1,2)\), \((2,-1)\), and \((-1,0)\), with the first coordinate in milligrams and the second in milliliters. These values are used only to illustrate how to form the cross-products.
The sums of squares are \(1^2+2^2+(-1)^2=6\) for the first response and \(2^2+(-1)^2+0^2=5\) for the second. The cross-product is \(1(2)+2(-1)+(-1)(0)=0\). Hence $$ E= \begin{pmatrix} 6&0\\ 0&5 \end{pmatrix}. $$ The diagonal entries have units of milligrams squared and milliliters squared; the off-diagonal entries have units of milligram-milliliters. A zero off-diagonal entry here means the products sum to zero, not that each case has zero residuals or that all forms of association between the responses are absent.
Common Mistakes and Why the Distinction Matters
- Using the number of responses as the residual degrees of freedom. A \(q\times q\) SSCP has \(q\) rows and columns, but the residual degrees of freedom are \(n-p\), determined by the design.
- Equating the rank of \(E\) with its degrees of freedom. The rank is at most \(\min(q,n-p)\); the degrees of freedom remain \(n-p\), even when the observed matrix is singular.
- Calling \(E\) a covariance estimate without scaling. Under the stated assumptions, \(E\) has expectation \((n-p)\Sigma\). It is \(E/(n-p)\), not \(E\), that is unbiased for \(\Sigma\).
- Using the covariance scaling without its assumptions. The expectation calculation requires independent error rows with a common covariance matrix. Correlation across cases or unequal row covariances can change the expectation.
Keeping these quantities separate is useful whenever residual variation across multiple outcomes is summarized or compared. The SSCP records the observed residual cross-products; the residual degrees of freedom describe how many independent directions in case space remain after fitting. Under the specified error assumptions, the scaled SSCP estimates the common response covariance. Those are related facts, but they answer different questions.
Check Your Understanding
Use the definitions and derivations above to answer these questions.
- If \(n=20\) and the full-rank design has \(p=5\) columns, what are the residual degrees of freedom?
- For \(q=4\) responses and \(\nu=2\), what is the largest possible rank of the residual SSCP matrix?
- What does an off-diagonal entry of \(E\) sum across cases, and what units does it have?
- Under what assumptions is \(E/(n-p)\) an unbiased estimate of \(\Sigma\)?
- Why can \(E\) be singular even when the residual degrees of freedom are positive?