Tutorials › Multivariate Statistics (Graduate) › Extra-Sum-of-Squares Tests for Nested Designs

Home  /  Tutorials  /  Hypothesis Testing in Linear Models

Hypothesis Testing in Linear Models · Tutorial 324 of 975

Extra-Sum-of-Squares Tests for Nested Designs

Use the difference between nested-model projections to measure the multivariate contribution of added predictor terms and build a test of that contribution.

Intermediate 11 min read

What You'll Learn

  • Define the incremental projection for nested reduced and full designs
  • Compute the hypothesis sum-of-squares matrix from added predictor terms
  • Relate its rank to the increase in design-space dimension
  • Separate the incremental hypothesis matrix from the full-model error matrix
  • Formulate a Gaussian multivariate likelihood-ratio test for the added terms

Why Compare Nested Designs?

In multivariate regression, a predictor may be added to explain several responses at once. The relevant question is not whether the full model fits the observed responses, but whether the added predictor space improves fit beyond what the reduced model already explains. Nested designs make this comparison precise: the extra sum-of-squares matrix records the part of the response variation explained by the full design but not by the reduced design.

Let \(Y\) be an \(n\times q\) response matrix, with one row per observation and one column per response. Let the reduced and full design matrices be \(X_R\) and \(X_F\). Assume their column spaces are nested, \(\mathcal C(X_R)\subseteq\mathcal C(X_F)\). Write \(P_R\) and \(P_F\) for the orthogonal projection matrices onto those spaces. These projections are well-defined even if a design matrix has redundant columns.

Definition: The incremental projection for the added design terms is \(D=P_F-P_R\). The corresponding extra-sum-of-squares hypothesis matrix is $$ H=Y'(P_F-P_R)Y=Y'DY. $$ The full-model error sum-of-squares matrix is $$ E=Y'(I-P_F)Y. $$ Both \(H\) and \(E\) are \(q\times q\) matrices. \(H\) measures the response variation attributable to the added design space; \(E\) measures variation left unexplained by the full design.

Projection Geometry and the Incremental Matrix

The nesting condition means that every vector in the reduced design space is also in the full design space. Consequently \(P_FP_R=P_RP_F=P_R\). This gives the key properties of the difference \(D\).

Result: If \(\mathcal C(X_R)\subseteq\mathcal C(X_F)\), then \(D=P_F-P_R\) is a symmetric idempotent matrix, and its rank is $$ d=\operatorname{rank}(X_F)-\operatorname{rank}(X_R). $$ It projects onto the part of the full design space orthogonal to the reduced design space.

Proof. Both projections are symmetric, so \(D'=D\). Using \(P_FP_R=P_RP_F=P_R\) and \(P_F^2=P_F,\ P_R^2=P_R\),

$$ D^2=(P_F-P_R)^2 =P_F-P_FP_R-P_RP_F+P_R =P_F-P_R =D. $$

Thus \(D\) is an orthogonal projection. Its range is the portion of \(\mathcal C(X_F)\) orthogonal to \(\mathcal C(X_R)\), so its dimension is the difference between the dimensions of those nested spaces. That difference is \(d\). \(\square\)

The same projection identities show how the reduced-model residual matrix splits into two parts:

$$ Y'(I-P_R)Y =Y'\{(P_F-P_R)+(I-P_F)\}Y =H+E. $$

This is the matrix-valued version of the extra-sum-of-squares decomposition. It is useful to keep the terms distinct: the extra matrix \(H\) is based on the difference between the projections, whereas the error matrix \(E\) is based on the residual projection from the full model.

Result: \(H\) and \(E\) are positive semidefinite, and \(\operatorname{rank}(H)\leq \min(q,d)\). In particular, the rank of \(H\) cannot exceed the number of new independent design directions.

Proof. For any \(a\in\mathbb R^q\), \(a'Ha=(Ya)'D(Ya)=\|DYa\|^2\geq0\), since \(D\) is a symmetric projection. The same argument with \(I-P_F\) proves that \(E\) is positive semidefinite. Finally, \(H=(DY)'(DY)\), and \(D\) has rank \(d\); therefore \(\operatorname{rank}(H)\leq d\). As \(H\) is \(q\times q\), its rank is also at most \(q\). \(\square\)

Worked Example: Computing the Incremental Matrix

Worked Example: Computing the Incremental Matrix

There are four observations and two responses. The reduced design contains only an intercept, and the full design adds the predictor \(z=(-1,0,0,1)'\). The response matrix is

$$ Y= \begin{pmatrix} 1&2\\ 3&0\\ 2&1\\ 4&3 \end{pmatrix}. $$

The intercept vector is orthogonal to \(z\), since \(\mathbf 1'z=0\). Also, \(z'z=2\), so the incremental projection is \(D=zz'/2\). First calculate the predictor-response cross-products:

$$ z'Y= \begin{pmatrix} -1&0&0&1 \end{pmatrix} \begin{pmatrix} 1&2\\ 3&0\\ 2&1\\ 4&3 \end{pmatrix} = \begin{pmatrix} 3&1 \end{pmatrix}. $$

It follows that

$$ H=Y'\frac{zz'}{2}Y =\frac{(z'Y)'(z'Y)}{2} =\frac12 \begin{pmatrix}3\\1\end{pmatrix} \begin{pmatrix}3&1\end{pmatrix} = \begin{pmatrix} 9/2&3/2\\ 3/2&1/2 \end{pmatrix}. $$

This matrix has rank one, as expected: the added predictor contributes only one design direction, even though there are two responses. Its off-diagonal entries reflect the joint variation explained across the two responses. The matrix is not two separate sums of squares; its diagonal entries are response-specific sums of squares, and its off-diagonal entries retain cross-response information.

Formulating the Multivariate Test

Suppose the full-model mean matrix is \(M=X_FB_F\), where \(B_F\) is a coefficient matrix. The null hypothesis that the added design terms have no effect on any response is

$$ H_0: DM=0 \qquad\text{against}\qquad H_A: DM\ne 0. $$

Under \(H_0\), the mean lies in the reduced design space, because \(DM=0\) says that its component in the added space is zero. The alternative allows a nonzero mean component in that space for at least one response. Thus the test is an omnibus test across responses: it does not require each response to show an effect separately.

For example, assume independent rows with \(Y\sim MN_{n,q}(X_FB_F,I_n,\Sigma)\), where \(\Sigma\) is positive definite. If the reduced and full models are fit by least squares, their residual sum-of-squares matrices are \(E+H\) and \(E\), respectively. Provided these matrices are positive definite, maximizing the Gaussian likelihood over the mean coefficients and covariance matrix gives the likelihood-ratio statistic

$$ \Lambda=\left(\frac{|E|}{|E+H|}\right)^{n/2}. $$

Small values of \(\Lambda\) count against the reduced model: they indicate that the full model's residual matrix is substantially smaller, in the determinant sense, than the reduced model's. Equivalently, one may use \(|E|/|E+H|\) as the statistic, rejecting for sufficiently small values. Under the Gaussian model, its null distribution supplies the calibration when the relevant residual matrices are nonsingular. In particular, with nonsingular \(\Sigma\), \(E\) is positive definite with probability one if the full-model residual degrees of freedom \(n-\operatorname{rank}(X_F)\) are at least \(q\).

The likelihood-ratio formula follows by profiling the Gaussian likelihood. For a fitted mean with residual matrix \(S\), the maximizing covariance estimate is \(S/n\); substituting it into the likelihood makes the part depending on \(S\) proportional to \(|S|^{-n/2}\). The restricted fit has \(S=E+H\), and the full fit has \(S=E\). Their likelihood ratio is therefore the displayed \(\Lambda\). This formulation is for the nested-design comparison; the specific finite-sample calibration and choice among classical multivariate criteria depend on the test procedure being used.

Worked Example: Recognizing Redundant Added Predictors

Suppose \(X_R=\mathbf 1\) and the full design is \(X_F=[\mathbf 1,\ z,\ 2z]\), using the predictor vector from the previous calculation. Although two columns have been appended, the columns \(z\) and \(2z\) span only one new direction.

Design-space increment. The reduced design has rank one; the full design has rank two. Thus \(d=2-1=1\), not two. The incremental projection remains \(D=zz'/2\).

Hypothesis matrix. The response matrix in the calculation above therefore gives \(H=\frac12\begin{pmatrix}9&3\\3&1\end{pmatrix}\), just as it did when the full design added only \(z\). Appending a duplicate direction cannot create another increment in explained variation.

Interpretation. It would be misleading to test two freely varying coefficients for \(z\) and \(2z\) as though both were separately identifiable. Only their combined contribution along the direction \(z\) is determined by the fitted mean. The projection-based test handles this redundancy directly through the rank difference.

Worked Example: A Multivariate Likelihood-Ratio Calculation

Consider six observations, with the reduced design containing an intercept and the full design adding \(z=(1,-1,0,0,0,0)'\). Define \(r_1=(1,1,-2,0,0,0)'\) and \(r_2=(1,1,1,-3,0,0)'\). Both residual vectors are orthogonal to the intercept and to \(z\), and \(r_1'r_2=0\). Let the response columns be

$$ y_1=2\mathbf 1+z+r_1,\qquad y_2=-\mathbf 1+2z+2r_2. $$

These definitions give the observed response matrix

$$ Y= \begin{pmatrix} 4&3\\ 2&-1\\ 0&1\\ 2&-7\\ 2&-1\\ 2&-1 \end{pmatrix}. $$

Because \(z'z=2\), the increment is \(D=zz'/2\). The cross-products are \(z'y_1=2\) and \(z'y_2=4\), so

$$ H=\frac12 \begin{pmatrix}2\\4\end{pmatrix} \begin{pmatrix}2&4\end{pmatrix} = \begin{pmatrix}2&4\\4&8\end{pmatrix}. $$

The full-model residuals are \(r_1\) and \(2r_2\). Since \(r_1'r_1=6\), \(r_2'r_2=12\), and \(r_1'r_2=0\),

$$ E= \begin{pmatrix} 6&0\\ 0&48 \end{pmatrix}, \qquad E+H= \begin{pmatrix} 8&4\\ 4&56 \end{pmatrix}. $$

The determinants are \(|E|=6\cdot48=288\) and \(|E+H|=8\cdot56-4\cdot4=432\). Hence

$$ \frac{|E|}{|E+H|}=\frac{288}{432}=\frac23, \qquad \Lambda=\left(\frac23\right)^{6/2}=\frac{8}{27}. $$

This calculation gives the test statistic, not a conclusion by itself. A decision requires comparing it with the appropriate null distribution or critical value for the specified test and design.

Common Mistakes and Practical Checks

  • Counting appended columns instead of new directions. The increment rank is \(d=\operatorname{rank}(X_F)-\operatorname{rank}(X_R)\). Redundant columns do not add degrees of freedom.
  • Using \(P_F\) alone for the hypothesis matrix. \(Y'P_FY\) includes variation already explained by the reduced model. The added-term matrix is \(Y'(P_F-P_R)Y\).
  • Using reduced-model residuals as the full-model error matrix. The correct error matrix for the nested comparison is \(E=Y'(I-P_F)Y\); the reduced residual matrix is \(E+H\).
  • Reading \(H\) as one test per response. The matrix contains cross-response terms as well as diagonal sums of squares. A multivariate test evaluates the added effect jointly.
  • Assuming the coefficient columns are uniquely meaningful when the design is rank deficient. The tested object is the added design-space contribution. Individual appended coefficients may not be identifiable.

Before applying a test, verify that the design spaces really are nested, compute their ranks, and check that the residual matrices meet the nonsingularity conditions required by the chosen statistic. These checks distinguish a valid nested comparison from a calculation that merely subtracts two unrelated fits.

Check Your Understanding

Use the projection definitions and rank logic above to answer the following.

  1. If the reduced and full design ranks are 3 and 7, respectively, what is the rank of the incremental projection?
  2. Why does \(Y'P_FY\) not, in general, equal the extra-sum-of-squares hypothesis matrix?
  3. If two appended predictor columns are scalar multiples of one another and are orthogonal to the reduced design space, how many new design directions do they contribute?
  4. State the null hypothesis for testing whether the added design space contributes to any response mean.
  5. In the six-observation example, why is the determinant ratio alone insufficient to decide whether to reject the null hypothesis?

Previous

Next

Back to Tutorials
Clinical Biostats