Rank Conditions for Classical Multivariate Tests

Why Rank Conditions Matter

Classical multivariate tests compare variation attributed to a hypothesis with residual variation. Their determinant and root calculations depend on that residual variation being usable in every response direction. A design can have many observations and still fail this requirement if it uses too many independent predictors relative to its sample size. A second problem arises when the requested hypothesis is not estimable from the design: in that case, there is no uniquely determined test of the stated effect.

The checks are related but distinct. The design rank determines the residual degrees of freedom; the number of responses determines how many independent residual directions are needed to estimate their joint covariance; and the hypothesis rank determines how many independent directions the test can attribute to the restrictions. Passing one check does not guarantee that the others pass.

Model, Hypothesis, and the Relevant Ranks

Let $Y$ be an $n\times p$ matrix of observations on $p$ responses, and let $X$ be an $n\times q$ design matrix. The multivariate linear model is $Y=XB+U$, where $B$ is a $q\times p$ coefficient matrix and $U$ is the error matrix. Write $r=\operatorname{rank}(X)$. The fitted-value space is the column space of $X$, and $P_X$ denotes the orthogonal projection onto it. The residual projection is $I-P_X$.

To describe a hypothesis, consider a reduced model whose fitted-value space is a subspace of the full model's fitted-value space. Let $P_0$ and $P_1$ be the projections onto the reduced and full spaces, respectively, so $P_0P_1=P_0$. Define $h=\operatorname{rank}(P_1-P_0)$, the number of independent fitted directions added by the full model. For a full model with design $X$, $P_1=P_X$, and $r=\operatorname{rank}(P_1)$.

Definition: The hypothesis and error sum-of-squares-and-cross-products matrices are $$ H=Y'(P_1-P_0)Y,\qquad E=Y'(I-P_1)Y. $$ The residual degrees of freedom are $\nu=n-r$, where $r=\operatorname{rank}(P_1)$. A hypothesis has effective rank $h=\operatorname{rank}(P_1-P_0)$. The number of responses is $p$.

Both matrices are $p\times p$ and positive semidefinite. The usual classical tests include calculations involving $E^{-1}$, determinants such as $|E|$, or roots formed from $H$ and $E$. Consequently, positive definiteness of $E$ is a central condition. The rank $h$ must also represent a genuine, estimable comparison: redundant restrictions do not create additional independent hypothesis directions.

Definition: A coefficient contrast $a'B$ is estimable if it has the same value for every coefficient matrix $B$ that gives the same fitted mean matrix $XB$. Equivalently, $a$ belongs to the row space of $X$. A matrix of contrasts is estimable when each of its rows belongs to that row space.

This matters when $X$ is rank deficient. The fitted mean $XB$ can still be uniquely determined, but individual coefficients need not be. A test of an unestimable coefficient contrast cannot be repaired just by having many residual degrees of freedom. For a hypothesis expressed as a reduced and full fitted-value space, estimability is built into the comparison: both spaces must be valid subspaces of the model's fitted-value space.

What the Hypothesis Rank Can Be

The matrix $P_1-P_0$ is itself an orthogonal projection: it projects onto the part of the full fitted-value space orthogonal to the reduced space. Its rank is $h$. The next result bounds the rank of $H$, regardless of the particular observed responses.

Theorem: For nested fitted-value spaces as above, $H$ is positive semidefinite and $$ \operatorname{rank}(H)\leq \min(p,h). $$ Thus no more than $\min(p,h)$ hypothesis directions can contribute nonzero roots to a classical multivariate comparison.

Proof. Let $Q$ be an $n\times h$ matrix whose columns are an orthonormal basis for the part of the full fitted-value space orthogonal to the reduced space. Then $P_1-P_0=QQ'$, so $$ H=Y'QQ'Y=(Q'Y)'(Q'Y). $$ For any $v\in\mathbb{R}^p$, $v'Hv=\|Q'Yv\|^2\geq 0$, proving positive semidefiniteness. Also, the rank of $(Q'Y)'(Q'Y)$ equals the rank of $Q'Y$, which cannot exceed either the number $h$ of rows of $Q'Y$ or the number $p$ of its columns. Hence $\operatorname{rank}(H)\leq\min(p,h)$. $\square$

Worked Example: Checking a Hypothesis Rank Bound

Suppose a full model adds three independent fitted directions to a reduced model, and there are two responses. Here $h=3$ and $p=2$. The theorem gives $\operatorname{rank}(H)\leq\min(2,3)=2$. Even though the model comparison adds three independent directions in the observation space, a $2\times2$ hypothesis matrix cannot have rank greater than two. In particular, the test cannot have three nonzero hypothesis roots.

If instead there were five responses with the same $h=3$, then $\operatorname{rank}(H)\leq\min(5,3)=3$. The bound is only an upper bound: the observed $H$ can have lower rank, for example if the responses have special linear dependencies in the added fitted directions.

When the Error Matrix Is Positive Definite

The residual projection has rank $\nu=n-r$. Since $E=Y'(I-P_1)Y$, its rank cannot exceed either the number of responses or the rank of the residual projection. Thus $\operatorname{rank}(E)\leq\min(p,\nu)$ for every data set. If $\nu<p$, this forces $E$ to be singular, without any distributional assumptions.

Theorem: Suppose the errors have independent rows with distribution $N_p(0,\Sigma)$, where $\Sigma$ is positive definite, and the fitted-value space has rank $r$. Then $E$ is positive definite with probability one if and only if $\nu=n-r\geq p$. If $\nu<p$, $E$ is singular for every data set.

Proof. Let $A$ be an $n\times\nu$ matrix whose columns form an orthonormal basis for the residual space, so $I-P_1=AA'$. The fitted mean lies in the full fitted-value space, so $A'XB=0$. Therefore $E=(A'Y)'(A'Y)=(A'U)'(A'U)$. Under the stated normal model, the rows of $A'U$ are independent $N_p(0,\Sigma)$ vectors: an orthonormal transformation preserves the joint Gaussian distribution, and their cross-covariances are zero. Thus $A'U$ is a $\nu\times p$ random matrix with a continuous density on its full matrix space. If $\nu<p$, its rank is at most $\nu<p$, so $E$ is singular.

If $\nu\geq p$, the first $p$ rows of $A'U$ form a square random matrix with a continuous density. Its determinant is a nonzero polynomial in the entries, whose zero set has Lebesgue measure zero. The square matrix is therefore nonsingular with probability one, so $A'U$ has column rank $p$ and $E=(A'U)'(A'U)$ is positive definite with probability one. $\square$

The theorem gives the basic sample-size condition for the usual determinant-based tests: $n-r\geq p$, or equivalently $n\geq r+p$. It is a condition on residual degrees of freedom, not merely on the raw sample size. When the condition holds, positive definiteness is guaranteed with probability one under the stated model, but a particular data set can still be singular if the observed residuals have an exact linear dependency. Without the normal model, the rank bound when $\nu<p$ remains certain, while the probability-one conclusion for $\nu\geq p$ requires additional assumptions on the data distribution.

Worked Example: Enough Residual Degrees of Freedom

A study has $n=18$ observations, a full design of rank $r=5$, and $p=4$ responses. The residual degrees of freedom are $$ \nu=n-r=18-5=13. $$ Since $13\geq4$, $E$ is positive definite with probability one under independent multivariate normal errors with positive-definite covariance. The sample-size check passes. If the tested comparison adds $h=2$ independent fitted directions, its hypothesis matrix has rank at most $\min(4,2)=2$. These are separate checks: this example passes the residual-rank condition, while the hypothesis rank still depends on the comparison actually specified.

Worked Example: Too Few Residual Directions

Now let $n=12$, $r=9$, and $p=4$. Then $$ \nu=12-9=3<4=p. $$ For any $12\times4$ response matrix, the residual projection has rank three, so $\operatorname{rank}(E)\leq3$. Hence $|E|=0$ and $E^{-1}$ does not exist. The usual determinant-based multivariate tests cannot be formed in their standard nonsingular form, regardless of how large the hypothesis rank is.

Adding observations or reducing the fitted model rank can increase $\nu$; reducing the response dimension can also change the comparison. A different method designed for singular or high-dimensional error structure may be appropriate, but it is not the usual determinant-based procedure with its standard calibration.

Design Rank and Estimability Are Separate Checks

A design matrix need not have full column rank for its fitted-value space to be well defined. However, a rank-deficient parameterization does not identify every coefficient separately. The test must concern an estimable contrast, and redundant restrictions should be reduced to independent ones before interpreting $h$.

Worked Example: Adequate Residual Rank but an Unestimable Contrast

Suppose the four columns of $X$ are $x_1,x_2,x_3,x_4$, with $x_4=x_2$, and the other three columns are linearly independent. Let $n=15$, $p=2$, and write the coefficient rows as $B_1,\ldots,B_4$. The design rank is $r=3$, so $\nu=15-3=12\geq2$. The residual-rank condition passes.

But the fitted contribution from the second and fourth columns is $x_2B_2+x_4B_4=x_2(B_2+B_4)$. The data identify their sum, not the two coefficient rows separately. In particular, the contrast $B_2$ is not estimable: changing $B_2$ by a row vector $d$ and $B_4$ by $-d$ leaves $XB$ unchanged but changes $B_2$. In contrast, $B_2+B_4$ is estimable because it is the coefficient of the identifiable fitted direction $x_2$. Thus adequate residual degrees of freedom do not make every requested hypothesis valid.

Practical Conditions for the Usual Tests

Before applying a classical determinant-based test, check the design, the restrictions, and the residual matrix in that order. A compact checklist is:

Find the fitted-space rank.
Compute $r=\operatorname{rank}(X)$, not just the number of columns in $X$. The residual degrees of freedom are $\nu=n-r$.

Check the response dimension.
For the usual nonsingular error-matrix procedures, require $\nu\geq p$, or $n\geq r+p$. If this fails, $E$ must be singular.

Check the hypothesis.
Confirm the restrictions are estimable and remove redundant restrictions. The effective comparison rank is $h$, and $\operatorname{rank}(H)\leq\min(p,h)$.

Inspect the observed error matrix.
Even when the model conditions imply $E$ is positive definite with probability one, check whether the computed $E$ is numerically or exactly singular before using an inverse or determinant-based statistic.

These conditions are necessary for the usual nonsingular calculations, but they are not a guarantee that every practical approximation or every scientific model is appropriate. For example, a model may satisfy $\nu\geq p$ while its residual matrix is nearly singular, making numerical calculations unstable. A very small eigenvalue can signal that some response combination has little residual variation, even if the determinant is technically nonzero.

Common Mistake: Counting Observations Instead of Residual Degrees of Freedom

A common incorrect argument is: “There are more observations than responses, so the error matrix must be invertible.” The comparison $n>p$ ignores how many independent fitted directions have been used. With $n=12$, $r=9$, and $p=4$, there are more observations than responses, but only $\nu=3$ residual directions. Therefore $\operatorname{rank}(E)\leq3$, and $E$ is necessarily singular.

The correction is to compare $p$ with $n-r$, using the actual rank of the full design. A related mistake is to assume that a full-rank error matrix makes every coefficient testable. It does not: estimability concerns the design and the specific contrast, whereas error-matrix invertibility concerns residual information across response dimensions.

Key takeaway: The usual determinant-based multivariate tests require enough residual directions to span the response space: $n-\operatorname{rank}(X)\geq p$. Separately, the tested restrictions must be estimable and nonredundant. Check both conditions before interpreting a test statistic.

Check Your Understanding

Use the ranks of the full design, residual space, response matrix, and hypothesis comparison to answer each question.

A model has $n=24$, design rank $r=18$, and $p=5$ responses. Does the residual-rank condition pass? Show the calculation.
Why must $E$ be singular if the residual degrees of freedom are smaller than the number of responses?
A comparison adds four independent fitted directions and has three responses. What is the largest possible rank of $H$?
Explain why a rank-deficient design can still have a well-defined fitted-value space but fail to identify an individual coefficient contrast.
Under independent multivariate normal errors with positive-definite covariance, what does $\nu\geq p$ imply about $E$, and what does it not imply for every possible observed data set?