Rank and Redundancy in Hypothesis Matrices

Why Rank Matters in a Hypothesis

A matrix hypothesis may be written with many rows, but the row count need not equal the number of independent restrictions. For instance, if one row of a restriction matrix is the sum of two others, that row adds no new information about the coefficient matrix—provided its target is the corresponding sum as well. If the targets disagree, however, the restrictions may be incompatible. Rank is therefore not just a computational detail: it determines what the hypothesis says and whether a proposed simplification preserves it.

This tutorial develops rank and redundancy for linear restrictions in a multivariate linear model. The key distinction is between the written number of equations and the effective rank of the linear map imposing them. A reduction is valid only when it preserves the full set of coefficient matrices satisfying the null hypothesis.

Definition: Let \(B\) be a \(q\times p\) coefficient matrix and consider the restrictions \(CB=\Theta\), where \(C\) is \(r\times q\) and \(\Theta\) is \(r\times p\). The effective rank of these restrictions is \(\operatorname{rank}(C)\). If \(C\) has rank \(k\), then at most \(k\) rows of \(C\) are linearly independent. The full system contains \(p\operatorname{rank}(C)\) independent scalar restrictions when the coefficient entries are unrestricted.

The target matrix \(\Theta\) matters just as much as \(C\). A dependence among rows of \(C\) must also hold among the corresponding rows of \(\Theta\) if all equations are to be jointly satisfiable. A set of independent rows is enough to describe the null hypothesis only after this compatibility has been checked.

When Can Dependent Rows Be Removed?

Suppose \(C\) has rank \(k\), and select \(k\) linearly independent rows to form a matrix \(C_0\). There is an \(r\times k\) matrix \(R\) such that \(C=RC_0\). The selected rows can be arranged first, so the corresponding rows of \(R\) form the \(k\times k\) identity matrix. Let \(\Theta_0\) contain the target rows corresponding to \(C_0\).

Theorem: With \(C=RC_0\) as above and \(\Theta_0\) the target rows for the selected independent rows, the systems \(CB=\Theta\) and \(C_0B=\Theta_0\) have exactly the same solution set if and only if \(\Theta=R\Theta_0\).

Proof. If \(\Theta=R\Theta_0\), then any \(B\) satisfying \(C_0B=\Theta_0\) also satisfies \(CB=RC_0B=R\Theta_0=\Theta\). Conversely, if \(CB=\Theta\), its selected rows give \(C_0B=\Theta_0\). Hence every solution to the full system solves the reduced system. If the two systems have the same solution set, substitute any solution of the reduced system into the full equations: \(CB=RC_0B=R\Theta_0\), so the full target must equal \(R\Theta_0\). If the reduced system has no solution, the full system cannot have one either; in that case both solution sets are empty precisely when \(\Theta=R\Theta_0\) does not hold. More directly, the full system is consistent exactly when \(\Theta=R\Theta_0\) and \(C_0B=\Theta_0\) is consistent. Thus removing rows preserves the null hypothesis only when every discarded target is dictated by the retained targets. \(\square\)

The theorem gives a practical rule: choose independent rows of \(C\), retain their targets, and reconstruct every discarded target using the same row combinations. If even one target fails this check, simply dropping the associated equation changes the hypothesis. The original system might instead be inconsistent, in which case its null hypothesis is empty.

Worked Example: Removing a Compatible Redundant Row

Let \(B\) be \(2\times2\), and consider

C=\begin{pmatrix}1&0\\0&1\\1&1\end{pmatrix}, \qquad \Theta=\begin{pmatrix}0&0\\1&0\\1&0\end{pmatrix}.

The third row of \(C\) is the sum of its first and second rows, so \(\operatorname{rank}(C)=2\). Select the first two rows as \(C_0=I_2\). Then

R=\begin{pmatrix}1&0\\0&1\\1&1\end{pmatrix}, \qquad C=RC_0.

The first two target rows form \(\Theta_0\). Multiplying them by \(R\) gives

R\Theta_0 = \begin{pmatrix}1&0\\0&1\\1&1\end{pmatrix} \begin{pmatrix}0&0\\1&0\end{pmatrix} = \begin{pmatrix}0&0\\1&0\\1&0\end{pmatrix} =\Theta.

The compatibility condition holds. The full equations say that the first row of \(B\) is \((0,0)\), the second row is \((1,0)\), and their sum is \((1,0)\). The third equation follows from the first two, so the reduced system \(C_0B=\Theta_0\) describes exactly the same null hypothesis. There are \(2\cdot2=4\) independent scalar restrictions, not \(3\cdot2=6\).

Worked Example: A Redundant Row With an Incompatible Target

Keep the same \(C\), but change the target to

\Theta=\begin{pmatrix}0&0\\1&0\\0&0\end{pmatrix}.

The first two equations require the rows of \(B\) to be \((0,0)\) and \((1,0)\). Their sum is \((1,0)\), but the third equation requires that sum to be \((0,0)\). Thus no \(B\) satisfies all three equations. In the theorem's notation, the selected targets give

R\Theta_0 = \begin{pmatrix}0&0\\1&0\\1&0\end{pmatrix} \ne \begin{pmatrix}0&0\\1&0\\0&0\end{pmatrix} =\Theta.

If the third equation is dropped, the reduced system is satisfied by \(B=\begin{pmatrix}0&0\\1&0\end{pmatrix}\). The reduced system is therefore not equivalent to the original one: it replaces an empty null hypothesis with a nonempty one. The row of \(C\) is redundant as a linear functional, but its mismatched target makes the full system inconsistent.

General Linear Restrictions and the Restriction Operator

A more general matrix restriction is \(CBM=\Theta\), where \(M\) is \(p\times s\) and \(\Theta\) is \(r\times s\). In this case, dependencies can occur through either \(C\) or \(M\). To count the independent scalar equations, write the matrix equation as a vector equation. Under column-wise vectorization,

\operatorname{vec}(CBM)=(M^{\mathsf T}\otimes C)\operatorname{vec}(B).

The matrix \(M^{\mathsf T}\otimes C\) is the restriction operator: it maps the \(qp\) entries of \(B\) to the \(rs\) entries of \(CBM\). Thus its rank, rather than the number \(rs\) of displayed equations, is the effective number of independent scalar restrictions.

Theorem: For \(C\) and \(M\) of the dimensions above, \(\operatorname{rank}(M^{\mathsf T}\otimes C)=\operatorname{rank}(M)\operatorname{rank}(C)\). Therefore \(CBM=\Theta\) imposes at most \(\operatorname{rank}(M)\operatorname{rank}(C)\) independent scalar restrictions.

Proof. Put \(k=\operatorname{rank}(C)\) and \(l=\operatorname{rank}(M)\). A rank factorization gives \(C=UV\), where \(U\) has \(k\) linearly independent columns and \(V\) has \(k\) linearly independent rows. Similarly write \(M=ST\), with \(S\) having \(l\) linearly independent columns and \(T\) having \(l\) linearly independent rows. Then

M^{\mathsf T}\otimes C =(T^{\mathsf T}\otimes U)(S^{\mathsf T}\otimes V).

The first factor has \(lk\) linearly independent columns: a left inverse for each of \(T^{\mathsf T}\) and \(U\) induces a left inverse for their Kronecker product. The second factor has \(lk\) linearly independent rows, by the corresponding right-inverse argument for \(S^{\mathsf T}\) and \(V\). Their product has rank \(lk\): it is a product of a full-column-rank matrix and a full-row-rank matrix with inner dimension \(lk\), so it has rank \(lk\). This proves the stated formula. \(\square\)

The rank calculation counts independent equations in the linear map, but it does not by itself guarantee that the target is compatible. For a system \(L\operatorname{vec}(B)=\operatorname{vec}(\Theta)\), compatibility means that \(\operatorname{vec}(\Theta)\) lies in the column space of \(L\). If it does not, the null set is empty. When it does, independent rows of \(L\), with their corresponding targets, can be retained to describe the same null set.

Worked Example: Counting Restrictions Through Both Factors

Let \(C\) be \(3\times2\) with rank 2, and let

M=\begin{pmatrix}1&2\\2&4\end{pmatrix}.

The second column of \(M\) is twice the first, so \(\operatorname{rank}(M)=1\). Although \(CBM=\Theta\) displays \(3\cdot2=6\) scalar equations, its restriction operator has rank

\operatorname{rank}(M^{\mathsf T}\otimes C) =\operatorname{rank}(M)\operatorname{rank}(C) =1\cdot2=2.

Only two scalar equations can be independent. This can also be seen directly: each column of \(CBM\) is determined by the other, because the second column is twice the first. Since \(C\) has rank 2, the first column of \(CBM\) can supply at most two independent scalar restrictions. For a particular target, the two columns of \(\Theta\) must have the same doubling relationship, and its first column must be attainable as \(CB\)'s first relevant contrast. If those compatibility conditions fail, the full null hypothesis is empty; the rank count alone does not make row or column removal valid.

Why Equivalent Reductions Preserve the Hypothesis Matrix

In the multivariate linear model \(Y=XB+U\), let \(Y\) be \(n\times p\), \(X\) be \(n\times q\) of full column rank, and let \(Y_0\) be the fitted matrix minimizing the residual sum-of-squares matrix subject to a given restriction. Let \(\widehat Y=X\widehat B\) be the unrestricted least-squares fit. The hypothesis sum-of-squares matrix for these nested fits is

H=(Y-Y_0)^{\mathsf T}(Y-Y_0)-(Y-\widehat Y)^{\mathsf T}(Y-\widehat Y).

Because \(Y-\widehat Y\) is orthogonal to the column space of \(X\), and \(\widehat Y-Y_0\) lies in that space, the cross-products vanish. Expanding the first term therefore gives

H=(\widehat Y-Y_0)^{\mathsf T}(\widehat Y-Y_0).

This identity shows why the exact set of allowed coefficient matrices matters. If two restriction systems have the same solution set for \(B\), they permit the same fitted matrices \(XB\). They therefore have the same restricted fit \(Y_0\), and the same \(H\). Removing dependent rows with compatible targets cannot change the hypothesis sum-of-squares matrix; removing a row with an incompatible target can change the restricted model and hence \(H\).

Key takeaway: First find the rank of the restriction operator. Then check that every discarded equation's target is implied by the retained equations. Only a reduction preserving the entire null set is guaranteed to preserve the restricted fit and its hypothesis sum-of-squares matrix.

Common Pitfall

A frequent mistake is to count the rows of \(C\) and call that the number of restrictions, or to remove a dependent row without checking its target. For \(CB=\Theta\), the number of independent scalar restrictions is \(p\operatorname{rank}(C)\), not \(rp\) unless \(C\) has full row rank. But rank alone is not a license to discard equations: dependent rows with incompatible targets can make the full null hypothesis empty. Reduce the operator and its targets together, and verify that the resulting system has exactly the same solution set.

Check Your Understanding

Use rank and target compatibility together when assessing a proposed reduction.

If \(C\) has four rows and rank two, how many independent scalar restrictions does \(CB=\Theta\) impose when \(B\) has three columns?
For a dependent row of \(C\), what relationship must its target row satisfy before that restriction can be removed?
Explain why the original and reduced systems in the incompatible-target example have different null sets.
If \(\operatorname{rank}(C)=3\) and \(\operatorname{rank}(M)=2\), what is the rank of the restriction operator for \(CBM=\Theta\)?
Why does an equivalent restriction system have the same hypothesis sum-of-squares matrix?