Rank Deficiency and Estimable Regression Functions

Why Rank Deficiency Changes What Coefficients Mean

In a regression model, a design matrix can contain redundant columns. For example, a predictor might be an exact sum of other predictors, or a factor might be represented with a redundant set of indicator columns. Then multiple coefficient matrices produce the same mean response. The data can still determine fitted values uniquely, but it cannot distinguish every coefficient separately.

The important question is therefore not simply whether a coefficient estimate can be calculated. It is whether the particular function of the coefficients is estimable: uniquely determined by the model's mean response. This distinction lets us use least squares and generalized inverses without treating a nonunique coefficient matrix as if it were uniquely identified.

If \(X\) has full column rank, \(XB_1=XB_2\) implies \(B_1=B_2\), so all coefficient functions are identified. If \(X\) is rank deficient, there is a nonzero vector \(h\) with \(Xh=0\). For any coefficient matrix \(B\), changing it to \(B+h c'\), for any \(q\)-vector \(c\), leaves the mean unchanged: \(X(B+h c')=XB+(Xh)c'=XB\). Thus some coefficient directions cannot be learned from the model's mean.

Characterizing Estimable Functions

For a single response, a linear coefficient function has the form \(a'\beta\), where \(a\) is a \(p\)-vector. It is estimable precisely when \(a\) belongs to the row space of \(X\). For multiple responses, apply the same condition to every row of \(A\).

Proof. Suppose \(A=CX\). If \(B_1\) and \(B_2\) give the same mean, then \(X(B_1-B_2)=0\), and \(A(B_1-B_2)=CX(B_1-B_2)=0\). Therefore \(AB_1=AB_2\), so \(AB\) is estimable.

Conversely, suppose \(AB\) is estimable. If \(h\) is in the null space of \(X\), then \(B\) and \(B+h c'\) give the same mean for every \(c\). Estimability requires \(A h c'=0\) for every \(c\), which implies \(Ah=0\). Thus every row of \(A\) is orthogonal to the null space of \(X\). In finite dimensions, the orthogonal complement of the null space of \(X\) is the row space of \(X\). Hence every row of \(A\) lies in that row space, so \(A=CX\) for some \(C\). This also proves the stated null-space equivalence. \(\square\)

The same condition has an estimation interpretation. A linear estimator \(CY\) has expectation \(CXB\), so it is unbiased for \(AB\) for every \(B\) exactly when \(CX=A\). Such a \(C\) exists precisely when the rows of \(A\) lie in the row space of \(X\). This is another way to see why a nonestimable coefficient function cannot be recovered by choosing a different linear estimation formula.

Least Squares and the Unique Fitted Values

Even when the coefficients are nonunique, least squares still determines a unique fitted response matrix. The fitted values are the orthogonal projection of \(Y\) onto the column space of \(X\), denoted \(\mathcal{C}(X)\). This column space is the set of all possible mean vectors for one response.

Proof. Write \(M=X'X\). For any vector \(y\), \(X'y\) belongs to the range of \(M\). To verify this, first note that \(Mv=0\) implies \(v'Mv=\|Xv\|^2=0\), and hence \(Xv=0\). Conversely, \(Xv=0\) implies \(Mv=0\). Thus \(M\) and \(X\) have the same null space, and their ranges, which are the orthogonal complements of their null spaces, are the same. In particular, \(X'y\), which belongs to the range of \(X'\), belongs to the range of \(M\).

Since \(X'y\) is in the range of \(M\), write \(X'y=Mu\) for some \(u\). The generalized-inverse condition gives \(MGX'y=MGMu=Mu=X'y\). Therefore \(b=GX'y\) satisfies the normal equations \(X'Xb=X'y\). These equations are the least-squares condition: for any vector \(d\), \(\|y-X(b+d)\|^2=\|y-Xb\|^2-2d'X'(y-Xb)+\|Xd\|^2\). The middle term is zero by the normal equations, and the final term is nonnegative. So \(b\) minimizes the squared error.

The normal equations also say that \(y-Xb\) is orthogonal to every vector in \(\mathcal{C}(X)\). Hence \(Xb\) is the orthogonal projection of \(y\) onto that space. This projected vector is unique: if \(u_1,u_2\in\mathcal{C}(X)\) both have residuals orthogonal to \(\mathcal{C}(X)\), then \(u_1-u_2\) belongs to \(\mathcal{C}(X)\) and is orthogonal to itself, so \(\|u_1-u_2\|^2=0\). Applying this argument to each response column proves the result for \(Y\). \(\square\)

The coefficient solution \(GX'Y\) can depend on the chosen generalized inverse, but its fitted matrix \(XGX'Y\) does not. A generalized inverse is a device for describing solutions to the normal equations, not a claim that the coefficient matrix is unique. In the full-rank case, the familiar inverse \((X'X)^{-1}\) is available; in the rank-deficient case, a generalized inverse replaces it.

Estimable Functions From a Generalized Inverse

A generalized inverse can also give an unbiased estimator of any estimable function. Let \(G\) satisfy \(MGM=M\), where \(M=X'X\). If \(A\) is estimable, its rows lie in the row space of \(X\), which equals the row space of \(M\). Thus \(A=DM\) for some \(D\), and \(AGM=DMGM=DM=A\). The coefficient solution \(\hat B=GX'Y\) therefore satisfies \(E[A\hat B]=AGX'XB=AGMB=AB\). The estimate \(A\hat B\) is the same for every least-squares coefficient solution, even though \(\hat B\) itself need not be unique.

Why This Distinction Matters

Rank deficiency often reflects how a model has been written rather than a failure of least squares. The model may still determine every fitted mean, while redundant columns make the coefficient representation nonunique. Inference about regression functions must therefore target estimable combinations, not arbitrary coordinates of one selected coefficient solution.

For generalized least squares with a positive-definite covariance matrix for the cases, the same identification condition holds: \(XB\) is still the mean matrix, so the estimable functions are unchanged. The fitting criterion changes the projection geometry: after weighting (or whitening) the cases, fitted means are obtained by projection in that weighted geometry. Rank deficiency still does not make the coefficient matrix unique.