Rank and Collinearity: A Worked Matrix Audit

Why a Matrix Audit Matters

A design matrix is small enough to inspect directly in many examples, but a quick visual check can miss important structure. Two columns may be exactly dependent, so distinct coefficient vectors produce the same linear predictor. Or columns may be very similar without being exactly dependent, leaving the coefficients uniquely determined in exact arithmetic but sensitive to small changes in the data. A matrix audit separates these cases and checks which coefficient combinations the design can support.

The audit here concerns the design matrix \(X\) in the linear predictor \(\eta=X\beta\). If there are \(n\) observations and \(p\) columns, then \(X\) is an \(n\times p\) matrix, \(\beta\) is a \(p\)-vector, and \(\eta\) is an \(n\)-vector. The column rank is the number of linearly independent columns of \(X\).

Definition: A design matrix has full column rank if its column rank is \(p\). It is rank deficient if its column rank is less than \(p\). An exact column dependency is a nonzero vector \(h\) such that \(Xh=0\). The set of all such vectors, including zero, is the null space of \(X\).

A coefficient vector describes how columns contribute to \(\eta\). If \(Xh=0\), then \(X(\beta+h)=X\beta\): moving the coefficients by \(h\) leaves every linear-predictor value unchanged. That is the central implication of exact collinearity for a matrix audit.

Worked Audit: Finding an Exact Dependency

Worked Example: A Four-Column Matrix With Rank Three

Consider four observations and four columns: an intercept, \(x\), \(z\), and \(w\). The design matrix is

X= \begin{pmatrix} 1&0&1&0\\ 1&1&3&1\\ 1&2&5&0\\ 1&3&7&1 \end{pmatrix}.

Check dimensions. There are four rows and four columns, so \(X\) is \(4\times4\). The coefficient vector must be \(4\times1\), and \(X\beta\) is \(4\times1\), as required for four linear-predictor values.

Check for a dependency. The \(z\) column is the intercept column plus twice the \(x\) column:

\begin{pmatrix}1\\3\\5\\7\end{pmatrix} = \begin{pmatrix}1\\1\\1\\1\end{pmatrix} + 2\begin{pmatrix}0\\1\\2\\3\end{pmatrix}.

Thus, for \(h=(-1,-2,1,0)^{\mathsf T}\), \(Xh=0\). The columns cannot all be independent, so the rank is at most three.

Verify the rank is three. The submatrix formed from columns 1, 2, and 4 and rows 1, 2, and 4 is

\begin{pmatrix} 1&0&0\\ 1&1&1\\ 1&3&1 \end{pmatrix}, \qquad \det\begin{pmatrix} 1&0&0\\ 1&1&1\\ 1&3&1 \end{pmatrix} =1(1-3)=-2\ne0.

These three columns are independent, so \(X\) has rank at least three. Together with the dependency, this proves that its rank is exactly three. It is a \(4\times4\) matrix, but it does not have full column rank.

For any coefficient vector \(\beta\), every vector \(\beta+t h\), with \(t\) any real number, gives the same linear predictor:

X(\beta+t h)=X\beta+tXh=X\beta.

Consequently, the four coefficients are not uniquely determined by the linear predictor. This does not mean every function of the coefficients is unidentifiable; the audit of contrasts below will distinguish the combinations that are preserved.

Rank, Uniqueness, and Estimability

The first result makes the connection between rank and coefficient uniqueness precise. Here, uniqueness means that for a fixed vector \(\eta\), at most one coefficient vector \(\beta\) satisfies \(X\beta=\eta\).

Result: The coefficients are unique from \(X\beta\) exactly when \(X\) has full column rank. If \(X\) is rank deficient, there are distinct coefficient vectors with the same linear predictor.

Proof. If \(X\) has full column rank, its null space contains only zero. Suppose \(X\beta_1=X\beta_2\). Then \(X(\beta_1-\beta_2)=0\), so \(\beta_1-\beta_2=0\) and \(\beta_1=\beta_2\). Conversely, if \(X\) is rank deficient, its null space contains some \(h\ne0\). For any \(\beta\), \(X(\beta+h)=X\beta\), although \(\beta+h\ne\beta\). Thus the coefficients are not unique. \(\square\)

Often the target is not the entire coefficient vector but a linear function \(c^{\mathsf T}\beta\), where \(c\) is a specified \(p\)-vector. Call this function estimable when its value is the same for every pair of coefficient vectors that produce the same linear predictor. This definition asks what the design identifies, without requiring that individual coefficients be uniquely identified.

Theorem: A linear function \(c^{\mathsf T}\beta\) is estimable if and only if \(c\) lies in the row space of \(X\). Equivalently, it is estimable if and only if \(c^{\mathsf T}h=0\) for every \(h\) in the null space of \(X\).

Proof. Two coefficient vectors \(\beta_1\) and \(\beta_2\) produce the same predictor precisely when \(X(\beta_1-\beta_2)=0\). Thus their difference can be any vector \(h\) in the null space. The contrast has the same value for all such coefficient vectors precisely when

c^{\mathsf T}(\beta+h)-c^{\mathsf T}\beta=c^{\mathsf T}h=0 \quad\text{for every }h\text{ such that }Xh=0.

This establishes the null-space criterion. Every vector in the row space can be written as \(X^{\mathsf T}a\) for some \(a\). If \(c=X^{\mathsf T}a\) and \(Xh=0\), then \(c^{\mathsf T}h=a^{\mathsf T}Xh=0\). Hence every row-space vector satisfies the criterion. Conversely, the row space is contained in the orthogonal complement of the null space by the same calculation. Both spaces have dimension \(\operatorname{rank}(X)\): the row space has that dimension by definition of rank, and the null space has dimension \(p-\operatorname{rank}(X)\) by the rank–nullity theorem, so its orthogonal complement has dimension \(\operatorname{rank}(X)\). The containment is therefore equality. This proves the row-space criterion. \(\square\)

Worked Audit: Which Contrasts Survive?

Worked Example: Testing Coefficient Combinations

Use the matrix from the preceding audit, with coefficients \(\beta=(\beta_0,\beta_x,\beta_z,\beta_w)^{\mathsf T}\) and null-space vector \(h=(-1,-2,1,0)^{\mathsf T}\). The theorem says that \(c^{\mathsf T}\beta\) is estimable exactly when \(c^{\mathsf T}h=0\).

Is \(\beta_z\) estimable? Its contrast vector is \(c=(0,0,1,0)^{\mathsf T}\), and \(c^{\mathsf T}h=1\). It is not zero. Indeed, moving from \(\beta\) to \(\beta+t h\) changes \(\beta_z\) by \(t\) while leaving \(X\beta\) unchanged.

Is \(\beta_0+\beta_z\) estimable? Here \(c=(1,0,1,0)^{\mathsf T}\), giving \(c^{\mathsf T}h=-1+1=0\). The contrast is estimable. One direct row-space check is that the first row of \(X\), transposed, is \((1,0,1,0)^{\mathsf T}\); therefore \(c=X^{\mathsf T}a\) for \(a=(1,0,0,0)^{\mathsf T}\).

Is \(\beta_x+2\beta_z\) estimable? Its vector is \(c=(0,1,2,0)^{\mathsf T}\), and \(c^{\mathsf T}h=-2+2=0\). It is estimable, even though neither \(\beta_x\) nor \(\beta_z\) is individually estimable. The design identifies certain combinations, not the separate contributions assigned to exactly dependent columns.

Near Collinearity Is Not Exact Collinearity

Exact dependency is an algebraic statement: a nonzero vector \(h\) satisfies \(Xh=0\). Near collinearity describes columns that are close to satisfying such a relation, but no universal numerical cutoff turns “near” into a mathematical category. Its practical assessment depends on the scale of the columns and on the precision of the computation. A small determinant alone is not a scale-free measure of near collinearity.

Worked Example: Full Rank With Nearly Dependent Columns

Let \(\delta=0.01\), and take three columns: an intercept, \(x\), and \(z\), where \(x=(0,1,2)^{\mathsf T}\) and \(z=x+\delta v\) for \(v=(0,0,1)^{\mathsf T}\). Then

X= \begin{pmatrix} 1&0&0\\ 1&1&1\\ 1&2&2.01 \end{pmatrix}, \qquad \det(X)=1(2.01-2)=0.01\ne0.

The nonzero determinant proves that this square matrix has rank three: the columns are not exactly dependent. Yet \(z-x=(0,0,0.01)^{\mathsf T}\), a small difference relative to the entries in the two columns.

The coefficient implications can be seen directly. Write the fitted vector as \(\alpha\mathbf{1}+a x+b z\). Since \(z=x+\delta v\),

\alpha\mathbf{1}+a x+b z =\alpha\mathbf{1}+(a+b)x+b\delta v.

To represent the response vector \(y=v\), set \(\alpha=0\), \(a+b=0\), and \(b\delta=1\). With \(\delta=0.01\), these equations give \(b=100\) and \(a=-100\). The matrix has a unique coefficient vector, but representing this response requires two large, opposing coefficients on similar columns. This illustrates why near collinearity can make coefficient values sensitive: uniqueness in exact arithmetic does not guarantee numerical stability.

A Practical Matrix-Audit Sequence

A useful audit proceeds from bookkeeping to structure, then to the inferential target. The calculations should be based on the actual columns supplied to the model, including any intercept or constructed predictor columns.

Check dimensions.
Count the observations and columns. Confirm that \(X\) is \(n\times p\), that \(\beta\) has \(p\) entries, and that \(X\beta\) has \(n\) entries.

Inspect exact dependencies.
Compare columns and test any suspected relation. A verified nonzero \(h\) with \(Xh=0\) proves rank deficiency.

Determine rank.
For a small matrix, find independent columns or a nonzero square minor of the relevant size. For larger matrices, use a rank-revealing numerical method and report the tolerance used.

Check the requested target.
For a contrast vector \(c\), test whether it is orthogonal to every null-space vector, or equivalently whether it lies in the row space.

Common Mistake: Treating Full Rank as a Guarantee of Stability

Wrong reasoning: “The determinant is nonzero, so the coefficients are unique; therefore the coefficients are reliable.” The determinant establishes nonsingularity for a square matrix, not numerical stability. In the near-collinearity example, the determinant is nonzero and the coefficients are unique, yet a response of modest size requires coefficients \(100\) and \(-100\) on the similar columns.

Correction: Separate two questions. Exact rank determines uniqueness and exact estimability; near dependencies concern sensitivity. For numerical work, inspect a suitable rank-revealing decomposition or condition measure while accounting for column scaling. When exact rank deficiency is present, state which coefficients are not individually identified and check whether the scientific contrast of interest is nevertheless estimable.

Key takeaway: A sound audit reports the matrix dimensions, its rank, any exact dependencies, and the estimability of the target coefficient combination. Full rank guarantees uniqueness, not stability; rank deficiency rules out unique coefficients, but does not rule out every estimable contrast.

Check Your Understanding

Use the definitions and criteria above to answer these questions.

If a design matrix has six rows and four columns, what are the dimensions of \(\beta\) and \(X\beta\)?
For the four-column matrix in the first worked example, verify directly that \(X(-1,-2,1,0)^{\mathsf T}=0\).
Why does a nonzero null-space vector imply that the coefficient vector is not unique?
For that same matrix, is \(\beta_0+\beta_z\) estimable? Give a null-space or row-space justification.
How can a matrix have full column rank but still have coefficients sensitive to small changes?

===