Tutorials › Generalized Linear Models › Column Rank and Uniqueness of Coefficients

Home  /  Tutorials  /  Design Matrices and Predictors

Design Matrices and Predictors · Tutorial 88 of 981

Column Rank and Uniqueness of Coefficients

Use column independence to determine when a linear predictor uniquely determines its coefficient vector, and check the criterion on small matrices.

Beginner 10 min read

What You'll Learn

  • Define full column rank for an \(n \times p\) design matrix
  • Prove the connection between column independence and coefficient uniqueness
  • Distinguish uniqueness of a solution from existence of a solution
  • Solve small systems of the form \(X\beta=\eta\)
  • Diagnose when a rank-deficient matrix gives multiple coefficient vectors

Why Column Rank Matters

In a generalized linear model, the linear predictor is written as \(\eta=X\beta\). Here \(X\) is a fixed design matrix with \(n\) rows, one for each observation, and \(p\) columns, one for each coefficient; \(\beta\) is a \(p\)-component coefficient vector, and \(\eta\) is an \(n\)-component vector of linear-predictor values. The columns of \(X\) are vectors in \(\mathbb{R}^n\).

A basic question is whether the predictor values determine the coefficients. If two different coefficient vectors produce the same \(\eta\), then the mapping from coefficients to predictors does not have a unique inverse. The relevant property of \(X\) is whether its columns are linearly independent, summarized by its column rank.

Definition: The column rank of \(X\) is the maximum number of linearly independent columns in \(X\). If \(X\) has \(p\) columns and its column rank is \(p\), it has full column rank. Equivalently, its columns are linearly independent: the only coefficients \(c_1,\ldots,c_p\) satisfying \(c_1x_1+\cdots+c_px_p=0\) are \(c_1=\cdots=c_p=0\), where \(x_j\) denotes column \(j\) of \(X\).

The Uniqueness Criterion

First separate two questions. Existence asks whether there is any coefficient vector \(\beta\) satisfying \(X\beta=\eta\). Uniqueness asks whether there is exactly one such vector. A particular \(\eta\) has a solution precisely when it is a linear combination of the columns of \(X\), that is, when it belongs to the column space of \(X\). Full column rank guarantees uniqueness if a solution exists; it does not guarantee that every possible \(\eta\) has a solution.

Lemma: Two coefficient vectors \(\beta\) and \(\gamma\) give the same linear predictor if and only if their difference gives a zero linear combination of the columns of \(X\).

Proof. Write \(x_j\) for column \(j\), and write \(\beta_j\) and \(\gamma_j\) for the corresponding coefficients. By the definition of matrix multiplication,

$$ X\beta=\sum_{j=1}^{p}\beta_jx_j \qquad\text{and}\qquad X\gamma=\sum_{j=1}^{p}\gamma_jx_j. $$

Subtracting these vectors gives

$$ X\beta-X\gamma =\sum_{j=1}^{p}(\beta_j-\gamma_j)x_j =X(\beta-\gamma). $$

Therefore \(X\beta=X\gamma\) exactly when \(X(\beta-\gamma)=0\), which is exactly a zero linear combination of the columns, with coefficients \(\beta_j-\gamma_j\). \(\square\)

Theorem: For a fixed \(\eta\), the equation \(X\beta=\eta\) has exactly one solution if and only if \(\eta\) belongs to the column space of \(X\) and \(X\) has full column rank. In particular, \(X\) has full column rank if and only if every attainable linear predictor has exactly one coefficient vector.

Proof. Suppose first that \(\eta\) belongs to the column space of \(X\), so at least one solution \(\beta\) exists, and that \(X\) has full column rank. If \(\gamma\) is another solution, then \(X\beta=X\gamma\). By the lemma, \(\beta-\gamma\) gives a zero linear combination of the columns. Full column rank means the only such coefficients are all zero, so \(\beta-\gamma=0\) and \(\beta=\gamma\). Thus the solution is unique.

Conversely, suppose \(X\) does not have full column rank. Then its columns are linearly dependent, so there is a vector \(c\ne 0\) such that \(Xc=0\). For any attainable \(\eta\), take a solution \(\beta\). For every real number \(t\),

$$ X(\beta+tc)=X\beta+tXc=\eta+t\,0=\eta. $$

Since \(c\ne 0\), the vectors \(\beta+tc\) are distinct for distinct \(t\). Thus every attainable \(\eta\) has more than one solution when the columns are dependent. Finally, if \(\eta\) is not in the column space, it has no solution, regardless of rank. These facts establish both claims. \(\square\)

Worked Matrix Examples

Worked Example: A Full-Column-Rank Matrix Gives Unique Coefficients

Let

$$ X= \begin{pmatrix} 1&0\\ 1&1\\ 1&2 \end{pmatrix}, \qquad \eta= \begin{pmatrix} 2\\ 5\\ 8 \end{pmatrix}. $$

The two columns are \(x_1=(1,1,1)^\mathsf{T}\) and \(x_2=(0,1,2)^\mathsf{T}\). To check independence, suppose \(a x_1+b x_2=0\). The first component gives \(a=0\); the second then gives \(b=0\). The columns are independent, so \(X\) has full column rank.

Now solve \(X\beta=\eta\), writing \(\beta=(\beta_1,\beta_2)^\mathsf{T}\). The equations are

$$ \beta_1=2,\qquad \beta_1+\beta_2=5,\qquad \beta_1+2\beta_2=8. $$

The first equation gives \(\beta_1=2\), and the second gives \(\beta_2=3\). The third checks: \(2+2(3)=8\). Hence \(\beta=(2,3)^\mathsf{T}\) is a solution, and full column rank proves that it is the only one.

Worked Example: A Rank-Deficient Matrix Gives Multiple Coefficients

Consider

$$ X= \begin{pmatrix} 1&2\\ 2&4\\ 0&0 \end{pmatrix}, \qquad \eta= \begin{pmatrix} 3\\ 6\\ 0 \end{pmatrix}. $$

The second column is twice the first, so the two columns are dependent and the column rank is \(1\), not \(2\). The equation \(X\beta=\eta\) reduces to

$$ \beta_1+2\beta_2=3, \qquad 2\beta_1+4\beta_2=6, \qquad 0=0. $$

The second equation is twice the first and adds no new restriction. Set \(\beta_2=t\), where \(t\) can be any real number. Then \(\beta_1=3-2t\), so every vector

$$ \beta= \begin{pmatrix} 3-2t\\ t \end{pmatrix} \quad\text{satisfies}\quad X\beta= \begin{pmatrix} (3-2t)+2t\\ 2(3-2t)+4t\\ 0 \end{pmatrix} = \begin{pmatrix} 3\\ 6\\ 0 \end{pmatrix}. $$

For example, \(t=0\) gives \((3,0)^\mathsf{T}\), while \(t=1\) gives \((1,1)^\mathsf{T}\). Both produce the same \(\eta\), so the coefficients are not unique.

Worked Example: Full Rank Does Not Guarantee a Solution for Every Target

Use the full-column-rank matrix from the first example, but now take \(\eta=(0,1,0)^\mathsf{T}\). The first row of \(X\beta=\eta\) requires \(\beta_1=0\). The second requires \(\beta_1+\beta_2=1\), so \(\beta_2=1\). The third row would then require

$$ \beta_1+2\beta_2=0+2(1)=2, $$

but the target’s third component is \(0\). The equations are inconsistent, so no coefficient vector produces this \(\eta\). The matrix still has full column rank; it ensures that any solution is unique, not that every \(n\)-component target can be reached.

Practical Interpretation and a Common Mistake

For an \(n\times p\) design matrix to have full column rank, its \(p\) columns must be independent vectors in \(\mathbb{R}^n\). In particular, \(p\le n\) is necessary, since there cannot be more than \(n\) independent vectors in \(\mathbb{R}^n\). This condition alone is not sufficient: a matrix with \(p\le n\) can still have dependent columns.

The rank test concerns the design matrix and therefore the mapping from coefficients to linear predictors. It does not depend on a particular response vector or on whether the coefficients have already been estimated. When the matrix has full column rank, two distinct coefficient vectors cannot yield the same vector of predictor values. When it does not, every predictor vector that the matrix can produce is produced by multiple coefficient vectors.

Common mistake: “The matrix has full column rank, so \(X\beta=\eta\) has exactly one solution for every \(\eta\).” This confuses uniqueness with existence. Full column rank rules out two different solutions, but the target \(\eta\) must still be in the column space of \(X\). For a target outside that space there is no solution at all. The correct statement is: full column rank gives at most one solution; if a solution exists, it is unique.

A second practical mistake is to count rows or columns and infer rank from the matrix dimensions alone. Dimensions give only the bound \(\operatorname{rank}(X)\le \min(n,p)\). To establish full column rank, one must verify that all \(p\) columns are independent, for example by checking the equations for a zero linear combination or by using row reduction.

Key takeaway: Coefficient uniqueness is a property of the design matrix’s columns. For a specified \(\eta\), there is exactly one \(\beta\) precisely when \(\eta\) is attainable and the columns of \(X\) are linearly independent.

Check Your Understanding

Use the distinction between rank and existence to answer these questions.

  1. What does full column rank mean for an \(n\times p\) design matrix?
  2. If \(X\) has full column rank and \(X\beta=\eta\) has a solution, why can there not be a second solution?
  3. If the columns of \(X\) are dependent and one coefficient vector produces \(\eta\), what can be said about the number of coefficient vectors that produce it?
  4. Can a matrix with full column rank fail to produce a particular target vector \(\eta\)? Explain.
  5. Why does \(p\le n\) not by itself prove that \(X\) has full column rank?

Previous

Next

Back to Tutorials
Clinical Biostats