Coefficient Identifiability and the Null Space

Why Different Coefficients Can Describe the Same Model

A coefficient vector is meaningful through the linear predictor it produces. For a fixed design matrix \(X\), the GLM linear predictor is \(\eta=X\beta\). If two different coefficient vectors produce the same \(\eta\), then the observed design cannot distinguish those coefficient vectors through the linear predictor. This is exact nonidentifiability, not merely a difficulty caused by noisy data or an unstable numerical calculation.

The key object is the null space of \(X\): the set of coefficient directions that \(X\) maps to zero. Adding any such direction to a coefficient vector leaves every entry of the linear predictor unchanged. This gives a direct algebraic description of all coefficient vectors that fit the same linear predictor.

Definition: For an \(n\times p\) design matrix \(X\), its null space is \(\operatorname{Null}(X)=\{v\in\mathbb{R}^p:Xv=0\}\). A coefficient vector \(\beta\in\mathbb{R}^p\) is identifiable from the linear predictor for this design if no other coefficient vector produces the same \(X\beta\). If distinct coefficient vectors do produce the same linear predictor, the coefficients are nonidentifiable for this design.

The null-space vectors are directions in coefficient space, not vectors of fitted values. A nonzero \(v\in\operatorname{Null}(X)\) may change several coefficients at once, but its effects cancel across the columns of \(X\) in every row. Thus it changes the numerical description \(\beta\) without changing the fitted linear predictor.

The Null-Space Characterization

The central equivalence is simple, but important: two coefficient vectors give the same linear predictor exactly when their difference lies in the null space. The result applies to any real matrix \(X\), so it does not depend on the response distribution.

Theorem: For \(\beta,\widetilde{\beta}\in\mathbb{R}^p\), \(X\beta=X\widetilde{\beta}\) if and only if \(\widetilde{\beta}-\beta\in\operatorname{Null}(X)\). Consequently, the full set of coefficient vectors producing \(X\beta\) is \(\{\beta+v:v\in\operatorname{Null}(X)\}\).

Proof. By linearity of matrix multiplication, \(X\beta=X\widetilde{\beta}\) if and only if \(X\widetilde{\beta}-X\beta=0\). The left-hand side is \(X(\widetilde{\beta}-\beta)\), so this equality holds if and only if \(\widetilde{\beta}-\beta\in\operatorname{Null}(X)\). Rearranging gives \(\widetilde{\beta}=\beta+v\) for some \(v\in\operatorname{Null}(X)\). Conversely, for any such \(v\), \(X(\beta+v)=X\beta+Xv=X\beta\). This proves both claims. \(\square\)

A coefficient vector is unique for a given linear predictor precisely when the null space contains only the zero vector. In matrix terms, this is the full-column-rank condition.

Theorem: The map \(\beta\mapsto X\beta\) is one-to-one if and only if \(\operatorname{Null}(X)=\{0\}\), which holds if and only if \(X\) has full column rank \(p\).

Proof. If \(\operatorname{Null}(X)=\{0\}\) and \(X\beta=X\widetilde{\beta}\), the preceding theorem gives \(\widetilde{\beta}-\beta=0\), hence \(\widetilde{\beta}=\beta\). So the map is one-to-one. If the map is one-to-one, then \(Xv=0=X0\) implies \(v=0\); therefore the null space contains only zero. Finally, the columns of \(X\) are linearly independent exactly when the only coefficients \(v\) satisfying \(Xv=0\) are all zero. Independent columns are equivalent to rank \(p\), or full column rank. \(\square\)

Worked Examples

Worked Example: Two Identical Predictor Columns

Suppose a design matrix has two identical columns:

X= \begin{pmatrix} 1&1\\ 2&2\\ 3&3 \end{pmatrix}.

For \(\beta=(\beta_1,\beta_2)^\mathsf{T}\), multiplication gives \(X\beta=(\beta_1+\beta_2,\,2\beta_1+2\beta_2,\,3\beta_1+3\beta_2)^\mathsf{T}\). Only the sum of the coefficients appears. To find a null-space direction, solve \(Xv=0\). Writing \(v=(v_1,v_2)^\mathsf{T}\), the first row requires \(v_1+v_2=0\), and the other rows impose the same equation. Thus \(v_2=-v_1\), and \(\operatorname{Null}(X)=\{t(1,-1)^\mathsf{T}:t\in\mathbb{R}\}\).

For example, \(\beta=(2,1)^\mathsf{T}\) and \(\widetilde{\beta}=(3,0)^\mathsf{T}\) differ by \((1,-1)^\mathsf{T}\), a null-space vector. Directly, \(X\beta=(3,6,9)^\mathsf{T}=X\widetilde{\beta}\). The data design identifies the sum \(3\) as the predictor contribution, but the two separate coefficients are not uniquely determined.

Worked Example: An Intercept and Two Exhaustive Group Indicators

Consider three observations, with the first in group A and the other two in group B. Use an intercept and indicators for both groups:

X= \begin{pmatrix} 1&1&0\\ 1&0&1\\ 1&0&1 \end{pmatrix}.

The first column is the intercept. In every row, the group-A indicator plus the group-B indicator equals the intercept column. Therefore, the coefficient vector \(v=(-1,1,1)^\mathsf{T}\) satisfies \(Xv=(-1+1,\,-1+1,\,-1+1)^\mathsf{T}=0\).

Take \(\beta=(5,2,4)^\mathsf{T}\). Its predictor is \(X\beta=(5+2,\,5+4,\,5+4)^\mathsf{T}=(7,9,9)^\mathsf{T}\). For any real \(t\), let \(\widetilde{\beta}=\beta+t(-1,1,1)^\mathsf{T}\). Then \(X\widetilde{\beta}=X\beta+tX(-1,1,1)^\mathsf{T}=X\beta\). For instance, at \(t=1\), \(\widetilde{\beta}=(4,3,5)^\mathsf{T}\), which also gives \((4+3,\,4+5,\,4+5)^\mathsf{T}=(7,9,9)^\mathsf{T}\). The group-specific linear predictors remain the same, although all three coefficients have changed.

Worked Example: A Third Column That Is the Sum of Two Others

Let

X= \begin{pmatrix} 1&0&1\\ 0&1&1\\ 1&1&2 \end{pmatrix}.

The third column is the sum of the first and second columns. Accordingly, \(v=(-1,-1,1)^\mathsf{T}\) is in the null space: \(Xv=(-1+1,\,-1+1,\,-1-1+2)^\mathsf{T}=(0,0,0)^\mathsf{T}\). Let \(\beta=(1,2,3)^\mathsf{T}\). Row-by-row multiplication gives \(X\beta=(1+3,\,2+3,\,1+2+2(3))^\mathsf{T}=(4,5,9)^\mathsf{T}\).

Adding the null vector gives \(\widetilde{\beta}=(0,1,4)^\mathsf{T}\). Its predictor is \(X\widetilde{\beta}=(0+4,\,1+4,\,0+1+2(4))^\mathsf{T}=(4,5,9)^\mathsf{T}\), the same result. The coefficient for the third column can be increased while both other coefficients are decreased by the same amount, with no change in the predictor.

What This Means for a GLM

In a GLM with a fixed design matrix, the response distribution is specified through the model's linear predictor and the chosen response-family and link structure. When two coefficient vectors give exactly the same \(\eta\), they give the same modeled response distribution, provided that the other model components are held fixed. In particular, their likelihoods are equal for every possible observed response:

X\widetilde{\beta}=X\beta \quad\Longrightarrow\quad L(\widetilde{\beta})=L(\beta).

This follows because the distribution used for each observation is unchanged when its linear predictor is unchanged; hence the joint distribution, and its likelihood at any data value, are unchanged. If a nonzero null-space vector exists, moving along it creates different coefficient vectors with the same predictor and likelihood. The likelihood therefore cannot select a unique coefficient vector on the basis of these data and this design alone.

This is coefficient nonidentifiability conditional on the specified design. It does not mean that the fitted linear predictors must be ambiguous: as the examples show, every coefficient vector in the same null-space-shifted set gives the same predictor. Nor does it mean that the numerical entries in a reported coefficient vector are meaningless under every possible convention. A fitting procedure may impose a constraint or choose one representative, but that choice does not make the original unrestricted coefficients uniquely identified by the design.

Exact Nonidentifiability Versus Near Collinearity

Exact nonidentifiability occurs when a nonzero vector \(v\) satisfies \(Xv=0\). Near collinearity is different: a direction may produce a small, but nonzero, change \(Xv\). In that case the coefficients can be highly sensitive to data or numerical perturbations, but the exact null-space argument does not apply. A useful diagnostic question is therefore: can the proposed coefficient change leave every predictor entry exactly unchanged, or does it merely change them by small amounts?

Write the proposed coefficient change.
Represent the change as a vector \(v\in\mathbb{R}^p\), with one entry for each coefficient.

Multiply by the design matrix.
Calculate \(Xv\), checking every row or using the column relation.

Interpret the result.
If \(v\ne0\) and \(Xv=0\), then \(\beta\) and \(\beta+v\) yield identical linear predictors. If \(Xv\ne0\), the change is not an exact nonidentifiability direction.

Common Mistakes

Confusing a zero predictor with a null-space vector. A null-space vector is a coefficient-space vector \(v\) for which \(Xv=0\). It is not itself a vector of fitted predictors.
Thinking that different coefficients must imply different fitted models. That is false whenever their difference lies in \(\operatorname{Null}(X)\). The correct check is to compare \(X\beta\) and \(X\widetilde{\beta}\).
Calling a small change an exact null-space direction. If \(Xv\) is small but not zero, the predictor changes. That may signal instability, but it is not exact coefficient nonidentifiability.
Assuming a software-reported coefficient is uniquely determined by the design. A reported value can reflect a computational convention or constraint. Check whether \(X\) has full column rank before interpreting the coefficients as unique.

Key takeaway: For a fixed design matrix, all coefficients producing a given linear predictor are exactly \(\beta+\operatorname{Null}(X)\). A nonzero null-space direction proves that distinct coefficient vectors have identical predictors; a trivial null space is equivalent to unique coefficients.

Check Your Understanding

Use the null-space characterization to answer these questions.

For \(X=\begin{pmatrix}1&2\\2&4\end{pmatrix}\), find a nonzero vector in \(\operatorname{Null}(X)\) and verify the multiplication.
If \(Xv=0\), explain why \(\beta\) and \(\beta+v\) produce the same linear predictor.
What condition on the columns of \(X\) guarantees that each linear predictor has a unique coefficient vector?
In a GLM with all other model components fixed, why do two coefficient vectors with the same linear predictor have the same likelihood?
How does exact nonidentifiability differ from near collinearity?