Why Exact Collinearity Makes Coefficients Unidentifiable

Why Exact Dependence Changes What Can Be Estimated

A linear regression model can describe the observed responses perfectly well even when it cannot assign a unique value to every coefficient. The issue is not necessarily a failure of least squares: if two predictor columns have an exact linear relationship, different coefficient vectors can produce exactly the same fitted response for every observation. The data then determine certain combinations of coefficients, but not the individual coefficients in that redundant parameterization.

The distinction matters when interpreting a model. A reported fitted value may be uniquely determined even though a coefficient table cannot give uniquely defined estimates for every predictor. The key tools are the rank and null space of the design matrix.

Definition: In a linear regression with \(n\) observations and \(p\) coefficients, write \(y=X\beta+\varepsilon\), where \(y\) is the \(n\)-vector of responses, \(X\) is the \(n\times p\) design matrix, \(\beta\) is the \(p\)-vector of coefficients, and \(\varepsilon\) is the error vector. The rank of \(X\) is the number of linearly independent columns of \(X\). The null space of \(X\) is the set of vectors \(h\) satisfying \(Xh=0\). Exact collinearity means that the columns of \(X\) are linearly dependent, so \(X\) has a nonzero null-space vector.

The first column of \(X\) is often a column of ones for the intercept. A dependence may involve that column as well as predictor columns. This distinction determines which combinations of intercept and slopes are identified.

Least Squares and the Rank Condition

For a fixed design matrix \(X\), ordinary least squares chooses a coefficient vector to minimize the residual sum of squares:

\operatorname{RSS}(\beta)=\|y-X\beta\|^2.

If \(X\) has full column rank, then \(X^{\mathsf T}X\) is invertible, and the unique least-squares coefficient vector is \(\hat{\beta}=(X^{\mathsf T}X)^{-1}X^{\mathsf T}y\). This formula requires full column rank. Exact collinearity makes \(X^{\mathsf T}X\) singular, so its ordinary inverse does not exist.

The least-squares minimization itself still has a solution. It is the coefficient representation of the closest vector to \(y\) in the column space of \(X\). If that representation is redundant, multiple coefficient vectors can describe that same closest vector.

Theorem: If \(h\ne 0\) and \(Xh=0\), then for any coefficient vector \(\beta\) and any scalar \(t\), \(X(\beta+th)=X\beta\). Consequently, \(\beta\) and \(\beta+th\) have identical fitted values and identical residual sums of squares. If a least-squares solution exists, adding any null-space vector to it gives another least-squares solution.

Why Fitted Values Can Be Unique When Coefficients Are Not

The least-squares fitted vector is the projection of \(y\) onto the column space of \(X\). That projected vector is unique, even if there are multiple coefficient vectors that represent it. The following argument explains why. Let \(\hat{y}\) be the closest vector to \(y\) in the column space, and let \(r=y-\hat{y}\). At the least-squares solution, \(r\) is perpendicular to every vector in that column space. For any other vector \(z\) in the column space, \(z-\hat{y}\) is also in the column space, so it is perpendicular to \(r\). Therefore:

\|y-z\|^2 =\|r-(z-\hat{y})\|^2 =\|r\|^2+\|z-\hat{y}\|^2.

The second term is nonnegative and equals zero only when \(z=\hat{y}\). Thus no different fitted vector can attain the same minimum residual sum of squares. The coefficient vectors that produce \(\hat{y}\), however, need not be unique.

This uniqueness concerns the fitted responses at the rows used to form \(X\). A prediction at a new predictor combination is uniquely determined from the fitted data only if its design row gives the same linear prediction for every coefficient vector that fits the observed data. In null-space terms, a new design row \(x_{\mathrm{new}}^{\mathsf T}\) must satisfy \(x_{\mathrm{new}}^{\mathsf T}h=0\) for every \(h\) in the null space of \(X\).

Which Coefficient Combinations Are Identifiable?

Suppose two coefficient vectors give the same fitted vector. Their difference \(h\) satisfies \(Xh=0\). A linear combination \(c^{\mathsf T}\beta\) is identifiable from the design if it has the same value for all coefficient vectors that produce those fitted values. This requires \(c^{\mathsf T}h=0\) for every null-space vector \(h\). Equivalently, \(c\) must belong to the row space of \(X\), the space spanned by the rows of \(X\).

This criterion is a practical way to read exact dependence: an individual coefficient may vary across equivalent solutions, while particular sums or contrasts remain fixed. The identifiable quantities depend on the actual column relation, including whether the intercept participates.

Find the dependence.
Express one or more design columns as a linear combination of the others, or find a nonzero vector \(h\) with \(Xh=0\).

Translate it to coefficients.
Use \(X(\beta+th)=X\beta\) to describe how the coefficients can change without changing any fitted value.

State what is determined.
A combination \(c^{\mathsf T}\beta\) remains identified exactly when \(c^{\mathsf T}h=0\) for every null-space direction.

Worked Example: A Redundant Slope With No Intercept Dependence

A model includes an intercept and two predictors \(x\) and \(w\), with observed rows \(x=(0,1,2)\) and \(w=(0,2,4)\). The responses are \(y=(1,4,7)\). Find the least-squares fitted vector and describe all coefficient vectors that attain it.

Method. Write the design columns and identify a null-space vector. This directly reveals which slope combination can be determined. No probability assumption is needed for this algebraic least-squares calculation.

The observed response is exactly \(1+3x\), so the fitted vector \((1,4,7)\) has zero residual sum of squares. Since \(w=2x\), the design matrix satisfies

X= \begin{pmatrix} 1&0&0\\ 1&1&2\\ 1&2&4 \end{pmatrix}, \qquad X \begin{pmatrix} 0\\-2\\1 \end{pmatrix} = \begin{pmatrix}0\\0\\0\end{pmatrix}.

Thus \(h=(0,-2,1)^{\mathsf T}\) is a nonzero null-space vector. One exact-fit coefficient vector is \((\beta_0,\beta_1,\beta_2)=(1,3,0)\). Adding \(t h\) gives every vector in this one-dimensional family:

(\beta_0,\beta_1,\beta_2)=(1,\ 3-2t,\ t),\qquad -\infty<t<\infty.

For every \(t\), the intercept remains \(1\), and \(\beta_1+2\beta_2=(3-2t)+2t=3\). Therefore the slope of the fitted response as a function of \(x\) is determined, but neither \(\beta_1\) nor \(\beta_2\) is individually determined. The fitted values remain \((1,4,7)\) for all \(t\). The common mistake avoided here is to interpret one convenient solution, such as \((1,3,0)\), as the unique coefficient estimate.

Worked Example: When the Intercept Participates in the Dependence

A different design has an intercept and predictors \(x,w\), with rows \(x=(0,1,2)\) and \(w=(3,4,5)\). The responses are \(y=(2,4,6)\). Find the coefficient combinations determined by the observations and give two distinct coefficient vectors with the same fitted values.

Method. Express the dependent column in terms of the intercept column and \(x\), then rewrite the fitted equation in terms of the identifiable combinations. This avoids treating an intercept-involving dependence as if it affected slopes only.

Here \(w=x+3\). For any row, the model's fitted response is

\beta_0+\beta_1x+\beta_2w =(\beta_0+3\beta_2)+(\beta_1+\beta_2)x.

The data follow \(y=2+2x\) at \(x=0,1,2\), giving fitted values \((2,4,6)\) and zero residual sum of squares. Thus the combinations \(\beta_0+3\beta_2=2\) and \(\beta_1+\beta_2=2\) are determined. Taking \(\beta_2=t\) yields

(\beta_0,\beta_1,\beta_2)=(2-3t,\ 2-t,\ t).

For \(t=0\), one solution is \((2,2,0)\). For \(t=1\), another is \((-1,1,1)\). Checking the second vector gives \(-1+x+(x+3)=2+2x\), so it produces exactly the same fitted values. Unlike the preceding example with \(w=2x\), this dependence involves the intercept column: both an intercept-like combination and a slope-like combination must be considered.

Worked Example: Nonzero Residuals but a Unique Fitted Vector

Use \(x=(0,1,2)\), \(w=(0,2,4)\), and responses \(y=(1,4,8)\) in a model with an intercept and both predictors. Find the least-squares fitted vector and describe the coefficient ambiguity.

Method. Since \(w=2x\), fit the response on an intercept and \(x\), then translate its slope to the two-predictor coefficient form. This demonstrates that coefficient nonuniqueness does not require a perfect fit.

The means are \(\bar{x}=1\) and \(\bar{y}=(1+4+8)/3=13/3\). The centered cross-product and sum of squares are

\sum_{i=1}^3(x_i-\bar{x})(y_i-\bar{y}) =(-1)(-10/3)+0(-1/3)+(1)(11/3)=7, \qquad \sum_{i=1}^3(x_i-\bar{x})^2=2.

The least-squares slope for \(x\) is \(7/2\), and the intercept is \(13/3-7/2=(26-21)/6=5/6\). Hence \(\hat{y}=5/6+(7/2)x\), giving fitted values \((5/6,13/3,47/6)\). The residuals are \(1/6,-1/3,1/6\), which sum to zero; their squared sum is \(1/36+1/9+1/36=1/6\). The nonzero residual sum of squares confirms that this is not an exact fit.

Because \(w=2x\), every coefficient vector with \(\beta_0=5/6\) and \(\beta_1+2\beta_2=7/2\) gives these same fitted values. For example, choosing \(\beta_2=0\) gives \((5/6,7/2,0)\), while choosing \(\beta_2=1\) gives \((5/6,3/2,1)\). The common mistake avoided is to infer that a nonzero residual sum of squares removes the coefficient ambiguity; it does not.

Common Exam Trap: Trying to Invert a Singular Matrix

A frequent error is to apply the full-rank formula \((X^{\mathsf T}X)^{-1}X^{\mathsf T}y\) after discovering exact collinearity. For the design with an intercept and \(w=2x\), the second predictor column is twice the first slope column. The columns are dependent, so \(X\) is not full column rank, and \(X^{\mathsf T}X\) is singular. Its ordinary inverse does not exist. One cannot use that formula to select unique individual coefficients.

A related interpretation error is to find one valid coefficient vector and call it the uniquely estimated vector. In the intercept-dependent example, \((2,2,0)\) is valid, but so is \((-1,1,1)\). Both produce the same fitted values because the observations satisfy \(w=x+3\). A chosen computational convention may select one solution, but that selection does not make the individual coefficients identifiable from the design.

Key takeaway: Exact collinearity creates a nonzero null space, allowing coefficients to change along null-space directions without changing the fitted vector. State the identifiable coefficient combinations from the actual column dependence; do not confuse a unique fitted vector with unique individual coefficients. This treatment is aligned with the SOA Exam SRM (Statistics for Risk Modeling) syllabus.

Check Your Understanding

For each problem, identify the rank or coefficient ambiguity and state exactly what is determined.

A model has an intercept and predictors \(x,w\), with \(w=3x\) in every observed row. Which slope combination is identifiable, and what null-space direction describes the coefficient ambiguity? (Answer: \(\beta_1+3\beta_2\) is identifiable; one null-space direction is \((0,-3,1)^{\mathsf T}\).)
In a design with intercept and predictors \(x,w\), suppose \(w=x+4\) for every row. Which two combinations can determine the fitted equation as a function of \(x\)? (Answer: \(\beta_0+4\beta_2\) and \(\beta_1+\beta_2\).)
Suppose \(X\) is rank deficient and \(\hat{\beta}\) is a least-squares solution. If \(Xh=0\), show what happens to the fitted vector and residual sum of squares for \(\hat{\beta}+5h\). (Answer: Both are unchanged, since \(X(\hat{\beta}+5h)=X\hat{\beta}+5Xh=X\hat{\beta}\).)
For \(w=x+2\), a fitted equation is \(3+5x\). Give a coefficient vector with \(\beta_2=2\), and verify it represents the same fitted equation. (Answer: \((\beta_0,\beta_1,\beta_2)=(-1,3,2)\); \(-1+3x+2(x+2)=3+5x\).)
A design has \(n=5\) observations and four columns, and its rank is three. How many independent null-space directions are there, and can the least-squares fitted vector still be unique? (Answer: There is one independent null-space direction; yes, the least-squares fitted vector is unique even though the coefficient vector is not.)

What You'll Learn