Exact Collinearity and Nonidentifiable Coefficients

When Predictors Contain Exactly the Same Information

A multiple regression model tries to describe an outcome using several predictors. If two predictors contain exactly redundant information, the data may still determine fitted outcomes, but they cannot determine a unique contribution for each predictor. This is not simply a matter of estimates being imprecise: the coefficient values themselves may not be identifiable from the model and data.

The issue arises when one predictor column is an exact linear combination of other predictor columns in the observed data. A duplicated variable is one example. Another is a predictor constructed as the sum of two others. Recognizing these cases lets us separate what the regression can determine from what it cannot.

Model and Definition

For \(n\) observations and \(p\) predictors, write the multiple linear regression model as

Y_i=\beta_0+\beta_1x_{i1}+\cdots+\beta_px_{ip}+\varepsilon_i,\qquad i=1,\ldots,n.

Here \(Y_i\) is the outcome for observation \(i\), \(x_{ij}\) is its value on predictor \(j\), \(\beta_0\) is the intercept, and \(\beta_j\) is the coefficient for predictor \(j\). The model for the conditional mean is \(E(Y_i\mid x_{i1},\ldots,x_{ip})=\beta_0+\beta_1x_{i1}+\cdots+\beta_px_{ip}\).

For the usual linear regression setting, observations are modeled as having errors with mean zero and constant variance; errors are also taken to be independent across observations. Normal errors are an additional assumption when exact small-sample \(t\)- or \(F\)-inference is required. These error assumptions describe variation around the mean model. They do not remove an exact redundancy in the predictor values.

Definition: A regression coefficient is identifiable from a given design if distinct coefficient values cannot produce the same vector of conditional means for the observed predictor values. If different coefficient vectors give the same means, those coefficients are not individually identifiable. A particular combination of coefficients may still be identifiable.

To express this precisely, let \(X\) be the design matrix: its first column consists of ones for the intercept, and its other columns contain the observed predictor values. Let \(\beta\) be the vector of coefficients. The vector of conditional means is \(X\beta\). The coefficient vector is uniquely determined by this mean vector only when the columns of \(X\) are linearly independent (equivalently, \(X\) has full column rank).

Worked Example: Two Duplicated Predictors

Suppose a hypothetical clinic records an outcome \(Y\), in points, and two columns intended to measure weekly activity, both in hours. In a small data set, the columns have the same values: \(x_1=x_2=(0,1,2)\). The observed outcomes are \((2,5,8)\) points. Consider the model \(Y_i=\beta_0+\beta_1x_{i1}+\beta_2x_{i2}+\varepsilon_i\).

Check the predictor relationship. Each observation has \(x_{i1}=x_{i2}\), so the two predictor columns are exact duplicates. The mean for any observation simplifies to

\beta_0+\beta_1x_{i1}+\beta_2x_{i2} =\beta_0+(\beta_1+\beta_2)x_{i1}.

Only the sum \(\beta_1+\beta_2\) appears in the mean model. The observed outcomes lie on the line \(2+3x\): at \(x=0\), the fitted value is 2 points; at \(x=1\), it is 5 points; and at \(x=2\), it is 8 points. Thus \(\beta_0=2\) and \(\beta_1+\beta_2=3\) produce these fitted values.

For instance, \((\beta_0,\beta_1,\beta_2)=(2,3,0)\) and \((2,1,2)\) both give the same fitted values: \(2,5,8\) points. More generally, every pair \((\beta_1,\beta_2)\) satisfying \(\beta_1+\beta_2=3\) gives those fitted values. The combined slope is determined in this example, but the separate coefficients for the two duplicate activity columns are not.

Why Exact Redundancy Prevents a Unique Solution

The key algebraic fact is that if a nonzero vector \(v\) satisfies \(Xv=0\), then for any coefficient vector \(\beta\) and any number \(t\),

X(\beta+tv)=X\beta+tXv=X\beta.

So changing the coefficients in the direction \(v\) leaves every fitted mean unchanged. There is then no unique coefficient vector associated with those means. This argument establishes nonidentifiability directly; no assumption about the distribution of the errors can make the alternative coefficient vectors distinguishable from the predictor columns.

For duplicated predictors, a vector \(v\) can increase one duplicate's coefficient and decrease the other's by the same amount. The coefficients change, but their sum—and therefore every fitted mean—stays fixed. The same reasoning applies to more complicated exact relationships.

Theorem: If the columns of the design matrix \(X\) are linearly dependent, there is a nonzero vector \(v\) such that \(Xv=0\). Consequently, whenever \(\beta\) gives a set of fitted means, every vector \(\beta+tv\) gives the same fitted means for any \(t\). The full coefficient vector is not uniquely identifiable.

Worked Example: A Predictor Defined as the Sum of Two Others

Suppose a hypothetical service records three predictors for each case: \(x_1\), hours of instruction; \(x_2\), hours of practice; and \(x_3\), total hours, with \(x_3=x_1+x_2\) for every case. All three are measured in hours. For three illustrative cases, let \((x_1,x_2,x_3)\) be \((1,0,1)\), \((0,1,1)\), and \((1,1,2)\), and let the outcomes be 5, 8, and 11 points. Fit a model with an intercept and all three predictors.

Rewrite the mean model. Since \(x_3=x_1+x_2\),

\beta_0+\beta_1x_1+\beta_2x_2+\beta_3x_3 =\beta_0+(\beta_1+\beta_3)x_1+(\beta_2+\beta_3)x_2.

Thus the data can determine the two combinations \(\beta_1+\beta_3\) and \(\beta_2+\beta_3\), rather than the three coefficients separately. In this example the outcomes follow \(2+3x_1+6x_2\): for the first case the fitted value is \(2+3(1)+6(0)=5\) points; for the second it is \(2+3(0)+6(1)=8\) points; and for the third it is \(2+3(1)+6(1)=11\) points.

One coefficient vector that gives those fitted values is \((\beta_0,\beta_1,\beta_2,\beta_3)=(2,3,6,0)\). Another is \((2,2,5,1)\): it gives the same two combinations, \(2+1=3\) and \(5+1=6\), and hence the same three fitted values. In general, replacing \((\beta_1,\beta_2,\beta_3)\) with \((\beta_1-t,\beta_2-t,\beta_3+t)\) leaves both combinations unchanged. Therefore the individual effects assigned to instruction, practice, and total hours cannot all be uniquely determined in this model.

What Remains Determined?

Nonidentifiability does not mean that every part of the regression is unknowable. It means that coefficient values cannot be distinguished wherever their changes leave the fitted means unchanged. In the duplicated-predictor example, the sum of the two slopes determines the overall slope against their shared predictor. In the sum-predictor example, two combinations of slopes determine the mean model's coefficients on \(x_1\) and \(x_2\).

The fitted values can also be unique even when the coefficient vector is not. In the examples, every coefficient vector in the stated family produces exactly the same fitted value for each case. The ambiguity is in how the model allocates a fitted association among redundant predictors, not in the fitted outcomes themselves. This distinction matters whenever someone wants to interpret a coefficient as the separate association for one predictor while holding the others fixed.

One practical response is to remove a redundant predictor when there is a defensible reason to do so, or to define a model using the meaningful combined predictor or coefficient combination. The model's scientific meaning should guide that choice: deleting a column makes a coefficient calculation possible, but it does not create information in the data about a distinction the predictor values never made.

Worked Example: Interpreting a Coefficient Output Carefully

A hypothetical analyst fits a model with two identical columns measuring a patient's weekly exercise hours. A software program returns an intercept of 4.0 points, a coefficient of 1.5 points per hour for the first column, and a coefficient of 1.5 points per hour for the second. The predictor values are identical for all observations.

Check what the fitted equation uses. If both columns have value \(x\) hours, the fitted mean is

$$ 4.0+1.5x+1.5x=4.0+3.0x. $$

The combined slope is 3.0 points per hour for the shared exercise-hours measurement. But the output's equal split is not evidence that the two columns each have a separate effect of 1.5 points per hour. For example, coefficients 2.0 and 1.0, or 0.0 and 3.0, give the same combined slope and fitted equation.

The predictor relation is checked directly from the columns, so the conclusion about nonidentifiability does not depend on a residual diagnostic or a significance test. The example does not establish that exercise causes a change in the outcome; it demonstrates only what this regression model can distinguish given these predictor values.

Common Mistake: Treating a Reported Split as Information

A common mistake is to see two coefficient estimates in software output and conclude that the data have identified the separate effects of two predictors. That conclusion does not follow when one predictor is an exact duplicate or exact linear combination of the others. A program may omit a redundant column, report one of many possible coefficient solutions, or display a warning; the particular behavior depends on the software and fitting method.

The correction is to inspect the predictor relationship and interpret only quantities that are identifiable under the model. In the duplicate case, the combined slope is the meaningful determined quantity. In the sum case, the two coefficient combinations in the rewritten mean model are determined, not all three original slopes. If a unique coefficient for each predictor is required, the data and model must provide distinct predictor information; a software setting cannot supply that information.

Key takeaway: Exact collinearity means there is a nonzero change to the coefficients that leaves every fitted mean unchanged. Individual coefficients may therefore be nonidentifiable even though some coefficient combinations and all fitted values are determined.

Check Your Understanding

Answer each question by using the predictor relationships, not by guessing from the coefficient values.

If \(x_1=x_2\) for every observation, which combination of their two slopes appears in the mean model?
Suppose \(x_3=2x_1+x_2\) for every observation. Rewrite the contribution \(\beta_1x_1+\beta_2x_2+\beta_3x_3\) using only \(x_1\) and \(x_2\). Which coefficient combinations appear?
Can two different coefficient vectors give the same fitted values? State the algebraic condition that guarantees they can.
Why do mean-zero, constant-variance error assumptions not resolve exact nonidentifiability?
A program reports two equal slopes for duplicate predictors. What can be concluded about their combined slope, and what cannot be concluded about the separate slopes?