Exact Collinearity and Redundant Predictors

Why Exact Collinearity Matters

A generalized linear model represents the linear predictor for the observations as \(\eta=X\beta\). Each column of \(X\) records one predictor across the observations, and each entry of \(\beta\) gives that column a coefficient. If one predictor column is exactly determined by other columns, changing its coefficient can be offset by changing the other coefficients. The linear predictor then stays the same, even though the coefficient vector changes.

This is exact collinearity. It is a structural property of the observed design matrix, not a claim that two predictors are merely strongly associated. Its consequence is important: the data and model cannot distinguish all the individual coefficients in a redundant parameterization. They can still determine a linear predictor, but more than one coefficient vector can represent it.

Definition: The columns of a design matrix \(X\) are exactly linearly dependent if there are numbers \(c_1,\ldots,c_p\), not all zero, such that \(c_1x_1+\cdots+c_px_p=0\), where \(x_j\) denotes column \(j\). A predictor column is redundant if it is an exact linear combination of the other columns.

The coefficients \(c_j\) in this definition are constants shared across rows. Thus, dependence can be checked by comparing entire columns, not by looking at whether two predictor values happen to be related for one observation. If a column is redundant, the design does not have full column rank, the condition associated with uniqueness of coefficients in the earlier material on column rank.

How Redundancy Changes the Coefficients

Suppose one column, \(x_j\), satisfies \(x_j=\sum_{k\ne j}a_kx_k\). Its contribution to the linear predictor can be rewritten using the other columns:

X\beta=\sum_{k\ne j}x_k\beta_k+x_j\beta_j =\sum_{k\ne j}x_k(\beta_k+a_k\beta_j).

The same vector of linear predictors is therefore obtained using coefficients \(\gamma_k=\beta_k+a_k\beta_j\) for the other columns and coefficient zero for column \(j\). More generally, many different coefficient vectors can give the same \(\eta\).

Theorem: Let \(X\) be a fixed \(n\times p\) design matrix. If its columns are exactly linearly dependent, then there are distinct coefficient vectors that produce the same linear predictor \(X\beta\). In a GLM whose response distribution, at fixed dispersion and other fixed parameters, is determined by the linear predictor, those coefficient vectors give the same response distribution and likelihood.

Proof. By exact dependence, there is a nonzero vector \(c=(c_1,\ldots,c_p)^\mathsf{T}\) such that \(Xc=0\). For any coefficient vector \(\beta\) and any real number \(t\), define \(\beta^{*}=\beta+tc\). Then

X\beta^{*}=X(\beta+tc)=X\beta+tXc=X\beta.

Since \(c\ne0\), choosing \(t\ne0\) gives \(\beta^{*}\ne\beta\). Thus the coefficient vectors are distinct but the linear predictors agree. In a GLM, equal linear predictors give equal means through the inverse link; with the same family and dispersion, they give the same response distribution. Consequently, for every possible observed response vector, the likelihood evaluated at \(\beta\) and at \(\beta^{*}\) is equal. The coefficients cannot be uniquely distinguished by that likelihood. \(\square\)

Worked Examples

Worked Example: One Predictor Is Twice Another

Suppose a model has an intercept, a numerical predictor \(x\), and a second predictor \(z\), with \(z=2x\) in every row. For four observations, let \(x=(0,1,2,3)^\mathsf{T}\). The design matrix is

X= \begin{pmatrix} 1&0&0\\ 1&1&2\\ 1&2&4\\ 1&3&6 \end{pmatrix}.

The third column is exactly twice the second column: \(z=2x\). For coefficients \((\beta_0,\beta_x,\beta_z)\), the linear predictor in row \(i\) is

\eta_i=\beta_0+\beta_x x_i+\beta_z z_i =\beta_0+(\beta_x+2\beta_z)x_i.

For example, take \((\beta_0,\beta_x,\beta_z)=(1,3,2)\). The resulting predictor is \(\eta_i=1+7x_i\), so \(\eta=(1,8,15,22)^\mathsf{T}\). Another choice, \((1,7,0)\), gives the same vector. Indeed, its coefficient on \(x\) is \(7\), equal to \(3+2(2)\), the combined contribution in the first parameterization. The model can determine the combined slope \(\beta_x+2\beta_z\), but it cannot assign that slope uniquely between \(\beta_x\) and \(\beta_z\).

Worked Example: An Intercept and Exhaustive Category Indicators

Suppose every patient belongs to exactly one of two groups, A or B. Let \(A_i=1\) for a patient in group A and \(0\) otherwise, and define \(B_i\) similarly. Because the groups are exhaustive and disjoint, every row satisfies \(A_i+B_i=1\). With an intercept, the design columns obey

\boldsymbol{1}=x_A+x_B.

Consider the four rows A, B, A, B. The design matrix has columns for the intercept, A, and B:

X= \begin{pmatrix} 1&1&0\\ 1&0&1\\ 1&1&0\\ 1&0&1 \end{pmatrix}.

For coefficients \((\beta_0,\beta_A,\beta_B)=(10,2,5)\), the predictor equals \(12\) in group A and \(15\) in group B. Now choose \((13,-1,2)\). The A predictor is \(13-1=12\), and the B predictor is \(13+2=15\). Both coefficient vectors produce the same values for all rows because the intercept column equals the sum of the two indicator columns. This is an exact dependency caused by the coding of the categories, not by a numerical accident.

One common parameterization uses an intercept and only one of the two indicators. That removes this particular redundancy while retaining a group-specific contrast. Which group is the reference determines how the remaining coefficients are described.

Worked Example: Removing a Redundant Predictor Without Changing the Linear Predictors

Suppose a study records two laboratory measurements \(u\) and \(v\), and a third column \(w=u+v\) has been included. For three observations, let \(u=(1,0,2)^\mathsf{T}\), \(v=(0,1,3)^\mathsf{T}\), and hence \(w=(1,1,5)^\mathsf{T}\). With an intercept, the predictor is

\eta_i=\beta_0+\beta_u u_i+\beta_v v_i+\beta_w w_i =\beta_0+(\beta_u+\beta_w)u_i+(\beta_v+\beta_w)v_i.

Take \((\beta_0,\beta_u,\beta_v,\beta_w)=(4,2,-1,3)\). The equivalent coefficients for the model with only the intercept, \(u\), and \(v\) are \((4,5,2)\), since \(2+3=5\) and \(-1+3=2\). The resulting predictor is

\eta= \begin{pmatrix} 4+5(1)+2(0)\\ 4+5(0)+2(1)\\ 4+5(2)+2(3) \end{pmatrix} = \begin{pmatrix} 9\\6\\20 \end{pmatrix}.

Direct calculation with all four columns gives \(4+2(1)-1(0)+3(1)=9\), \(4+2(0)-1(1)+3(1)=6\), and \(4+2(2)-1(3)+3(5)=20\), confirming the same vector. Removing \(w\) does not change which linear predictors this design can produce: its contribution has been absorbed into the coefficients of \(u\) and \(v\).

A General Way to Remove Exact Redundancy

The previous example illustrates a general reparameterization. If \(x_j=\sum_{k\ne j}a_kx_k\), then

\sum_{k=1}^{p}x_k\beta_k =\sum_{k\ne j}x_k(\beta_k+a_k\beta_j).

Thus every linear predictor made with all columns can also be made after dropping column \(j\), by using coefficients \(\gamma_k=\beta_k+a_k\beta_j\) on the retained columns. In the other direction, any coefficients on the retained columns are obtained in the full design by setting \(\beta_j=0\). The two designs therefore generate exactly the same set of linear predictors.

Result: If one column of \(X\) is an exact linear combination of the remaining columns, dropping that column preserves the set of possible values of \(X\beta\). It removes a redundant coefficient, but it does not add information to the data or change the fitted linear predictors available to the model.

This algebra gives a practical remedy, but the choice of which column to remove can affect how the remaining coefficients are interpreted. In the two-group example, keeping the intercept and dropping one indicator sets up a reference group. In other settings, a scientific convention or a deliberate coding scheme can guide the choice. Any such change should be documented so that coefficient meanings remain clear.

How to Audit a Design for Exact Redundancy

For a small design, compare columns directly or solve for constants that express one column in terms of the others. For a larger design, a rank calculation or exact row reduction can reveal dependencies. The key word is exact: a relation must hold for every row. A relation that holds only approximately is not an exact redundancy, even if it raises other numerical concerns.

Write out the columns.
Include the intercept and every indicator or numerical predictor that is actually in the design.

Check for an exact relation.
Ask whether a column equals a constant multiple or a fixed linear combination of other columns in every row.

Show the consequence algebraically.
Substitute the relation into \(X\beta\) and identify which coefficient combinations determine the predictor.

Choose a nonredundant representation.
Remove a redundant column or change the coding, then state how the retained coefficients are interpreted.

Common Mistakes and Why the Issue Matters

Calling a strong association exact collinearity. Two columns can be highly related without being exact linear combinations. For exact redundancy, the proposed relation must hold row by row with no discrepancy.
Assuming a unique fitted predictor requires unique coefficients. The examples show the opposite: different coefficient vectors can give exactly the same \(\eta\). Equal predictors also imply equal model distributions when the other GLM parameters are held fixed.
Thinking the data select one coefficient vector from the equivalent choices. If the likelihood is unchanged across those choices, maximizing it cannot distinguish them. A software package may impose a convention or report an error, but that convention does not create information that the design lacks.
Dropping a column without considering interpretation. Removing a redundant column can preserve all possible linear predictors while changing the role and meaning of the remaining coefficients. Check that the reduced coding answers the intended scientific question.

Exact collinearity matters because a reported coefficient is often interpreted as the effect associated with changing one predictor while holding the others fixed. When predictors are redundant, those separate coefficient effects cannot all be determined from the design. A nonredundant parameterization makes the model’s coefficients uniquely representable under the full-column-rank condition, while preserving the model’s intended linear predictors.

Check Your Understanding

Use the column relationships and coefficient algebra to answer these questions.

If a design has columns \(x_1\) and \(x_2\) with \(x_2=3x_1\), what combination of their coefficients determines their total contribution to \(\eta\)?
For an intercept and two indicators that sum to one in every row, state the exact column relation that makes the design redundant.
Why does an exact dependency produce distinct coefficient vectors with the same linear predictor?
If \(x_3=x_1+x_2\), express the contribution \(\beta_1x_1+\beta_2x_2+\beta_3x_3\) using only \(x_1\) and \(x_2\).
Explain why two strongly associated columns are not necessarily exactly collinear.