Near Collinearity and Coefficient Instability

Why Nearly Redundant Predictors Cause Trouble

A GLM linear predictor is built from design-matrix columns: each coefficient weights one column's contribution to the fitted linear predictors. If two columns are nearly parallel, their contributions look similar across the observations. The data can then have difficulty distinguishing how much of the fitted pattern belongs to one predictor and how much belongs to the other.

This is near collinearity. Unlike exact collinearity, it does not make the coefficients mathematically non-unique. Instead, the coefficients may be uniquely determined but estimated imprecisely. Small changes in responses, or other small changes in the data, can then produce substantial changes in the separate coefficient estimates.

Definition: Design-matrix columns are nearly collinear when one or more columns can be approximated closely by linear combinations of the others, but the relevant dependence is not exact. With two nonzero columns, a simple diagnostic is their correlation after centering: an absolute correlation close to one indicates near collinearity. The degree of concern depends on scaling, the full design, and the analysis; there is no universal correlation cutoff.

Near collinearity is about the geometry of the design, not whether a particular coefficient happens to be large. It can make individual coefficients uncertain even when the model's fitted linear predictors are comparatively stable. The two-predictor case makes this distinction precise.

Two Predictors: Information in the Weak Direction

Consider a Gaussian linear model with fixed design matrix $X$, two centered columns $x_1,x_2$, and independent errors with mean zero and variance $\sigma^2$. Suppose each column has squared length $n$, and let their correlation be $r$. If $|r|<1$, the columns are linearly independent, and

X^\mathsf{T}X = n \begin{pmatrix} 1&r\\ r&1 \end{pmatrix}.

The eigenvectors point in the sum and difference directions of the coefficient pair. When $r$ is positive and close to one, the columns vary together; changing the coefficients in opposite directions has little effect on the fitted values. The following result quantifies the resulting loss of precision.

Theorem: In the model $y=X\beta+\varepsilon$, assume $X$ is fixed with the two columns just described, $|r|<1$, $E[\varepsilon]=0$, and $\operatorname{Cov}(\varepsilon)=\sigma^2 I$. Then the least-squares estimator has covariance matrix $$ \operatorname{Cov}(\hat{\beta}\mid X) = \frac{\sigma^2}{n(1-r^2)} \begin{pmatrix} 1&-r\\ -r&1 \end{pmatrix}. $$ In particular, each coefficient has variance $\sigma^2/[n(1-r^2)]$.

Proof. Since $|r|<1$, the determinant of $X^\mathsf{T}X$ is $n^2(1-r^2)>0$, so the least-squares estimator is $\hat{\beta}=(X^\mathsf{T}X)^{-1}X^\mathsf{T}y$. Substituting $y=X\beta+\varepsilon$ gives $\hat{\beta}-\beta=(X^\mathsf{T}X)^{-1}X^\mathsf{T}\varepsilon$. Taking its covariance and using $\operatorname{Cov}(\varepsilon)=\sigma^2 I$,

\operatorname{Cov}(\hat{\beta}\mid X) =\sigma^2(X^\mathsf{T}X)^{-1}X^\mathsf{T}X(X^\mathsf{T}X)^{-1} =\sigma^2(X^\mathsf{T}X)^{-1}.

The inverse of the displayed $2$-by-$2$ matrix is \[ (X^\mathsf{T}X)^{-1} = \frac{1}{n(1-r^2)} \begin{pmatrix} 1&-r\\ -r&1 \end{pmatrix}. \] Combining the equations gives the stated covariance. $\square$

For $r$ near one, the variance is inflated by the factor $1/(1-r^2)$ compared with the case of uncorrelated columns of the same lengths. Equivalently, the eigenvalues of $X^\mathsf{T}X$ are $n(1+r)$ and $n(1-r)$ when $r>0$. The second is small: the data contain little information about the difference between the two coefficients. The ratio of the larger eigenvalue to the smaller, called the condition number here, is $(1+r)/(1-r)$.

Worked Example: Standard Errors When the Correlation Is 0.99

Suppose the two centered predictor columns each have squared length $n=100$, their correlation is $r=0.99$, and the Gaussian error variance is $\sigma^2=1$. The variance of either coefficient is

\operatorname{Var}(\hat{\beta}_j\mid X) =\frac{1}{100(1-0.99^2)} =\frac{1}{100(0.0199)} =\frac{1}{1.99} \approx 0.5025.

Thus the standard error is $\sqrt{0.5025}\approx 0.709$. If the columns had the same lengths but correlation zero, the variance would be $1/100=0.01$ and the standard error would be $0.1$. Here the standard error is about seven times as large. The condition number is $1.99/0.01=199$, indicating a much weaker direction in the design than in the uncorrelated case.

Small Data Changes, Large Coefficient Changes

The same weak direction explains sensitivity. In least squares, a response change $\Delta y$ changes the coefficients by $(X^\mathsf{T}X)^{-1}X^\mathsf{T}\Delta y$. A change aligned with the weakly informed direction can be magnified in the coefficients. Yet the resulting change in fitted values can remain small, because that direction changes the fitted vector only slightly.

Worked Example: A Small Response Change Shifts the Coefficients

Let $u$ and $v$ be orthogonal unit vectors, and set $\delta=0.01$. Use the two unit-length design columns

x_1=u,\qquad x_2=\frac{u+\delta v}{\sqrt{1+\delta^2}}.

Their correlation is $u^\mathsf{T}x_2=1/\sqrt{1+\delta^2}\approx 0.99995$. For response $y=u$, the exact fit has coefficients $(1,0)$, since $u=1x_1+0x_2$. Now change the response by just $0.01v$, giving $y^\ast=u+\delta v$. The new response has changed by a vector of length $0.01$. But $y^\ast=\sqrt{1+\delta^2}\,x_2$, so its exact-fit coefficients are $(0,\sqrt{1+\delta^2})\approx(0,1.00005)$.

The coefficient pair has changed by approximately $(-1,1)$, even though the response change has length only $0.01$. Both fits reproduce their responses exactly. This example isolates the instability: the nearly parallel columns allow the same fitted pattern to be represented by substantially different allocations between the two coefficients.

More generally, if $X$ has full column rank, the least-squares change obeys the bound $\|\Delta\hat{\beta}\|\leq \|\Delta y\|/s_{\min}(X)$, where $s_{\min}(X)$ is the smallest singular value of $X$. To see this, write the singular value decomposition $X=U\Sigma V^\mathsf{T}$. Then the response-to-coefficient map is $X^+=V\Sigma^{-1}U^\mathsf{T}$, whose operator norm is $1/s_{\min}(X)$. Therefore $\|\Delta\hat{\beta}\|=\|X^+\Delta y\|\leq\|\Delta y\|/s_{\min}(X)$. Near collinearity makes this smallest singular value small, so the bound permits substantial coefficient changes even for small response changes.

How the Same Issue Appears in a GLM

For an independent-response GLM, a regular model with linear predictor $\eta=X\beta$, mean $\mu_i=g^{-1}(\eta_i)$, and variance $\operatorname{Var}(Y_i)=\phi V(\mu_i)$ has expected Fisher information for $\beta$

\mathcal{I}(\beta)=X^\mathsf{T}WX,\qquad W_{ii}=\frac{(d\mu_i/d\eta_i)^2}{\phi V(\mu_i)}.

This expression applies when the derivatives and variances are defined and the usual likelihood regularity conditions hold. The diagonal weights describe how informative the observations are at the fitted means. If the design columns are nearly dependent, weighting them does not generally create a strong distinction between their contributions. A weak information direction can therefore lead to large estimated standard errors, subject to the model and the usual likelihood-based uncertainty approximation.

Worked Example: Near Collinearity in Logistic Regression

Consider a logistic regression with two predictor columns, no intercept, and coefficients equal to zero at the point where expected information is evaluated. Let $n=50$; suppose both columns are centered with squared length 50 and correlation $r=0.98$. At $\beta=0$, each fitted probability is $0.5$. For Bernoulli responses with the logit link, the information weight is $p_i(1-p_i)=0.25$, so

\mathcal{I}(\beta) =0.25X^\mathsf{T}X =12.5 \begin{pmatrix} 1&0.98\\ 0.98&1 \end{pmatrix}.

The diagonal entries of the inverse information matrix are

\bigl[\mathcal{I}(\beta)^{-1}\bigr]_{jj} =\frac{1}{12.5(1-0.98^2)} =\frac{1}{12.5(0.0396)} \approx 2.0202.

The corresponding local, information-based standard error is approximately $\sqrt{2.0202}=1.421$. With the same column lengths and correlation zero, it would be $\sqrt{1/12.5}\approx 0.283$. These are standard errors from expected information at the specified parameter value, not a claim about a particular observed data set. The comparison shows how the design can weaken coefficient information in a GLM as well as in a Gaussian model.

Interpretation, Diagnosis, and a Common Mistake

Large standard errors do not mean that every aspect of the fitted model is equally uncertain. With positively correlated predictors, the sum direction is well represented by the columns together, while the difference direction is weakly represented. As a result, the fitted linear predictors or predictions for cases similar to those observed may be much more stable than the separate coefficients. Predictions for new cases with a different combination of predictor values may not share that stability.

A common mistake is to conclude, “The coefficients are unique, so they must be reliable.” Uniqueness only says there is one exact coefficient vector for the specified design and response. It does not say that small changes in the data leave that vector nearly unchanged. The correction is to inspect the design's near dependencies and the uncertainty of the coefficients, and to interpret individual effects cautiously when the data distinguish them poorly.

Another mistake is to treat a correlation cutoff as a universal rule. Pairwise correlations can help reveal a nearly parallel pair, but a column can also be approximated by a combination of several other columns. Further, scaling and GLM information weights affect the numerical conditioning relevant to estimation. Diagnose the full design in context rather than treating one pairwise threshold as a definitive test.

Key takeaway: Near collinearity is not exact redundancy. It leaves coefficients mathematically distinct but makes some coefficient combinations weakly informed. This can inflate standard errors and make individual coefficients sensitive to small data changes, even while fitted linear predictors remain comparatively stable.

Check Your Understanding

Answer each question using the design geometry and uncertainty arguments above.

For two centered columns of equal squared length $n$, what happens to the smaller eigenvalue of $X^\mathsf{T}X$ as their positive correlation approaches one?
In the Gaussian two-predictor result, how does the coefficient variance change when the correlation increases from zero toward one?
Why can a small response change produce a large coefficient change but only a small change in fitted values?
In the logistic regression example, why is the information weight $0.25$ at $\beta=0$?
Why does a unique coefficient estimate not guarantee that the estimate is stable?