Near Collinearity and the Geometry of Predictor Data

Why Predictor Geometry Matters

A regression coefficient describes the association of one predictor with the outcome while the other predictors are held fixed. That interpretation depends on the data containing enough variation to distinguish the predictors. If two predictors move almost together, the observed data may provide little evidence about which predictor accounts for a particular part of the fitted outcome.

This is the distinction between exact collinearity and near-collinearity. With exact collinearity, one predictor column is an exact linear combination of other columns, so the coefficients are not uniquely identifiable. With near-collinearity, the columns are not exactly redundant, and least squares can produce a unique estimate, but small changes in the fitted values may correspond to large changes in individual coefficients.

Definition: Predictors are near-collinear when one or more predictor columns are close to a linear combination of other predictor columns. The design is still full rank, so the coefficients are identifiable, but the observed predictor data provide weak information for separating some coefficient effects.

A Model and Its Geometry

For \(n\) observations and \(p\) predictors, write the multiple linear regression model as

Y = \beta_0\mathbf{1} + Z\boldsymbol{\beta} + \boldsymbol{\varepsilon}.

Here \(Y\) is the \(n\)-vector of outcomes, \(\mathbf{1}\) is an \(n\)-vector of ones, \(Z\) is the \(n \times p\) matrix containing only the predictor columns, and \(\boldsymbol{\beta}\) is the \(p\)-vector of slopes. The full design matrix is \(X=[\mathbf{1}\; Z]\), and the full coefficient vector is \((\beta_0,\boldsymbol{\beta}^{T})^{T}\). For the uncertainty calculations below, treat the predictor values as fixed and assume the errors have mean zero, common variance \(\sigma^2\), and are uncorrelated with one another. Normal errors are not needed for the geometric argument or the variance formula.

The columns of \(Z\) are vectors in \(n\)-dimensional observation space. The model uses their weighted combinations to create the fitted outcomes. If two columns point in nearly the same direction, increasing one slope and decreasing the other by a similar amount can leave the fitted values nearly unchanged. The data then distinguish the combined contribution more clearly than the two separate contributions.

To make this concrete for two predictors, suppose their columns have been centered (each column sums to zero) and scaled so each has sum of squares \(n-1\). Their sample correlation is \(r\). The inner-product matrix for these two predictor columns—using \(Z\), not the full design matrix \(X\)—is

Z^{T}Z=(n-1) \begin{pmatrix} 1&r\\ r&1 \end{pmatrix}.

Because the columns are centered, each has inner product zero with the intercept column. Thus the full matrix \(X^{T}X\) has a separate intercept entry \(n\), with zero intercept-predictor entries, alongside the predictor block \(Z^{T}Z\). Keeping these matrices distinct prevents confusing the geometry of the predictors with that of the full design.

Worked Example: Reading Near-Alignment in a Predictor Plot

Suppose a hypothetical data set contains 51 observations on two predictors, each centered and scaled to have sum of squares 50. A scatterplot of predictor 2 against predictor 1 shows a tight upward-sloping cloud, and their sample correlation is \(r=0.98\). How does this describe the geometry?

The inner product of the columns is \(50(0.98)=49\), while each column has squared length 50. Their cosine is therefore

\frac{Z_1^{T}Z_2}{\sqrt{Z_1^{T}Z_1}\sqrt{Z_2^{T}Z_2}} = \frac{49}{\sqrt{50}\sqrt{50}} =0.98.

The columns point in almost the same direction. In a predictor scatterplot, this appears as observations clustered close to a straight, positively sloped line. The plot does not say that the predictors are identical: \(r=0.98\) is less than 1. It does indicate that the data contain relatively little variation in one predictor that is not accompanied by variation in the other.

Strong and Weak Directions in the Coefficients

The two-predictor inner-product matrix has two important directions. The vector \((1,1)^T\) corresponds to changing the slopes together; \((1,-1)^T\) corresponds to increasing one while decreasing the other. Multiplication by \(Z^TZ\) gives

\begin{pmatrix} 1&r\\ r&1 \end{pmatrix} \begin{pmatrix} 1\\ 1 \end{pmatrix} =(1+r) \begin{pmatrix} 1\\ 1 \end{pmatrix}, \qquad \begin{pmatrix} 1&r\\ r&1 \end{pmatrix} \begin{pmatrix} 1\\ -1 \end{pmatrix} =(1-r) \begin{pmatrix} 1\\ -1 \end{pmatrix}.

These directions are eigenvectors, with eigenvalues \(1+r\) and \(1-r\), respectively; the factor \(n-1\) in \(Z^TZ\) scales both eigenvalues. When \(r\) is close to 1, \(1-r\) is small. The data therefore provide weak information about the contrast between the two slopes. When \(r\) is close to \(-1\), the weak direction instead corresponds to changing the slopes together. More generally, with several predictors, a weakly supported direction can involve a combination of three or more columns, not just a pair.

Worked Example: A Coefficient Change With Little Effect on Fits

Continue with the hypothetical centered predictors having \(n=51\) and \(r=0.98\). Consider changing the slope vector by \(\Delta\boldsymbol{\beta}=(5,-5)^T\), with the intercept unchanged. The resulting change in fitted values is \(Z\Delta\boldsymbol{\beta}\), because \(Z\) contains the predictor columns only.

The squared length of the change in fitted values is

\begin{aligned} \lVert Z\Delta\boldsymbol{\beta}\rVert^2 &= \Delta\boldsymbol{\beta}^{T}Z^{T}Z\Delta\boldsymbol{\beta}\\ &= 50\left(5^2+(-5)^2+2(0.98)(5)(-5)\right)\\ &= 50(50-49)\\ &=50. \end{aligned}

Thus the length of the fitted-value change is \(\sqrt{50}\approx 7.071\). For comparison, changing both slopes by 5 gives \(\Delta\boldsymbol{\beta}=(5,5)^T\), and

\begin{aligned} \lVert Z\Delta\boldsymbol{\beta}\rVert^2 &= 50\left(5^2+5^2+2(0.98)(5)(5)\right)\\ &= 50(99)=4950, \end{aligned}

so the fitted-value change has length \(\sqrt{4950}\approx 70.36\). In this design, the same-sized slope change has a much smaller effect on the fitted outcomes when it moves in the contrast direction \((1,-1)^T\). This is the geometric reason the individual slopes are difficult to separate.

Why Weak Geometry Means Greater Coefficient Uncertainty

When the full design matrix \(X\) has full column rank, the least-squares estimate is \(\hat{\boldsymbol{\gamma}}=(X^TX)^{-1}X^TY\), where \(\boldsymbol{\gamma}=(\beta_0,\boldsymbol{\beta}^{T})^{T}\). Under the stated error assumptions, its conditional covariance matrix is

\operatorname{Cov}(\hat{\boldsymbol{\gamma}}\mid X)=\sigma^2(X^TX)^{-1}.

For centered predictors, the intercept is orthogonal to their columns, so the slope covariance block is \(\sigma^2(Z^TZ)^{-1}\). Inverting the two-predictor matrix gives

\operatorname{Cov}(\hat{\boldsymbol{\beta}}\mid Z) = \frac{\sigma^2}{(n-1)(1-r^2)} \begin{pmatrix} 1&-r\\ -r&1 \end{pmatrix}.

The diagonal entries show that, in this standardized two-predictor setting, the conditional variance of either estimated slope is \(\sigma^2/((n-1)(1-r^2))\). The formula makes precise how a small eigenvalue of the predictor geometry corresponds to a large uncertainty in some coefficient direction.

A short derivation explains the connection. Since \(Y=X\boldsymbol{\gamma}+\boldsymbol{\varepsilon}\), the least-squares estimation error is \(\hat{\boldsymbol{\gamma}}-\boldsymbol{\gamma}=(X^TX)^{-1}X^T\boldsymbol{\varepsilon}\). Taking its covariance and using \(\operatorname{Cov}(\boldsymbol{\varepsilon}\mid X)=\sigma^2 I\) gives

\begin{aligned} \operatorname{Cov}(\hat{\boldsymbol{\gamma}}\mid X) &=(X^TX)^{-1}X^T(\sigma^2 I)X(X^TX)^{-1}\\ &=\sigma^2(X^TX)^{-1}. \end{aligned}

The inverse gives greater weight to directions with small eigenvalues. At exact collinearity, one eigenvalue is zero and the inverse does not exist; the coefficients cannot be uniquely estimated. Near-collinearity is different: the eigenvalue remains positive, but its small size makes the corresponding coefficient combination uncertain.

Worked Example: Comparing Slope Uncertainty

Assume the hypothetical data above have error standard deviation \(\sigma=2\) outcome units. The centered, scaled predictor columns have \(n-1=50\) squared length and correlation \(r=0.98\). The conditional variance of either slope estimate is

\operatorname{Var}(\hat{\beta}_j\mid Z) = \frac{2^2}{50(1-0.98^2)} = \frac{4}{50(0.0396)} = \frac{4}{1.98} \approx 2.020.

The corresponding standard error is \(\sqrt{2.020}\approx 1.421\) outcome units per one standard-deviation increase in the predictor. If the same columns instead had \(r=0\), their slope variance would be \(4/50=0.0800\), with standard error \(\sqrt{0.0800}\approx 0.2828\) in the same units. The large difference arises from predictor geometry, not from a change in the assumed error standard deviation or the number of observations.

What to Conclude—and What Not to Conclude

Near-collinearity is principally a limitation on separating individual coefficient effects. It does not by itself imply that the regression model is incorrect, that least squares is biased, or that predictions must be poor. If the predictors tend to occur in the same combinations at which predictions are needed, their combined contribution may be estimated more precisely than the separate slopes. The focus here is the information in the predictor arrangement, not a judgment that a model should automatically be discarded.

A predictor scatterplot can reveal that two variables are nearly related, but a plot of one pair at a time cannot display every possible relationship among several predictors. Geometry also depends on the predictor columns after accounting for the intercept and other columns. Thus a visually modest pairwise relationship does not rule out a weakly supported combination in a larger model.

Key takeaway: Near-collinearity means that some changes in coefficients produce only small changes in fitted values. Those coefficient directions are weakly identified by the observed predictor data and consequently have greater sampling uncertainty, even though the coefficients remain identifiable when the design has full rank.

Common Mistake

A common mistake is to reason, “The regression software returned two coefficients, so the data must have clearly separated the two predictor effects.” A software result only shows that the calculation produced estimates; it does not show that those estimates are precise. If two centered predictor columns are almost parallel, increasing one slope while decreasing the other can leave fitted values nearly the same. The correction is to distinguish existence of a unique estimate from strength of information about it: full rank provides the former, while the predictor geometry determines how well individual coefficient directions are supported.

Check Your Understanding

Use the geometry and calculations in this tutorial to answer the following questions.

In a two-predictor model with centered columns and \(r\) close to 1, which slope direction is weakly supported: changing the slopes together or increasing one while decreasing the other?
Why does a correlation of \(0.98\), rather than exactly 1, still allow a unique least-squares estimate when the rest of the design is full rank?
If the predictors are centered and \(r=0\), what is the variance of either slope estimate when \(n=41\) and \(\sigma=3\), assuming each column has sum of squares \(n-1\)?
Explain why the fitted values can be relatively stable even when the separate slope estimates are uncertain.
Why can a scatterplot of one predictor pair fail to reveal a weakly supported combination involving several predictors?