Eigenvalues and Condition Indices for Collinearity

Why Eigenvalues Help Diagnose Collinearity

A regression can include predictors that are not identical yet still carry nearly the same information. In that case, the design matrix contains a direction in which the predictors vary very little. Eigenvalues quantify how much variation the predictor data contain in different directions: a very small eigenvalue indicates a direction with little variation and therefore a potential near-linear dependence among predictors.

Condition indices summarize the relative size of these directions. They do not, by themselves, identify which predictors form a near-dependence, nor do they establish that a regression model is unsuitable. To interpret them, first define the matrix being diagnosed and then connect its eigenvalues to combinations of predictor columns.

Definition: Let $Z$ be a matrix of $p$ centered and scaled predictor columns, with the intercept excluded. A common scaling gives each column unit sum of squares, so the scaled cross-product matrix $G=Z^\mathsf{T}Z$ has unit diagonal. Write the eigenvalues of $G$ in descending order as $\lambda_1\geq\lambda_2\geq\cdots\geq\lambda_p\geq0$. The condition index for the direction associated with $\lambda_j$ is $$ \kappa_j=\sqrt{\frac{\lambda_1}{\lambda_j}},\qquad \lambda_j>0. $$ The largest eigenvalue's condition index is 1. If any eigenvalue is zero, the corresponding condition index is infinite, indicating exact linear dependence in the scaled predictor columns.

Some presentations use a correlation matrix instead of $Z^\mathsf{T}Z$. Dividing all of $G$ by the same positive constant changes every eigenvalue by that constant but leaves each condition index unchanged. What matters is that the predictor columns are put on comparable scales before their directions are compared. If predictors are not scaled, a measurement unit such as dollars versus thousands of dollars can affect the eigenvalues and make the resulting diagnostics misleading.

What a Small Eigenvalue Means

An eigenvector identifies a weighted combination of predictor columns. If $q_j$ is a unit eigenvector of $G$ with eigenvalue $\lambda_j$, then the data values of the combination $Zq_j$ have squared length $\lambda_j$. Thus, a small $\lambda_j$ means that this particular combination varies little across the observations. It does not necessarily mean that any one pair of predictors has a large correlation; the near-dependence can involve several predictors at once.

The square root in the condition-index formula is important. Eigenvalues describe squared lengths, while the ratio of lengths is the square root of the ratio of squared lengths. Consequently, the condition index is not the raw ratio $\lambda_1/\lambda_j$.

Derivation: Suppose $Gq_j=\lambda_jq_j$, where $q_j^\mathsf{T}q_j=1$. The squared length of the predictor combination $Zq_j$ is $$ \|Zq_j\|^2=(Zq_j)^\mathsf{T}(Zq_j) =q_j^\mathsf{T}Z^\mathsf{T}Zq_j =q_j^\mathsf{T}Gq_j =\lambda_j q_j^\mathsf{T}q_j =\lambda_j. $$ Therefore, its length is $\sqrt{\lambda_j}$. The ratio between the largest such length and the length in direction $q_j$ is $\sqrt{\lambda_1}/\sqrt{\lambda_j}=\sqrt{\lambda_1/\lambda_j}$, which gives the condition index. A small eigenvalue corresponds to a short, low-variation predictor direction.

For a full-rank ordinary least squares model with independent errors of common variance $\sigma^2$, the coefficient covariance matrix, conditional on the design, is $\sigma^2G^{-1}$ when the model is written using the scaled predictor columns. Since $G^{-1}$ has eigenvalues $1/\lambda_j$ in the same eigenvector directions, small eigenvalues also correspond to directions in which the coefficients are poorly determined. This connection helps explain why near collinearity can make coefficient estimates unstable even when fitted values are not obviously poor.

Calculating Condition Indices

Worked Example: Computing Indices From Three Eigenvalues

A hypothetical pricing analysis uses three centered, scaled predictors. The eigenvalues of their scaled cross-product matrix are $2.4,\ 0.54,\ 0.06$. Calculate all condition indices to four decimal places, and identify the most concerning direction by its index.

Method. The eigenvalues are already in descending order, so $\lambda_1=2.4$. Apply $\kappa_j=\sqrt{\lambda_1/\lambda_j}$ separately to each eigenvalue.

For the largest eigenvalue, $$ \kappa_1=\sqrt{\frac{2.4}{2.4}}=\sqrt{1}=1. $$

For the second eigenvalue, $$ \kappa_2=\sqrt{\frac{2.4}{0.54}} =\sqrt{\frac{40}{9}} =2.108185\ldots \approx 2.1082. $$

For the smallest eigenvalue, $$ \kappa_3=\sqrt{\frac{2.4}{0.06}} =\sqrt{40} =6.324555\ldots \approx 6.3246. $$

The indices are $1.0000,\ 2.1082,\ 6.3246$, in the same order as the eigenvalues. The third direction has the largest index and is the direction to investigate for a near-dependence. The indices alone do not identify which predictors form that combination; the associated eigenvector is needed for that interpretation.

Check and common mistake. Squaring the last index should recover $2.4/0.06=40$. Using the unrooted ratio would give 40, not the condition index. Rounding the square root to four decimals gives 6.3246.

Using Eigenvectors to Find the Combination

A small eigenvalue signals a low-variation direction, but its eigenvector supplies the weights of that direction. The sign of an eigenvector is arbitrary: if $q$ is an eigenvector, then $-q$ is also one. The relative signs and magnitudes of its entries, rather than the overall sign, indicate the relevant predictor combination.

Worked Example: Interpreting a Two-Predictor Near-Dependence

A hypothetical insurer studies two standardized measures of asset exposure, $Z_1$ and $Z_2$. Their correlation is $0.96$. Find the eigenvalues and condition indices of the two-predictor correlation matrix, then state which combination has little variation.

Method. For two standardized predictors with correlation $r$, the matrix is $$ G= \begin{pmatrix} 1&r\\ r&1 \end{pmatrix}. $$ The vectors $(1,1)^\mathsf{T}$ and $(1,-1)^\mathsf{T}$ are eigenvectors: multiplying by $G$ gives eigenvalues $1+r$ and $1-r$, respectively.

With $r=0.96$, the eigenvalue in the $(1,1)^\mathsf{T}$ direction is $1+0.96=1.96$, and the eigenvalue in the $(1,-1)^\mathsf{T}$ direction is $1-0.96=0.04$. The ordered eigenvalues are $1.96$ and $0.04$.

The first condition index is $\sqrt{1.96/1.96}=1$. The second is $$ \kappa_2=\sqrt{\frac{1.96}{0.04}} =\sqrt{49} =7. $$

The small eigenvalue belongs to the difference direction, proportional to $Z_1-Z_2$. The two predictors tend to move together, so their difference varies little. The condition indices are $1$ and $7$.

Check and common mistake. The trace of the matrix is 2, and the eigenvalues sum to $1.96+0.04=2$, as required. A common misinterpretation is to say that the small eigenvalue means $Z_1+Z_2$ varies little. Here the sum direction has the large eigenvalue; it is the difference direction that has little variation.

A Near-Dependence Can Involve Several Predictors

Pairwise correlations can miss a near-dependence involving three or more variables. An eigenvector can expose such a combination directly. The following example uses a symmetric correlation matrix so that its eigenvalues and relevant direction can be found without numerical software.

Worked Example: Finding a Three-Predictor Combination

A hypothetical risk study has three centered, standardized predictors $Z_1,Z_2,Z_3$. Each pair has correlation $-0.49$, giving $$ G= \begin{pmatrix} 1&-0.49&-0.49\\ -0.49&1&-0.49\\ -0.49&-0.49&1 \end{pmatrix}. $$ Find the eigenvalues, condition indices, and the combination associated with the smallest eigenvalue.

Method. For the vector $(1,1,1)^\mathsf{T}$, each row of $G$ sums to $1-0.49-0.49=0.02$. Therefore, this vector is an eigenvector with eigenvalue $0.02$.

For any vector whose entries sum to zero, the common off-diagonal terms imply that multiplying by $G$ scales that vector by $1-(-0.49)=1.49$. There are two independent directions with entries summing to zero, so the other two eigenvalues are both $1.49$. In descending order, the eigenvalues are $1.49,\ 1.49,\ 0.02$.

The condition indices are $$ \kappa_1=1,\qquad \kappa_2=\sqrt{\frac{1.49}{1.49}}=1,\qquad \kappa_3=\sqrt{\frac{1.49}{0.02}} =\sqrt{74.5} =8.631338\ldots \approx 8.6313. $$

The smallest eigenvalue corresponds to the direction proportional to $(1,1,1)^\mathsf{T}$. Thus, the sum $Z_1+Z_2+Z_3$ varies little. Each pair has a moderate negative correlation, yet all three together have a near-dependence. The eigenvalue sum provides a check: $1.49+1.49+0.02=3$, equal to the trace of $G$.

Check and common mistake. The diagonal entries are all 1, so the trace is 3; the calculated eigenvalues satisfy this. Checking only whether any single pair is extremely correlated would miss the three-variable combination. Use the eigenvector of the small eigenvalue to identify the direction.

Practical Interpretation and Common Traps

A condition index is a diagnostic, not a universal pass-or-fail test. Larger values indicate more imbalance between predictor directions, but the interpretation depends on the scaling, model, and purpose of the analysis. A small eigenvalue and its eigenvector are useful clues; they do not by themselves tell whether a predictor should be removed or how a model should be changed.

Several details matter when calculating and interpreting the indices:

Scale the predictor columns. A change from dollars to thousands of dollars should not create a spurious diagnosis. Centering and scaling the predictor columns makes the comparison about their relationships rather than their measurement units.
Keep the intercept separate. For diagnosing dependence among predictors, exclude the intercept from the scaled predictor matrix. Otherwise, its scale and the predictor means can affect the eigenstructure being inspected.
Use the square root. The condition index is $\sqrt{\lambda_1/\lambda_j}$. The raw ratio is not the condition index.
Inspect the eigenvector. The smallest eigenvalue indicates a direction, not a particular pair of variables. Read the corresponding eigenvector to see which predictors contribute and with what relative signs.
Distinguish exact from near dependence. A zero eigenvalue means exact dependence among the columns in the matrix being analyzed. A positive but small eigenvalue indicates a near-dependence, with severity judged in context.

For a scaled matrix with unit-length columns, the eigenvalues sum to $p$, the number of predictors, because their sum equals the matrix trace. This is a useful arithmetic check, as in the three-predictor example. It does not establish that the eigenvalues or eigenvectors have been interpreted correctly, so also verify the eigenvalue ordering and the direction associated with the smallest eigenvalue.

Key takeaway: Order the eigenvalues of a scaled predictor matrix from largest to smallest, calculate $\sqrt{\lambda_1/\lambda_j}$ for each, and use the eigenvector of any small eigenvalue to identify the near-dependent predictor combination. The indices measure relative direction lengths; they do not name the variables involved.

Check Your Understanding

For each problem, use eigenvalues in descending order and report condition indices to four decimal places when needed.

A scaled design matrix for a hypothetical mortality study has eigenvalues $3.0,\ 0.75,\ 0.03$. Calculate all condition indices and identify the direction with the largest index. (Answer: $1.0000,\ 2.0000,\ 10.0000$; the direction associated with $0.03$.)
Two standardized predictors in a hypothetical claims model have correlation $0.90$. Find the eigenvalues and condition indices, and state whether the sum or difference direction has less variation. (Answer: Eigenvalues $1.90,\ 0.10$; indices $1.0000,\ 4.3589$; the difference direction has less variation.)
A scaled predictor matrix has eigenvalues $2.2,\ 0.7,\ 0.1$. Calculate the three condition indices. (Answer: $1.0000,\ 1.7728,\ 4.6904$.)
Three standardized predictors have the same pairwise correlation, $-0.45$. Find the eigenvalues and indices, and identify the small-eigenvalue combination. (Answer: Eigenvalues $1.45,\ 1.45,\ 0.10$; indices $1.0000,\ 1.0000,\ 3.8079$; the sum of the three predictors is the small-eigenvalue direction.)
A researcher reports eigenvalues $4,\ 1,\ 0.04$ for a scaled matrix and claims the last condition index is $100$. Correct the claim and explain the error. (Answer: The last index is $\sqrt{4/0.04}=\sqrt{100}=10.0000$; $100$ is the unrooted eigenvalue ratio, not the condition index.)

What You'll Learn