Using VIF to Diagnose Collinearity

Why Collinearity Matters

In multiple regression, a coefficient describes the change in the fitted conditional mean associated with changing one predictor while holding the other included predictors fixed. If a predictor is closely related to the others, the data may contain little information about that isolated change. The coefficient can then be estimated imprecisely, even when the model's overall predictions are reasonably stable.

A variance inflation factor, or VIF, measures this loss of precision from the relationships among the predictors. It is a property of the predictor design, not of the response values. A large VIF signals that the individual coefficient may have a large standard error relative to what it would have under an otherwise comparable design with no linear redundancy.

Definition: Consider a linear regression with an intercept and $p$ predictors $X_1,\ldots,X_p$. For predictor $X_j$, regress its observed values on an intercept and all the other predictors. Let $R_j^2$ be the coefficient of determination from this auxiliary regression. The variance inflation factor for $X_j$ is $$ \operatorname{VIF}_j=\frac{1}{1-R_j^2}. $$ The auxiliary regression uses $X_j$ as its response; it is not the original regression of the outcome on the predictors.

With an intercept, $R_j^2$ is between 0 and 1. If $R_j^2=0$, the other predictors do not linearly explain $X_j$, and $\operatorname{VIF}_j=1$. As $R_j^2$ approaches 1, the VIF grows without bound. If $R_j^2=1$, there is exact linear dependence: the coefficient for $X_j$ cannot be separately estimated in the usual multiple regression.

From VIF to Coefficient Standard Error

To connect VIF to uncertainty, suppose the data follow the linear model $Y=X\beta+\varepsilon$, with a full-rank design matrix that includes an intercept. For the classical variance formula, assume $E(\varepsilon\mid X)=0$, $\operatorname{Var}(\varepsilon\mid X)=\sigma^2 I$, and a correctly specified linear conditional mean. The equal-variance and uncorrelated-error assumptions give the usual conditional covariance matrix $\sigma^2(X^\mathsf{T}X)^{-1}$. The VIF describes a part of that variance attributable to the predictor relationships.

\operatorname{Var}(\hat{\beta}_j\mid X) =\frac{\sigma^2}{SST_j(1-R_j^2)} =\operatorname{VIF}_j\frac{\sigma^2}{SST_j}, \qquad SST_j=\sum_{i=1}^n(x_{ij}-\bar{x}_j)^2.

Here, $SST_j$ measures the total centered variation in predictor $X_j$. The comparison $\sigma^2/SST_j$ is the coefficient variance that would result if $X_j$ had the same total variation and were orthogonal to the other predictors. Therefore, holding that variation and the error variance fixed, VIF multiplies the coefficient variance; its square root multiplies the standard error.

Remove the other predictors.
Regress $X_j$ on the other predictors, including an intercept, and let $z_j$ be the residual vector. It contains the part of $X_j$ not linearly explained by those predictors.

Measure the remaining variation.
For a regression with an intercept, the residual sum of squares in that auxiliary regression is $z_j^\mathsf{T}z_j=(1-R_j^2)SST_j$.

Relate it to coefficient variance.
The Frisch–Waugh–Lovell result says the coefficient on $X_j$ is obtained by regressing $Y$ on this residualized predictor. Under the stated error assumptions, its conditional variance is $\sigma^2/(z_j^\mathsf{T}z_j)$, giving the displayed VIF formula.

The key reasoning in the last step can be seen directly. Residuals $z_j$ are orthogonal to the intercept and the other predictors. Thus, residualizing $Y$ against those same terms does not change its inner product with $z_j$. In the model, the other predictors' contributions also have zero inner product with $z_j$, leaving $z_j^\mathsf{T}Y=\beta_j z_j^\mathsf{T}z_j+z_j^\mathsf{T}\varepsilon$. The resulting estimate has variance $\operatorname{Var}(z_j^\mathsf{T}\varepsilon\mid X)/(z_j^\mathsf{T}z_j)^2=\sigma^2/(z_j^\mathsf{T}z_j)$, as claimed.

Interpretation: A VIF of $c$ means that, under the classical variance comparison above, the coefficient variance is $c$ times the orthogonal-design comparison. The standard error is multiplied by $\sqrt{c}$, not by $c$. VIF alone does not give the coefficient's standard error; the error variance and predictor variation also matter.

Worked VIF Calculations

Worked Example: Translating an Auxiliary R-Squared into a Standard Error

A hypothetical insurer models the log of annual claim expense using several rating predictors. In the auxiliary regression for a particular predictor, the reported $R_j^2$ is 0.84. An otherwise comparable orthogonal-design calculation gives a standard error of 0.040 for its coefficient. Find the VIF and the corresponding standard error under the observed predictor relationships.

Method. Apply the VIF definition to the auxiliary $R_j^2$, then multiply the comparison standard error by the square root of VIF.

First calculate the unexplained fraction of the predictor variation: $1-R_j^2=1-0.84=0.16$.

Then $\operatorname{VIF}_j=1/0.16=6.25$. The standard-error multiplier is $\sqrt{6.25}=2.5$.

The corresponding standard error is $0.040(2.5)=0.100$. Thus the coefficient variance is 6.25 times the orthogonal-design comparison, while the standard error is 2.5 times as large. Check: squaring the standard-error multiplier gives $2.5^2=6.25$, the calculated VIF. Common mistake avoided: multiplying the standard error by 6.25 would confuse the variance multiplier with the standard-error multiplier.

Worked Example: Two Correlated Predictors and a Coefficient Test

A hypothetical pricing analysis includes two standardized predictors whose sample correlation is 0.80. For one coefficient, the standard error in the orthogonal-design comparison is 0.060, and the estimated coefficient is 0.30. Calculate the VIF, the adjusted standard error under the classical variance comparison, and the resulting $t$-statistic for testing that the coefficient is zero.

Method. With just two predictors, the auxiliary regression of either predictor on the other has $R_j^2=r^2$. This relationship holds because the auxiliary regression includes an intercept and the predictors are standardized.

The auxiliary $R^2$ is $0.80^2=0.64$. Therefore, $\operatorname{VIF}_j=1/(1-0.64)=1/0.36=25/9\approx2.7778$. The standard-error multiplier is $\sqrt{25/9}=5/3\approx1.6667$.

The adjusted standard error is $0.060(5/3)=0.100$. The test statistic is $t=0.30/0.100=3.00$. For comparison, using the orthogonal-design standard error would give $0.30/0.060=5.00$. The coefficient estimate itself has not been multiplied or divided by VIF; the change here is to its estimated uncertainty. Check: the adjusted standard error is $1.6667$ times the comparison value, and its squared ratio is approximately 2.7778. Common mistake avoided: using the original regression's response $R^2$ instead of the auxiliary predictor $R^2$.

Worked Example: Multiple Predictors in an Auxiliary Regression

In a hypothetical risk model, three standardized predictors have pairwise correlations $r_{12}=0.60$, $r_{13}=0.60$, and $r_{23}=0$. For the coefficient on $X_1$, an orthogonal-design standard error comparison is 0.080. Find the auxiliary $R_1^2$, the VIF, and the corresponding standard error.

Method. Regress standardized $X_1$ on standardized $X_2$ and $X_3$, with an intercept. Because $X_2$ and $X_3$ are uncorrelated, their regression coefficients are their separate correlations with $X_1$: both are 0.60. The fitted value is therefore $0.60X_2+0.60X_3$.

Each standardized predictor has variance 1, and $X_2$ and $X_3$ have covariance 0. The variance of the fitted value is $0.60^2(1)+0.60^2(1)+2(0.60)(0.60)(0)=0.36+0.36=0.72$. Since $X_1$ has variance 1, the auxiliary coefficient of determination is $R_1^2=0.72$. The supplied correlations are internally consistent: the determinant of their correlation matrix is $1-0.36-0.36=0.28>0$.

The VIF is $1/(1-0.72)=1/0.28=25/7\approx3.5714$. The standard-error multiplier is $\sqrt{25/7}=5/\sqrt{7}\approx1.8898$. Thus the corresponding standard error is $0.080(1.8898)\approx0.1512$, rounded to four decimal places. Check: $(0.1512/0.080)^2$ is approximately 3.57, consistent with the VIF subject to rounding. Common mistake avoided: using only the largest pairwise correlation. The auxiliary $R^2$ accounts for the combined linear explanation from all the other predictors.

What VIF Does—and Does Not—Diagnose

VIF diagnoses linear redundancy among included predictors. It does not test whether a predictor is statistically significant, measure the response model's overall fit, or establish whether a chosen model is appropriate. The auxiliary regression deliberately uses one predictor as its response. A high $R_j^2$ in that regression can coexist with a low or high $R^2$ in the original response regression.

Under the stated conditional-mean assumption $E(\varepsilon\mid X)=0$, collinearity does not by itself bias ordinary least-squares coefficient estimates. It reduces the predictor variation available to identify an individual coefficient, which can increase its sampling variance and standard error. Large standard errors can make tests less precise and coefficient estimates more sensitive to changes in the data. If the conditional-mean assumption fails—for example, because of an omitted relevant factor—bias can occur, but a high VIF alone does not establish that failure.

The usual exact variance multiplier relies on the classical homoskedastic, uncorrelated-error variance formula. With heteroskedasticity or correlated errors, a robust or otherwise appropriate covariance estimate may be needed; the VIF remains a description of predictor redundancy, but it does not generally act as an exact multiplier for that alternative standard error. A VIF also has no universal cutoff that automatically decides whether a predictor must be removed. Its practical importance depends on the inferential goal, the coefficient's standard error, the available predictor variation, and the rest of the model.

For a model with an intercept, exact collinearity gives $R_j^2=1$, so the usual VIF is undefined as a finite quantity and the full coefficient vector is not identifiable. Near-exact collinearity produces a large but finite VIF. If a model deliberately omits an intercept, the auxiliary regression and its $R^2$ convention must be handled consistently; the centered-$SST_j$ formula used here assumes an intercept.

Common Exam Traps

Using the wrong $R^2$. Use the auxiliary regression's $R_j^2$, where $X_j$ is regressed on the other predictors, not the original outcome model's $R^2$.
Inflating the standard error by VIF itself. VIF multiplies variance; the standard-error multiplier is $\sqrt{\operatorname{VIF}_j}$.
Assuming pairwise correlations tell the whole story. A predictor may be explained by a combination of several others even when no single pairwise correlation is extreme.
Calling collinearity bias. Under the correct conditional-mean model, collinearity primarily increases uncertainty; it does not itself make least-squares estimates biased.
Treating a rule of thumb as a decision rule. A threshold can flag a case for review, but it is not a mathematical boundary between acceptable and unacceptable models.

Key takeaway: Calculate $\operatorname{VIF}_j=1/(1-R_j^2)$ from the auxiliary regression for $X_j$. Interpret its square root as the standard-error multiplier under the classical variance comparison, and remember that collinearity concerns precision—not, by itself, bias.

Check Your Understanding

For each item, use the auxiliary $R^2$ and the stated standard-error comparison where provided.

An auxiliary regression has $R_j^2=0.75$. Find the VIF and standard-error multiplier. (Answer: VIF $=4$; standard-error multiplier $=2$.)
An auxiliary regression has $R_j^2=0.91$. An orthogonal-design standard error is 0.045. Find the VIF, standard-error multiplier, and corresponding standard error. (Answer: VIF $\approx11.1111$; multiplier $\approx3.3333$; standard error $=0.150$.)
A model has two standardized predictors with correlation $-0.50$. For either predictor, find the auxiliary $R^2$, VIF, and standard-error multiplier. (Answer: auxiliary $R^2=0.25$; VIF $=4/3\approx1.3333$; multiplier $\approx1.1547$.)
For one predictor in a three-predictor model, the auxiliary regression has $R_j^2=0.64$, and the comparison standard error is 0.12. Find the VIF and corresponding standard error. (Answer: VIF $=25/9\approx2.7778$; corresponding standard error $=0.20$.)
A predictor has $R_j^2=0.96$ in its auxiliary regression, and its comparison standard error is 0.020. Find the VIF and corresponding standard error. Does the VIF alone prove that the least-squares coefficient is biased under $E(\varepsilon\mid X)=0$? (Answer: VIF $=25$; corresponding standard error $=0.100$; no, VIF alone does not prove bias.)

What You'll Learn