Multicollinearity and Variance Inflation Factors

Why Predictor Overlap Matters

In multiple regression, each slope describes the association between a predictor and the response holding the other predictors fixed. That qualification becomes difficult to use when predictors tend to move together. For example, a model might include both weekly exercise time and weekly activity-session count. If these predictors largely convey the same information, the data may contain little evidence about what would happen to the response as one changes while the other stays fixed.

This situation is called multicollinearity. It can make individual coefficient estimates imprecise even when the model’s predictions or overall fit look strong. A variance inflation factor (VIF) measures how much the overlap among predictors increases the variance of one coefficient estimate relative to a design in which that predictor is uncorrelated with the other predictors, with its own spread unchanged.

Definition: For predictor $x_j$, regress its observed values on all the other predictors in the multiple-regression model, including an intercept. Let $R_j^2$ be the coefficient of determination from this auxiliary regression. The variance inflation factor for $x_j$ is $$ \mathrm{VIF}_j=\frac{1}{1-R_j^2}. $$ A larger $R_j^2$ means the other predictors explain more of the variation in $x_j$, and therefore its VIF is larger. If $R_j^2=1$, the predictor is an exact linear combination of the others, and its individual coefficient cannot be estimated uniquely.

What a VIF Measures

A VIF belongs to a predictor, not to the response model as a whole. To obtain it, treat one predictor in turn as the response in an auxiliary regression and use the remaining predictors as explanatory variables. The auxiliary regression’s $R_j^2$ is not the $R^2$ for predicting the original response.

For example, $\mathrm{VIF}_j=4$ means the estimated variance of the coefficient for $x_j$ is four times the variance it would have if $x_j$ had the same spread but were uncorrelated with the other predictors. Since standard errors are square roots of variances, its standard error is twice as large. A VIF does not say that the coefficient itself is multiplied, nor does it measure how much the response changes.

The standard interpretation follows from the usual multiple linear regression model. For the variance calculation, assume the model is linear in its coefficients, the errors have mean zero, constant variance $\sigma^2$, and are uncorrelated across observations. Treat the predictor values as fixed (or interpret the calculation conditional on the observed predictors). The predictor columns must not have exact linear dependence; otherwise the individual coefficients are not separately identifiable. Normal errors are additionally used for the usual exact $t$-based inference in small samples, but are not needed for the variance-inflation identity itself.

Worked Example: Calculating a VIF From an Auxiliary Regression

Suppose a hypothetical model predicts a clinic’s average wait time from three predictors, including the number of appointments scheduled per day. In the auxiliary regression for appointments per day using the other two predictors, suppose $R_j^2=0.75$. Find the VIF and explain what it means.

Substitute the auxiliary $R_j^2$ into the definition:

\mathrm{VIF}_j=\frac{1}{1-0.75} =\frac{1}{0.25} =4.

Thus, the other predictors explain 75% of the observed variation in appointments per day. The variance of its regression-coefficient estimate is four times the variance for a predictor with the same spread but no linear association with the other predictors. Its standard error is inflated by the square root of four:

\sqrt{\mathrm{VIF}_j}=\sqrt{4}=2.

If that comparison design would have given a standard error of 0.20 minutes per appointment, the VIF-adjusted standard error is $0.20\times2=0.40$ minutes per appointment. The VIF describes the predictor relationships in the design; it does not, by itself, tell us the coefficient estimate or whether the predictor is scientifically useful.

Why the Standard Error Is Inflated

The underlying reason can be seen by separating a predictor’s values into the part explained by the other predictors and the part that remains. In the auxiliary regression for $x_j$, write each centered observed value as its fitted value plus a residual. The residuals are the portion of $x_j$ not linearly explained by the other predictors. In ordinary least squares, the coefficient for $x_j$ is identified by this remaining variation: if almost none remains, there is little information for distinguishing its contribution from theirs.

Let $SST_j=\sum_i(x_{ij}-\bar{x}_j)^2$ be the total centered sum of squares for predictor $j$, and let $u_{ij}$ be the residual from its auxiliary regression. By the definition of $R_j^2$, the residual sum of squares is $\sum_i u_{ij}^2=(1-R_j^2)SST_j$. The least-squares coefficient can be expressed using these residualized predictor values. Because the auxiliary residuals are orthogonal to the other predictors, the conditional variance of that coefficient is $\sigma^2$ divided by their sum of squares:

\operatorname{Var}(\hat{\beta}_j\mid X) =\frac{\sigma^2}{\sum_i u_{ij}^2} =\frac{\sigma^2}{(1-R_j^2)SST_j}.

If the same predictor had the same $SST_j$ but were uncorrelated with the other predictors, its auxiliary $R_j^2$ would be zero. Its coefficient variance in that comparison would be $\sigma^2/SST_j$. Dividing the actual variance by this comparison variance gives

\frac{\sigma^2/[(1-R_j^2)SST_j]}{\sigma^2/SST_j} =\frac{1}{1-R_j^2} =\mathrm{VIF}_j.

This establishes why a VIF multiplies the variance, while $\sqrt{\mathrm{VIF}_j}$ multiplies the standard error. In practice, $\sigma^2$ is estimated from the regression’s residual variation; the VIF factor describes the additional uncertainty due to the predictor geometry.

Worked Example: Comparing VIFs Across Predictors

In a hypothetical regression of daily energy use on building area, number of rooms, and outdoor temperature, the auxiliary regression for area has $R_j^2=0.90$. The auxiliary regression for temperature has $R_j^2=0.20$. Calculate and compare their VIFs.

For area:

\mathrm{VIF}_{\text{area}} =\frac{1}{1-0.90} =\frac{1}{0.10} =10.

For temperature:

\mathrm{VIF}_{\text{temperature}} =\frac{1}{1-0.20} =\frac{1}{0.80} =1.25.

The standard-error multipliers are $\sqrt{10}\approx3.162$ for area and $\sqrt{1.25}\approx1.118$ for temperature. Area is much more predictable from the other predictors, so its separate coefficient is less precisely estimated than it would be without that overlap. The temperature coefficient has comparatively little inflation from linear overlap. These comparisons do not establish whether either predictor should be included; that depends on the model’s purpose and scientific context.

Diagnosing Unstable Individual Estimates

Multicollinearity is often suspected when the predictors are strongly related and individual coefficient estimates have large standard errors relative to their magnitudes. A coefficient may then have a wide confidence interval or a $t$-statistic near zero, even when the model’s $R^2$ is substantial. This is not a contradiction: a model can predict the response well using information shared by several predictors without estimating each predictor’s distinct partial association precisely.

VIFs are one diagnostic, not a universal pass-or-fail test. There is no single cutoff that makes a model acceptable or unacceptable in every setting. A VIF of 5 corresponds to $R_j^2=0.80$, while a VIF of 10 corresponds to $R_j^2=0.90$; these values flag substantial predictability from other predictors, but the practical consequences depend on the precision needed and the scientific question. A modest VIF can matter when precise individual effects are essential, while a larger VIF may be tolerable when the goal is prediction and the model performs adequately on relevant data.

Worked Example: Interpreting a Large VIF and an Uncertain Coefficient

A hypothetical regression uses 50 observations and three predictors to estimate a response. The fitted model has $R^2=0.82$. For predictor $x_1$, the auxiliary regression has $R_1^2=0.9375$, and the coefficient table reports $\hat{\beta}_1=0.30$ response units per unit of $x_1$. Suppose its standard error is 0.25 response units per unit of $x_1$. Interpret these results.

First calculate the VIF and its standard-error multiplier:

\mathrm{VIF}_1=\frac{1}{1-0.9375} =\frac{1}{0.0625} =16, \qquad \sqrt{\mathrm{VIF}_1}=4.

The other predictors explain 93.75% of the observed variation in $x_1$. The estimated standard error is therefore four times the comparison standard error for a predictor with the same spread but no linear association with the other predictors. The corresponding comparison standard error is $0.25/4=0.0625$ response units per unit of $x_1$.

The coefficient’s $t$-statistic for testing $H_0:\beta_1=0$ is

t=\frac{\hat{\beta}_1-0}{SE} =\frac{0.30}{0.25} =1.20.

With an intercept and three predictors, the residual degrees of freedom are $50-3-1=46$. A two-sided $t$-test with $t=1.20$ and $df=46$ has a p-value of about 0.236. This result does not provide strong evidence against $H_0$ at common significance levels. It is possible for the response model’s $R^2$ to be 0.82 while this individual coefficient is imprecise: the overall fit and the ability to separate $x_1$’s partial association from the other predictors are different questions.

What to Do When VIFs Are Large

A large VIF is a reason to investigate, not an instruction to delete a predictor. Start by checking which variables are involved and whether their overlap is expected. For example, a model may include two measures of a closely related construct because each has a meaningful role. Removing one without considering the research question can change what the model is designed to estimate.

Clarify the goal. If the goal is prediction, assess predictive performance as well as coefficient precision. If the goal is interpreting separate associations, substantial overlap may directly limit the conclusions.
Review the predictors. Check units, data collection, and whether two measures are redundant or represent distinct concepts that happen to be related.
Consider the model specification carefully. A theoretically justified choice of predictors or a meaningful combined measure may help, but should not be chosen only to make a VIF smaller.
Report the uncertainty. Describe coefficient estimates and their standard errors or intervals; do not present an unstable point estimate as precise evidence.

Changing units by multiplying a predictor by a constant changes the numerical scale of its coefficient but does not remove its linear predictability from the other predictors, so it does not change its VIF. Likewise, a high VIF does not show that the regression assumptions about the errors are violated. It concerns the relationships among predictor columns and the resulting precision of individual coefficient estimates.

Common Mistakes

Using the response model’s $R^2$ as a predictor’s VIF. The VIF uses $R_j^2$ from a separate auxiliary regression in which predictor $x_j$ is the response.
Calling a VIF of 4 a fourfold increase in standard error. It is a fourfold variance increase; the standard error is multiplied by $\sqrt{4}=2$.
Treating a cutoff as a universal decision rule. A large VIF signals potential imprecision, but whether it matters depends on the model’s purpose and required precision.
Assuming high VIFs mean the whole model is useless. They directly concern individual coefficient precision; a model may still predict well.
Assuming a nonsignificant coefficient proves there is no association. With substantial overlap, the estimate may be too imprecise to distinguish its partial association clearly from zero.

Key takeaway: For predictor $x_j$, $\mathrm{VIF}_j=1/(1-R_j^2)$, where $R_j^2$ comes from regressing that predictor on the other predictors. Its square root is the standard-error inflation factor. Use VIFs to understand coefficient precision and interpret them in light of the model’s purpose, not as automatic deletion rules.

Check Your Understanding

Answer each question using the definitions and relationships in this tutorial.

An auxiliary regression for a predictor has $R_j^2=0.60$. Calculate its VIF and standard-error inflation factor.
Why can a regression have a high $R^2$ for its response model but an imprecise estimate for one individual coefficient?
A VIF is 9. By what factor is the coefficient variance inflated, and by what factor is its standard error inflated relative to the stated comparison design?
Which regression’s $R^2$ is used to calculate the VIF for $x_j$: the original response model or the auxiliary regression for $x_j$? Explain.
Why is it not generally correct to remove every predictor with a VIF above a chosen cutoff?