Deriving the Variance Inflation Factor

Why Regress One Predictor on the Others?

In multiple linear regression, a coefficient describes the association between the response and one predictor while the other included predictors are held fixed. If that predictor is strongly explained by the other predictors, the data contain relatively little independent variation with which to estimate its separate coefficient. The variance inflation factor (VIF) measures this loss of information relative to a benchmark in which the predictor is uncorrelated with the others.

The key calculation is an auxiliary regression: regress the predictor of interest on all the other predictors. Its \(R^2\) measures how much of the predictor’s variation is explained by those other predictors. The VIF is the reciprocal of the unexplained fraction. The derivation below shows why that reciprocal also multiplies the coefficient’s variance.

Model and assumptions: Consider the fixed-design multiple linear regression model \(y=\beta_0\mathbf{1}+\sum_{k=1}^{p}\beta_k x_k+\varepsilon\), with an intercept. Conditional on the predictor values, assume \(E[\varepsilon]=0\) and \(\operatorname{Var}(\varepsilon)=\sigma^2 I\): errors have common variance \(\sigma^2\) and are uncorrelated. For the usual normal-theory inference, errors are also normally distributed. The predictors must have enough independent variation for the coefficients to be estimable. The standard variance formula derived here does not apply unchanged when errors are heteroskedastic or correlated.

For a selected predictor \(x_j\), its auxiliary regression includes an intercept and every other predictor \(x_k\), \(k\ne j\), as explanatory variables. Write its coefficient of determination as \(R_j^2\). With an intercept, this \(R_j^2\) compares the auxiliary regression’s residual sum of squares with the centered total sum of squares of \(x_j\). Define \(SST_j=\sum_{i=1}^n(x_{ij}-\bar{x}_j)^2\).

Definition: The variance inflation factor for predictor \(x_j\) is \(VIF_j=1/(1-R_j^2)\), provided \(R_j^2<1\). It is the factor by which the variance of the estimated coefficient \(\hat{\beta}_j\) exceeds the variance it would have if \(x_j\) were orthogonal to the other predictors, with its own centered sum of squares and the error variance held fixed.

Deriving the Coefficient Variance

The derivation isolates the part of \(x_j\) that cannot be linearly predicted from the other predictors. Let \(Z\) be the design matrix containing an intercept and all predictors except \(x_j\). Let \(P_Z\) be the projection matrix onto the columns of \(Z\), and let \(M_Z=I-P_Z\). Regressing \(x_j\) on \(Z\) leaves residual vector \(r=M_Zx_j\). By construction, \(r\) is orthogonal to every column of \(Z\).

Remove the variation explained by the other predictors.
The residualized predictor is \(r=M_Zx_j\). Because \(Z\) includes an intercept, the residuals sum to zero. Its squared length \(r'r\) is the auxiliary regression residual sum of squares.

Express the coefficient using that residual.
After removing the contribution of \(Z\) from both the response and \(x_j\), the least-squares coefficient is \(\hat{\beta}_j=(r'y)/(r'r)\). The residual \(r\) is orthogonal to \(Z\), so the other predictors’ terms in \(y\) contribute zero to \(r'y\).

Calculate its conditional variance.
Under the stated error model, \(r'y=\beta_j r'r+r'\varepsilon\), and \(\operatorname{Var}(r'\varepsilon\mid X)=\sigma^2r'r\). Dividing by \((r'r)^2\) gives \(\operatorname{Var}(\hat{\beta}_j\mid X)=\sigma^2/(r'r)\).

To finish, relate \(r'r\) to the auxiliary \(R^2\). The intercept-only fit to \(x_j\) has total sum of squares \(SST_j\), while the full auxiliary regression has residual sum of squares \(r'r\). Thus \(R_j^2=1-r'r/SST_j\), or \(r'r=(1-R_j^2)SST_j\). Substitution gives the coefficient-variance formula:

\operatorname{Var}(\hat{\beta}_j\mid X) =\frac{\sigma^2}{r'r} =\frac{\sigma^2}{SST_j(1-R_j^2)} =\underbrace{\frac{\sigma^2}{SST_j}}_{\text{orthogonal-predictor benchmark}} \underbrace{\frac{1}{1-R_j^2}}_{VIF_j}.

The benchmark follows from the same formula when the selected predictor is orthogonal to the other predictors after accounting for the intercept. Then its auxiliary \(R^2\) is zero, so \(r'r=SST_j\). This shows that the VIF is a variance ratio, not a standard-error ratio. The corresponding standard error is multiplied by \(\sqrt{VIF_j}\).

Worked Calculations

Worked Example: Two Correlated Rating Predictors

A hypothetical insurer fits a multiple regression for annual claim cost using centered vehicle age \(x_1\) and centered annual mileage \(x_2\), with an intercept. Treat the predictor values as fixed and assume independent errors with common variance \(\sigma^2=4\) cost-units squared. The centered cross-products are \(x_1'x_1=40\), \(x_2'x_2=25\), and \(x_1'x_2=30\). Find the VIF for vehicle age and the conditional variance of its coefficient.

Method. In the auxiliary regression of \(x_1\) on an intercept and \(x_2\), the fitted centered values are the projection of \(x_1\) onto \(x_2\). Its explained sum of squares is \((x_1'x_2)^2/(x_2'x_2)=30^2/25=36\).

The total centered sum of squares is \(SST_1=40\). Therefore \(R_1^2=36/40=0.90\), and the auxiliary residual sum of squares is \(40-36=4\). The VIF is \(1/(1-0.90)=10\).

The coefficient variance is \(\sigma^2/r'r=4/4=1\) cost-units squared. The orthogonal-predictor benchmark is \(4/40=0.10\); multiplying by the VIF gives \(0.10(10)=1\), as required. The standard error is \(\sqrt{1}=1\) cost-unit, compared with benchmark standard error \(\sqrt{0.10}\approx0.3162\) cost-unit. Check: the variance ratio is \(1/0.10=10\), exactly the VIF. This calculation avoids confusing the variance multiplier with the standard-error multiplier.

Worked Example: Multiple Predictors in the Auxiliary Regression

In a hypothetical regression of claim frequency on exposure-related measures, a selected predictor \(x_2\) has centered total sum of squares \(SST_2=100\). Regressing \(x_2\) on an intercept and all the other predictors gives auxiliary residual sum of squares 36. Assume the original model’s errors are uncorrelated with common variance \(\sigma^2=9\) frequency-units squared. Find \(R_2^2\), \(VIF_2\), the coefficient variance, and its standard error.

Method. The auxiliary \(R^2\) is one minus the ratio of auxiliary residual sum of squares to \(SST_2\): \(R_2^2=1-36/100=0.64\). The unexplained fraction is \(1-R_2^2=0.36\), so \(VIF_2=1/0.36=25/9\approx2.7778\).

Using the auxiliary residual sum of squares directly, the coefficient variance is \(9/36=0.25\) frequency-units squared. The benchmark is \(9/100=0.09\); indeed \(0.09(25/9)=0.25\). The standard error is \(\sqrt{0.25}=0.50\) frequency-units. The benchmark standard error is \(\sqrt{0.09}=0.30\), and the ratio \(0.50/0.30=5/3=\sqrt{25/9}\). Check: squaring the standard-error ratio gives \(25/9\), the variance inflation factor.

Worked Example: Recovering VIF from Centered Cross-Products

A hypothetical model for the processing time of insurance applications includes centered predictors \(x_1\) and \(x_2\), along with an intercept. Their cross-products are \(x_1'x_1=36\), \(x_2'x_2=25\), and \(x_1'x_2=24\). Assume the regression errors are uncorrelated with common variance \(\sigma^2=9\) time-units squared. Find the VIF for \(x_1\), its coefficient variance, and its standard error.

Method. The auxiliary regression has one explanatory predictor, \(x_2\). The explained sum of squares for \(x_1\) is \(24^2/25=576/25=23.04\). Hence \(R_1^2=23.04/36=0.64\), leaving residual sum of squares \(36-23.04=12.96\). The VIF is \(1/(1-0.64)=25/9\approx2.7778\).

The coefficient variance is \(9/12.96=25/36\approx0.6944\) time-units squared. As a check, the benchmark is \(9/36=0.25\), and \(0.25(25/9)=25/36\). The standard error is \(\sqrt{25/36}=5/6\approx0.8333\) time-units. Without inflation it would be \(0.50\) time-units, and \(0.8333/0.50\) is approximately \(\sqrt{25/9}\). Check: both calculations give the same coefficient variance.

Conditions, Boundaries, and Common Traps

For an estimable predictor, \(0\le R_j^2<1\), so \(VIF_j\ge1\). A VIF of 1 means the predictor is orthogonal to the other predictors after accounting for the intercept. As \(R_j^2\) approaches 1, the unexplained predictor variation shrinks and the coefficient variance increases. If \(R_j^2=1\), then \(r'r=0\): the predictor is exactly a linear combination of the other predictors and the coefficient is not separately estimable. The formula’s denominator is zero; this is not a finite VIF attached to an estimable coefficient.

The formula uses the ordinary auxiliary \(R^2\) from a regression that includes an intercept. Omitting the intercept changes the auxiliary regression and its sum-of-squares convention, so one must not apply the centered-\(SST_j\) derivation automatically to a no-intercept model. Also, if errors are heteroskedastic or correlated, the stated OLS coefficient variance need not hold, even though the auxiliary \(R^2\) can still be calculated.

A frequent exam error is to use the unexplained fraction as the VIF. For example, if \(R_j^2=0.80\), the wrong calculation is \(VIF_j=1-R_j^2=0.20\). That quantity is the fraction of predictor variation left unexplained, not the inflation factor. The correction is \(VIF_j=1/(1-0.80)=5\). Thus the coefficient variance is five times its benchmark, while its standard error is multiplied by \(\sqrt{5}\), not by 5.

Key takeaway: Regress \(x_j\) on the other predictors to find how much independent variation remains. With an intercept and the stated homoskedastic, uncorrelated-error model, \(\operatorname{Var}(\hat{\beta}_j\mid X)=\sigma^2/[SST_j(1-R_j^2)]\). The reciprocal unexplained fraction is the VIF, the coefficient-variance multiplier relative to the orthogonal-predictor benchmark.

Check Your Understanding

For each problem, identify the auxiliary-regression information needed before calculating the VIF.

A centered predictor has \(SST_j=50\), and its auxiliary regression on the other predictors has \(R_j^2=0.36\). If the original model has \(\sigma^2=9\), calculate the VIF and coefficient variance. (Answer: \(VIF_j=1.5625\); \(\operatorname{Var}(\hat{\beta}_j\mid X)=0.28125\).)
A predictor has centered total sum of squares 120 and auxiliary residual sum of squares 30. With \(\sigma^2=25\), calculate its auxiliary \(R^2\), VIF, and coefficient variance. (Answer: \(R_j^2=0.75\); \(VIF_j=4\); variance \(=5/6\approx0.8333\).)
Two centered predictors have \(x_1'x_1=64\), \(x_2'x_2=100\), and \(x_1'x_2=40\). For \(x_1\), calculate the auxiliary \(R^2\), VIF, and coefficient variance if \(\sigma^2=9\). (Answer: \(R_1^2=0.25\); \(VIF_1=4/3\); variance \(=3/16=0.1875\).)
A centered predictor has \(SST_j=200\), auxiliary \(R_j^2=0.90\), and the regression error variance is 4. Find the coefficient variance and standard error, then compare the standard error with its orthogonal-predictor benchmark. (Answer: variance \(=0.20\); standard error \(\approx0.4472\); benchmark standard error \(\approx0.1414\), so the standard-error multiplier is \(\sqrt{10}\).)
An auxiliary regression gives \(R_j^2=0.96\), with \(SST_j=75\) and \(\sigma^2=12\). Calculate the VIF and coefficient variance. What changes if the auxiliary \(R^2\) is exactly 1? (Answer: \(VIF_j=25\); variance \(=4\); at \(R_j^2=1\), the coefficient is not separately estimable because of exact collinearity.)

What You'll Learn