Deriving Variance Inflation from an Auxiliary Regression

Why an Auxiliary Regression Explains Coefficient Uncertainty

In multiple regression, a coefficient describes the association between the outcome and one predictor while the other predictors are held fixed. If the predictor of interest is largely predictable from those other predictors, there may be little independent variation left to distinguish its contribution. The coefficient can then be estimated less precisely, even if the outcome model and sample size are unchanged.

An auxiliary regression makes that overlap measurable: regress the predictor of interest on the other predictors, and examine the auxiliary regression’s \(R^2\). This tutorial derives how that \(R^2\) determines the variance inflation factor for the coefficient.

Model and Assumptions

Consider a multiple linear regression with an intercept and \(p\) predictors, observed for \(n\) cases:

We focus on the coefficient \(\beta_j\) of predictor \(X_j\). For the variance derivation, treat the observed predictor values as fixed, or equivalently condition on them. Assume the conditional mean is correctly represented by the linear model, the errors have mean zero, and their variances are all \(\sigma^2\), with zero covariance between distinct cases. Also assume the design matrix has full column rank, so no predictor is an exact linear combination of the intercept and the other predictors. These conditions give an identifiable least-squares coefficient and the variance formula derived below. Normal errors are not needed for this variance result.

The auxiliary regression for predictor \(X_j\) includes an intercept and all the other predictors:

Let \(R_j^2\) be the usual coefficient of determination from this auxiliary regression. It is the fraction of the centered sum of squares of \(X_j\) explained by its linear relationship with the other predictors. It is not the \(R^2\) from the regression of the outcome \(Y\) on all predictors.

Deriving the Variance Formula

The key quantity is the part of \(X_j\) that remains after its linear relationship with the other predictors has been removed. Let \(Z\) be the design matrix containing the intercept and all predictors except \(X_j\). Let \(P_Z\) be the projection matrix onto the columns of \(Z\), and define \(M_Z=I-P_Z\). Applying \(M_Z\) to a vector removes its fitted component from the regression on \(Z\), leaving its residual component.

Write \(x_j\) for the vector of observed values of predictor \(X_j\), and \(y\) for the outcome vector. After removing from both \(x_j\) and \(y\) the components explained by \(Z\), the least-squares coefficient of \(X_j\) is:

This expression is the one-predictor least-squares slope after accounting for all the columns of \(Z\). Since \(M_ZZ=0\), the terms in the outcome model involving the intercept and the other predictors disappear when multiplied by \(M_Z\). Thus \(M_Zy=\beta_jM_Zx_j+M_Z\varepsilon\), giving:

Conditional on the predictors, the denominator is fixed. Under the stated error assumptions, the variance of \(x_j'M_Z\varepsilon\) is \(\sigma^2x_j'M_Zx_j\): \(M_Z\) is symmetric and idempotent, so \(M_ZM_Z=M_Z\). Dividing by the squared denominator yields:

The denominator \(x_j'M_Zx_j\) is the residual sum of squares from the auxiliary regression. With an intercept, the total centered sum of squares for \(X_j\) is \(S_{jj}=\sum_{i=1}^n(X_{ij}-\bar{x}_j)^2\). By the definition of \(R_j^2\), the auxiliary residual sum of squares is \(S_{jj}(1-R_j^2)\). Substituting this identity gives the result:

If \(X_j\) were orthogonal to the other predictors, the auxiliary regression would explain none of its centered variation, so \(R_j^2=0\). Its coefficient variance would then be \(\sigma^2/S_{jj}\). The ratio of the general variance to that reference variance is \(1/(1-R_j^2)\), the variance inflation factor. Because a standard error is the square root of a variance, the corresponding standard-error multiplier is \(\sqrt{1/(1-R_j^2)}\), not \(1/(1-R_j^2)\).

What the Auxiliary \(R^2\) Tells Us

The formula separates two influences on coefficient variance. The error variance \(\sigma^2\) describes noise in the outcome model, while \(S_{jj}\) measures the total centered spread in the predictor of interest. The factor \(1/(1-R_j^2)\) describes how much of that predictor spread is unavailable as independent variation once the other predictors are accounted for.

As \(R_j^2\) increases toward 1, the residual variation \(S_{jj}(1-R_j^2)\) approaches zero, and the coefficient variance grows. If \(R_j^2=1\), the predictor is exactly explained by the other predictors in the auxiliary regression. Then \(x_j'M_Zx_j=0\), the coefficient cannot be separated from the other coefficients, and the variance formula has a zero denominator. This is exact collinearity, not merely a large but finite variance.

A larger variance inflation factor does not by itself say that the coefficient is biased or that the overall model predicts poorly. It describes a loss of precision for this coefficient under the model and design assumptions. The size of a coefficient’s variance also depends on the predictor’s scale and spread, so the inflation factor is a relative comparison rather than a stand-alone variance.

A Common Mistake

A frequent error is to say, “The auxiliary \(R^2\) is 0.80, so the coefficient variance is inflated by 80%.” That treats the explained fraction as the inflation factor. The derivation shows the correct relationship is:

When \(R_j^2=0.80\), the variance inflation factor is \(1/0.20=5\), not \(0.80\) or \(1.80\). The standard-error multiplier is then \(\sqrt{5}\), because standard error is the square root of variance. Another important distinction is that the \(R^2\) in this formula comes from predicting \(X_j\) using the remaining predictors, not from predicting \(Y\).