Leverage and Residual Variance

Why Residual Size Depends on Leverage

A residual is the difference between an observed response and its fitted value. It is natural to view a large residual as evidence that an observation is unusual, but the same-sized residual does not have the same scale at every predictor value. In ordinary least-squares regression, a case with high leverage has a smaller residual variance than a case with low leverage, under the usual constant-variance model. Comparing raw residuals alone can therefore be misleading.

The key is to distinguish two questions. A residual describes the discrepancy between an observation and the fitted regression. Its variance describes how much that discrepancy would vary across repeated samples under a specified model. Leverage affects the second quantity: the fitted value is especially responsive to the observed response at high-leverage predictor values, leaving less variation in the residual itself.

Model and Assumptions

To derive the result, write the regression model in matrix form as \(y=X\beta+\varepsilon\). Here \(y\) is the vector of responses, \(X\) is the design matrix of predictor values (including a column of ones if the model has an intercept), and \(\beta\) is the vector of regression coefficients. The number of columns of \(X\), assumed linearly independent, is \(p\). The error vector is \(\varepsilon\).

The derivation conditions on the observed predictor values. It requires the errors to have conditional mean zero and common variance, with no correlation between errors:

Here \(I\) is the identity matrix. These conditions state that, given the predictors, the response follows the specified linear mean model, and the errors are uncorrelated with equal variance \(\sigma^2\). Normal errors are not needed for the variance derivation. The result is not guaranteed if the mean model is misspecified or the errors have unequal variances or are correlated.

The hat matrix is \(H=X(X^\mathsf{T}X)^{-1}X^\mathsf{T}\), so the fitted response vector is \(\hat{y}=Hy\). Define \(M=I-H\). Then the residual vector is \(e=y-\hat{y}=My\). The leverage \(h_{ii}\) is the diagonal entry \(H_{ii}\); under the usual full-rank least-squares setup, \(0\le h_{ii}\le1\).

Deriving the Residual Variance

Because \(MX=0\), the fitted linear mean contributes nothing to the residual vector:

The covariance matrix of a linear transformation \(M\varepsilon\), conditional on \(X\), is \(M\operatorname{Var}(\varepsilon\mid X)M^\mathsf{T}\). The matrix \(H\) is symmetric and idempotent, so \(M=I-H\) is also symmetric and idempotent: \(M^\mathsf{T}=M\) and \(M^2=M\). Substituting the error covariance gives:

The variance of residual \(e_i\) is the \(i\)th diagonal entry of this covariance matrix. Since the \(i\)th diagonal entry of \(I-H\) is \(1-h_{ii}\), the desired result follows:

The standard deviation is the square root of the variance. As leverage increases, \(1-h_{ii}\) decreases, so the residual’s standard deviation decreases too. For example, if \(h_{ii}=0.75\), the residual standard deviation is \(\sigma\sqrt{0.25}=0.5\sigma\), not \(\sigma\). This does not mean the observation is less influential; leverage and residual variance describe different features of a case.

The residuals are also generally correlated with one another: the off-diagonal entries of \(\operatorname{Var}(e\mid X)\) are \(-\sigma^2h_{ij}\) for \(i\ne j\). The diagonal result concerns each residual’s variance, not an assumption that residuals are independent.

Using an Estimated Error Variance

In practice, \(\sigma^2\) is usually unknown. Under the stated model, a common unbiased estimator is the residual mean square:

The denominator \(n-p\) accounts for the \(p\) fitted coefficients. Replacing \(\sigma^2\) by \(\hat{\sigma}^2\) gives an estimated residual variance \(\hat{\sigma}^2(1-h_{ii})\), and an estimated residual standard deviation \(\hat{\sigma}\sqrt{1-h_{ii}}\). These are estimated scales, not known population quantities.

Why Raw Residual Comparisons Can Mislead

A raw residual is measured in the response’s units and remains useful for describing the actual prediction error in the fitted sample. But residuals from different leverage levels do not have the same conditional variance under the model. A residual of 5 units might be modest at low leverage and comparatively large at high leverage, depending on the estimated error scale.

For a fairer scale comparison, compare each residual with its own estimated residual standard deviation, rather than using the same denominator for every case. The quantity \(e_i/[\hat{\sigma}\sqrt{1-h_{ii}}]\) makes that adjustment. It is a scale-adjusted residual; further residual diagnostics define and study particular standardized versions. The adjustment alone does not account for every issue in diagnosing unusual observations, and it should not be treated as a universal cutoff rule.

Conditions and Common Pitfalls

The leverage adjustment follows from the OLS fit and the assumed error covariance. If the error variances differ across observations, then \(\operatorname{Var}(\varepsilon\mid X)\) is not \(\sigma^2I\), and the simple formula \(\sigma^2(1-h_{ii})\) does not generally apply. Likewise, a wrong mean model can leave systematic structure in the residuals; scaling by leverage does not repair that problem.

• Comparing only raw residuals. A larger raw residual is not automatically more unusual when the cases have different leverages.
• Assuming high leverage means a large residual. Leverage describes the predictor location and its role in the fit. The observed residual can be small or large.
• Assuming a small high-leverage residual is reassuring on its own. High leverage reduces the residual’s expected variance, so a modest raw residual can still be sizeable relative to its scale.
• Using the formula under unequal error variance without qualification. The result requires common conditional error variance and uncorrelated errors, as well as a correctly specified linear mean.
• Confusing a variance with a standard deviation. The residual variance is \(\sigma^2(1-h_{ii})\); the residual standard deviation is \(\sigma\sqrt{1-h_{ii}}\).

The practical lesson is not to replace the residual with leverage or to judge a case from one number. Use residuals to describe response discrepancies, and account for the leverage-dependent residual scale when comparing their magnitudes. This gives a more defensible starting point for regression diagnostics.