Large Residual With Low Leverage

Why a Large Residual Does Not Automatically Mean High Influence

A data point can be surprising in its outcome without being unusual in its predictor values. For example, a clinic might have a typical staffing level for the clinics in a data set but a much longer-than-expected wait time. In a regression, this case may have a large residual while having low leverage. Those are different features of the observation, and neither one alone tells the whole story about its effect on the fitted model.

The distinction matters because a large residual describes the vertical discrepancy between an observed response and a fitted response. Leverage describes how unusual the predictor values are relative to the predictor values used to fit the model. Influence describes how much the fitted model changes when a case is removed. A case with ordinary predictors can be far from the fitted line, yet the fitted line may change only modestly when that case is omitted.

Model and Key Relationships

For the calculations below, assume an ordinary least-squares regression with an intercept and \(p\) fitted coefficients. In simple linear regression, the model for the conditional mean is \(E(Y_i\mid x_i)=\beta_0+\beta_1x_i\). The standard regression model assumes a correctly specified conditional mean, independent errors with mean zero and constant variance, and a full-rank predictor design. Normal errors are additionally used for exact small-sample inference, but are not needed for the algebraic identities below.

Leverage is determined by the predictor values, not by the observed responses. In simple linear regression, it has the form

Thus a predictor value near \(\bar{x}\) has leverage near \(1/n\), the minimum possible value in this setting. In a regression with \(p\) coefficients and an intercept, the average leverage is \(p/n\). “Low” should therefore be understood relative to the design and the other cases; it is not a judgment about the response.

A residual can be standardized to account for its estimated scale and leverage. The internally studentized residual is

where \(s=\sqrt{\text{SSE}/(n-p)}\) is the residual standard error from the full fit. A large raw residual may or may not be large on this standardized scale. Also, because \(s\) is estimated using all cases, an extreme residual can itself increase \(s\); the externally studentized deleted residual instead estimates the error scale after omitting the case.

How Leverage Limits a Case’s Fitted-Value Change

The exact case-deletion identity makes the leverage-residual distinction concrete. Let \(\hat{\beta}\) be the full-sample coefficient vector and \(\hat{\beta}_{(i)}\) the coefficient vector after case \(i\) is omitted. Write \(\mathbf{x}_i\) for the predictor vector of case \(i\), including its intercept term. Provided that deleting the case leaves the design full rank, the deleted residual \(e_{(i)}=y_i-\mathbf{x}_i^\mathsf{T}\hat{\beta}_{(i)}\) satisfies

Here is a justification. The full-sample normal equations are \(X^\mathsf{T}X\hat{\beta}=X^\mathsf{T}y\). Removing case \(i\) subtracts \(\mathbf{x}_i\mathbf{x}_i^\mathsf{T}\) from the left-hand design cross-product and subtracts \(\mathbf{x}_iy_i\) from the right-hand side. Subtracting the deleted normal equations from the full ones gives

Multiplying by \((X^\mathsf{T}X)^{-1}\) and then by \(\mathbf{x}_i^\mathsf{T}\) shows that the full-fit prediction at case \(i\) minus the deleted-fit prediction there is \(h_i e_{(i)}\). Since \(e_i=e_{(i)}-h_i e_{(i)}\), rearranging yields the identity. In particular,

When \(h_i\) is low, this change in the fitted value at the case’s own predictor position is a relatively small fraction of its residual. That does not guarantee that every coefficient or every prediction changes negligibly, but it explains why a large vertical discrepancy need not pull the fit strongly. Influence also depends on the model’s scale and the direction of the predictor vector.

Worked Examples

What to Conclude—and What Not to Conclude

A large residual at low leverage is evidence that the observed response is far from the fitted conditional mean at an ordinary predictor position. It can flag an unusual outcome, a data-recording problem, omitted structure in the model, or ordinary random variation. The residual alone cannot distinguish among these explanations. Nor does it show that the observation is invalid or should be deleted.

The effect on the fitted model depends on more than whether the residual looks large. Leverage enters the exact fitted-value change, while the model’s residual scale affects standardized diagnostics and Cook’s distance. A careful interpretation therefore keeps three questions separate:

• Is the outcome unusual? Inspect the residual in context and, where appropriate, on a standardized scale.
• Are the predictors unusual? Assess leverage from the predictor design, not from the response.
• Does the case change the fit? Examine a suitable influence measure or compare the full fit with the case-deleted fit.

A common mistake is to reason, “This point has the largest residual, so it must control the regression.” That confuses vertical discrepancy with influence. The correction is to check the predictor leverage and a measure of fitted-model change as well. The reverse mistake is also possible: “Its leverage is low, so it cannot matter.” A sufficiently large residual can still produce meaningful influence, so low leverage is not a guarantee of negligible effect.