Why High Leverage Need Not Mean High Influence
An observation far from the center of the predictor values can have substantial potential to affect a regression fit. That potential is called leverage. But potential is not the same as an actual change: if the observation lies close to the fitted line, its residual may be small, and deleting it may change the fitted regression only slightly.
This distinction matters when reviewing diagnostic plots. A point at an extreme predictor value deserves attention because it has high leverage, but leverage alone does not establish that the point has strongly changed the estimated line. Conversely, a small residual does not prove that deletion has no effect: leverage can amplify the effect of even a modest residual. The relevant question is how the two features combine, and what the deletion diagnostics show.
What a Small Residual Does—and Does Not—Tell Us
Leverage describes where an observation sits in predictor space; a residual describes its vertical distance from the fitted line in response units. These are different properties. In simple linear regression with an intercept, the leverage at predictor value \(x_i\) is
A predictor value far from \(\bar{x}\) has greater leverage. The average hat value is \(p/n\), because the hat values sum to \(p\); an unusually large value relative to the others signals an unusual predictor position. No single cutoff turns leverage into a verdict about influence.
For a full-rank least-squares model, let \(\hat{y}_{i(i)}\) denote the fitted response at observation \(i\)'s predictor values after observation \(i\) has been omitted. When \(h_{ii}<1\), the leave-one-out identities are
The first quantity is the deleted residual: the observed response minus the prediction from the fit that left the observation out. The second is the signed change in prediction, defined here as deleted-fit prediction minus full-fit prediction. Its sign depends on the sign of \(e_i\); its magnitude is \(h_{ii}|e_i|/(1-h_{ii})\). Thus, a small residual can yield a small deletion change, but the factor \(1/(1-h_{ii})\) grows when leverage approaches 1.
Why the Deletion Formula Has This Form
The relationship between leverage and the deleted fit follows from the least-squares normal equations. Let \(X\) be the design matrix, let \(\boldsymbol{x}_i\) be the column vector of predictors for observation \(i\) (including the intercept column if the model has an intercept), and let \(C=(X^\mathsf{T}X)^{-1}\). The full-fit coefficient vector is \(\hat{\boldsymbol{\beta}}\), and the deleted-fit vector is \(\hat{\boldsymbol{\beta}}_{(i)}=\hat{\boldsymbol{\beta}}+\boldsymbol{d}\). The hat value is \(h_{ii}=\boldsymbol{x}_i^\mathsf{T}C\boldsymbol{x}_i\).
Subtract the full-data normal equations from the equations for the data with case \(i\) deleted. Since the omitted case contributes \(\boldsymbol{x}_i y_i\) to \(X^\mathsf{T}\boldsymbol{y}\) and \(\boldsymbol{x}_i\boldsymbol{x}_i^\mathsf{T}\) to \(X^\mathsf{T}X\), the difference gives
For \(h_{ii}<1\), set \(\boldsymbol{d}=-C\boldsymbol{x}_i e_i/(1-h_{ii})\). Substitution verifies this solution: multiplying by the matrix on the left gives
At observation \(i\)'s predictor values, the prediction change is \(\boldsymbol{x}_i^\mathsf{T}\boldsymbol{d}=-h_{ii}e_i/(1-h_{ii})\), as stated. This is an algebraic result for least squares; it does not require normally distributed errors.
Worked Examples: High Leverage and a Very Small Residual
Worked Example: A Distant Predictor Value Close to the Fitted Line
Consider the following hypothetical simple linear regression data. The response is in points, and the predictor is measured in hours.
| \(x_i\) (hours) | \(y_i\) (points) | Residual from the full fit (points) |
|---|---|---|
| 0 | 21.04 | 1.04 |
| 1 | 19.95 | -2.05 |
| 2 | 25.00 | 1.00 |
| 5 | 30.01 | 0.01 |
The proposed full-data line is \(\hat{y}=20+2x\). Its fitted values at \(x=0,1,2,5\) are 20, 22, 24, and 30 points, giving the residuals shown. They sum to zero, and their predictor-weighted sum is \(0(1.04)+1(-2.05)+2(1)+5(0.01)=0\). These are the least-squares orthogonality conditions for a model with an intercept, so the line is indeed the full-data least-squares fit.
The predictor mean is \(\bar{x}=2\), and \(\sum (x_j-\bar{x})^2=14\). For the observation at \(x_i=5\),
This is the largest hat value in this data set: the other three are \(0.5357\), \(0.3214\), and \(0.2500\). Thus the point at \(x=5\) is far from the predictor center and has high leverage relative to the other observations. Yet its residual is only \(0.01\) point.
The residual sum of squares is \(1.04^2+(-2.05)^2+1^2+0.01^2=6.2842\) points squared. There are \(n-p=4-2=2\) residual degrees of freedom, so \(MSE=6.2842/2=3.1421\) points squared. The deleted residual at \(x=5\) is
The full-fit prediction there is \(30.00\) points; the deleted-fit prediction is \(30.01-0.0933\approx29.9167\) points. The signed prediction change is
The negative sign is consistent with the actual predictions: deleting the case lowers the prediction from \(30.00\) to about \(29.9167\) points. The change is small relative to the response and residual scales here, despite the high leverage.
Cook’s Distance Combines Residual Size and Leverage
A coefficient-based summary of deletion change is Cook’s distance. For a least-squares model with \(p\) coefficients, full-model mean squared error \(MSE\), residual \(e_i\), and hat value \(h_{ii}\), it is
The residual is scaled by the model’s residual variation, and the leverage factor can magnify the result. Consequently, the word “small” should be judged in context: a residual’s size in response units alone does not determine whether it has a substantial effect. Cook’s distance is a diagnostic summary, not a formal test or an automatic instruction to remove a point.
Worked Example: Checking Influence With Cook’s Distance
Use the hypothetical data and full fit from the previous example. There are \(p=2\) coefficients, \(MSE=3.1421\) points squared, \(e_i=0.01\) point, and \(h_{ii}=0.8929\). Substituting into Cook’s distance gives
For clarity, the two factors are approximately \(0.0000159\) and \(77.8\), whose product is about \(0.00124\). The high leverage substantially multiplies the scaled residual contribution, but the contribution remains small because the residual is tiny relative to the overall residual variation. This supports the conclusion from the deleted prediction: this high-leverage point has little influence on the fit by this diagnostic.
Worked Example: An Exactly Zero Residual
Consider a second hypothetical data set with \(x=0,1,2,5\) and \(y=21,20,25,30\), respectively. The line \(\hat{y}=20+2x\) gives residuals \(1,-2,1,0\). They sum to zero, and their predictor-weighted sum is \(0(1)+1(-2)+2(1)+5(0)=0\), confirming that this is the least-squares line.
The point at \(x=5\) has the same leverage as in the preceding four-point example:
Its residual is exactly zero, so the leave-one-out identities give a deleted residual of \(0/(1-0.8929)=0\) and a deleted-minus-full prediction change of \(-0.8929(0)/(1-0.8929)=0\) points. Its Cook’s distance is also zero, since its squared residual is zero. Here, deleting the point leaves the least-squares line unchanged. This exact case illustrates the principle particularly clearly: high leverage indicates potential, while the residual determines whether this observation pulls the fitted line away from the fit to the other observations.
Interpreting the Pattern Without Overreaching
A high-leverage point close to the fitted line can be reassuring in one sense: its small residual means it agrees with the line fitted to the full data, and deletion diagnostics may show only minor changes. But agreement with the fitted line does not establish that the observation is correct, representative, or generated by the same process as the others. Check the data and the study context before drawing a substantive conclusion.
Also distinguish “small residual” from “no possible influence.” When \(h_{ii}\) is very close to 1, the factor \(1/(1-h_{ii})\) can be large. A nonzero residual that looks modest in isolation can then lead to a meaningful deleted residual or prediction change. Examine a suitable influence summary, such as Cook’s distance, and consider the scale and context of the changes rather than relying on leverage or residual size alone.
- High leverage, small residual, small deletion changes: the point is unusual in predictor space but may have little influence on the fitted model.
- High leverage, non-negligible residual: the combination can produce larger deletion changes, especially when \(h_{ii}\) is near 1.
- Small residual in response units: this is not enough by itself to judge influence; compare it with residual variation and inspect deletion diagnostics.
Check Your Understanding
Use the definitions and deletion relationships in this tutorial to answer the following questions.
- In simple linear regression with an intercept, why does a predictor value far from \(\bar{x}\) tend to have greater leverage?
- A point has a small residual and high leverage. What additional information would help you judge whether it is influential?
- Suppose \(h_{ii}=0.80\) and \(e_i=0.10\) response units. Calculate the deleted residual and the deleted-minus-full prediction change, including their signs.
- Why is it incorrect to call high leverage proof that an observation has strongly changed the fitted line?
- If an observation has an exactly zero residual and \(h_{ii}<1\), what do the deletion identities imply about its deleted residual and its prediction change at its own predictor values?