Why a Small Residual Can Hide an Important Case
A point can sit close to a fitted regression line and still have an unusual predictor value. That combination matters: because the point is far from the predictor center, it can help determine the line’s slope or position. If removing it substantially changes the fitted model, the point is influential, even though its residual in the full-data fit is modest.
The essential distinction is between where an observation lies in predictor space and how far its response lies from the fitted value. Leverage describes the first; the residual describes the second. Influence concerns how much the fitted model changes when the observation is omitted. A high leverage value alone does not establish influence, and a modest residual alone does not rule it out.
For simple linear regression with an intercept and one predictor, \(p=2\), where \(p\) is the number of fitted coefficients. The leverage at predictor value \(x_i\) is \(h_{ii}=1/n+(x_i-\bar{x})^2/S_{xx}\), where \(S_{xx}=\sum_{j=1}^n(x_j-\bar{x})^2\). The average leverage is \(p/n=2/n\). Thus, a predictor value far from \(\bar{x}\) has higher leverage. This geometric fact does not depend on the observed responses.
What Deleting a High-Leverage Case Reveals
The following results apply to ordinary least squares with a full-rank design matrix. For standard regression diagnostics, the usual model assumes \(Y=X\beta+\varepsilon\), with errors having mean zero and constant variance; independence is also commonly assumed. Normal errors are needed for exact small-sample \(t\)-based inference, but not for calculating leverage or the OLS deletion identities below.
A useful deletion identity connects the residual, leverage, and fitted value at the deleted observation:
Here, \(\hat{y}_{i(-i)}\) is the prediction at \(x_i\) from a model fitted after observation \(i\) is omitted. For \(h_{ii}<1\), the identity shows why a high leverage value can magnify the effect of a nonzero residual: the change is proportional to \(h_{ii}/(1-h_{ii})\). When \(h_{ii}\) is close to one, that multiplier can be large even if \(e_i\) is not.
Record the fitted value, residual, and leverage for the case under review.
Compare the leverage with the average \(p/n\) and, if useful, a stated diagnostic guideline. Treat a guideline as a screening aid, not a deletion rule.
Consider the residual relative to the residual standard error and the remaining model assumptions.
Examine how fitted values, coefficients, or a measure such as Cook’s distance change when the case is omitted.
For a model with \(p\) fitted coefficients and residual mean square \(MSE\), Cook’s distance for case \(i\) is
This expression combines residual size and leverage. It is not a probability, and its magnitude should be interpreted in context. Diagnostic cutoffs are prompts for investigation, not proof that a case is erroneous or should be removed. If \(h_{ii}=1\), the case cannot be assessed by ordinary leave-one-out fitting in the usual way: deleting it makes the fitted design rank-deficient or leaves no residual information at that predictor direction.
Worked Example: A High-Leverage Point With a Modest Residual
A hypothetical pricing analyst fits simple linear regression of annual service cost \(y\), in thousands of dollars, on a policy feature \(x\). The six records are \((x,y)=(0,2),(1,6),(2,9),(3,12),(4,10),(10,33)\). Identify the leverage and residual for the last record, then assess its influence. Use OLS with an intercept.
Method and model fit. The data have \(\bar{x}=20/6=10/3\), \(\sum x_i^2=130\), and \(S_{xx}=130-20^2/6=190/3\). The responses can be written as \(y_i=2+3x_i+e_i\), with deviations \(0,1,1,1,-4,1\). Their sum is \(0\), and their predictor-weighted sum is \(0+1+2+3-16+10=0\). Therefore the OLS intercept and slope are exactly \(2\) and \(3\). The last fitted value is \(2+3(10)=32\), so its residual is \(33-32=1\).
Leverage. For the last record,
The average leverage is \(p/n=2/6=1/3\), so this case has high leverage relative to the average. Its predictor value is far from the predictor mean; its leverage does not say that its response is unusual.
Residual scale and influence. The residual sum of squares is \(0^2+1^2+1^2+1^2+(-4)^2+1^2=20\). With \(n-p=4\) residual degrees of freedom, \(MSE=20/4=5\). The internally studentized residual is \(1/\{\sqrt{5}\sqrt{1-33/38}\}=\sqrt{38}/5\approx1.2329\), which is not large by common screening conventions.
Cook’s distance is
The case-deletion identity gives \(\hat{y}_i-\hat{y}_{i(-i)}=(33/38)(1)/(5/38)=6.6\). Thus the model fitted without this case predicts \(32-6.6=25.4\) at \(x=10\), a substantial shift from the full-data prediction. Directly, the deleted-model coefficients are an intercept of \(3.4\) and slope of \(2.2\), which give \(3.4+2.2(10)=25.4\). The check agrees with the deletion identity.
Interpretation and trap avoided. The last point is far from the predictor center, but its residual is only \(1\) thousand dollars relative to a residual standard error of \(\sqrt{5}\). Yet deleting it changes the fitted line substantially. The mistake would be to call it harmless because it lies close to the full-data line; the correction is to assess its deletion effect as well as its residual.
Reading Leverage and Influence Together
The example illustrates an important feature of fitted regression lines: a high-leverage observation can pull the line toward itself. Its residual is calculated after that pull has already happened. Consequently, the residual in the full-data fit may understate how different the fitted value would be without the case. This does not mean every high-leverage point is influential. If its residual is exactly zero, the deletion identity gives no change in the fitted value at that point, and the coefficient deletion formula likewise gives no coefficient change.
Worked Example: A Small Residual and a Large Deletion Shift
A hypothetical regression has \(n=5\) observations and two coefficients, including the intercept. The predictor values are \(1,2,3,4,9\). For the case at \(x=9\), the fitted-model output gives residual \(e_i=0.4\) and \(MSE=4\). Calculate its leverage, internally studentized residual, and change in prediction when it is omitted.
Method. The predictor mean is \(19/5\), and \(S_{xx}=111-19^2/5=194/5\). Therefore
The residual standard error is \(\sqrt{MSE}=2\). The internally studentized residual is \(0.4/\{2\sqrt{1-87/97}\}=0.2\sqrt{97/10}\approx0.6229\). It is modest on this scale. However, the fitted-value change on deletion is
The deleted model’s prediction at \(x=9\) is therefore 3.48 response units below the full-model fitted value. This calculation uses the residual from the full model and the leverage of that same case; reversing the sign would instead describe deleted prediction minus full prediction. The arithmetic checks because the deletion multiplier is \(h_{ii}/(1-h_{ii})=87/10=8.7\), and \(8.7(0.4)=3.48\).
Interpretation and trap avoided. The point has very high leverage and a small studentized residual, yet its deletion changes the prediction considerably. Do not infer low influence from a small residual without considering how close leverage is to one.
Worked Example: Comparing Cook’s Distance for Two Cases
A hypothetical regression with \(p=2\) coefficients and \(MSE=4\) has two cases under review. Case A has leverage \(0.9\) and residual \(0.3\); case B has leverage \(0.2\) and residual \(1.2\). Compare their Cook’s distances.
Method. For case A,
For case B,
The ratio is \(D_A/D_B=1.0125/0.05625=18\). Thus case A has eighteen times the Cook’s distance, even though its raw residual is smaller. The comparison is internally consistent: the same \(p\) and \(MSE\) apply to both cases, and the difference comes from their residuals and leverages together.
Interpretation and trap avoided. Ranking cases by raw residual alone would make case B seem more influential. Cook’s distance also accounts for leverage, so it identifies case A as having the greater overall effect on the fitted model. This comparison is diagnostic, not a decision to delete either case.
Common Mistakes and Practical Interpretation
- Calling leverage a response outlier measure. Leverage comes from the predictors, not from the observed response. A point can have high leverage and a small residual.
- Equating high leverage with influence. Leverage creates the potential for influence. The residual and the resulting deletion effect also matter.
- Using the residual alone to judge influence. The full-data residual is measured after the observation has helped determine the fitted line. Check a deletion diagnostic when predictor leverage is high.
- Treating a cutoff as an automatic deletion rule. A large diagnostic value signals a case to investigate. Check data entry, measurement, model specification, and the case’s substantive context before deciding how to proceed.
In applied work, retaining a valid observation may be more appropriate than deleting it, even when it is influential. It can be useful to report how key conclusions change with and without that case, while explaining why the case is present. The central question is not simply whether a diagnostic crosses a threshold; it is whether the fitted model and resulting conclusions are robust enough for the intended use.
This treatment is aligned with the SOA Exam SRM (Statistics for Risk Modeling) syllabus.
Check Your Understanding
For each problem, use the stated diagnostic definitions and keep the direction of any deletion difference clear.
- A simple linear regression with an intercept has \(n=10\) observations. What is its average leverage? A case has leverage \(0.45\). Is its leverage above average? (Answer: Average leverage \(=2/10=0.20\); yes.)
- A case has \(h_{ii}=0.75\) and full-model residual \(e_i=-0.6\). Calculate \(\hat{y}_i-\hat{y}_{i(-i)}\). (Answer: \(-1.8\).)
- A regression has \(p=3\), \(MSE=9\), and a case with \(h_{ii}=0.6\), \(e_i=1.5\). Calculate Cook’s distance. (Answer: \(0.3125\).)
- Two cases in the same regression have \(p=2\) and the same \(MSE\). Case A has \(h=0.7,e=0.2\); case B has \(h=0.1,e=1\). Which Cook’s distance is larger, and by what factor? (Answer: Case A’s is larger by a factor of \(2.52\).)
- A case has \(h_{ii}=0.75\), \(e_i=0.5\), and \(MSE=4\). Calculate its internally studentized residual \(e_i/\{\sqrt{MSE}\sqrt{1-h_{ii}}\}\). (Answer: \(0.5\).)