When One Observation Shapes the Line
A regression line can look convincing at a glance and still describe much of the data poorly. One reason is that an observation far from the others in its predictor value can pull the fitted line toward itself. The line may then pass relatively close to that observation while leaving a systematic pattern of residuals among the rest.
This problem is easiest to miss if attention is focused only on the unusual observation’s residual. Residuals must be read alongside the scatterplot, the residual pattern, and the distribution of predictor values. A point can have a modest residual in the full fit precisely because it helped determine that fit.
A response outlier is an observation with a response that is far from the fitted line, as assessed by its residual or a related diagnostic. A high-leverage point, by contrast, is unusual in the horizontal direction of a scatterplot. These are different properties. An observation can have high leverage but a small residual, a large residual but ordinary leverage, both, or neither.
Leverage and the Pull of an Extreme Predictor
For a least-squares line with an intercept, the leverage of observation \(i\) is
The first term is shared by all observations. The second grows as \(x_i\) moves farther from \(\bar{x}\). Thus, observations near the ends of the predictor range tend to have more leverage than observations near its center. Leverage depends only on the predictor values, not on the responses.
A high-leverage observation can strongly affect the line because least squares chooses its slope and intercept to minimize the total squared residuals. Moving the line toward a point at an extreme \(x\)-value can change its predictions across the predictor range. The point’s own residual may consequently be smaller than it would be under a fit based on the other observations.
Leverage is a diagnostic, not a verdict. Rules of thumb for flagging leverage vary, especially with small samples, and crossing a threshold does not establish that an observation is harmful. The key question here is whether the full-data line gives an adequate description across the data, particularly among observations with ordinary predictor values.
Worked Example: A High-Leverage Point Hides a Poor Fit
Consider hypothetical data with predictor values \(x=0,1,2,3,4,10\) and responses \(y=0,0,0,0,0,10\). The first five responses are all zero; the last observation is far from the others in \(x\). Calculate the least-squares line and examine what the residuals show.
The needed sums are \(n=6\), \(\sum x_i=20\), \(\sum y_i=10\), \(\sum x_i^2=130\), and \(\sum x_i y_i=100\). Therefore,
The slope \(b_1\) and intercept \(b_0\) are
So the fitted line is \(\hat{y}=-35/19+(20/19)x\). Its residuals \(e_i=y_i-\hat{y}_i\), in order of increasing \(x\), are
For example, at \(x=10\) the fitted value is \(-35/19+200/19=165/19\), so its residual is \(10-165/19=25/19\), about \(1.316\) response units. That residual may not look dramatic beside the full-data fit. But the first five residuals change systematically from positive to negative, and the line predicts values ranging from about \(-1.842\) to \(2.368\) for observations whose responses are all zero. The fit among those five observations is poor.
The leverage of the point at \(x=10\) is
This is much larger than the leverage of an observation near the center of the \(x\)-values. The scatterplot would show the same story visually: the extreme point helps pull the line upward, while the five observations at ordinary \(x\)-values do not follow that rising line.
Why the High-Leverage Point Can Look Well Fitted
The connection between leverage and the residual can be made precise by comparing the full-data fit with a fit that omits one observation. Write \(\hat{y}_{i,(-i)}\) for the fitted prediction at \(x_i\) from the regression fitted without observation \(i\). If the full design is of full rank and deleting observation \(i\) leaves a full-rank design, then \(h_{ii}<1\), and
The left side is the residual for observation \(i\) against the line fitted without it. It can be much larger in magnitude than the full-data residual when \(h_{ii}\) is close to 1. In other words, the full-data fit can look close to a high-leverage point partly because that point helped set the line.
A short justification shows where the factor comes from. Let \(X\) be the design matrix, with an intercept column, and let \(x_i^\mathsf{T}\) be the row for observation \(i\). The full-data fitted values are \(X\hat{\beta}\), and the leverage is \(h_{ii}=x_i^\mathsf{T}(X^\mathsf{T}X)^{-1}x_i\). The least-squares coefficient update after deleting observation \(i\) is
This update follows from the least-squares normal equations for the full and deleted fits; the denominator is defined under the stated full-rank deletion condition. Multiplying by \(x_i^\mathsf{T}\) gives the change in the prediction at \(x_i\):
Since \(e_i=y_i-\hat{y}_i\), subtracting the deleted-fit prediction from \(y_i\) yields
This relation is not a claim that every high-leverage point has a small residual. Rather, for a given full-data residual, a high \(h_{ii}\) means the discrepancy from the fit without that observation can be larger. If deleting a point makes the design rank deficient, the deleted-fit coefficients are not uniquely determined and this identity must not be used.
Worked Example: Comparing the Full and Deleted Fits
Use the six observations from the previous example. The full-data residual at \(x=10\) is \(e_i=25/19\), and its leverage is \(h_{ii}=33/38\). Deleting this observation leaves five distinct \(x\)-values, so a line with an intercept can still be fitted with full rank.
The identity gives the residual against the deleted-fit prediction:
Indeed, the first five observations all have response zero, so their fitted line is \(\hat{y}=0\). At \(x=10\), that line predicts zero, and the omitted observation’s response is \(10\); its deleted-fit residual is \(10-0=10\). The full-data residual, about \(1.316\), is far smaller because the observation pulled the line toward itself. This comparison makes the masking effect visible without treating the full-data residual as the whole story.
Use the Scatterplot and Residual Plot Together
Residual plots are useful for detecting structure that a numerical summary can conceal. In this setting, inspect residuals against the predictor value, or against fitted values, while also checking where the observations lie in the original scatterplot. A pattern among the ordinary predictor values can signal poor fit even when an extreme point has a modest residual.
Look for residuals that vary systematically with \(x\), such as a sequence that stays positive at one end and negative at the other, or a curved pattern. Such structure indicates that a straight line may not capture the relationship over that region. Do not infer curvature merely from one extreme residual or from the presence of a high-leverage point; assess the pattern across several observations and in context.
Worked Example: High Leverage Does Not Automatically Mean Poor Fit
Consider a different hypothetical data set with the same predictor values, \(0,1,2,3,4,10\), but responses given by \(y=5+2x\): \(5,7,9,11,13,25\). The point at \(x=10\) still has high leverage because leverage depends only on the \(x\)-values.
Here \(\sum y_i=70\) and \(\sum x_i y_i=360\). Using \(S_{xx}=190/3\) from the predictor values,
Thus \(\hat{y}=5+2x\), and every residual is zero. The leverage at \(x=10\) remains \(33/38\approx0.8684\), but this particular high-leverage point does not conceal poor fit: all observations follow the same line. The conclusion depends on the residual pattern and the scatterplot, not on leverage alone.
A Practical Diagnostic Sequence
Inspect the scatterplot and the distribution of \(x\). A point separated from the main group horizontally is a candidate for high leverage, not automatically a problem.
Look for systematic residual behavior among the observations in the main predictor range. A small residual at an extreme point does not establish good fit elsewhere.
When appropriate, compare the fitted line with and without the observation. Confirm that the deleted design remains full rank before using a deletion-fit prediction or identity.
Describe the range where the line fits poorly and how the unusual observation affects the conclusion. Do not remove a valid observation solely to make the residual plot look better.
Common Mistake
Wrong reasoning: “The extreme point has a small residual, so it is well behaved and the line fits the data.” This judges the point only against a line it helped determine. A small full-data residual does not show that the line fits the other observations well.
Correction: Separate predictor unusualness from response unusualness. Examine the scatterplot and residuals across the ordinary predictor range, and consider a deletion comparison only when the deleted fit is identifiable. High leverage is a reason to inspect fit carefully, not a reason by itself to discard an observation or choose a different model.
Check Your Understanding
Use the definitions and diagnostic ideas above to answer these questions.
- In simple linear regression, what makes an observation high leverage, and which values determine its leverage?
- Why can a high-leverage observation have a modest residual in the full-data fit even when its deleted-fit residual is large?
- For the data \(x=0,1,2,3,4,10\), \(y=0,0,0,0,0,10\), the point at \(x=10\) has \(h_{ii}=33/38\) and \(e_i=25/19\). Calculate its deleted-fit residual using the stated identity.
- Does high leverage by itself prove that a point is influential or that the regression line fits poorly? Explain.
- What evidence in a residual plot could show poor fit among ordinary predictor values even if the extreme observation’s full-data residual is modest?