How Leverage Can Conceal a Response Outlier

When a Small Residual Hides an Unusual Response

Residuals are often used to look for responses that do not fit a regression pattern. But each residual is calculated from a fitted model, and the fit itself uses the observed responses. If one case has an unusually large predictor value, the least-squares line can move toward that case. The result can be a small residual even when the response is far from the pattern suggested by the other cases.

This effect is a consequence of leverage: how unusual a case’s predictor value is relative to the other predictor values. Leverage concerns the predictor locations, not the response values. It does not mean that a case must have an unusual response, or that it must have a large effect on the fitted line. It does mean that the fitted value at that case can be especially responsive to its observed response.

Definition: In simple linear regression with an intercept, the leverage of case $i$ is $h_{ii}$, where $$ h_{ii}=\frac{1}{n}+\frac{(x_i-\bar{x})^2}{S_{xx}}, \qquad S_{xx}=\sum_{j=1}^{n}(x_j-\bar{x})^2. $$ A case has high leverage when its predictor value is far from the mean of the observed predictor values relative to their overall spread. Its fitted residual is $e_i=y_i-\hat{y}_i$.

The setting here is ordinary least-squares regression of a quantitative response $y$ on a quantitative predictor $x$, with an intercept. For the usual regression model, the mean response is linear in $x$, and errors have mean zero and constant variance; independent errors are also commonly assumed for statistical inference. The concealment effect below is an algebraic property of the least-squares fit: it does not require normally distributed errors. The predictor values must not all be equal, so that a slope can be fitted.

Why Leverage Can Shrink a Residual

Imagine first that the responses follow a simple straight-line pattern. Now change just one response, at case $i$, by an amount $\delta$, while keeping every predictor value fixed. The least-squares fitted values change in a way determined by the predictor locations. At case $i$, the fitted value changes by $h_{ii}\delta$. The observed response changes by the full amount $\delta$. Consequently, the residual at that case changes by only the remainder:

\text{change in residual at }i = \delta-h_{ii}\delta = (1-h_{ii})\delta.

This relation explains the central idea. If $h_{ii}$ is close to 1, most of the response change is absorbed by a change in the fitted value, leaving only a small change in the residual. In simple regression with an intercept, $h_{ii}$ is at least $1/n$, and the leverages sum to 2. A predictor value far from $\bar{x}$ has larger leverage because the squared distance $(x_i-\bar{x})^2$ contributes to its value.

The relation can be justified directly using the least-squares projection. For fixed predictor values, the vector of fitted values is $Hy$, where $H$ is the projection matrix with diagonal entry $h_{ii}$. Changing only response $i$ by $\delta$ changes the response vector by $\delta$ in position $i$, so the fitted value at $i$ changes by $h_{ii}\delta$. Since the residual is observed response minus fitted response, its change is $\delta-h_{ii}\delta$. This is an exact calculation, not an approximation.

Worked Example: A High-Leverage Response Outlier

Suppose five hypothetical cases have predictor values $0,1,2,3,10$. The first four responses follow $y=2+3x$, while the response at $x=10$ is 40 rather than the value 32 suggested by that pattern. Find the least-squares line using all five cases and compare the response at $x=10$ with its fitted value.

The data are $(0,2),(1,5),(2,8),(3,11),(10,40)$. The first four cases define a reference pattern: at $x=10$, the line $y=2+3x$ predicts $2+3(10)=32$. Thus the response 40 is 8 response units above that pattern. To see what fitting all five cases does, calculate the least-squares slope and intercept.

\bar{x}=\frac{16}{5}=3.2,\qquad \bar{y}=\frac{66}{5}=13.2,

S_{xx}=\sum x_i^2-\frac{(\sum x_i)^2}{n} =114-\frac{16^2}{5} =62.8,

S_{xy}=\sum x_i y_i-\frac{(\sum x_i)(\sum y_i)}{n} =454-\frac{16(66)}{5} =242.8.

Therefore the fitted slope is $b_1=242.8/62.8\approx3.866$, and the intercept is $b_0=13.2-(242.8/62.8)(3.2)\approx0.828025$. The fitted line is approximately $\hat{y}=0.8280+3.866x$. At $x=10$, it predicts $\hat{y}\approx39.490$, so the residual is only $40-39.490\approx0.5096$ response units.

Now calculate leverage for the case at $x=10$:

h_{ii}=\frac{1}{5}+\frac{(10-3.2)^2}{62.8} =0.2+\frac{46.24}{62.8} \approx0.9363.

The four other cases follow the reference line exactly, and the response at $x=10$ is 8 units above its prediction from that line. The residual after fitting all five cases is $(1-h_{ii})(8)\approx(0.06369)(8)=0.5096$, matching the calculation from the fitted line. The extreme predictor location gave the response substantial leverage to pull the line toward itself.

Schematic plot:
Response $y$
40   |                                 ● $(10,40)$
    |                                × $\hat{y}\approx39.49$
32   |                                   ○ reference prediction $32$
    | ● ● ● ●                                 /
    +------------------------------------------------ Predictor $x$
        0   1   2   3                       10

The sketch emphasizes the distinction: relative to the main group’s pattern, the response at $x=10$ is far above its reference prediction. Relative to the line fitted to all five cases, it sits very close to the fitted value.

The Predictor Location, Not the Response, Sets Leverage

A useful check is to hold the predictor values fixed and imagine an unusual response at a different location. The leverage formula depends only on $x$, $\bar{x}$, and $S_{xx}$; changing a response does not change leverage. The amount of response unusualness that remains in the residual, however, depends on $1-h_{ii}$.

Worked Example: A More Extreme Response at the Same High-Leverage Location

Keep the five predictor values from the previous example and the reference pattern $y=2+3x$, but suppose the response at $x=10$ is 52. This response is 20 units above the reference prediction of 32. What residual does the full least-squares fit leave at $x=10$?

The leverage is unchanged because the $x$-values have not changed: $h_{ii}\approx0.9363$. The response departure is now $\delta=52-32=20$. Thus the residual is

e_i=(1-h_{ii})\delta =(1-0.9363)(20) \approx1.274.

The fitted value at $x=10$ is consequently about $52-1.274=50.726$. The residual is larger than in the first example, but it is still far smaller than the 20-unit discrepancy from the reference pattern. A larger response departure does not guarantee a large fitted residual when leverage is high.

Worked Example: The Same Departure at a Less Extreme Predictor Value

Use the same predictor values $0,1,2,3,10$, but now place an 8-unit response departure at $x=2$, whose reference response is $2+3(2)=8$. The response there is therefore 16, while the other responses remain on the reference line. Compare the leverage and residual with the high-leverage case.

At $x=2$, the leverage is

h_{ii}=\frac{1}{5}+\frac{(2-3.2)^2}{62.8} =0.2+\frac{1.44}{62.8} \approx0.2229.

The residual is therefore

e_i=(1-h_{ii})(8) \approx(1-0.2229)(8) \approx6.217.

In this comparison, an 8-unit departure at $x=2$ leaves a residual of about 6.217, while the same departure at $x=10$ leaves a residual of about 0.5096. The predictor values and the size of the response departure are otherwise the same. The difference comes from leverage: the middle case has much less ability to pull the fitted line toward itself.

How to Read the Evidence

A residual is the distance from an observed response to the fitted line at that case’s predictor value. It is not the distance from the response to a pattern that would have been fitted without that case. That distinction matters because ordinary least squares chooses the line using all the cases. For a high-leverage case, the fitted value may already have shifted toward an unusual response.

A small residual is therefore not, by itself, evidence that the response is unremarkable. When a case is far from the others in predictor space, it is useful to examine the predictor distribution and the scatterplot as well as residuals. Ask whether the response looks unusual relative to the pattern of the other observations, and whether the fitted line is being strongly anchored by the distant predictor value. These are diagnostic questions, not automatic reasons to discard a case.

The effect also depends on the scale of the discrepancy: a high-leverage point can retain a noticeable residual if its response is sufficiently far from the reference pattern. Conversely, high leverage alone does not establish a response outlier. A case may be far out in $x$ and still follow the relationship well. The relevant evidence combines predictor location and response behavior.

Common Mistake

Wrong reasoning: “The residual is small, so this response cannot be unusual.”
Correction: The fitted value was calculated using the response itself. High leverage can pull that fitted value toward the observation and shrink its residual. Compare the response with the broader data pattern and check whether its predictor location is extreme.
Another wrong reasoning: “A point far out in predictor space must be a response outlier.”
Correction: Leverage describes the predictor location, not the response’s agreement with the regression pattern. A high-leverage case can have a small or large residual.

Key takeaway: In simple least-squares regression, a response departure $\delta$ at case $i$ changes its residual by $(1-h_{ii})\delta$. When leverage $h_{ii}$ is near 1, even a substantial response departure may leave a deceptively small residual.

Check Your Understanding

Try these questions using the definitions and calculations above.

What feature of a case determines its leverage in simple regression with an intercept: its response value or its predictor value?
A response departs by 10 units from a reference pattern at a case with leverage $h_{ii}=0.8$. What residual departure does the relation $(1-h_{ii})\delta$ predict?
Explain why a high-leverage case can have a small residual even when its response is unusual relative to the other cases.
Can a case have high leverage and still follow the regression pattern closely? Explain.
In the five-case example, why does the 8-unit departure at $x=10$ leave a smaller residual than the same departure at $x=2$?