Outliers, High Leverage, and Influential Points

Three Different Questions About an Observation

A regression diagnostic can flag an observation for more than one reason. A response may be far from the fitted line, a predictor value may be far from the center of the observed predictor values, or removing an observation may substantially change the fitted regression. These are related concerns, but they are not interchangeable. In particular, an unusual predictor value does not necessarily have an unusual response, and a large residual does not automatically make a point influential.

This tutorial considers ordinary least squares in a simple linear regression with an intercept. The model is $Y_i=\beta_0+\beta_1x_i+\varepsilon_i$, for $i=1,\ldots,n$. The standard regression model assumes that the errors are independent, have mean zero and common variance $\sigma^2$, and that the conditional mean of $Y_i$ is linear in $x_i$. Normality is additionally used for exact small-sample inference, but the diagnostic distinctions here are based on residuals, predictor locations, and changes in the fitted model.

Definition: A vertical outlier has a response unusually far from the fitted regression value at its predictor value, as assessed using residual diagnostics. A high-leverage observation has an unusual predictor location relative to the other observed predictor values. An influential observation is one whose inclusion or removal substantially changes the fitted regression or its conclusions.

The terms describe different properties. A vertical outlier concerns the response direction, leverage concerns the predictor direction, and influence concerns the effect on the fit. Influence is usually assessed by comparing the full-data fit with a fit in which one observation is omitted.

Residuals and Leverage Measure Different Things

For observation $i$, the fitted response is $\hat{Y}_i=\hat{\beta}_0+\hat{\beta}_1x_i$, and the residual is $e_i=Y_i-\hat{Y}_i$. A large residual in the original units can be hard to judge because the overall residual scale and the observation’s leverage both matter. One common scale-adjusted diagnostic is the internally standardized residual:

r_i=\frac{e_i}{s\sqrt{1-h_{ii}}},

where $s=\sqrt{\mathrm{MSE}}$ is the residual standard error and $h_{ii}$ is the leverage of observation $i$. An externally studentized residual instead estimates the error scale after omitting observation $i$. It is often preferred for formal outlier checks because the observation being checked does not contribute to that scale estimate. A large absolute studentized residual is evidence of a potentially unusual response, not proof that the observation is erroneous.

In simple linear regression with an intercept, leverage has the form

h_{ii}=\frac{1}{n}+\frac{(x_i-\bar{x})^2}{S_{xx}}, \qquad S_{xx}=\sum_{j=1}^{n}(x_j-\bar{x})^2.

Leverage is determined by the predictor values, not by $Y_i$. It is nonnegative, and the leverages sum to the number of fitted coefficients, which is $q=2$ in simple linear regression with an intercept. A predictor value far from $\bar{x}$ tends to produce high leverage. High leverage gives an observation an opportunity to pull the fitted line toward itself, but whether it actually does so depends in part on its residual.

Diagnostic question	Typical quantity	What it indicates
Is the response unusual given the predictor?	Studentized residual	Vertical departure from the fitted line
Is the predictor location unusual?	Leverage $h_{ii}$	Distance in predictor space from the center of the data
Does the observation materially change the fit?	Cook’s distance or deletion comparison	Combined effect of residual size and leverage

Cook’s Distance Connects Residual Size and Leverage

Cook’s distance measures the overall change in the fitted regression when observation $i$ is omitted. For a model with $q$ fitted coefficients, a common definition is

D_i= \frac{(\hat{\boldsymbol{\beta}}-\hat{\boldsymbol{\beta}}_{(i)})' X'X(\hat{\boldsymbol{\beta}}-\hat{\boldsymbol{\beta}}_{(i)})} {q\,\mathrm{MSE}},

where $\hat{\boldsymbol{\beta}}$ is the coefficient vector from all observations, $\hat{\boldsymbol{\beta}}_{(i)}$ is the coefficient vector after omitting observation $i$, and $X$ is the regression design matrix. A large value means that the deleted fit differs appreciably from the full fit, relative to the residual variation and number of coefficients. For ordinary least squares, this can also be written

D_i=\frac{e_i^2}{q\,\mathrm{MSE}}\frac{h_{ii}}{(1-h_{ii})^2}.

This expression shows why neither residual size nor leverage alone settles the question of influence. A large residual raises Cook’s distance, and high leverage raises it as well. When leverage approaches one, the denominator also makes the diagnostic especially sensitive.

Derivation: In ordinary least squares, deleting observation $i$ changes the coefficient vector by $\hat{\boldsymbol{\beta}}-\hat{\boldsymbol{\beta}}_{(i)}=(X'X)^{-1}x_i e_i/(1-h_{ii})$, where $x_i$ is the row of predictors including the intercept. Substituting this difference into the quadratic form in the definition of $D_i$ gives $$ \frac{e_i^2}{(1-h_{ii})^2} x_i'(X'X)^{-1}X'X(X'X)^{-1}x_i = \frac{e_i^2h_{ii}}{(1-h_{ii})^2}. $$ Dividing by $q\,\mathrm{MSE}$ yields the residual-and-leverage formula. Thus Cook’s distance combines the two diagnostics because the coefficient change depends on both the residual and the predictor location.

Rules of thumb are screens, not definitions. For example, leverage above $2q/n$ is sometimes flagged for review, and Cook’s distance may be compared with $4/n$; values near or above 1 are also often treated as a reason to investigate. Likewise, an absolute studentized residual around 2 can prompt review, while values around 3 or more are a stronger signal. These cutoffs depend on context, sample size, and how many observations are being checked. They do not establish that a point is an error or dictate that it be deleted.

Worked Examples: The Categories Need Not Coincide

Worked Example: High Leverage Without a Large Residual

In a hypothetical study of equipment operating time, a simple regression has $n=20$ observations, $q=2$ fitted coefficients, residual standard error $s=10$ hours, and $S_{xx}=100$ hours-squared. One unit has $x_i-\bar{x}=\sqrt{20}$ hours and residual $e_i=4.1833$ hours. Calculate its leverage, standardized residual, and Cook’s distance, then classify it.

Method. Calculate the leverage from the predictor location, then use the residual and leverage formulas. Keep in mind that the predictor deviation is given in hours, so its square is $20$.

First, $h_{ii}=1/20+20/100=0.05+0.20=0.25$. This exceeds the screening value $2q/n=4/20=0.20$, so the observation has high leverage by that rule.

The standardized residual is $r_i=4.1833/[10\sqrt{1-0.25}]=4.1833/[10\sqrt{0.75}]\approx 0.483$. It is not large. Its Cook’s distance is $D_i=4.1833^2/[2(10^2)]\times 0.25/(0.75^2)\approx 17.5/200\times 0.4444=0.0389$.

Conclusion and check. This observation is high leverage, but its response is close to the fitted line and its Cook’s distance is small. It is not thereby a vertical outlier or an influential point. The mistake to avoid is labeling every unusual predictor value influential without checking how far its response is from the fit.

Worked Example: A Vertical Outlier With Low Leverage

A hypothetical regression study has $n=50$, $q=2$, and $s=4$ units. For one observation, $h_{ii}=0.02$ and $e_i=12$ units. Calculate its internally standardized residual and Cook’s distance. Use $2q/n$ and $4/n$ only as screening references.

Method. The residual scale adjustment uses $\sqrt{1-h_{ii}}$, while Cook’s distance includes both $e_i^2$ and $h_{ii}$.

The standardized residual is $r_i=12/[4\sqrt{0.98}]=12/[4(0.98995)]\approx 3.030$. That is a strong signal of a potentially unusual response. The leverage reference is $2q/n=4/50=0.08$; since $0.02<0.08$, the point has low leverage by this screen.

For influence, $D_i=12^2/[2(4^2)]\times 0.02/(0.98^2) =144/32\times 0.02/0.9604 \approx 4.5(0.02082)=0.0937$. The Cook’s distance reference is $4/n=4/50=0.08$, so this value merits review, though it is not large by every convention.

Conclusion and check. The observation has a large standardized residual and low leverage; its Cook’s distance is modest but above the stated screening reference. A vertical outlier need not have high leverage, and its influence should be assessed separately rather than inferred from its residual alone.

Worked Example: High Leverage and Influence With a Moderate Residual

In a hypothetical regression of maintenance cost on annual operating hours, $n=20$, $q=2$, and $s=5$ cost units. An observation has $h_{ii}=0.45$ and residual $e_i=7$ cost units. Calculate its standardized residual and Cook’s distance, then interpret the diagnostics.

Method. Compare the standardized residual with common outlier screens and compare Cook’s distance with $4/n$. These are guidelines rather than fixed decision rules.

The standardized residual is $r_i=7/[5\sqrt{0.55}]=7/[5(0.74162)]\approx 1.888$, which is not especially large. The leverage screen is $2q/n=4/20=0.20$, and $0.45>0.20$, so this is a high-leverage observation.

Its Cook’s distance is $D_i=7^2/[2(5^2)]\times 0.45/(0.55^2) =49/50\times 0.45/0.3025 =0.98(1.4876)\approx 1.458$. This is well above $4/n=0.20$ and also above 1, so it warrants close examination as influential.

Conclusion and check. The observation is influential despite having only a moderate standardized residual. Its unusual predictor position gives that moderate departure substantial scope to change the fitted regression. The mistake to avoid is equating “not a vertical outlier” with “not influential.”

Interpreting the Diagnostics Together

A useful interpretation keeps the three questions separate, then considers them jointly. A high-leverage point with a small residual often lies near the extended trend of the other data and may have little effect on the fit. A low-leverage point can have a large residual but still have limited ability to rotate or shift the line. A high-leverage point with a substantial residual is a common pattern for an influential observation, although the actual change should still be checked.

The word “outlier” can be used broadly, so be precise in written work. If the concern is a response far from the regression line, say “vertical outlier” and support the assessment with a residual diagnostic. If the predictor is unusual, say “high leverage.” If deleting the point appreciably changes the coefficients or fitted values, say “influential” and cite Cook’s distance or the deletion comparison.

A diagnostic flag is an invitation to investigate, not an instruction to remove data. Confirm the observation’s units and recording, consider whether it belongs to the population and model being studied, and compare the analysis with and without it when appropriate. A valid observation can be influential; deleting it merely to obtain a preferred result can distort the analysis.

Key takeaway: A vertical outlier is identified through its response residual, high leverage through its predictor location, and influence through the change in the fitted model. Check all three; none is a synonym for another.

Check Your Understanding

For each case, distinguish the residual evidence, leverage evidence, and influence evidence. Treat the stated cutoffs as screening guidelines only.

A regression has $n=30$ and $q=2$. An observation has $h_{ii}=0.22$, internally standardized residual $0.7$, and $D_i=0.06$. Which diagnostic category is most clearly indicated? (Answer: High leverage; there is little residual evidence of a vertical outlier and little evidence of influence.)
A regression has $n=50$, $q=2$, $s=4$, $h_{ii}=0.02$, and $e_i=12$. Calculate the standardized residual and Cook’s distance, then classify the point using $2q/n$ and $4/n$. (Answer: $r_i\approx3.030$, $D_i\approx0.0937$; low leverage, strong vertical-outlier signal, and Cook’s distance just above the $0.08$ screening reference.)
A regression has $n=25$, $q=2$, $s=4$, $h_{ii}=0.40$, and $e_i=5$. Calculate $r_i$ and $D_i$. Is it high leverage, a strong vertical outlier, or potentially influential under the stated screens? (Answer: $r_i=5/[4\sqrt{0.60}]\approx1.614$; $D_i=25/32\times0.40/0.36\approx0.868$. It is high leverage and potentially influential since $D_i>4/25=0.16$, but its residual is not a strong outlier signal.)
Two observations have the same large absolute studentized residual, but one has leverage $0.03$ and the other has leverage $0.35$. Which one is generally more likely to have the larger Cook’s distance, all else equal? (Answer: The observation with leverage $0.35$; Cook’s distance depends on both residual size and leverage.)
An observation is far from the center of the predictor values, but its response lies almost exactly on the fitted line. Should it automatically be called influential? (Answer: No. It has high leverage, but a small residual may make its Cook’s distance and deletion effect small.)

What You'll Learn