Two Different Kinds of Unusual Observation
A regression case can be unusual in two fundamentally different ways. Its predictor values may lie far from the main body of the data, or its observed response may lie far from the response predicted by the fitted model. The first feature is measured by leverage; the second is assessed using the residual and, often, a standardized residual. These are related diagnostics, but they do not identify the same cases.
This distinction matters when reviewing a regression diagnostic. A point can be far out in predictor space and still sit close to the fitted regression surface. Conversely, a point can have a large response residual while having predictor values near the center of the data. Neither description alone establishes that the case has materially changed the fitted model. In this tutorial, “influence” means an effect a case has on the fitted model; assessing that effect directly requires additional diagnostics.
Leverage and Residuals Measure Different Features
Assume the ordinary least squares linear model is appropriate, with a full-rank design matrix \(X\), \(n\) observations, and \(p\) estimated regression coefficients, including the intercept when one is in the model. The hat matrix is \(H=X(X^\mathsf{T}X)^{-1}X^\mathsf{T}\). Its diagonal entry \(h_{ii}\) is the leverage of observation \(i\). The fitted responses are \(H\mathbf{y}\), and the residuals are \(\mathbf{e}=(I-H)\mathbf{y}\). Thus leverage comes from the predictor design, whereas a residual depends on both the observed responses and the fitted model.
For simple linear regression with an intercept and one predictor, the leverage at predictor value \(x_i\) has the especially useful form below. Here \(\bar{x}\) is the mean predictor value and \(S_{xx}=\sum_{j=1}^n(x_j-\bar{x})^2\). The formula is specific to this simple-regression design; in multiple regression, leverage depends on the case’s position in the full predictor space.
Leverage is nonnegative, and the leverages sum to \(p\). Their mean is therefore \(p/n\). A commonly used screening rule calls leverage notably high when it exceeds \(2p/n\), but this is a heuristic rather than a universal decision rule. It flags predictor-space position; it does not test whether the response is unusual or prove that the observation is influential.
For comparing residual sizes across observations, the internally standardized residual is
This standardization accounts for the fact that fitted residuals have different variances across observations: under the usual constant-variance linear model, the variance of \(e_i\), conditional on \(X\), is \(\sigma^2(1-h_{ii})\). A large absolute standardized residual can be evidence of a response outlier, but its interpretation depends on the model and the diagnostic context. A cutoff is a screening aid, not an automatic deletion instruction.
Why the Simple-Regression Leverage Formula Holds
With an intercept, the fitted value at \(x_i\) is the sample mean response plus the estimated slope multiplied by \(x_i-\bar{x}\). The OLS slope is \(\sum_j(x_j-\bar{x})y_j/S_{xx}\). The coefficient of \(y_i\) in that fitted value is therefore \(1/n+(x_i-\bar{x})^2/S_{xx}\): the mean-response part contributes \(1/n\), and the slope part contributes \((x_i-\bar{x})^2/S_{xx}\). That coefficient is the \(i\)th diagonal entry of the hat matrix, giving the stated leverage formula.
The decomposition also explains the interpretation. Every observation has the baseline contribution \(1/n\), while the additional contribution increases with squared distance from \(\bar{x}\). A distant predictor value can consequently have high leverage even if its response happens to agree almost exactly with the fitted line.
Worked Example: High Leverage With a Zero Residual
A hypothetical insurer fits a simple linear regression of a claim-cost index \(y\) on a risk score \(x\) for five portfolios. The observed pairs are \((0,11),(1,10),(2,15),(3,16),(10,30)\). Assume the model includes an intercept and is fitted by ordinary least squares. Find the fitted line, the leverage values, and the internally standardized residuals. Identify the high-leverage case and the case with the largest absolute residual.
Method and fitted line. The predictor mean is \(\bar{x}=(0+1+2+3+10)/5=3.2\). The response values can be written as \(y_i=10+2x_i+e_i\), with proposed residuals \(1,-2,1,0,0\). They sum to zero, and their predictor-weighted sum is \(0(1)+1(-2)+2(1)+3(0)+10(0)=0\). These are the OLS orthogonality conditions for an intercept and slope, so the fitted line is \(\hat{y}=10+2x\). The residuals are exactly \(1,-2,1,0,0\), and \(\operatorname{SSE}=1^2+(-2)^2+1^2=6\). With \(n=5\) and \(p=2\), \(s=\sqrt{6/(5-2)}=\sqrt{2}\).
Leverage calculations. The sum of squared predictor deviations is \(S_{xx}=(-3.2)^2+(-2.2)^2+(-1.2)^2+(-0.2)^2+(6.8)^2=10.24+4.84+1.44+0.04+46.24=62.8\). The cutoff \(2p/n=4/5=0.8\) is used here only as a screening heuristic.
The unrounded leverages sum to \(2\), as required because \(p=2\); the displayed rounded values also sum to \(2.0000\). The point at \(x=10\) exceeds the heuristic cutoff and is clearly the high-leverage case.
Standardized residuals. Substituting each residual and leverage into \(r_i=e_i/(\sqrt{2}\sqrt{1-h_{ii}})\) gives, to three decimals, \(0.886,-1.663,0.802,0.000,0.000\), respectively. The largest absolute raw residual and largest absolute standardized residual are both at \(x=1\), not at \(x=10\). The \(x=10\) case has zero residual: it is far out in predictor space but lies exactly on the fitted line. This avoids the common mistake of calling a high-leverage case a response outlier just because its predictor value is unusual.
Check. The fitted responses are \(10,12,14,16,30\); subtracting these from the observed responses gives \(1,-2,1,0,0\), confirming the residual calculations. Leverage identifies the unusual predictor position, not an unusual response or an influence conclusion.
A Large Residual Does Not Require High Leverage
The reverse distinction is just as important. A case can be near the center of predictor space and still have a response far from the fitted value. Such a case may deserve investigation as a response outlier, but the fact that its residual is large does not, by itself, show how much the fitted model would change if the case were altered or omitted.
Worked Example: A Response Outlier at a Central Predictor Value
A hypothetical pricing team models an experience index \(y\) using predictor \(x\), with an intercept. For \(x=-4,-3,-2,-1,0,1,2,3,4,5\), the observed responses are \(5,5.5,6,7.5,5,8.5,8,8.5,9,9.5\), respectively. Assess which observation has the largest residual and whether it has high leverage.
Method and fitted line. The responses follow \(y_i=7+0.5x_i+e_i\), with residuals equal to \(1\) at \(x=-1\), \(-2\) at \(x=0\), \(1\) at \(x=1\), and zero elsewhere. Their sum is \(1-2+1=0\), and the predictor-weighted sum is \((-1)(1)+0(-2)+1(1)=0\). Hence the OLS line is \(\hat{y}=7+0.5x\), and the stated residuals are confirmed. The sum of squared residuals is \(1+4+1=6\), so with \(n=10\) and \(p=2\), \(s=\sqrt{6/8}=\sqrt{0.75}=0.8660\).
Leverage at the largest-residual case. The predictor mean is \(\bar{x}=0.5\), and \(S_{xx}=82.5\). At \(x=0\), the residual is \(-2\), the largest in absolute value, while \(h_{ii}=1/10+(0-0.5)^2/82.5=0.1+0.25/82.5=0.1030\). The heuristic \(2p/n=0.4\) is much larger than this leverage, so the case is not high leverage by that screening rule.
Its internally standardized residual is \(r_i=-2/(0.8660\sqrt{1-0.1030})=-2.438\), rounded to three decimals. This supports examining the response as unusual relative to the residual variation; it does not establish influence. The example also shows why looking only at predictor position would miss a potentially unusual response.
Check. The residuals satisfy both OLS orthogonality conditions, and their sum of squares is \(6\), matching the value used to calculate \(s\). The largest absolute residual occurs at a predictor value near the mean, not at an extreme predictor value.
Use Both Diagnostics, but Do Not Confuse Them With Influence
A useful diagnostic review separates three questions. First, is the case unusual in predictor space? Examine leverage in relation to the model’s predictor design and the other cases. Second, is its response unusual given the fitted model? Examine the residual relative to the appropriate residual scale, including a standardized residual when appropriate. Third, is there evidence that the fitted model is materially affected by the case? Leverage and residual size can help motivate that question, but neither alone answers it.
Use leverage to determine whether the predictors are unusual relative to the fitted design. In simple regression, compare \(h_{ii}\) with the distribution of leverages; in multiple regression, consider the case’s position across all predictors.
Examine the residual and its standardized version. A large absolute residual is a raw discrepancy; standardization accounts for the case-specific residual variance under the model.
Report high leverage and a large residual as distinct observations about the case. Do not infer influence or delete the case solely from either feature.
The four broad combinations clarify what a diagnostic does and does not say:
| Leverage | Residual size | What the combination indicates |
|---|---|---|
| Low or ordinary | Small | Neither predictor position nor response deviation is especially unusual by these diagnostics. |
| High | Small | The predictors are unusual, but the response is close to the fitted model. |
| Low or ordinary | Large | The response is unusual relative to the fit, without unusual predictor position. |
| High | Large | Both features warrant attention; influence still requires a direct assessment. |
Common Exam Traps
- Calling high leverage a response outlier. In the first example, the case with \(x=10\) has leverage \(0.9363\) but residual zero. Predictor-space position does not measure vertical distance from the fitted line.
- Calling every large residual influential. A large residual flags a response discrepancy. It does not quantify the change to the fitted regression when the case is removed or changed.
- Using raw residuals alone to compare cases. The internally standardized residual divides by \(s\sqrt{1-h_{ii}}\). Cases with different leverage have different residual scales under the model.
- Treating a leverage cutoff as a verdict. The rule \(2p/n\) is a screening heuristic, not a formal test and not a rule for automatic deletion.
- Using the wrong predictor dimension. In multiple regression, a case can be unusual through a combination of predictors even if no single predictor value looks extreme. Leverage concerns the full design.
Check Your Understanding
For each problem, distinguish what the leverage and residual information establishes from what it does not establish.
- A simple regression has \(n=8\) observations and \(p=2\) coefficients. A case has leverage \(0.71\) and residual zero. Find the average leverage and the \(2p/n\) screening cutoff. What can you conclude about this case? (Answer: Average leverage \(=0.25\); cutoff \(=0.50\). It is high leverage by this heuristic but is not a response outlier on the basis of its zero residual; influence is not established.)
- In a fitted linear model, \(s=3\), \(h_{ii}=0.20\), and \(e_i=6\). Calculate the internally standardized residual and compare leverage with the \(2p/n\) cutoff when \(n=6,p=2\). (Answer: \(r_i=6/[3\sqrt{0.8}]=2.236\) to three decimals; the cutoff is \(0.667\), so leverage is not high by that heuristic, while the standardized residual is relatively large.)
- A case has leverage \(0.85\) and residual \(0\). State whether the case is high leverage, a response outlier, or necessarily influential. (Answer: It is high leverage; its zero residual does not indicate a response outlier, and it is not necessarily influential.)
- Two cases have the same residual \(e=4\) in a model with \(s=2\). Case A has \(h=0.75\), and case B has \(h=0.10\). Calculate each internally standardized residual. (Answer: \(r_A=4/[2\sqrt{0.25}]=4.000\); \(r_B=4/[2\sqrt{0.90}]=2.108\) to three decimals. The same raw residual produces different standardized residuals.)
- A multiple regression has \(n=20\), \(p=4\), and a case with \(h_{ii}=0.45\) and \(e_i=0.6s\). Calculate the screening cutoff and internally standardized residual, then interpret them. (Answer: \(2p/n=0.40\), so leverage is high by the heuristic; \(r_i=0.6/\sqrt{0.55}=0.809\) to three decimals, which is not a large standardized residual. High leverage alone does not establish a response outlier or influence.)