Why Residual Size Is Not the Same as Influence
A residual describes how far an observation lies vertically from a fitted regression line. Influence asks a different question: how much would the fitted model change if this case were not included? A point can be far from the line without substantially changing it, or it can lie close to the line while still helping determine where the line goes.
The distinction matters because unusual data are not all unusual in the same way. A response value may be surprising given its predictor value, while a predictor value may be far from the rest of the data. Those features can occur together, separately, or not at all. Neither one by itself tells us how sensitive the regression fit is to that case.
The amount of influence depends on what part of the fit we examine. A case may change the slope but have little effect on an intercept, or alter predictions at one end of the predictor range more than elsewhere. Therefore, “influential” is a description of sensitivity in a stated model and for a stated feature of that model, not a permanent label attached to an observation.
Model and Calculation Framework
The examples use ordinary least-squares simple linear regression with one quantitative predictor \(x\), one quantitative response \(y\), an intercept, and equal weight for every case. The fitted line is \(\hat{y}=b_0+b_1x\), where the coefficients minimize the sum of squared residuals. These examples are about changes in a fitted line, not hypothesis tests or confidence intervals; no normality or constant-variance assumption is needed for the least-squares calculations themselves.
For \(n\) cases, define \(S_{xx}=\sum (x_i-\bar{x})^2\) and \(S_{xy}=\sum (x_i-\bar{x})(y_i-\bar{y})\). When \(S_{xx}>0\), the least-squares slope and intercept are \(b_1=S_{xy}/S_{xx}\) and \(b_0=\bar{y}-b_1\bar{x}\).
These formulas follow directly from minimizing the squared residuals. For a fixed slope, differentiating \(\sum (y_i-b_0-b_1x_i)^2\) with respect to \(b_0\) and setting the result to zero gives \(\sum(y_i-b_0-b_1x_i)=0\). Thus \(b_0=\bar{y}-b_1\bar{x}\). Substituting this expression into the squared-error sum and setting its derivative with respect to \(b_1\) to zero gives \(\sum(x_i-\bar{x})(y_i-\bar{y})=b_1\sum(x_i-\bar{x})^2\). Dividing by \(S_{xx}\) gives the slope formula.
For simple regression with an intercept, the leverage of case \(i\) can be calculated as \(h_i=1/n+(x_i-\bar{x})^2/S_{xx}\). This formula makes the role of predictor location explicit: cases far from the predictor mean tend to have greater leverage. Leverage is not itself a measure of how much the fitted line changes; the response value also matters.
Worked Comparisons
Worked Example: A Large Residual With Low Leverage
Suppose five baseline cases lie exactly on \(y=10+2x\), with predictor values \(-2,-1,0,1,2\) and responses \(6,8,10,12,14\). Add a sixth case with \(x=0\) and \(y=22\). Compare its residual and leverage in the full fit with the change in the line when it is removed.
Fit using all six cases. The totals are \(\sum x_i=0\), \(\sum y_i=72\), \(\sum x_i^2=10\), and \(\sum x_iy_i=20\). Hence \(\bar{x}=0\), \(\bar{y}=72/6=12\), \(S_{xx}=10\), and \(S_{xy}=20\). The fitted coefficients are
At the added case’s predictor value \(x=0\), the fitted response is \(12+2(0)=12\). Its residual is \(22-12=10\) response units, a large vertical discrepancy in this small data set.
The case has \(x=\bar{x}=0\), so its leverage is \(h_i=1/6+(0-0)^2/10=1/6\), or about \(0.1667\). It is at the center of the predictor values, not at an extreme predictor location.
Refit without the case. The five remaining observations have mean predictor \(0\), mean response \(10\), \(S_{xx}=10\), and \(S_{xy}=20\). Their fitted line is \(\hat{y}=10+2x\). Removing the case therefore changes the intercept from \(12\) to \(10\), a change of \(2\) response units, while the slope remains \(2\) response units per predictor unit.
This case has a large residual but low leverage. It does affect the intercept, so “low leverage” does not mean “no influence.” But it does not change the slope in this example. Influence depends on the feature of the fit being considered, and a large residual alone does not tell us how much that feature will change.
Worked Example: A Small Residual With High Leverage
Now take five baseline cases at \(x=0,1,2,3,4\), with responses \(y=2,3,4,5,6\). They lie on \(y=2+x\). Add a case at \(x=10,\ y=16\). Calculate its residual and leverage in the six-case fit, then compare the fitted line with the line after deletion.
For all six cases, \(\sum x_i=20\), \(\sum y_i=36\), \(\sum x_i^2=130\), and \(\sum x_iy_i=210\). Thus \(\bar{x}=20/6=10/3\) and \(\bar{y}=36/6=6\). The centered sums are
The slope is \(90/(190/3)=27/19\), about \(1.421\), and the intercept is \(6-(27/19)(10/3)=24/19\), about \(1.263\). The full-data fitted line is therefore \(\hat{y}=24/19+(27/19)x\). At \(x=10\), its fitted response is \(294/19\), about \(15.47\), so the case’s residual is \(16-294/19=10/19\), about \(0.5263\) response units.
Despite that small residual, its leverage is high:
Without the case, the five baseline observations give the line \(\hat{y}=2+x\), with slope \(1\) and intercept \(2\). Deleting the case changes the slope from about \(1.421\) to \(1\), a difference of about \(0.421\) response units per predictor unit, and changes the intercept from about \(1.263\) to \(2\), a difference of about \(0.737\) response units.
The added case was four response units above the baseline line at \(x=10\), since that line predicts \(12\). But the full-data fit has already been pulled toward the added case, leaving it only about \(0.5263\) units above the full-data line. This example shows why a residual measured from the fitted line can be small even when the case has substantially helped determine that line.
Worked Example: An Extreme Predictor Value Without Influence
Use the same five baseline cases from the previous example, but add \(x=10,\ y=12\). This predictor value is far from the other five. Determine whether it changes the fitted line.
For the six cases, \(\sum x_i=20\), \(\sum y_i=32\), \(\sum x_i^2=130\), and \(\sum x_iy_i=170\). Therefore \(\bar{x}=10/3\), \(\bar{y}=16/3\), \(S_{xx}=190/3\), and
The six-case line is \(\hat{y}=2+x\), exactly the baseline line. At \(x=10\), the fitted response is \(12\), so the added case’s residual is \(0\). Its leverage remains \(33/38\), because the predictor values are the same as in the previous example. Removing the case leaves the line unchanged.
This case has high leverage but does not change the fitted line: its response agrees exactly with the trend established by the other observations. Predictor extremeness by itself is not enough to establish influence.
How to Think About the Comparison
The examples separate three features that are easy to conflate. A residual measures vertical disagreement with the fitted line. Leverage describes how far the predictor value lies from the center of the observed predictor values. Influence is assessed by comparing fits, such as the full-data fit and the fit after deleting a case. The first two features help identify cases worth examining, but the deletion comparison addresses whether the fitted model is sensitive to that case.
A case with both high leverage and a response inconsistent with the pattern among the other cases can be especially influential: its predictor position gives it an opportunity to affect the line, and its response pulls the fit in a particular direction. But the full-data residual may not appear large because the fitted line has moved toward that observation. This is why it is useful to distinguish the residual from the full fit from the discrepancy relative to a line estimated without the case.
Conversely, a large residual at an ordinary predictor value can signal a response that is poorly predicted without greatly changing the slope. The first worked example illustrates this: the added response raised the intercept but left the slope unchanged. The size and direction of a fit change depend on where the case lies and on what the other observations establish.
A deletion comparison is a diagnostic, not an instruction to remove a case. A large change prompts questions: Is the measurement correct? Does the case belong to the population and model being studied? Is the chosen line an adequate summary across the predictor range? A valid, influential observation may reveal that conclusions depend strongly on a limited part of the data. Removing it solely because the result becomes more convenient would discard information and can distort the analysis.
Common Mistakes
- Equating a large residual with high influence. A large residual signals vertical discrepancy, but a case can have a large residual and make only a limited change to a particular part of the fit. Compare the model with and without the case.
- Calling every extreme predictor value influential. The third example has high leverage but lies exactly on the line set by the remaining data, so removing it does not change that line.
- Assuming a small full-fit residual rules out influence. A case can help pull the fitted line toward itself, making its residual from that line small. Check the fit after deletion when sensitivity is the question.
- Treating influence as a reason to delete a case. First investigate data quality and model context. Influence describes sensitivity; it does not decide whether an observation is erroneous or should be excluded.
Check Your Understanding
Use the definitions and examples above to answer these questions.
- In simple regression, what does a residual measure, and what does leverage describe?
- Why did the added case with \(x=10,\ y=16\) have a small residual in the full fit even though it was four response units above the line based on the other five cases?
- In the first worked example, which coefficient changed when the case was removed, and which coefficient stayed the same?
- Can a case have high leverage and leave the fitted line unchanged? Explain using the third example.
- Why is deleting a case solely because it changes a regression fit substantially not a sound general rule?