Residuals as Evidence of Unusual Responses

Why Residual Size Needs Context

A residual is the difference between an observed response and the response predicted by a fitted regression model. Its sign and size can reveal that an observation is poorly described by the model. But a raw residual, measured in the response's units, does not by itself say how unusual that discrepancy is. Residual variability depends in part on where an observation sits in the predictor space, so comparisons based only on raw residual size can be misleading.

This tutorial compares raw residuals with two scaled versions: standardized residuals and studentized residuals. These quantities help identify responses that warrant investigation. They do not, on their own, establish that a case is influential, that the model is wrong, or that an observation should be removed.

Model and Residual Definitions

Let \(n\) observations be fitted by an ordinary least-squares linear regression with \(p\) estimated coefficients, including an intercept when the model has one. Write the response vector as \(y\), the design matrix as \(X\), and assume \(X\) has full column rank. The fitted values are \(\hat{y}=Hy\), where \(H=X(X^\mathsf{T}X)^{-1}X^\mathsf{T}\) is the hat matrix. For observation \(i\), write \(e_i=y_i-\hat{y}_i\) for its residual and \(h_{ii}\) for the \(i\)th diagonal element of \(H\), called its leverage.

For the scaled-residual results below, use the regression model \(y=X\beta+\varepsilon\), with fixed \(X\), independent errors having mean zero and common variance \(\sigma^2\). For exact \(t\)-distribution results, also assume the errors are normally distributed. The residual mean square is \(s^2=\mathrm{SSE}/(n-p)\), where \(\mathrm{SSE}=\sum_{j=1}^n e_j^2\), and \(s\) estimates the error standard deviation \(\sigma\).

The diagonal leverage values satisfy \(0\le h_{ii}\le 1\). A leverage value of one is possible even when \(X\) has full column rank and the model includes an intercept. Whenever a formula below divides by \(1-h_{ii}\), it therefore requires the additional condition \(h_{ii}<1\) for the observation being assessed. In a full-rank design, this is equivalent to deleting that observation without losing full column rank.

Why Residuals Have Different Variability

Under the stated model, \(\hat{y}=Hy\) and the residual vector is \(e=(I-H)y=(I-H)\varepsilon\), because \((I-H)X\beta=0\). The hat matrix is symmetric and idempotent, so \(I-H\) is also symmetric and idempotent. The residual vector therefore has mean zero and covariance matrix \(\sigma^2(I-H)\). In particular, the variance of an individual residual depends on its leverage:

This result explains why raw residuals do not all have the same natural scale. A high-leverage observation has a smaller residual variance under the model, while a lower-leverage observation has a larger one. The statement concerns residual variability under the assumed model; it does not mean that high leverage makes an observation harmless.

Dividing by the estimated residual standard deviation for that observation gives the internally standardized residual:

Many introductory sources call this a standardized residual; it is also called an internally studentized residual. Its denominator uses \(s\) computed from the full fit, including observation \(i\). The adjustment by \(\sqrt{1-h_{ii}}\) accounts for the residual's model-based variance, while \(s\) estimates the unknown error scale.

Studentizing by Leaving the Case Out

The internally standardized residual uses a scale estimate that includes the case being assessed. An observation with a large residual contributes to \(\mathrm{SSE}\), and can increase \(s\), making its own internally standardized residual smaller. An externally studentized residual instead estimates the error scale after deleting observation \(i\) and refitting the model.

Provided \(h_{ii}<1\), deleting observation \(i\) leaves the design matrix full rank, and the deleted residual sum of squares can be written as \(\mathrm{SSE}_{(i)}=\mathrm{SSE}-e_i^2/(1-h_{ii})\). Define \(s_{(i)}^2=\mathrm{SSE}_{(i)}/(n-p-1)\). When \(n-p-1>0\), the externally studentized residual is

The deleted fit estimates the response for observation \(i\) without using its observed response. Its prediction error is \(e_i/(1-h_{ii})\); the corresponding prediction standard error is \(s_{(i)}/\sqrt{1-h_{ii}}\). Their ratio is the same \(t_i\) shown above. Under the fixed-design normal-error model, this statistic has a \(t\) distribution with \(n-p-1\) degrees of freedom when the required deletion and scale conditions hold. Without normal errors, it can still be used as a diagnostic scale comparison, but that exact reference distribution is not justified.

What Residual Size Can and Cannot Establish

A large absolute standardized or studentized residual is evidence that the response is unusual relative to the fitted model. As a rough screening convention, values with absolute size above 2 may merit a closer look, and values above 3 are often treated as stronger flags. These are not universal decision boundaries. The model, sample size, error assumptions, and consequences of a false flag all matter. When many observations are screened, some may look unusual by chance, so a threshold should not be treated as a stand-alone formal test.

The word evidence matters. A flagged response might reflect a recording or measurement error, a legitimate but unusual case, a predictor or group omitted from the model, or a pattern showing that the model does not describe the data adequately. The residual alone cannot tell which explanation applies. Check the record and the data-collection context, and assess whether the model is appropriate before drawing conclusions.

Most importantly, an unusual response is not automatically an influential observation. A residual describes the vertical discrepancy between an observed response and a fitted response. Influence concerns how much the fitted model changes when a case is removed. A case can have a large residual without substantially changing the fitted coefficients; a high-leverage case can affect a fit even with a small residual. Residual size alone does not settle that question.

Common Mistake

Wrong reasoning: “This observation has the largest raw residual, so it must be the most unusual case and must be driving the regression.” The first part ignores that residual variability depends on leverage; the second confuses an unusual response with influence.

Correction: Start with the raw residual to see the discrepancy in context and response units. Then use an internally standardized or externally studentized residual, when its conditions are satisfied, to compare the discrepancy with an estimated residual scale. Treat a large scaled value as a reason to investigate, not as proof of error or influence. Assessing how the fit changes when cases are deleted is a separate question.