Diagnosing Large Residuals With Low Leverage

When a Large Residual Does Not Come With High Leverage

A case can have an unusual response even when its predictor values are ordinary. In a hypothetical model of claim-processing time, for example, a policy with typical recorded characteristics might have a processing time far above its fitted value. That case may be a response outlier, but it does not follow that it has high leverage or substantially changes the fitted regression.

These are separate diagnostic questions. A residual measures the gap between an observed response and its fitted value. Leverage describes how unusual the predictor values are relative to the design matrix. Influence measures assess how much a fitted result changes when a case is removed. A sound diagnosis keeps these questions distinct, then combines their answers.

Definition: In an ordinary least squares model with \(n\) observations and \(p\) fitted coefficients, let \(e_i=y_i-\hat{y}_i\) be the residual for case \(i\), \(h_{ii}\) its diagonal hat-matrix value, and \(MSE=SSE/(n-p)\). The internally studentized residual is \(r_i=e_i/\sqrt{MSE(1-h_{ii})}\). If \(s_{(i)}^2\) is the residual mean square after case \(i\) is deleted, the externally studentized residual is \(t_i=e_i/[s_{(i)}\sqrt{1-h_{ii}}]\).

Here, \(p\) includes the intercept when the model has one. The average leverage is \(p/n\), and a common screening guideline flags values above \(2p/n\) for review; it is not a universal decision threshold. A low-leverage case has predictor coordinates that are not especially unusual for the fitted design. Low leverage does not mean that its response must be ordinary.

What Studentization Adds

A raw residual is measured in response units and cannot be compared meaningfully across models with different response scales. Studentization scales it by an estimate of its standard deviation. Under the usual fixed-design linear model with independent normal errors having mean zero and common variance \(\sigma^2\), the internally studentized residual accounts for the fact that fitted values are more variable at high-leverage observations. Its denominator contains \(\sqrt{1-h_{ii}}\), so a given raw residual is more unusual when leverage is higher.

The externally studentized residual also estimates the error variance without using case \(i\). Under those same assumptions, and when \(n>p+1\), \(t_i\) has a \(t\) distribution with \(n-p-1\) degrees of freedom if the model and case \(i\) are otherwise as specified. This makes it useful for assessing whether a response is unusually far from the fitted value. The exact reference distribution relies on the normal-error model; the diagnostic’s basic purpose does not require claiming that real errors are exactly normal.

Result: The deleted residual sum of squares satisfies \(SSE_{(i)}=SSE-e_i^2/(1-h_{ii})\), and \(s_{(i)}^2=SSE_{(i)}/(n-p-1)\). A large externally studentized residual is evidence of response outlyingness relative to the fit formed without that case. It is not, by itself, evidence that the case has high leverage or that it strongly affects every coefficient.

The deleted-variance calculation is particularly informative when the ordinary residual variance is inflated by the case under review. Internally studentized residuals use the full-model \(MSE\); externally studentized residuals use the case-deleted estimate. Thus, the two measures can differ substantially. A diagnostic cutoff is a prompt for investigation, not a rule that a data point is erroneous or should be removed.

Why Residual Size and Influence Are Different

Cook’s distance summarizes the change in the fitted values across the design points when one case is deleted. With a full-rank design matrix \(X\), ordinary least squares gives \(\hat{\boldsymbol{\beta}}=(X^\mathsf{T}X)^{-1}X^\mathsf{T}\mathbf{y}\). Deleting case \(i\) changes the coefficient vector by

\hat{\boldsymbol{\beta}}-\hat{\boldsymbol{\beta}}_{(i)} =\frac{(X^\mathsf{T}X)^{-1}\mathbf{x}_i e_i}{1-h_{ii}}.

The resulting difference in fitted values over all design rows has squared length \(e_i^2h_{ii}/(1-h_{ii})^2\). Dividing by \(pMSE\) gives Cook’s distance. This explains why a large residual matters, but leverage also matters: with the same residual and error scale, a lower-leverage case generally has less overall effect on the fitted values.

D_i=\frac{e_i^2}{pMSE}\frac{h_{ii}}{(1-h_{ii})^2}.

DFFITS focuses instead on the change in the fitted value for case \(i\) itself, standardized using the deleted variance:

\operatorname{DFFITS}_i=t_i\sqrt{\frac{h_{ii}}{1-h_{ii}}}.

Both formulas include leverage as well as residual information. Therefore a large studentized residual can coexist with a less striking influence measure when leverage is low. Conversely, an ordinary-sized residual at an unusually high-leverage point can still merit attention. These measures complement one another rather than providing interchangeable labels.

Check the response gap.
Record the residual’s sign and size, then assess its internally or externally studentized value using the appropriate variance estimate.

Check the predictor location.
Compare \(h_{ii}\) with the average \(p/n\) and any chosen screening guideline. A response outlier need not be a leverage point.

Assess impact separately.
Use a measure such as Cook’s distance or DFFITS to ask whether the fitted results change materially when the case is deleted.

Investigate before acting.
Check the data record, context, and model adequacy. Keep a valid observation unless there is a defensible reason to correct or exclude it.

Worked Diagnostics

Worked Example: A Low-Leverage Case With an Extreme Deleted Residual

A hypothetical simple regression has \(n=9\) observations, \(p=2\) coefficients including the intercept, \(SSE=28.5\), and \(MSE=28.5/7\). One case has residual \(e_i=5\) and leverage \(h_{ii}=1/9\). Calculate its internal and external studentized residuals, Cook’s distance, and DFFITS. The stipulated regression summaries are compatible with a low-leverage case; no interest or mortality assumptions apply to this statistical example.

Method and leverage check. The average leverage is \(p/n=2/9\), and the common screening guideline \(2p/n\) is \(4/9\). Since \(1/9\) is below both, this case is not high leverage by either comparison.

Internal studentization. First, \(MSE=28.5/7=4.0714286\), and \(\sqrt{MSE(1-h_{ii})}=\sqrt{(28.5/7)(8/9)}=\sqrt{3.6190476}\approx1.9024\). Therefore \(r_i=5/1.9024\approx2.629\).

External studentization. The deleted sum of squares is \(SSE_{(i)}=28.5-25/(8/9)=28.5-28.125=0.375\). The deleted degrees of freedom are \(9-2-1=6\), so \(s_{(i)}^2=0.375/6=0.0625\) and \(s_{(i)}=0.25\). Thus \(t_i=5/[0.25\sqrt{8/9}]=5/0.2357023\approx21.213\).

Influence calculations. Cook’s distance is \(D_i=[25/(2(28.5/7))][(1/9)/(8/9)^2]\). The first factor is \(25/[2(28.5/7)]=175/57\approx3.0702\); the second is \(9/64=0.140625\). Therefore \(D_i=(175/57)(9/64)=1575/3648\approx0.4317\). DFFITS is \(21.2132\sqrt{(1/9)/(8/9)}=21.2132/\sqrt{8}=7.500\).

Interpretation and check. The large external studentized residual signals a response far from the case-deleted fit, while leverage is low. The large DFFITS also indicates a substantial change in this case’s fitted value. The example illustrates that low leverage does not guarantee low impact when the residual is very large. The corrected arithmetic checks: the Cook factor is \(3.0702\), not \(3.0691\), and Cook’s distance rounds to \(0.4317\), not \(0.4316\). The unusual deleted variance also warrants checking the other observations and the data record before drawing a conclusion.

Worked Example: Correcting an External Studentization Calculation

For another hypothetical regression case, suppose \(e_i=4\), \(h_{ii}=0.04\), and the deleted variance estimate is \(s_{(i)}^2=3.493161\), so \(s_{(i)}=1.8690\). Find the external studentized residual and DFFITS. These are supplied diagnostic summaries; no insurance interest or mortality assumptions are involved.

External studentized residual. The leverage factor is \(\sqrt{1-h_{ii}}=\sqrt{0.96}\approx0.9797959\). Hence \(t_i=4/(1.8690\times0.9797959)\approx4/1.8312385\approx2.1843\). If the square root is rounded first to \(0.9798\), the displayed denominator is \(1.8690\times0.9798=1.8312462\), which gives the same result to four decimals.

DFFITS. The multiplier is \(\sqrt{0.04/0.96}=\sqrt{1/24}\approx0.204124\). Thus \(\operatorname{DFFITS}_i=2.1843(0.204124)\approx0.4459\).

Interpretation and check. The external residual is appreciably larger in magnitude than the DFFITS value because the case has low leverage. The residual assesses how unusual the response is; DFFITS also reflects the limited leverage. A common arithmetic trap is to report \(t_i=2.183\). The denominator is about \(1.8312\), so the quotient is approximately \(2.1843\), not \(2.183\).

Worked Example: Comparing Residual Size and Leverage in Cook’s Distance

Consider two hypothetical diagnostic profiles from models with \(n=40\), \(p=2\), \(MSE=4\), and residual \(e_i=3\). One has leverage \(0.05\), and the other has leverage \(0.40\). Compare their leverage flags and Cook’s distances. Treat the profiles as a formula comparison, not as claims about an actual insurance portfolio.

Leverage screening. The average leverage is \(2/40=0.05\), and the guideline \(2p/n\) is \(0.10\). The first case, with leverage \(0.05\), is not flagged by that guideline; the second, with leverage \(0.40\), is.

Cook’s distances. The residual and scale factor shared by both profiles is \(e_i^2/(pMSE)=9/(2\times4)=1.125\). For the low-leverage profile, \(D_i=1.125(0.05/0.95^2)=1.125(0.0554017)\approx0.0623\). For the high-leverage profile, \(D_i=1.125(0.40/0.60^2)=1.125(1.111111)\approx1.2500\).

Interpretation and check. The residuals are equal, but the high-leverage profile has a much larger Cook’s distance. The lower-leverage case can still have an unusual response; it simply does not receive the same leverage multiplier in this influence measure. The calculations check by substituting each leverage value into the same formula, with no change to \(e_i\), \(p\), or \(MSE\).

Common Traps and Practical Interpretation

Calling every large residual influential. A residual addresses the response gap. Check leverage and an influence measure before claiming that coefficients or fitted values change substantially.
Calling every low-leverage case harmless. A large residual can still produce a large studentized residual or meaningful influence. Low leverage limits one component of influence; it does not erase the residual.
Mixing the variance estimates. The internally studentized residual uses the full-model \(MSE\); the externally studentized residual uses \(s_{(i)}^2\). State which one is being calculated.
Using a cutoff as an automatic deletion rule. Cutoffs are screening aids. Verify data entry, consider whether the observation belongs to the population of interest, and examine whether the model assumptions are reasonable before changing the analysis.

For a low-leverage case with a large studentized residual, the practical conclusion is deliberately limited: the observed response is unusual relative to the fitted relationship. Whether it is a recording error, a valid but rare outcome, or evidence that the model does not describe the case well requires contextual investigation. Report how sensitive the analysis is to the case when that sensitivity matters, and distinguish that assessment from the decision to retain or remove it.

Key takeaway: Diagnose the response and the predictor location separately. Studentized residuals assess response outlyingness; leverage describes predictor unusualness; Cook’s distance and DFFITS assess aspects of influence. A low-leverage case may have a large residual, and no single diagnostic statistic determines what to do with it.

Check Your Understanding

For each problem, calculate the requested diagnostic or state the interpretation. Assume ordinary least squares and use the formulas given above.

A case has \(e_i=-3\), \(h_{ii}=0.04\), and \(MSE=4\). Calculate its internally studentized residual. (Answer: \(r_i\approx-1.5309\))
A model has \(p=3\), \(MSE=2\), and a case with \(e_i=2\) and \(h_{ii}=0.08\). Calculate Cook’s distance. (Answer: \(D_i\approx0.0630\))
A case has \(e_i=3\), \(h_{ii}=0.19\), and deleted standard error \(s_{(i)}=1.5\). Calculate its externally studentized residual. (Answer: \(t_i=2.2222\))
A model has \(n=50\) and \(p=4\). A case has \(h_{ii}=0.07\), \(e_i=3\), and \(MSE=4\). Compare its leverage with the guideline \(2p/n\), then calculate Cook’s distance. (Answer: \(0.07<0.16\), so it is below the guideline; \(D_i\approx0.0455\))
A model has \(n=14\), \(p=2\), \(SSE=48\), and a case has \(e_i=3\), \(h_{ii}=0.10\). Calculate the deleted variance estimate and external studentized residual. (Answer: \(SSE_{(i)}=38\), \(s_{(i)}^2=38/11\approx3.4545\), and \(t_i\approx1.7014\))

What You'll Learn