Why Integrated Diagnosis Matters
A regression diagnosis is not a search for one number that declares a model “good” or “bad.” Residual plots, formal tests, leverage, studentized residuals, and influence measures each highlight a different feature of the fitted data. A case can be unusual in one respect and ordinary in another: a large residual need not have high leverage, and high leverage need not imply a large effect on the fitted model.
The aim is to combine these clues, identify which modeling assumptions may be questionable, and decide what follow-up is justified. The calculations below use a hypothetical regression of claim cost on two recorded risk characteristics. The data and software results are invented for instruction and do not describe an actual insurer.
What Each Diagnostic Can Tell You
A residual-versus-fitted plot is a broad check of the mean pattern and spread. A roughly horizontal cloud centered around zero is consistent with a suitable mean form and constant variance. A curve suggests that the mean pattern may be inadequate; a widening or narrowing band suggests nonconstant variance. These patterns are clues, not proof: a plot can be inconclusive, especially with a small sample.
A normal Q–Q plot compares ordered residuals with the pattern expected under normality. Strong departures at the ends can indicate unusually heavy tails or outlying responses. A residual-versus-order plot can reveal runs, trends, or cycles that raise concern about independence when the order has a meaningful interpretation. If observations have no meaningful sequence, an apparent order pattern may not be informative.
For response outlyingness, the internally studentized residual is \(r_i=e_i/[s\sqrt{1-h_{ii}}]\). It scales the residual using the full-fit error estimate. The externally studentized residual, also called the deleted or leave-one-out studentized residual, uses the error estimate after case \(i\) is omitted. With \(s_{(i)}\) denoting that deleted-fit error estimate, it is \(t_i=e_i/[s_{(i)}\sqrt{1-h_{ii}}]\). Under the normal linear model, \(t_i\) has a \(t\) distribution with \(n-p-1\) degrees of freedom when the model is fixed and case \(i\) is assessed under the stated assumptions. A large absolute value signals an unusual response relative to the fitted model, but not by itself an influential case.
Leverage is a feature of the predictor values, not of the response. In a model with an intercept, the leverages sum to \(p\), so their mean is \(p/n\). Values appreciably above the mean merit inspection; rules such as \(2p/n\) or \(3p/n\) are screening heuristics, not hypothesis-test boundaries.
Worked Integrated Case
The following example combines plots, formal checks, and case-level measures. In each calculation, the model has \(n=24\) observations and \(p=3\) coefficients: an intercept and two predictor coefficients. Therefore the full-fit residual degrees of freedom are \(24-3=21\).
Worked Example: Combining Plots, Tests, and Influence Measures
A hypothetical claim-cost regression has \(\operatorname{SSE}=84\), so \(s^2=84/21=4\) and \(s=2\). Case 17 has \(h_{17,17}=0.35\) and \(r_{17}=2.40\). The residual-versus-fitted plot shows a wider spread at larger fitted costs. A normal Q–Q plot is approximately straight, and a residual-versus-order plot shows no evident trend. Suppose a Shapiro–Wilk test reports \(W=0.96,\ p=0.46\), while a Breusch–Pagan test reports \(LM=6.40\) on 2 degrees of freedom, \(p=0.041\). A case-deletion calculation gives \(\hat{\beta}_{1}=0.80\) for the full fit and \(\hat{\beta}_{1(17)}=0.62\) without case 17; the DFBETAS scaling denominator for this coefficient is \(0.10\). Assess the evidence and calculate the case’s externally studentized residual, Cook’s distance, DFFITS, and DFBETAS for \(\hat{\beta}_1\).
1. Read the plots and tests in context. The wider spread at larger fitted values raises concern about constant error variance. The Q–Q plot and Shapiro–Wilk result do not show strong evidence against normality; \(p=0.46\) is not a reason to reject the normality null at a 5% level. This does not prove normality. The Breusch–Pagan test’s null is constant variance; its \(p=0.041\) is evidence against that null at the 5% level. The plot and test point in a consistent direction, so the variance pattern deserves follow-up. No trend in the order plot is reassuring only if the ordering is meaningful and the plot is capable of revealing dependence.
2. Recover the raw residual and deleted error estimate. From the definition of \(r_i\), \(e_{17}=r_{17}s\sqrt{1-h_{17,17}}\). Thus \(e_{17}=2.40(2)\sqrt{0.65}=4.80(0.80623)=3.8699\). The deleted residual sum of squares is the full \(\operatorname{SSE}\) less \(e_i^2/(1-h_{ii})\). Here \(e_{17}^2/(1-h_{17,17})=r_{17}^2s^2=5.76(4)=23.04\). Therefore \(\operatorname{SSE}_{(17)}=84-23.04=60.96\), and \(s_{(17)}^2=60.96/(21-1)=3.048\).
3. Calculate the externally studentized residual. Since \(\sqrt{3.048}=1.74585\) and \(\sqrt{0.65}=0.80623\), \(t_{17}=3.8699/[1.74585(0.80623)]=2.749\), to three decimals. Its reference \(t\) distribution has \(21-1=20\) degrees of freedom under the normal linear model. This is evidence that case 17 has a relatively unusual response, but it does not establish that the case should be removed.
4. Calculate overall influence and fitted-value influence. Cook’s distance can be written using the internally studentized residual as \(D_i=(r_i^2/p)[h_{ii}/(1-h_{ii})]\). Therefore \(D_{17}=(2.40^2/3)(0.35/0.65)=1.92(0.53846)=1.034\), to three decimals. DFFITS is \(t_i\sqrt{h_{ii}/(1-h_{ii})}\). Thus \(\operatorname{DFFITS}_{17}=2.749\sqrt{0.35/0.65}=2.749(0.73380)=2.017\), to three decimals.
5. Calculate coefficient-specific influence. With the stated DFBETAS convention, \(\operatorname{DFBETAS}_{ij}=(\hat{\beta}_j-\hat{\beta}_{j(i)})/\{s_{(i)}\sqrt{C_{jj}}\}\), where the supplied denominator is the applicable scaling term for coefficient \(j\). For coefficient 1, the numerator is \(0.80-0.62=0.18\), so \(\operatorname{DFBETAS}_{17,1}=0.18/0.10=1.80\). This indicates a noticeable change in that coefficient on the stated scale.
Conclusion. The leverage \(0.35\) exceeds the screening value \(2p/n=6/24=0.25\), though it is below \(3p/n=0.375\). Case 17 also has a relatively large externally studentized residual and substantial values of Cook’s distance, DFFITS, and coefficient-specific DFBETAS. The combined evidence warrants checking the case’s data and context and examining how conclusions change with and without it. It does not warrant automatic deletion. The variance evidence applies to the model’s overall residual pattern, not necessarily to this one case alone.
Why the Influence Formulas Combine Residual and Leverage
The formulas for Cook’s distance and DFFITS make the role of leverage explicit. In ordinary least squares, omitting case \(i\) changes its fitted value by an amount proportional to \(e_i h_{ii}/(1-h_{ii})\). A residual contributes to that change, but the leverage factor determines how strongly the case can affect its own fitted value. This is why a large residual at low leverage and a modest residual at high leverage can lead to different diagnostic patterns.
For Cook’s distance, the squared change across all fitted values is scaled by the number of fitted coefficients and the residual variance. Equivalently, using the internally studentized residual gives \(D_i=(r_i^2/p)[h_{ii}/(1-h_{ii})]\). The expression has two components: \(r_i^2\) for response discrepancy and \(h_{ii}/(1-h_{ii})\) for predictor-space position. DFFITS instead focuses on the change in the fitted value for the omitted case, scaled by its deleted-fit standard error; this yields \(\operatorname{DFFITS}_i=t_i\sqrt{h_{ii}/(1-h_{ii})}\). These formulas explain why no single residual or leverage cutoff completely describes influence.
Read residual-versus-fitted, Q–Q, and—when meaningful—residual-versus-order plots. Describe the visible pattern before assigning a cause.
Use studentized residuals to flag response outlyingness and leverage to flag unusual predictor combinations.
Review Cook’s distance, DFFITS, or DFBETAS alongside the case’s data quality and the effect of case deletion on conclusions.
Two Contrasting Case Patterns
Worked Example: A Large Residual With Low Leverage
A second hypothetical regression has \(n=30\), \(p=3\), \(h_{ii}=0.04\), and internally studentized residual \(r_i=3.00\) for one observation. Calculate its Cook’s distance and DFFITS, then interpret their relationship to its large residual.
The case has low leverage relative to the mean \(p/n=3/30=0.10\). Its Cook’s distance is \(D_i=(3.00^2/3)(0.04/0.96)=3(0.041667)=0.125\). The DFFITS calculation first obtains the externally studentized residual. Here \(n-p=27\), so \(t_i=3.00\sqrt{(27-1)/(27-3.00^2)} =3.00\sqrt{26/18}=3.606\), to three decimals. Then \(\operatorname{DFFITS}_i=3.606\sqrt{0.04/0.96} =3.606(0.20412)=0.736\), to three decimals.
The absolute studentized residual is large, while low leverage limits the case’s overall effect. For orientation, the heuristic Cook’s-distance screen \(4/(n-p)\) is \(4/27=0.148\), so this Cook’s distance is below that screen; the heuristic DFFITS screen \(2\sqrt{p/n}\) is \(2\sqrt{3/30}=0.632\), so its DFFITS is above it. Different measures can therefore flag different aspects of the same observation. The appropriate conclusion is to investigate the response and its context, not to dismiss the case because one influence measure is below a heuristic screen.
Worked Example: High Leverage Without a Large Residual
In another hypothetical model, \(n=20\), \(p=2\), and a case has \(h_{ii}=0.30\) and \(r_i=0.80\). Calculate Cook’s distance and decide what the numbers do and do not establish.
The mean leverage is \(p/n=2/20=0.10\), so \(0.30\) is distinctly elevated; it also exceeds \(2p/n=0.20\). The residual itself is modest. Cook’s distance is \(D_i=(0.80^2/2)(0.30/0.70) =0.32(0.42857)=0.137\), to three decimals.
This value is below the screening heuristic \(4/(n-p)=4/18=0.222\), but the case’s predictor combination is unusual enough to merit inspection. A modest residual in the full fit does not guarantee that the case has little effect on every coefficient or on predictions elsewhere in the predictor space. A deletion comparison or a coefficient-specific measure can answer a different question. The case should be checked, not automatically excluded.
Formal Checks, Cautions, and Conclusions
Formal tests can supplement plots, but their conclusions depend on the test’s null hypothesis, sample size, and assumptions. A large \(p\)-value means the data did not provide strong evidence against the tested null; it is not proof that the assumption holds. A small \(p\)-value identifies evidence of a discrepancy, not its cause. For example, a Breusch–Pagan test can indicate nonconstant variance, but it does not identify whether the pattern comes from a missing mean structure, a variance process, a data problem, or another feature.
Likewise, an unusual studentized residual is not automatically an error in the data. Verify the record, measurement units, and predictor definitions before deciding how to proceed. If the observation is valid, report how sensitive the substantive conclusions are to its inclusion. If the case is influential, do not present a deleted fit as the only result without a defensible reason for the exclusion.
Common heuristics such as \(2p/n\) for leverage, \(4/(n-p)\) for Cook’s distance, or \(2\sqrt{p/n}\) for DFFITS are screening devices. They are not universal significance tests, and they can produce different flags. Multiple diagnostics applied to many cases also make it unsurprising that some values appear unusual by chance. A sound conclusion states the observed evidence, the diagnostic’s scope, and the follow-up needed.
Check Your Understanding
For each hypothetical case, identify what the stated diagnostics support. Treat numerical cutoffs as screening heuristics, not automatic decisions.
- A regression has \(n=30\) and \(p=4\). A case has leverage \(0.24\). Find the mean leverage and the \(2p/n\) screen. Is the case above that screen? (Answer: Mean leverage \(=0.1333\); \(2p/n=0.2667\); no.)
- A model has \(n=18\), \(p=2\), \(r_i=-2.00\), and \(h_{ii}=0.05\). Calculate Cook’s distance. (Answer: \(D_i=(4/2)(0.05/0.95)=0.1053\), approximately.)
- A case has externally studentized residual \(t_i=1.20\) and leverage \(h_{ii}=0.30\). Calculate DFFITS. (Answer: \(1.20\sqrt{0.30/0.70}=0.7856\), approximately.)
- A Shapiro–Wilk test gives \(p=0.22\), while a Breusch–Pagan test gives \(p=0.01\). At a 5% level, what is the warranted conclusion? (Answer: The normality test does not provide evidence against its null; the Breusch–Pagan test provides evidence against constant variance. Neither result proves its corresponding assumption.)
- A case has high leverage, a modest studentized residual, and a large DFBETAS for one coefficient. What should be checked before drawing a conclusion about that coefficient? (Answer: Verify the observation and predictor values, then compare the coefficient and relevant inference with and without the case; high leverage alone does not establish a response outlier or justify deletion.)