Why Compare Diagnostic Measures?
A regression diagnostic is useful only when its question is clear. A case can have an unusual response relative to its fitted value, occupy an unusual position in predictor space, or substantially change the fitted model when omitted. These are related but different features. No single diagnostic summarizes all of them.
This comparison uses \(p\) for the number of fitted regression coefficients, including the intercept, and \(n\) for the number of observations. Write \(e_i=y_i-\hat{y}_i\) for the ordinary residual, \(h_{ii}\) for the leverage, \(s^2=\mathrm{SSE}/(n-p)\) for the full-model residual mean square, and \(s_{(i)}^2\) for the residual mean square after deleting case \(i\). For a full-rank ordinary least squares model, \(h_{ii}<1\). The standard algebraic diagnostics below do not require normally distributed errors; exact small-sample \(t\)-inference based on studentized residuals additionally uses the normal-error model.
The measures answer distinct questions. A studentized residual focuses on the response discrepancy, leverage on predictor position, and the influence measures on consequences of deletion. Influence is not itself proof that a record is erroneous: a valid but unusual observation can matter to the fitted model.
A Compact Comparison of the Measures
For the usual least-squares diagnostic definitions, the internally studentized residual, externally studentized residual, and Cook’s distance are
The case-deletion residual sum of squares and the DFFITS definition are
For coefficient \(j\), let \(c_{jj}\) be the \(j\)th diagonal element of \((X^\mathsf{T}X)^{-1}\). With the full-model coefficient estimate minus the deleted-case estimate in the numerator, the standard DFBETAS definition is
The sign of a DFFITS or DFBETAS value retains direction: it indicates whether the full-data estimate is above or below its case-deletion counterpart. Cook’s distance is nonnegative and has no direction. The value of an internally studentized residual and an externally studentized residual retains the sign of the ordinary residual.
Worked Comparison in a Small Regression
Worked Example: Comparing Five Diagnostics for One Case
A hypothetical insurer fits a simple linear regression of a policy-level response \(y\) on a numerical predictor \(x\), using the six observations below. Treat this as an ordinary least squares model with an intercept. For case \(i\) at \(x=1\), calculate its leverage, both studentized residuals, Cook’s distance, DFFITS, and DFBETAS for the intercept and slope.
Data and fitted line. The observations are \((x,y)=(0,11),(1,10),(2,15),(3,16),(4,18),(5,20)\). The response values equal \(10+2x\) plus residuals \(1,-2,1,0,0,0\). Those residuals sum to zero and their products with \(x\) sum to \(0(1)+1(-2)+2(1)=0\). Thus the least-squares line is \(\hat{y}=10+2x\), and the ordinary residuals are exactly \(1,-2,1,0,0,0\). Therefore \(\mathrm{SSE}=1^2+(-2)^2+1^2=6\). Here \(n=6\), \(p=2\), and \(s^2=6/(6-2)=1.5\).
Leverage. The predictor mean is \(\bar{x}=15/6=2.5\), and \(S_{xx}=\sum(x_i-\bar{x})^2=17.5\). In simple regression with an intercept, \(h_{ii}=1/n+(x_i-\bar{x})^2/S_{xx}\). For \(x_i=1\),
Studentized residuals. The ordinary residual for this case is \(e_i=-2\). Its internally studentized residual is
To calculate the external version, first obtain the deleted-case sum of squares:
The deleted fit has \(n-p-1=3\) residual degrees of freedom, so \(s_{(i)}^2=(12/37)/3=4/37\) and \(s_{(i)}=2/\sqrt{37}\). Hence
Cook’s distance and DFFITS. Cook’s distance is
DFFITS uses the external studentized residual and the leverage multiplier:
DFBETAS. In simple regression, the coefficient covariance factor is \((X^\mathsf{T}X)^{-1}\). Its intercept and slope diagonal elements are \(c_{00}=11/21\) and \(c_{11}=2/35\), respectively. The deletion identity gives
For this case, \((X^\mathsf{T}X)^{-1}\boldsymbol{x}_i=(8/21,-3/35)^\mathsf{T}\) and \(e_i/(1-h_{ii})=-105/37\). Thus the intercept difference is \(-40/37\), and the slope difference is \(9/37\). The coefficient-specific standardized changes are
The comparison is the main result: the studentized residual flags a response discrepancy relative to the fitted line, leverage measures predictor position, Cook’s distance summarizes the overall fitted-value change, DFFITS focuses on the fitted value at the deleted observation, and the two DFBETAS values reveal different coefficient effects. The unusually small deleted-model residual variance makes the external studentized residual especially large here. All displayed diagnostic values are rounded to three decimal places.
Method and mistake avoided. The method is to calculate the full-model fit and leverage first, then use the deletion quantities for external studentization and case-deletion measures. A common mistake is to use \(s\) instead of \(s_{(i)}\) in DFFITS or DFBETAS. That would answer a different, internally scaled question and would miss how strongly this case’s deletion changes the residual variance estimate.
Why the Formulas Relate as They Do
The measures are not unrelated scores. Their connections follow from the least-squares case-deletion identity. For a full-rank design matrix \(X\), let \(\boldsymbol{x}_i\) be the predictor vector for case \(i\), including the intercept, and let \(\hat{\boldsymbol{\beta}}\) be the full-data coefficient vector. Deleting one observation changes the coefficient vector by
This identity explains why an ordinary residual alone cannot determine influence. The deletion change also depends on the design and on \(1-h_{ii}\). To see how Cook’s distance follows, the change in the fitted response at observation \(k\) is \(\boldsymbol{x}_k^\mathsf{T}(\hat{\boldsymbol{\beta}}-\hat{\boldsymbol{\beta}}_{(i)})\). Summing its square over all \(k\) gives
Dividing this total squared fitted-value change by \(p s^2\) yields Cook’s distance as defined above. In contrast, DFFITS standardizes just the change in the fitted value at case \(i\), and DFBETAS standardizes one coordinate of the coefficient change. This is why the influence measures can rank cases differently: they summarize different aspects of the same deletion.
Use studentized residuals to assess response discrepancy and leverage to describe predictor position.
Use Cook’s distance for an overall fitted-value change, DFFITS for the deleted case’s fitted value, and DFBETAS for a selected coefficient.
Check the data and model, then consider whether the case is valid and whether conclusions are sensitive to its inclusion.
Comparing Results Without Turning Them Into Rules
A diagnostic cutoff is a screening aid, not a decision rule that labels a case “bad.” Values should be read with the sample size, number of fitted coefficients, model purpose, and other diagnostics in mind. A case that stands out on one measure may not stand out on another because each measure targets a different quantity. In particular, a large studentized residual is not itself a calculation of coefficient change, and a large DFBETAS for one coefficient does not say that every fitted value changed equally.
A practical comparison reports the measure and the question it addresses. For example: “The case has a large externally studentized residual, suggesting a response discrepancy; its DFBETAS for the exposure coefficient is also large, so that coefficient merits a sensitivity check.” This is more informative than reporting a list of numbers without interpretation. Follow-up should include checking for data-entry or measurement issues, examining the model specification, and comparing conclusions with and without the case when appropriate. Do not delete a valid observation solely because a diagnostic crosses a conventional cutoff.
Check Your Understanding
For each original problem, identify the diagnostic quantity requested and show the indicated calculation before checking the answer.
- A model has \(n=24\), \(p=3\), \(e_i=6\), \(h_{ii}=0.15\), and \(s=5\). Calculate the internally studentized residual. (Answer: \(1.301\), to three decimals.)
- In a model with \(p=2\), a case has \(e_i=4\), \(h_{ii}=0.25\), and \(s=2\). Calculate Cook’s distance. (Answer: \(0.889\), to three decimals.)
- A case has externally studentized residual \(t_i=-2.4\) and leverage \(h_{ii}=0.20\). Calculate DFFITS, retaining its sign. (Answer: \(-1.200\).)
- For a selected coefficient, suppose \(\hat{\beta}_j-\hat{\beta}_{j(i)}=-0.6\), \(s_{(i)}=3\), and \(c_{jj}=0.04\). Calculate its DFBETAS and state what the negative sign compares. (Answer: \(-1.000\); the full-data coefficient estimate is below the deleted-case estimate.)
- For one case, \(|t_i|=0.5\), \(D_i=0.04\), and the absolute DFBETAS for a particular coefficient is \(2.1\). Which measure most directly identifies a comparatively large deletion effect on that individual coefficient? (Answer: DFBETAS.)