Choosing Influence Screening Cutoffs

Why Screening Cutoffs Need Careful Interpretation

Influence diagnostics summarize different features of an observation. Leverage describes how unusual its predictor values are; a studentized residual describes how unusual its response is relative to the fitted model; and measures such as Cook’s distance, DFFITS, and DFBETAS describe aspects of how the fitted results change when the observation is omitted. Screening cutoffs help identify cases for closer examination, but they do not decide whether a case is wrong, influential in a practically important sense, or safe to remove.

The cutoffs are guidelines rather than universal boundaries. Some change with the sample size, some with both sample size and model size, and some are fixed rules of thumb. Because they measure different things, it is possible for a case to cross one guideline but not another. A useful analysis records which rule was used and treats its result as a prompt for investigation.

Model Size and the Guidelines

Use the ordinary least squares regression model \(Y=X\beta+\epsilon\), where \(n\) is the number of observations and \(X\) is the design matrix. Assume \(X\) has full column rank, and let \(q\) be the number of fitted coefficients, including the intercept if one is fitted. Thus \(n>q\) is needed to estimate the full-model residual variance; deleted-fit diagnostics that estimate variance after removing an observation require \(n>q+1\). For exact \(t\)-distribution results for externally studentized residuals, also assume independent normally distributed errors with mean zero and constant variance. The screening rules below are heuristics, not tests whose validity depends on a chosen significance level.

Write \(h_{ii}\) for observation \(i\)’s leverage, \(e_i\) for its ordinary residual, and \(t_i\) for its externally studentized deleted residual. Write \(D_i\) for its Cook’s distance, \(DFFITS_i\) for its standardized effect on its own fitted value, and \(DFBETAS_{ij}\) for its standardized effect on coefficient \(j\). These diagnostics have different scales and interpretations, so their numerical cutoffs should not be compared as if they were interchangeable.

For these conventions, the externally studentized deleted residual is the residual for observation \(i\), scaled using a residual standard deviation estimated after omitting that observation and adjusted for its leverage. DFFITS measures the case’s effect on its own fitted value, standardized by its deleted-fit uncertainty. DFBETAS measures the change in a particular coefficient, standardized by its deleted-fit uncertainty. Cook’s distance summarizes the change in the fitted values across the observations. Their values may be calculated by software; the key task here is applying and interpreting the guidelines consistently.

Why Leverage and Other Cutoffs Depend on Sample Size

The leverage rule is tied to the average leverage. For a full-rank design matrix, the hat matrix is \(H=X(X'X)^{-1}X'\), and \(h_{ii}\) is its \(i\)th diagonal entry. The sum of the diagonal entries is its trace:

The last equality follows because \((X'X)^{-1}X'X\) is the \(q\)-by-\(q\) identity matrix, whose trace is \(q\). Therefore the average leverage is \(q/n\). The \(2q/n\) and \(3q/n\) rules flag observations whose leverage is two or three times that average. This also explains why adding fitted coefficients while holding \(n\) fixed raises these leverage reference values, while increasing \(n\) with \(q\) fixed lowers them.

The \(2\sqrt{q/n}\) DFFITS guideline also changes with both \(q\) and \(n\). The DFBETAS guideline \(2/\sqrt{n}\) changes with \(n\), but not directly with \(q\). By contrast, the common studentized-residual reference values 2 and 3 do not explicitly contain either \(n\) or \(q\), and the Cook’s-distance rule \(4/n\) depends explicitly on \(n\). These differences are one reason the same observation can be flagged by one rule and not another.

A Numerical Example of Sample-Size Dependence

For fixed \(q\), larger samples generally lower the leverage, DFFITS, DFBETAS, and \(4/n\) screening thresholds. This does not mean that a larger sample automatically makes cases more influential. It means only that the numerical reference used by these guidelines has changed. The following hypothetical comparison illustrates how the same diagnostic values can be classified differently under different sample sizes.

Cutoffs Are Not Decisions

A screening flag is not a formal hypothesis test and does not provide a probability that a case is erroneous or influential. In particular, comparing a studentized residual with 2 does not mean that there is a 5% chance the case is an outlier. The heuristic threshold is not a significance level, and screening several observations with several diagnostics can produce flags even when no individual data point is a mistake.

Nor does crossing a cutoff tell you what to do next. Check the observation’s original record and units; assess whether it comes from the population or process of interest; inspect its predictor values and residual; and consider which diagnostic changed and what part of the fitted model it concerns. A large DFBETAS value for one coefficient, for example, has a coefficient-specific interpretation, while a large leverage value alone indicates unusual predictor geometry, not necessarily a large effect on the fit.

Different guidelines can also disagree because they summarize different features. A case may have unusual predictor values but a small residual, or a large residual without unusual leverage. A combined measure can respond differently from either feature alone. The appropriate conclusion is not that one diagnostic must be right and the others wrong; instead, use the disagreement to guide a more careful review.

Common Mistakes

• Wrong: “The case exceeds \(4/n\), so it must be removed.” Correction: \(4/n\) is a screening guideline for Cook’s distance, not a deletion rule. Verify the data and evaluate the case’s relevance before deciding how to report or model it.
• Wrong: “A residual cutoff of 2 means the observation is statistically significant at the 5% level.” Correction: The cutoff is a rule of thumb, not a test at a specified significance level. Do not assign it a p-value or treat it as a probability that the case is unusual.
• Wrong: “All diagnostics should flag the same cases.” Correction: Leverage, residual size, fitted-value change, coefficient change, and overall fitted-value change are distinct features. Agreement is not required.
• Wrong: “The number \(2q/n\) can be used without defining \(q\).” Correction: Here \(q\) counts all fitted coefficients, including the intercept if present. Different conventions for counting model size can lead to different thresholds.