Cook's Distance for Influential Observations

Why Consider Cook’s Distance?

A point can matter to a regression fit for more than one reason. Its response may be far from the value predicted by the fitted line, or its predictor value may be unusual relative to the other cases. A large residual signals a poor fit for that case; high leverage means its predictor position gives it the potential to pull the fitted line. Neither feature by itself tells the whole story about how much the fitted regression may be affected.

Cook’s distance combines these features into one measure of a case’s potential influence on the fitted regression. It is useful for screening observations for closer examination, but it does not establish that a case is erroneous or that it should be removed. The measure is best understood alongside the regression context and other diagnostics.

Definition and Calculation

Consider a least-squares regression with \(n\) observations and \(p\) estimated coefficients, including the intercept. For example, simple linear regression estimates an intercept and a slope, so \(p=2\). Let \(e_i=y_i-\hat{y}_i\) be the residual for case \(i\), \(h_{ii}\) its leverage, and \(MSE=SSE/(n-p)\) the residual mean square from the fitted model.

This formula uses the model fitted to all \(n\) observations. The squared residual \(e_i^2\) measures how far the response is from its fitted value. The leverage term \(h_{ii}/(1-h_{ii})^2\) increases the contribution when the predictor position is unusual. The factor \(p\) accounts for the number of fitted coefficients, and division by \(MSE\) scales the residual relative to the model’s typical squared error.

The regression model should be a sensible representation of the relationship being studied. In the usual linear model, the response has a linear mean in the predictors, errors have mean zero and constant variance, and errors are independent across observations. These conditions support interpretation of the fitted regression and its diagnostics. Normal errors are not required merely to calculate Cook’s distance. As with leverage, a case’s predictor values determine its leverage; the residual reflects how well the response fits the model.

An equivalent form uses the internally standardized residual \(r_i=e_i/\sqrt{MSE(1-h_{ii})}\):

This form makes the combination especially visible: Cook’s distance increases with the squared standardized residual and with leverage. A modest residual can still contribute to a large distance when leverage is very high, while a large residual at ordinary leverage may also warrant attention.

Why Residual Size and Leverage Both Matter

A case with a large residual is poorly predicted by the fitted line, but if its predictor value is near the center of the other predictor values, it may not shift the fitted line much. Conversely, a case far from the center of the predictor values has high leverage, but if it lies close to the pattern followed by the other observations, it may have only a small residual. Cook’s distance is designed to combine both ingredients rather than label either one alone as influence.

In the equivalent expression, \(r_i^2\) represents the response discrepancy on a scale that accounts for the residual variation and leverage. The multiplier \(h_{ii}/(1-h_{ii})\) then gives more weight to high-leverage cases. The result summarizes potential impact on the fitted regression as a whole; it is not a directional measure. It does not indicate whether a slope will increase or decrease.

Screening Guidelines and Their Limits

There is no universal Cook’s-distance threshold that separates influential from non-influential observations in every data set. A common screening guideline flags values greater than \(4/n\); another rough guideline treats values near or above 1 as deserving attention. These are rules of thumb, not hypothesis tests. A value below a guideline does not prove that the case has no effect, and crossing one does not prove that the case is invalid.

A Short Algebraic Check

The equivalence of the two computational forms follows directly from the definition of the internally standardized residual. Start with \(r_i=e_i/\sqrt{MSE(1-h_{ii})}\). Squaring gives \(r_i^2=e_i^2/[MSE(1-h_{ii})]\), so \(e_i^2/MSE=r_i^2(1-h_{ii})\). Substituting this equality into the original Cook’s-distance formula gives:

This identity is useful as a calculation check. Small rounding differences between the two forms are expected when intermediate values are rounded.

Common Mistakes

• Calling every large residual influential. A large residual indicates a response that the model predicts poorly, but influence also depends on leverage. Use Cook’s distance to combine both features.
• Equating high leverage with influence. An unusual predictor value gives a case potential to affect the fit, but a small residual can limit its Cook’s distance.
• Treating \(4/n\) or 1 as a decisive cutoff. These are screening guidelines, not formal tests. A flag calls for examination, not automatic removal.
• Interpreting Cook’s distance as a probability or direction. It is a nonnegative, dimensionless diagnostic measure. It does not give the probability a case is wrong or say which way a coefficient would move.

A careful analysis uses Cook’s distance as one piece of evidence about the fitted regression. A flagged case may reveal a recording error, a model limitation, a legitimately unusual observation, or a combination of these. The appropriate response depends on the evidence and the study question; the numerical diagnostic alone cannot decide.