Influence Diagnostic Plots

Why Plot Residual Size Against Leverage?

An observation can be unusual in two different ways. Its response may be far from the value predicted by the fitted regression, giving it a large residual. Or its predictor values may be unusual, giving it high leverage. Neither feature alone tells the whole story about influence: a case is especially important to assess when its residual and leverage combine to make the fitted regression sensitive to that case. A residual-versus-leverage plot displays these two ingredients together, with Cook’s distance contours summarizing their combined effect.

The plot is a screening and interpretation tool, not a formal hypothesis test. Its purpose is to help identify cases whose influence may merit attention. A point outside a commonly drawn contour is not automatically an error, and the plot alone does not establish why that case differs from the others.

Quantities Shown on the Plot

Consider an ordinary least-squares regression with \(n\) observations and \(p\) fitted coefficients, counting the intercept if the model includes one. Write the model as \(y=X\beta+\varepsilon\), where \(X\) is the design matrix, \(\beta\) is the coefficient vector, and \(\varepsilon\) is the error vector. For observation \(i\), let \(y_i\) be its observed response, \(\hat{y}_i\) its fitted response, and \(e_i=y_i-\hat{y}_i\) its residual.

The hat matrix is \(H=X(X'X)^{-1}X'\), when \(X\) has full column rank. Its diagonal entry \(h_{ii}\) is the leverage of observation \(i\). Leverage describes how unusual the observation’s predictor values are relative to the predictor configuration of the fitted sample; it does not measure how far its response is from the fitted value. In a full-rank model, the leverages sum to \(p\), so their average is \(p/n\). This average is a reference point, not a universal cutoff for high leverage.

Residuals have the units of the response. To compare residual size across observations, the plot commonly uses the internally studentized residual. Let \(s^2=\sum_{j=1}^{n}e_j^2/(n-p)\), the residual mean square, and let \(s=\sqrt{s^2}\).

The factor \(\sqrt{1-h_{ii}}\) accounts for the fact that, under the usual equal-variance regression model, residuals at observations with higher leverage have smaller variance. Thus \(r_i\) expresses the residual relative to its estimated standard deviation. Its sign indicates whether the observed response is above or below its fitted value; its magnitude indicates the standardized residual size.

Conditions for a Defined Diagnostic

For the usual OLS diagnostic interpretation, assume a full-rank design matrix and errors that are independent, have mean zero, and have common variance \(\sigma^2\). Normality is not needed simply to construct this plot or compute Cook’s distance. The calculation also requires the following conditions: \(n>p\), a positive residual mean square \(s^2>0\), and, for every plotted observation, \(0<h_{ii}<1\). The condition \(h_{ii}<1\) ensures that deleting that case leaves a full-rank design, so the deletion comparison underlying Cook’s distance is defined. The condition \(h_{ii}>0\) ensures the case has a nonzero predictor-space contribution and allows the contour formula below to be written at that leverage value.

These qualifications matter. If \(n=p\), there are no residual degrees of freedom for estimating \(s^2\). If \(h_{ii}=1\), then the internally studentized residual has a zero denominator, and deletion of that case does not leave a full-rank design. Such a point cannot be interpreted using the ordinary plotted coordinates and Cook’s-distance contours described here.

Cook’s Distance Contours

Cook’s distance compares the fitted regression with the fit obtained after deleting one observation. In the standard definition, the squared differences between the two sets of fitted values are summed and divided by \(p s^2\), using the full-sample residual standard deviation \(s\). Larger \(D_i\) indicates a larger change in fitted values relative to the scale of residual variation.

For a full-rank design, the OLS case-deletion identity gives the coefficient change \(\hat{\beta}-\hat{\beta}_{(i)}=(X'X)^{-1}x_i e_i/(1-h_{ii})\), where \(x_i\) is the predictor row for case \(i\), written as a column vector. This identity applies when deletion leaves a full-rank design, as required above. It leads directly to a useful link between the plot and Cook’s distance.

For the first equality, the vector of fitted-value changes is \(X(\hat{\beta}-\hat{\beta}_{(i)})\), whose squared length is the quadratic form in the numerator. Substituting the deletion identity into that quadratic form gives the second line, using \(x_i'(X'X)^{-1}x_i=h_{ii}\). For the final line, substitute \(r_i^2=e_i^2/[s^2(1-h_{ii})]\). This shows why neither the residual nor leverage alone determines Cook’s distance: both enter the expression.

A contour for a fixed Cook’s distance value \(c>0\) consists of points satisfying

The two signs produce a contour above and below zero because positive and negative residuals of the same magnitude have the same Cook’s distance. A plot may show reference contours such as \(D_i=0.5\) and \(D_i=1\). These are visual guides, not significance levels or universal decision rules; interpreting a case also requires the model, data, and purpose of the analysis.

How to Read the Plot

Read each point using both axes and the contours. A point farther to the right has higher leverage; one farther from the horizontal axis has a larger absolute internally studentized residual. A point on a Cook’s distance contour has the value of \(D_i\) named by that contour. If a point falls between two contours, its Cook’s distance falls between their values. Contours are usually symmetric, so the sign of the residual distinguishes whether the observed response is above or below the fitted response, not whether Cook’s distance is large.

A useful reading sequence is to locate the point, note its vertical and horizontal positions, and then compare it with the displayed contours. The reference \(p/n\) can help describe leverage relative to the sample average, but it does not set a universal flagging threshold. Likewise, contour values help compare the combined influence indicated by the plot, but they do not replace knowledge of the application or a reasoned follow-up.

Why the Plot Matters—and a Common Mistake

A residual-versus-leverage plot provides a compact view of two distinct ingredients in case influence. It can reveal cases that might be overlooked by a residual-only display, and its contours translate the combination into Cook’s distance. This is useful because influence concerns changes to the fitted regression, rather than unusualness in only the response or only the predictors.

Common mistake: “This case has a large residual, so it must be highly influential.” A large residual can contribute to influence, but Cook’s distance also depends on leverage. The correction is to assess both coordinates and their Cook’s distance contours. The reverse mistake is also possible: a case with a modest residual is not guaranteed to be unimportant if its leverage is high. In either direction, a flagged case is a prompt to examine the analysis, not an automatic instruction to delete an observation or a claim that the model is wrong.