Influence Plots Combining Residuals and Leverage

Why Combine Residuals and Leverage?

A regression diagnostic can flag an observation for different reasons. Its response may be unusual relative to the fitted model, its predictor values may be far from the rest of the data, or it may affect the fitted regression substantially. These are related but distinct features. An influence plot brings two of them together by placing studentized residuals against leverage, often adding Cook’s-distance contours or point sizes.

The plot is useful because neither coordinate alone gives the full picture. A large residual at low leverage may have little effect on the fitted regression. A high-leverage case with a small residual may fit the model closely and have modest influence. Cook’s distance summarizes how residual size and leverage combine to measure a case’s effect on the fitted coefficients.

What the Plot Displays

Assume the ordinary least-squares model is fitted using a full-rank design matrix \(X\), with \(n\) rows and \(p\) columns, including the intercept column if the model has an intercept. The hat matrix is \(H=X(X^\mathsf{T}X)^{-1}X^\mathsf{T}\); the diagonal entry \(h_{ii}\) is observation \(i\)’s leverage. Leverage depends on the predictor values, not on the observed response.

Let \(e_i=y_i-\hat y_i\) be the ordinary residual, and let \(s^2=\mathrm{SSE}/(n-p)\) be the full-model residual mean square. The internally studentized residual is \(r_i=e_i/[s\sqrt{1-h_{ii}}]\). It accounts for the fact that observations with higher leverage tend to have smaller residual variance under the model. An externally studentized residual instead uses the residual standard error \(s_{(i)}\) estimated after omitting observation \(i\): \(t_i=e_i/[s_{(i)}\sqrt{1-h_{ii}}]\).

Many influence plots use the externally studentized residual, but conventions vary. Check the plot’s legend or documentation before interpreting numerical contours. The direct Cook’s-distance contour formula below uses the internally studentized residual. When the vertical axis is externally studentized, the conversion is different.

Cook’s-Distance Contours

For the full least-squares fit, Cook’s distance for observation \(i\) is

where \(\hat{\boldsymbol{\beta}}\) is the full-model coefficient estimate and \(\hat{\boldsymbol{\beta}}_{(i)}\) is the estimate after deleting observation \(i\). The first expression measures the coefficient change on a scale that accounts for the design and residual variance. The second expression shows why Cook’s distance can be read from the residual and leverage coordinates together.

For a fixed Cook’s-distance contour \(D_i=c\), the internally studentized-residual magnitude at leverage \(h_{ii}\) is

As leverage increases, a smaller residual magnitude can reach the same Cook’s-distance contour. Conversely, at low leverage, the residual must be larger to produce the same value of \(D_i\). This is the central visual message of the contours.

Why the Formula Holds

The case-deletion identity for least squares gives \(\hat{\boldsymbol{\beta}}-\hat{\boldsymbol{\beta}}_{(i)} =(X^\mathsf{T}X)^{-1}\mathbf{x}_i e_i/(1-h_{ii})\), where \(\mathbf{x}_i\) is the predictor vector for observation \(i\). Substituting this coefficient change into the defining quadratic form yields

The last equality follows from the definition \(h_{ii}=\mathbf{x}_i^\mathsf{T}(X^\mathsf{T}X)^{-1}\mathbf{x}_i\). Dividing by \(p s^2\), then using \(r_i^2=e_i^2/[s^2(1-h_{ii})]\), gives \(D_i=r_i^2h_{ii}/[p(1-h_{ii})]\). This algebraic identity holds for the least-squares fit when the deletion is defined; its interpretation as a measure of model influence does not require normally distributed errors.

When the Plot Uses Externally Studentized Residuals

If the vertical coordinate is the externally studentized residual \(t_i\), do not substitute \(t_i\) directly into the internally studentized contour formula. Let \(\nu=n-p\). The deleted residual variance satisfies \(s_{(i)}^2/s^2=(\nu-r_i^2)/(\nu-1)\), and therefore \(t_i^2=r_i^2(\nu-1)/(\nu-r_i^2)\). Solving for \(r_i^2\) gives

Thus the corresponding Cook’s distance, expressed using the external residual, is \(D_i=\dfrac{h_{ii}}{p(1-h_{ii})}\dfrac{t_i^2\nu}{\nu-1+t_i^2}\). Under the usual independent, constant-variance normal-error model, the externally studentized residual also has a \(t\) distribution with \(\nu-1\) degrees of freedom when the model is correct. That distributional result is useful for residual testing, but it is not needed for the conversion or the Cook’s-distance identity.

Reading the Plot Without Overstating It

A practical reading separates the two axes before considering the Cook’s-distance contours. First note which points have large absolute studentized residuals. Then identify points with unusual leverage relative to the rest of the design. Finally, look for points that also lie near or beyond substantial Cook’s-distance contours, or that are drawn with conspicuously large Cook’s-distance bubbles.

Reference values such as \(D_i=1\) or \(4/n\), and leverage rules based on multiples of \(p/n\), are screening conventions rather than universal decision boundaries. A point just below a reference can still matter in context, and a point above one is not automatically invalid. Check the data for recording problems, reconsider whether the model is appropriate, and report diagnostic findings with the analysis.

Common Reading Errors

• Equating a response outlier with an influential case. A large residual is one ingredient; leverage also affects Cook’s distance.
• Equating leverage with influence. A high-leverage point can have a small residual and modest Cook’s distance.
• Ignoring the residual definition. The simple contour equation uses internally studentized residuals. Convert an external residual first, or use a contour defined for that residual type.
• Treating a contour as a deletion rule. A contour identifies a diagnostic signal to investigate; it does not prove an observation should be removed.