Cook's Distance from Deleted Fits

Why Summarize a Deleted Fit?

A regression fit can change when one observation is removed. The changes may occur in several coefficients and may alter predictions at many predictor settings. Cook’s distance combines those changes into one number: it compares the full-model fitted values with the fitted values from a model refitted without the observation, then scales the total squared change by the model size and the full-model residual mean square.

The key is to compare predictions at the same set of predictor rows in both fits. The sum includes a prediction at the deleted observation’s predictor values as well as predictions at the other observed predictor values. Cook’s distance is a descriptive measure of how much the fitted values change under this deletion; its calculation does not require a p-value or a decision cutoff.

Definition and Least-Squares Setup

Suppose there are \(n\) observations and \(p\) regression coefficients, including the intercept if the model has one. Write the least-squares model in matrix form as \(y=X\beta+\varepsilon\), where \(X\) has full column rank. The full-data coefficient estimate is \(\hat{\beta}\), and the fitted-value vector is \(\hat{y}=X\hat{\beta}\). Let \(e_i=y_i-\hat{y}_i\) be the residual for observation \(i\), and let \(h_{ii}\) be its diagonal hat-matrix value. The full-model residual mean square is \(s^2=\mathrm{SSE}/(n-p)\), where \(\mathrm{SSE}=\sum_{j=1}^n e_j^2\).

For each observation \(i\), \(\hat{y}_{j(i)}\) denotes the fitted value at predictor row \(j\) from the model refitted after deleting observation \(i\). The subscript \(j(i)\) marks that the fit omits \(i\); it does not mean that row \(j\) is deleted.

This definition uses the ordinary least-squares setup: the same specified predictors are used in both fits, the full design matrix has rank \(p\), \(n>p\), and deleting observation \(i\) leaves a design matrix of rank \(p\). The last condition is equivalent to \(h_{ii}<1\). The formula is an algebraic diagnostic. For the usual statistical interpretation of \(s^2\) as an estimate of error variance, the classical linear model additionally assumes a correctly specified linear conditional mean and errors with mean zero, constant variance, and independence. Normality is not needed for the algebraic identity below.

From Deleted Fits to a Least-Squares Formula

Refitting the model separately for every possible deletion is not necessary. Least-squares algebra gives the change in the coefficients, and therefore in every fitted value, directly from the full-model residual and leverage.

Here is a justification of both parts. The full least-squares normal equations are \(X^\mathsf{T}X\hat{\beta}=X^\mathsf{T}y\). After deleting row \(i\), the matrix and right-hand side in the normal equations become \(X^\mathsf{T}X-x_ix_i^\mathsf{T}\) and \(X^\mathsf{T}y-x_i y_i\). Let \(A_i=X^\mathsf{T}X-x_ix_i^\mathsf{T}\). Applying the deleted normal equations to the full estimate gives

Subtracting the first equation from the second gives \(A_i(\hat{\beta}_{(i)}-\hat{\beta})=-x_i e_i\). The hat-matrix diagonal is \(h_{ii}=x_i^\mathsf{T}(X^\mathsf{T}X)^{-1}x_i\). Direct multiplication shows that

Thus \(A_i^{-1}x_i=(X^\mathsf{T}X)^{-1}x_i/(1-h_{ii})\), which yields the coefficient identity above. The full-minus-deleted fitted-value difference vector is \(X(\hat{\beta}-\hat{\beta}_{(i)})=e_i H_{\cdot i}/(1-h_{ii})\), where \(H=X(X^\mathsf{T}X)^{-1}X^\mathsf{T}\) and \(H_{\cdot i}\) is its \(i\)th column. Since \(H\) is symmetric and idempotent, the squared length of that column is

Therefore the sum of squared fitted-value changes is \(e_i^2h_{ii}/(1-h_{ii})^2\). Dividing it by \(p\,s^2\) gives the stated Cook’s distance formula.

Worked Calculations

The definition and the shortcut formula describe the same quantity. The first example calculates it from actual full and deleted fits; the next two use least-squares quantities directly.

How the Components Work Together

The shortcut formula separates the calculation into identifiable parts:

The squared residual compares the observation’s vertical discrepancy with the fitted line to the overall residual scale. The leverage factor accounts for the predictor location and for how much the deleted fit can move when that observation is removed. The divisor \(p\) scales the total change relative to the number of fitted coefficients. These pieces explain why Cook’s distance is not merely a residual measure: the residual and leverage jointly determine the fitted-value change.

For a compact equivalent expression, the internally studentized residual is \(r_i=e_i/[s\sqrt{1-h_{ii}}]\). Substitution gives \(D_i=(r_i^2/p)\{h_{ii}/(1-h_{ii})\}\). This is an algebraic rewrite, not a different definition; the deleted-fit version remains the basis for understanding what Cook’s distance measures.

Common Mistake: Reversing the Prediction Differences

A common error is to state the signs of the differences as deleted-fit predictions minus full-fit predictions while describing them as full-fit minus deleted-fit. In the first worked example, the deleted prediction at \(x=-1\) is \(8.6667\), and the full prediction is \(8\). Therefore, deleted minus full is \(+0.6667\), whereas full minus deleted is \(-0.6667\). The signs must match the stated subtraction order.

Reversing every difference does not change Cook’s distance, because \((a-b)^2=(b-a)^2\). But inconsistent signs make the explanation wrong and can conceal an arithmetic error. State the order explicitly, check at least one pair of predictions, and then square the differences. Also use the full-model \(s^2\) and the full-model number of coefficients \(p\) in the standard definition, rather than silently substituting quantities from the deleted fit.