Cook’s Distance For GLM Coefficients

Why Measure a Case’s Effect on Coefficients?

A large residual identifies a response that is unusual relative to its fitted mean; high leverage identifies a design point that can have unusual geometric influence. Neither fact alone says how much the fitted regression coefficients would change if that case were removed. Cook’s distance for coefficients combines the coefficient change with the information available in the full fit, giving a scale-aware measure of how consequential a deletion is.

For a generalized linear model (GLM), the natural scale is the Fisher information, or equivalently the inverse of the estimated coefficient covariance. The exact diagnostic compares the full-data and case-deleted estimates. A useful approximation follows by taking one Newton step from the full fit toward the case-deleted fit; this expresses the distance in terms of a case’s leverage and Pearson residual.

Definition on the Information Scale

Suppose $Y_1,\ldots,Y_n$ are independent observations in a GLM with a $p$-dimensional coefficient vector $\beta$ and full-rank design matrix $X$. Let $\hat\beta$ be the full-data estimate and $\hat\beta_{(i)}$ the estimate after deleting case $i$. Write $\widehat{\operatorname{Cov}}(\hat\beta)$ for the estimated covariance matrix from the full fit. The deletion estimate must exist and be finite for the diagnostic to be defined.

Definition: The coefficient Cook’s distance for case $i$ is $$ D_i=\frac{1}{p}(\hat\beta-\hat\beta_{(i)})^\mathsf{T} \left\{\widehat{\operatorname{Cov}}(\hat\beta)\right\}^{-1} (\hat\beta-\hat\beta_{(i)}). $$ It is the squared distance between the two coefficient estimates, measured in full-fit information units and averaged over the $p$ coefficients.

For a GLM with variance function $V(\mu_i)$, link $\eta_i=g(\mu_i)=x_i^\mathsf{T}\beta$, and observation weights $a_i$, use the convention $\operatorname{Var}(Y_i)=\phi V(\mu_i)/a_i$. At the full fit define $w_i=a_i\{d\mu_i/d\eta_i\}^2/V(\mu_i)$, $W=\operatorname{diag}(w_1,\ldots,w_n)$, and $A=X^\mathsf{T}WX$. The expected Fisher information for $\beta$ is $A/\phi$, so, when $\phi$ is known, the covariance approximation is $\phi A^{-1}$. Thus the distance can be written $$ D_i=\frac{1}{p\phi}(\hat\beta-\hat\beta_{(i)})^\mathsf{T} A(\hat\beta-\hat\beta_{(i)}). $$ When dispersion is estimated, use the full-fit estimate $\hat\phi$ in this expression. For binomial and Poisson models with their usual dispersion fixed at one, no estimated scale factor is needed.

Deriving the One-Step Approximation

The derivation uses the score and expected information at the full-data estimate. Let $u_i(\beta)$ be the score contribution from case $i$, and $U(\beta)=\sum_j u_j(\beta)$ the full score. For a regular interior maximum-likelihood estimate, $U(\hat\beta)=0$. The case contribution and full information in a GLM are

u_i(\hat\beta)=\frac{x_i q_i}{\phi}, \qquad q_i=\frac{a_i(y_i-\hat\mu_i)}{V(\hat\mu_i)} \left.\frac{d\mu_i}{d\eta_i}\right|_{\hat\beta}, \qquad I(\hat\beta)=\frac{A}{\phi}.

Assume the link and mean model are differentiable, the full-fit information is nonsingular, and the case-deleted estimate is sufficiently close to the full estimate for a first-order Taylor approximation to be useful. Also assume deleting case $i$ leaves nonsingular information. The deleted score at $\hat\beta$ is $-u_i(\hat\beta)$, because the full score is zero. A Newton step using the deleted information therefore gives

\hat\beta_{(i)}-\hat\beta \ \approx\ -\left(\frac{A-w_i x_i x_i^\mathsf{T}}{\phi}\right)^{-1} u_i(\hat\beta) = -\left(A-w_i x_i x_i^\mathsf{T}\right)^{-1}x_iq_i.

Define the GLM hat matrix $H=W^{1/2}X(X^\mathsf{T}WX)^{-1}X^\mathsf{T}W^{1/2}$ and its diagonal element $h_{ii}$. Then $h_{ii}=w_i x_i^\mathsf{T}A^{-1}x_i$. If $h_{ii}<1$, the Sherman–Morrison identity gives

\left(A-w_i x_i x_i^\mathsf{T}\right)^{-1}x_i = \frac{A^{-1}x_i}{1-h_{ii}}.

Substituting this expression into the one-step change and measuring that change with the full information yields a compact approximation. Define the Pearson residual $r_{Pi}=(y_i-\hat\mu_i)/\sqrt{\hat\phi V(\hat\mu_i)/a_i}$. Since $q_i=w_i(y_i-\hat\mu_i)/(d\mu_i/d\eta_i)$, direct substitution gives

D_i \ \approx\ \frac{q_i^2}{p\phi}\, \frac{x_i^\mathsf{T}A^{-1}x_i}{(1-h_{ii})^2} = \frac{r_{Pi}^2 h_{ii}}{p(1-h_{ii})^2}.

In the last equality, use $h_{ii}=w_i x_i^\mathsf{T}A^{-1}x_i$ and $q_i^2/(w_i\phi)=a_i(y_i-\hat\mu_i)^2/\{\phi V(\hat\mu_i)\}$. If dispersion is estimated, the residual in the displayed approximation uses $\hat\phi$. The approximation depends on the local full-fit quantities; it is not an identity for the exact deletion distance.

Theorem: Under the regularity and nonsingularity conditions above, the one-step coefficient change is $$ \hat\beta_{(i)}-\hat\beta\ \approx\ -\frac{A^{-1}x_iq_i}{1-h_{ii}}, $$ and its full-information Cook distance is approximately $$ D_i\ \approx\ \frac{r_{Pi}^2 h_{ii}}{p(1-h_{ii})^2}. $$ The minus sign in the coefficient change indicates that deleting a case moves the estimate opposite to that case’s score contribution. It disappears from the squared distance.

Worked Examples

Worked Example: Exact and Approximate Distance in a Poisson Model

Fit an intercept-only Poisson log-link model to the counts $1,2,3,10$. For case $i$ with count 10, compare the exact coefficient Cook’s distance with the one-step approximation. Here $p=1$ and $\phi=1$.

The full-data maximum-likelihood estimate of the mean is $\hat\mu=(1+2+3+10)/4=4$, so $\hat\beta=\log 4$. After deleting the count 10, the mean is $\hat\mu_{(i)}=(1+2+3)/3=2$, giving $\hat\beta_{(i)}=\log 2$. The full-fit information for the intercept is $A=\sum_j\hat\mu_j=4(4)=16$. Thus

D_i=16(\log 4-\log 2)^2 =16(\log 2)^2 \approx 7.687.

For the approximation, $w_i=\hat\mu_i=4$ and $A=16$, so $h_{ii}=w_i/A=1/4$. The Pearson residual is $r_{Pi}=(10-4)/\sqrt{4}=3$. Therefore

D_i\approx \frac{3^2(1/4)}{1(1-1/4)^2} =\frac{2.25}{0.5625}=4.

The one-step approximation is smaller than the exact distance here. It captures the direction and scale of the deletion effect, but the actual coefficient shift, $\log 2\approx0.693$, is not especially small. A substantial gap is a reason to compute the exact deletion rather than treat the approximation as exact.

Worked Example: Leverage Changes the Effect of the Same Residual

Consider a Gaussian identity-link regression with known dispersion $\phi=1$, intercept and slope, and four design values $x=(-1,0,1,2)$. Take the observed responses to be $y=(1.5,2.5,1.5,3.5)$. The fitted line is $\hat\mu=2+0.5x$, with residuals $(0,0.5,-1,0.5)$. Their sum is zero and their inner product with $x$ is zero, confirming that these are the least-squares residuals.

For this design, the leverage at $x_i$ is $h_{ii}=1/4+(x_i-\bar x)^2/\sum_j(x_j-\bar x)^2$, where $\bar x=0.5$ and $\sum_j(x_j-\bar x)^2=5$. Both the case at $x=0$ and the case at $x=2$ have residual $0.5$, so their Pearson residuals are also $0.5$. Their leverages are $h_{x=0}=1/4+0.25/5=0.3$ and $h_{x=2}=1/4+2.25/5=0.7$. Since this is a Gaussian model with known dispersion, the one-case deletion relation is exact, and $p=2$. For $x=0$,

D_{x=0}= \frac{(0.5)^2(0.3)}{2(1-0.3)^2} =\frac{0.075}{0.98} \approx0.0765.

For $x=2$, with the same residual but greater leverage,

D_{x=2}= \frac{(0.5)^2(0.7)}{2(1-0.7)^2} =\frac{0.175}{0.18} \approx0.9722.

The larger distance is not due to a larger response discrepancy: the residuals match. It arises because the high-leverage design point has a greater potential to change the fitted coefficients when deleted.

Worked Example: A Logistic Fit’s Local Approximation

In a Bernoulli logistic regression with an intercept and one covariate, consider a case with $x_i=(1,0)^\mathsf{T}$, $y_i=1$, and fitted probability $\hat\mu_i=0.5$. Suppose the full-fit information matrix is $A=\operatorname{diag}(1.25,1)$. The case’s weight is $w_i=\hat\mu_i(1-\hat\mu_i)=0.25$, so $h_{ii}=0.25(1/1.25)=0.2$. The Pearson residual is $r_{Pi}=(1-0.5)/\sqrt{0.5(1-0.5)}=1$. With $p=2$ and $\phi=1$, the one-step Cook distance is

D_i\approx \frac{1^2(0.2)}{2(1-0.2)^2} =\frac{0.2}{1.28} =0.15625.

The formula separates two ingredients: the residual contributes $r_{Pi}^2=1$, while the leverage and deletion adjustment contribute $h_{ii}/(1-h_{ii})^2=0.3125$. The resulting value is an approximation based on the supplied full-fit information, not a substitute for refitting the logistic model with this case removed.

Interpretation and Common Pitfalls

The approximation makes the roles of discrepancy and design explicit. Holding leverage fixed, the distance grows with the squared Pearson residual. Holding the residual fixed, it grows with leverage, with the factor $(1-h_{ii})^{-2}$ making the approximation especially sensitive as $h_{ii}$ approaches one. If $h_{ii}=1$, deleting the case makes the information matrix singular, so the displayed deletion approximation does not apply; the coefficient may not be identifiable from the remaining observations.

Dispersion matters because Cook’s distance compares a coefficient change with its estimated uncertainty. For a fixed coefficient shift and fixed $A$, a larger $\phi$ reduces the information-scaled distance. When $\phi$ is estimated, its estimate also changes Pearson residuals and the covariance scale. Comparisons across models with different dispersion assumptions should therefore be made cautiously.

Mistake: treating the approximation as the exact deletion distance. The one-step formula linearizes the deleted score near the full estimate. If deleting a case moves the coefficients substantially, the nonlinear fit can depart from that step. Correction: distinguish the approximate formula from the definition and refit when an influential case is consequential.
Mistake: calling every high-leverage case influential. Leverage describes the design’s potential to move the fit, not the realized size of the coefficient change. A high-leverage case with a small response discrepancy can have modest Cook distance. Correction: assess leverage and residual discrepancy together.
Mistake: applying a universal cutoff. The distance depends on model dimension, information, dispersion, and the use of an approximation. No single threshold has the same interpretation in every GLM and data setting. Correction: use it to rank and investigate cases, alongside the model context and refitting checks, rather than as an automatic deletion rule.

Key takeaway: GLM coefficient Cook’s distance measures a full-fit information-scaled change under case deletion. Its one-step approximation is $r_{Pi}^2 h_{ii}/\{p(1-h_{ii})^2\}$: a large value can reflect response discrepancy, leverage, or both.

Check Your Understanding

Use the definition and derivation to answer these questions.

Why is the coefficient difference measured with the inverse covariance matrix rather than ordinary Euclidean distance?
State the conditions needed for the displayed one-step approximation to be meaningful, including the role of $h_{ii}$.
If two cases have equal Pearson residuals but different leverages, what does the approximate formula imply about their Cook distances?
Why can the exact deletion distance differ substantially from its one-step approximation?
In an estimated-dispersion GLM, where does the dispersion estimate enter the information-scaled distance and Pearson residual?