Exact Case Deletion In Generalized Linear Models

What Exact Case Deletion Measures

A fitted generalized linear model depends on all the observations used to estimate its coefficients. To ask how much a particular observation matters, remove it and fit the same model again. The resulting change is an exact case-deletion change: “exact” means the reduced-data model is refitted, not that a formula approximating the refit is applied.

The comparison requires two fits to be genuinely comparable. Keep the model specification, covariates, link, and known offsets fixed; change only the data by omitting observation $i$. Any estimation choices beyond the coefficients, such as whether a dispersion parameter is fixed or estimated, should also follow the same stated procedure in both fits.

Definition: Let $\ell(\beta)=\sum_{j=1}^n \ell_j(\beta)$ be the log-likelihood for independent observations in a GLM, where $\ell_j$ is observation $j$'s log-likelihood contribution. The case-deleted log-likelihood and coefficient estimate are $$ \ell_{(i)}(\beta)=\sum_{j\ne i}\ell_j(\beta)=\ell(\beta)-\ell_i(\beta), \qquad \hat{\beta}_{(i)}\in\operatorname*{arg\,max}_{\beta}\ell_{(i)}(\beta). $$ When both estimates exist, the exact coefficient change on deleting $i$ is $\Delta\hat{\beta}_{(i)}=\hat{\beta}_{(i)}-\hat{\beta}$, where $\hat{\beta}$ maximizes the full-data log-likelihood.

The definition presumes that a suitable maximum exists. A deleted-data likelihood can have no finite maximizer, as can happen in logistic regression after separation, or can have multiple maximizers if the remaining design does not identify all coefficients. In those cases the coefficient change is not an ordinary, uniquely defined finite vector. It is important to detect this rather than report a numerical fit as if it were a valid deletion estimate.

Exact Changes in Fitted Means

A coefficient change is measured on the coefficient scale. To see what it does to predictions, specify a design vector $x$ and the corresponding known offset $o(x)$. Use the same design vector and offset in both fits. This can be an observed design point or a new point at which both models are to be compared.

Definition: For a GLM with link $g$, the full and case-deleted fitted means at design vector $x$ are $g^{-1}(x^\mathsf{T}\hat{\beta}+o(x))$ and $g^{-1}(x^\mathsf{T}\hat{\beta}_{(i)}+o(x))$, respectively. Their exact change is $$ \Delta\hat{\mu}_{(i)}(x) = g^{-1}(x^\mathsf{T}\hat{\beta}_{(i)}+o(x)) - g^{-1}(x^\mathsf{T}\hat{\beta}+o(x)). $$ The offset $o(x)$ is fixed and identical in both predictors.

This is a change in the fitted mean, not necessarily a change in the observed response at $x$. In particular, when $x=x_i$, the predictor still uses the design information for observation $i$, but its response is absent from the deleted fit. A prediction comparison at another design point is defined in the same way, with that point's offset held fixed.

Worked Example: Exact Deletion in an Intercept-Only Poisson Model

Suppose $Y_1,Y_2,Y_3$ are independent Poisson counts with a common mean $\mu=\exp(\beta)$, and the observed counts are $1,2,3$. Find the full and case-3-deleted coefficient estimates and compare their fitted means.

Ignoring terms that do not depend on $\beta$, the log-likelihood is $$ \ell(\beta)=\sum_{j=1}^3(y_j\beta-\exp(\beta)) =6\beta-3\exp(\beta). $$ Its derivative is $6-3\exp(\beta)$, so the maximum satisfies $\exp(\hat{\beta})=2$, giving $\hat{\beta}=\log 2$. The second derivative is $-3\exp(\beta)<0$, confirming a unique maximum.

After deleting $y_3=3$, the log-likelihood is $\ell_{(3)}(\beta)=3\beta-2\exp(\beta)$. Its derivative is $3-2\exp(\beta)$, so $\exp(\hat{\beta}_{(3)})=3/2$ and $\hat{\beta}_{(3)}=\log(3/2)$. The coefficient change is $$ \Delta\hat{\beta}_{(3)} =\log(3/2)-\log 2 =\log(3/4)\approx -0.288. $$ The fitted mean falls from $2$ to $1.5$, a change of $-0.5$. Both fits are finite and unique because each likelihood is strictly concave and its score has a finite root.

The Deleted Score Equation

The exact refit can be characterized without approximation by its score equation. Write $U(\beta)=\nabla\ell(\beta)$ and $U_i(\beta)=\nabla\ell_i(\beta)$. The key fact is that at the deleted estimate, the score contributed by the observations retained in the fit sums to zero. The following result also makes clear why evaluating the full-data score at the deleted estimate is generally not the same as evaluating it at the full-data estimate.

Theorem: Suppose each $\ell_j$ is differentiable, $\hat{\beta}_{(i)}$ is a finite interior maximizer of $\ell_{(i)}$, and the full likelihood is the sum of the independent observation contributions. Then $$ \nabla\ell_{(i)}(\hat{\beta}_{(i)})=0, \qquad U(\hat{\beta}_{(i)})=U_i(\hat{\beta}_{(i)}). $$ If the model has a canonical link and independent exponential-family observations with fixed dispersion $\phi$, then $$ U_i(\beta)=\frac{x_i\{y_i-b'(\eta_i)\}}{a(\phi)}, \qquad \eta_i=x_i^\mathsf{T}\beta+o_i, $$ for the usual exponential-family log-likelihood contribution $\ell_i(\beta)=[y_i\eta_i-b(\eta_i)]/a(\phi)+c(y_i,\phi)$.

Proof. Since $\ell_{(i)}(\beta)=\sum_{j\ne i}\ell_j(\beta)$, differentiating this finite sum gives $\nabla\ell_{(i)}(\beta)=\sum_{j\ne i}\nabla\ell_j(\beta)=U(\beta)-U_i(\beta)$. An interior maximum of a differentiable function has zero gradient, so at $\hat{\beta}_{(i)}$, $$ 0=\nabla\ell_{(i)}(\hat{\beta}_{(i)}) =U(\hat{\beta}_{(i)})-U_i(\hat{\beta}_{(i)}). $$ Rearranging yields the stated score identity. For the canonical exponential-family contribution, the offset is fixed, so $\partial\eta_i/\partial\beta=x_i$. Differentiating $[y_i\eta_i-b(\eta_i)]/a(\phi)+c(y_i,\phi)$ with respect to $\beta$ gives $x_i[y_i-b'(\eta_i)]/a(\phi)$. This proves both claims. $\square$

For a canonical model, the identity says that the full-data score at the deleted estimate equals the deleted observation's own score evaluated at that same estimate. It does not give the deleted estimate directly: the score still depends on the unknown coefficient vector. Solving the reduced-data likelihood, rather than substituting a residual into a formula, is what defines the exact change.

Worked Example: Exact Deletion in an Intercept-Only Logistic Model

Consider four independent Bernoulli observations with a common success probability $p=\operatorname{logit}^{-1}(\beta)$, and outcomes $1,1,0,0$. Compare the full fit with the fit after deleting one success.

For the full data, the likelihood is proportional to $p^2(1-p)^2$. Its log-likelihood is $2\log p+2\log(1-p)$; setting its derivative $2/p-2/(1-p)$ to zero gives $\hat p=1/2$ and $\hat\beta=\log\{\hat p/(1-\hat p)\}=0$. The log-likelihood is strictly concave as a function of $\beta$, and this finite stationary point is the unique maximum.

After deleting one success, the outcomes are $1,0,0$, so the likelihood is proportional to $p(1-p)^2$. Its log-likelihood derivative with respect to $p$ is $1/p-2/(1-p)$, which is zero at $\hat p_{(i)}=1/3$. Therefore $\hat\beta_{(i)}=\log\{(1/3)/(2/3)\}=-\log 2$, and $$ \Delta\hat{\beta}_{(i)}=-\log 2\approx -0.693, \qquad \Delta\hat p_{(i)}=\frac13-\frac12=-\frac16. $$ The deleted fit is finite: it contains both a success and a failure, so the likelihood does not increase toward a boundary value $p=0$ or $p=1$.

Existence, Uniqueness, and a Refitting Result

A zero score alone does not always certify a maximum. Concavity supplies a useful sufficient condition. This result is also a practical check on numerical refits: for a concave reduced-data log-likelihood, a finite interior stationary point is already a global maximizer.

Theorem: If $\ell_{(i)}$ is differentiable and concave on a convex parameter space and $\hat{\beta}_{(i)}$ is an interior point satisfying $\nabla\ell_{(i)}(\hat{\beta}_{(i)})=0$, then $\hat{\beta}_{(i)}$ is a global maximizer. If $\ell_{(i)}$ is strictly concave, this maximizer is unique.

Proof. For a differentiable concave function, the first-order concavity inequality gives, for every $\beta$ in the parameter space, $$ \ell_{(i)}(\beta)\le \ell_{(i)}(\hat{\beta}_{(i)}) +\nabla\ell_{(i)}(\hat{\beta}_{(i)})^\mathsf{T} (\beta-\hat{\beta}_{(i)}). $$ The gradient term is zero, so $\ell_{(i)}(\beta)\le\ell_{(i)}(\hat{\beta}_{(i)})$ for every $\beta$. Thus the stationary point is a global maximizer. If the function is strictly concave, two distinct points cannot both attain its maximum, so the maximizer is unique. $\square$

Worked Example: Exact Deletion with a Poisson Offset

Suppose independent counts $Y_j$ follow Poisson distributions with means $t_j\exp(\beta)$, where exposures $t_j$ are known. This is an intercept-only log-link model with offset $o_j=\log t_j$. Take counts $2,4,9$ and exposures $1,2,3$. Find the full and case-3-deleted fits, then compare predicted counts at exposure $t=4$.

Up to terms not involving $\beta$, the full log-likelihood is $$ \ell(\beta)=\sum_j\{y_j(\log t_j+\beta)-t_j\exp(\beta)\}. $$ Its score is $\sum_j y_j-\exp(\beta)\sum_jt_j=15-6\exp(\beta)$. Hence $\exp(\hat\beta)=15/6=2.5$ and $\hat\beta=\log(2.5)$. The second derivative is $-6\exp(\beta)<0$, so this is the unique maximum.

After deleting the third observation, the retained count total is $2+4=6$, and retained exposure total is $1+2=3$. The deleted-data score is $6-3\exp(\beta)$, giving $\exp(\hat\beta_{(3)})=2$ and $\hat\beta_{(3)}=\log 2$. Thus $\Delta\hat\beta_{(3)}=\log 2-\log(2.5)=\log(0.8)\approx-0.223$.

At exposure $t=4$, the offset is $\log 4$, so the full fitted mean is $\exp(\log 4+\log 2.5)=10$. The deleted fitted mean is $\exp(\log 4+\log 2)=8$. The exact fitted-mean change is $8-10=-2$. The offset is included identically in both predictors; omitting it would instead compare rates, not expected counts at exposure four.

Why Exact Changes Are a Useful Benchmark

Exact deletion answers a direct question: what coefficients and fitted means result when the model is estimated from all observations except $i$? It is therefore a natural reference against which to assess a faster approximation to the same deletion. The target is unambiguous: first specify the response contribution to remove, then refit the same model, then compare coefficients or predictions at explicitly stated design points.

The benchmark is especially useful because coefficient changes and fitted-mean changes need not tell the same story. A change in one coefficient can affect predictions differently at different covariate values, and the inverse link can further alter the scale of the change. Always state whether a comparison concerns $\Delta\hat\beta_{(i)}$, a fitted mean at $x_i$, or a prediction at another point.

Common Mistake

A frequent error is to treat deleting a case as if it simply subtracted that case's residual from the full-data estimate. That reasoning confuses an observation's discrepancy with the result of fitting the entire reduced likelihood. In general GLMs, the score contributions and fitted means depend on the coefficients, so removing one contribution changes the solution to a system of equations. The correction is to define $\ell_{(i)}=\ell-\ell_i$ and obtain $\hat\beta_{(i)}$ by maximizing it (or solving its score equation when the stated regularity conditions apply). Also, when offsets are present, retain the same offset in both prediction formulas.

Key takeaway: Exact case deletion is defined by refitting the same GLM after removing one observation. Coefficient and fitted-mean changes are direct comparisons between those two fits; they are not residuals and do not depend on an approximation.

Check Your Understanding

Use these questions to check whether the deletion target and comparison are clear.

Write the case-deleted log-likelihood in terms of the full log-likelihood and the contribution from observation $i$.
What assumptions allow the deleted score equation to be written as $U(\hat{\beta}_{(i)})=U_i(\hat{\beta}_{(i)})$?
Why can a logistic-regression case-deleted estimate fail to be finite even when the full-data estimate is finite?
For a Poisson log-link model with offset, what must be held fixed when comparing full and deleted fitted means at the same prediction point?
Why is an exact refit a useful reference for evaluating an approximate case-deletion calculation?