Deriving One-Step Case-Deletion Approximations

Why Linearize After Deletion?

Deleting an observation changes every coefficient estimate because the remaining observations must be fitted again. An exact refit solves the reduced score equations, but repeated refitting can be computationally expensive when many cases are being assessed. A one-step case-deletion approximation instead uses the score and curvature at the full-data estimate to estimate the change without fully solving those equations.

The approximation is local: it treats the deleted-case score as nearly linear near the full-data estimate. It can work well when the coefficient shift is small and the score curvature changes little over that shift. High leverage can magnify the shift, while strong curvature or a deleted fit near a boundary can make a local approximation unreliable.

Score Equations With One Case Removed

Consider independent observations with log-likelihood contributions \(\ell_j(\beta)\), where \(\beta\) is a \(p\)-dimensional coefficient vector. Write \(u_j(\beta)=\partial\ell_j(\beta)/\partial\beta\) for the contribution of observation \(j\) to the score, and let \(U(\beta)=\sum_{j=1}^n u_j(\beta)\). Assume the full-data estimate \(\hat\beta\) is an interior solution of \(U(\hat\beta)=0\). The score after deleting case \(i\) is \(U_{(-i)}(\beta)=\sum_{j\ne i}u_j(\beta)\).

Since the full score is zero, \(U_{(-i)}(\hat\beta)=-u_i(\hat\beta)\). Define the observed information from the remaining cases, evaluated at the full estimate, by \(J_{(-i)}(\hat\beta)=-\partial U_{(-i)}(\beta)/\partial\beta^\mathsf{T}\big|_{\hat\beta}\). If this matrix is nonsingular, the first-order Taylor expansion gives \(U_{(-i)}(\hat\beta+\Delta_i)\approx-u_i(\hat\beta)-J_{(-i)}(\hat\beta)\Delta_i\). Setting this linear expression to zero gives the one-step approximation

The minus sign matters: the deleted-case score at the full estimate is \(-u_i(\hat\beta)\), not \(u_i(\hat\beta)\). The formula is a Newton step for the deleted-case score equation starting from \(\hat\beta\). It uses the deleted-data curvature evaluated at the full fit; it is not, in general, the exact deletion change.

When Does the Linearization Approximate a Deleted Fit?

A small first step alone does not guarantee accuracy. The score must also remain close to its linear approximation over the relevant neighborhood, and a finite deleted solution must exist there. The following local result makes these requirements explicit. Use the Euclidean norm for vectors and its induced operator norm for matrices.

Proof. Taylor’s formula with an integral remainder gives \(G(\hat\beta+\delta)=-u_i(\hat\beta)-J\delta+R(\delta)\), where \(R(\delta)=\int_0^1[DG(\hat\beta+t\delta)-DG(\hat\beta)]\delta\,dt\). The Lipschitz condition implies \(\|R(\delta)\|\le\int_0^1 Mt\|\delta\|^2dt=M\|\delta\|^2/2\). A root is therefore a fixed point of \(T(\delta)=\Delta_i^{(1)}+J^{-1}R(\delta)\). On the ball \(\|\delta\|\le2a\), the remainder bound gives \(\|T(\delta)\|\le a+2\|J^{-1}\|Ma^2<2a\), so \(T\) maps that ball into itself. For two points \(\delta,\gamma\) in the ball, the derivative of \(R\) is \(DR(\delta)=DG(\hat\beta+\delta)-DG(\hat\beta)\), so \(\|DR(\delta)\|\le2Ma\). Integrating this derivative along the line segment between the points gives \(\|T(\delta)-T(\gamma)\|\le2\|J^{-1}\|Ma\|\delta-\gamma\|\). The assumed strict inequality makes \(T\) a contraction. The contraction mapping theorem gives a fixed point in the closed ball, hence a root of \(G\). Finally, at that root, \(\Delta_i-\Delta_i^{(1)}=J^{-1}R(\Delta_i)\), so \(\|\Delta_i-\Delta_i^{(1)}\|\le\|J^{-1}\|M(2a)^2/2=2\|J^{-1}\|Ma^2\). \(\square\)

This is a local guarantee, not a promise that every deleted GLM has a finite estimate. The bound shows two ways the approximation can deteriorate: a large proposed step \(a\), or a large curvature factor \(M\) relative to the conditioning of \(J\). If the deleted score has no finite root, no local method can approximate a finite deleted estimate.

A Leverage-Adjusted Approximation

For a GLM with independent observations, linear predictor \(\eta_j=x_j^\mathsf{T}\beta\) (with any offset held fixed), mean \(\mu_j\), and variance \(\phi V(\mu_j)\), the expected information contribution at the full fit is \(w_jx_jx_j^\mathsf{T}\), where \(w_j=(d\mu_j/d\eta_j)^2/\{\phi V(\mu_j)\}\). This expression assumes the usual differentiability and positive variance conditions at the fitted means. Define \(Q=\sum_jw_jx_jx_j^\mathsf{T}\), and suppose \(Q\) is positive definite. The expected information after removing case \(i\), with weights frozen at the full fit, is \(Q_{(-i)}=Q-w_ix_ix_i^\mathsf{T}\). This is an information-based approximation to the observed curvature, not an identity for every GLM’s observed Hessian.

The scalar \(h_i=w_ix_i^\mathsf{T}Q^{-1}x_i\) is the weighted leverage of case \(i\) in this frozen-weight geometry. If \(h_i<1\), the rank-one inverse identity yields \(Q_{(-i)}^{-1}x_i=Q^{-1}x_i/(1-h_i)\). If \(u_i(\hat\beta)=x_iq_i\), as in the usual GLM score contribution, the resulting leverage-adjusted one-step approximation is

To verify the inverse identity, put \(v=Q^{-1}x_i\). Then \((Q-w_ix_ix_i^\mathsf{T})v/(1-h_i) =[x_i-w_ix_i(x_i^\mathsf{T}Q^{-1}x_i)]/(1-h_i)=x_i\). Thus \(v/(1-h_i)\) is the solution to \(Q_{(-i)}z=x_i\). The formula requires \(Q_{(-i)}\) to be nonsingular; under positive-definite \(Q\), this holds precisely when \(h_i<1\). As \(h_i\) approaches one, the downdated information can become poorly conditioned and the estimated change can be strongly amplified. The adjustment accounts for a rank-one information loss, but does not by itself account for changing weights or nonlinear score curvature.

Worked Examples

Interpreting Reliability and Avoiding Pitfalls

The Gaussian example is unusually favorable because its score has constant curvature. In nonlinear GLMs, the score curvature generally changes as \(\beta\) moves, and the weights used to form \(Q\) change as the fitted means change. The local theorem describes the role of curvature; the leverage formula is a practical approximation that highlights the effect of information loss. Neither calculation replaces checking whether a deleted fit exists and whether the proposed shift stays in a neighborhood where the model behaves smoothly.

Common mistake: treating \(-Q_{(-i)}^{-1}u_i\) as the exact deleted coefficient change merely because the deleted information is invertible. Invertibility makes the linear system solvable, but does not make the score linear. The correction is to call this a one-step approximation, check the size of the predicted change and the leverage, and compare with an actual deleted refit when accuracy matters. A small leverage is not a guarantee: the Poisson example has \(h_i=1/3\), yet deletion removes the only positive count and leaves no finite estimate.