First-Order Case-Deletion Approximation

Why Approximate a Case-Deletion Refit?

Deleting one subject and refitting a model gives a direct measure of how much the fitted parameters depend on that record. In a selection model, this refit can be costly: each likelihood evaluation may involve integrating over unobserved outcomes, and the likelihood can have multiple local maxima. The exact case-deletion fit remains an important benchmark, as discussed in the earlier material on exact case deletion. A Hessian-based approximation can often indicate the direction and approximate size of the change at much lower computational cost.

The approximation is local. It uses the score contribution of the deleted subject and the curvature of the full fitted likelihood to estimate the movement needed to restore the score equation after deletion. It is not a replacement for refitting when deletion produces a large change, changes the optimizer’s basin of attraction, or exposes numerical problems.

Likelihood Contributions and the Deletion Change

Let \(\theta\) be the vector of parameters in a selection model, and write the observed-data log-likelihood as a sum of subject contributions, \(\ell(\theta)=\sum_{i=1}^{n}\ell_i(\theta)\). A contribution \(\ell_i\) is based on what was observed for subject \(i\). If some outcomes are missing, the observed-data likelihood contribution accounts for that missingness according to the specified outcome and selection models; it is not generally the likelihood evaluated at an invented value for the missing outcome.

Let \(\hat{\theta}\) be an interior maximum of the full-data-set log-likelihood, and define the subject score \(u_i(\theta)=\partial\ell_i(\theta)/\partial\theta\). Let \(H(\theta)\) be the Hessian of the full log-likelihood and \(H_i(\theta)\) the Hessian of subject \(i\)'s contribution. The observed information at the fitted value is \(J=-H(\hat{\theta})\).

Definition: The case-deletion change for subject \(i\) is \(\Delta_i=\hat{\theta}_{(-i)}-\hat{\theta}\), where \(\hat{\theta}_{(-i)}\) maximizes the log-likelihood with subject \(i\)'s contribution removed. The full-Hessian first-order approximation is \(\widehat{\Delta}_i^{\,\mathrm{FO}}=-J^{-1}u_i(\hat{\theta})\). Thus the approximate deleted estimate is \(\hat{\theta}+\widehat{\Delta}_i^{\,\mathrm{FO}}\).

The formula requires a differentiable likelihood near \(\hat{\theta}\), a fitted value satisfying the full score equation, and nonsingular observed information. For a stable local maximum, \(J\) should be positive definite, at least in the parameter directions being approximated. The approximation also relies on the deletion changing the solution only modestly, so that a first-order Taylor expansion is informative.

Deriving the First-Order Formula

The full-model score is \(U(\theta)=\sum_j u_j(\theta)\), and \(U(\hat{\theta})=0\). After deleting subject \(i\), the score is \(U_{(-i)}(\theta)=U(\theta)-u_i(\theta)\). Write the deleted estimate as \(\hat{\theta}+\Delta_i\). A first-order Taylor expansion of the deleted score around \(\hat{\theta}\) gives

0 \approx U_{(-i)}(\hat{\theta})+H_{(-i)}(\hat{\theta})\Delta_i = -u_i(\hat{\theta})+\{H(\hat{\theta})-H_i(\hat{\theta})\}\Delta_i .

This equation also shows what is omitted in the simplest approximation: the deleted likelihood has a different Hessian from the full likelihood. Since \(H(\hat{\theta})=-J\), replacing the deleted Hessian by the full Hessian gives

0\approx -u_i(\hat{\theta})-J\Delta_i \quad\Longrightarrow\quad \widehat{\Delta}_i^{\,\mathrm{FO}}=-J^{-1}u_i(\hat{\theta}).

The sign follows from the score equation. If a subject has a positive score in a parameter direction, removing that subject initially leaves the remaining-data score negative in that direction. The fitted value must then move in the direction that raises the remaining-data score back to zero; under positive information, that movement has the opposite sign to the subject score.

Retaining the deleted Hessian in the same Taylor expansion gives a related one-step approximation, \(\widetilde{\Delta}_i=-(J+H_i(\hat{\theta}))^{-1}u_i(\hat{\theta})\), when that matrix is nonsingular. It accounts for the deleted subject’s curvature but still uses a local linearization of the deleted score. The full-Hessian formula is the simpler first-order approximation; the two should not be confused with an exact deleted-data refit.

Justification: If deleting one subject changes the estimate by a small amount, then the quadratic and higher-order Taylor terms are small relative to the score and Hessian terms shown above. When the full information is of order \(n\), a typical subject score is of order one, so the first-order parameter change is often of order \(1/n\). This scaling explains why the approximation can work well for a non-dominant subject in a well-behaved large sample; it does not guarantee accuracy for an influential subject or an unstable likelihood.

Worked Calculations

Worked Example: A Scalar Approximation From Fitted-Model Output

Suppose a hypothetical fitted selection model has one parameter \(\gamma\), with observed information \(J=25\) at \(\hat{\gamma}=0.60\). For a particular subject, the fitted casewise score is \(u_i(\hat{\gamma})=1.50\). Calculate the first-order deletion change.

In one dimension, the formula is \(\widehat{\Delta}_i^{\,\mathrm{FO}}=-u_i/J\). Substituting the fitted values gives

\widehat{\Delta}_i^{\,\mathrm{FO}}=-\frac{1.50}{25}=-0.0600, \qquad \hat{\gamma}_{(-i)}\approx 0.60-0.0600=0.540 .

The approximation says that deleting this subject would lower the fitted parameter by about 0.060 in the parameter’s units. The calculation uses only the full-model information and the subject’s score; it does not establish that the exact refit will equal 0.540. A large score relative to the available information, or strong curvature over this 0.060 change, would make an exact refit especially important.

Worked Example: Cross-Parameter Movement

Suppose a hypothetical selection-model fit has two parameters, \(\theta=(\beta,\gamma)^\mathsf{T}\), where \(\beta\) is an outcome-model coefficient and \(\gamma\) is a selection-model coefficient. At the fitted value, let

J= \begin{pmatrix} 10 & 2\\ 2 & 5 \end{pmatrix}, \qquad u_i(\hat{\theta})= \begin{pmatrix} 1\\ -0.5 \end{pmatrix}.

The determinant of \(J\) is \(10(5)-2(2)=46\), so it is invertible, and

J^{-1}=\frac{1}{46} \begin{pmatrix} 5 & -2\\ -2 & 10 \end{pmatrix}.

Multiplying the inverse information by the subject score and changing the sign gives

\widehat{\Delta}_i^{\,\mathrm{FO}} =-\frac{1}{46} \begin{pmatrix} 5 & -2\\ -2 & 10 \end{pmatrix} \begin{pmatrix} 1\\ -0.5 \end{pmatrix} =-\frac{1}{46} \begin{pmatrix} 6\\ -7 \end{pmatrix} = \begin{pmatrix} -0.1304\\ 0.1522 \end{pmatrix}.

The approximate deletion lowers \(\beta\) by 0.1304 and raises \(\gamma\) by 0.1522, in their respective parameter units. The nonzero off-diagonal entries of \(J\) mean that the parameters’ fitted changes are coupled: a score on one parameter can contribute to the approximate change in the other. Computing each parameter as if its information were independent would miss this feature.

Worked Example: Comparing With an Exact Selection-Component Refit

Consider a hypothetical one-parameter response-selection component with \(R_i\sim\text{Bernoulli}(p)\) and \(p=\operatorname{logit}^{-1}(\gamma)\). This simple component illustrates the deletion calculation; it is not a claim that a response-only model represents every joint selection model. Suppose \(n=20\) records include 12 responses and 8 nonresponses. The fitted value is \(\hat{p}=12/20=0.6000\), so \(\hat{\gamma}=\log(12/8)=0.4055\).

For a responder, the log-likelihood contribution is \(\ell_i(\gamma)=\log(p)\). Its score at the fit is \(u_i=1-\hat{p}=0.4000\). The full observed information for the intercept is \(J=n\hat{p}(1-\hat{p})=20(0.6)(0.4)=4.800\). Therefore the first-order approximation for deleting one responder is

\widehat{\Delta}_i^{\,\mathrm{FO}} =-\frac{0.4000}{4.800}=-0.08333, \qquad \hat{\gamma}_{(-i)}^{\,\mathrm{FO}}\approx 0.4055-0.08333=0.3221 .

The exact refit has 11 responses and 8 nonresponses, so its fitted probability is \(11/19\) and its logit is \(\log(11/8)=0.3185\). The exact change is \(0.3185-0.4055=-0.08701\). Here the approximation is close, though not exact.

For comparison, retaining the deleted subject’s curvature gives \(H_i=-\hat{p}(1-\hat{p})=-0.2400\). The deleted-Hessian one-step value is

\widetilde{\Delta}_i =-\frac{0.4000}{4.800-0.2400} =-0.08772 .

This one-step value is also close to the exact change, but it is not the exact refit. The small discrepancy in both calculations is consistent with the score changing nonlinearly as \(\gamma\) moves away from the full-data fit.

Checking the Approximation

In a fitted selection model, evaluate the approximation in the same parameterization and scaling used for the fit. The likelihood derivatives must correspond to the observed-data likelihood, including the integration or summation required by the selection model. Derivatives from an inaccurately evaluated likelihood can produce a precise-looking but misleading change. Numerical integration accuracy and optimization stability therefore matter to this calculation.

A useful diagnostic is to compare \(\hat{\theta}+\widehat{\Delta}_i^{\,\mathrm{FO}}\) with an exact deletion fit for subjects selected across a range of approximate changes, including subjects with large score-based changes. Compare parameter changes on interpretable scales, and check whether both fits converge to the same local maximum. Agreement for a few cases supports local accuracy in those cases, not universal accuracy across subjects.

Large approximate movement: a large norm of \(J^{-1}u_i\), relative to meaningful parameter scales, signals that the local expansion may be inadequate.
Poor curvature: a nearly singular or indefinite information matrix makes the inverse unstable or unsuitable for a local-maximum approximation.
Changed optimization behavior: a deleted fit that converges to another maximum, reaches a boundary, or fails to converge is not well described by a small perturbation around the original solution.
Numerical disagreement: approximate and exact changes that vary materially with quadrature accuracy or optimization settings may reflect computational error rather than a reliable influence signal.

The main alternative is an exact deletion refit, which directly evaluates the model after removal but may require substantial computation and careful checks of convergence. The first-order formula is cheaper and useful for screening, but depends on local curvature. The deleted-Hessian one-step calculation includes one additional curvature term, yet it remains local. None of these calculations establishes that a subject is erroneous or that its record should be removed; they describe sensitivity of the fitted parameters to a specified deletion.

Common Mistake

Common mistake: Using \(+J^{-1}u_i\) because the subject’s score is being subtracted from the full score. The deleted score at the original fit is \(-u_i\), and the score changes with parameter movement at rate approximately \(-J\). Solving \(0\approx -u_i-J\Delta_i\) gives \(\Delta_i\approx -J^{-1}u_i\), not \(+J^{-1}u_i\). Check the sign by writing the deleted score equation before interpreting the direction of change.

A second pitfall is treating a good first-order approximation as proof that the selection model is correctly specified, or as evidence that the deleted subject is an outlier. The calculation is conditional on the fitted model and the observed-data likelihood used. It diagnoses local sensitivity of an estimate, not the cause of that sensitivity.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions.

A fitted scalar model has \(J=12\) and a subject score of \(-0.60\). What is the first-order deletion change?
Why can deleting a subject change two correlated parameters even if its score is nonzero for only one of them?
In the response-component example, what is the exact change after deleting a nonresponder, and how does it compare with the first-order approximation?
Name two conditions that make the first-order approximation questionable.

Solutions. (1) \(\widehat{\Delta}^{\,\mathrm{FO}}=-(-0.60)/12=0.0500\), an increase of 0.0500 parameter units. (2) Off-diagonal entries in \(J^{-1}\) transmit score information across parameter coordinates, so the product \(J^{-1}u_i\) can have multiple nonzero components. (3) The exact fit after deletion has 12 responses and 7 nonresponses, giving \(\gamma_{(-i)}=\log(12/7)=0.5390\). The exact change is \(0.5390-0.4055=0.1335\). The nonresponder’s score is \(-\hat p=-0.6000\), so the first-order change is \(-(-0.6000)/4.800=0.1250\), somewhat smaller than the exact change. (4) Examples include a nearly singular or indefinite observed information matrix, a large approximate parameter change, a deleted fit that moves to a different likelihood maximum, and sensitivity to numerical integration accuracy.