Case-Weight Perturbations And Influence Functions

Why Perturb a Case Weight?

Exact case deletion asks how a fitted model changes after an observation is removed and the model is refit. A related question is more local: how rapidly would the estimate move if that observation’s contribution to the likelihood were reduced slightly? Differentiating the score equations answers this question. The resulting derivative describes infinitesimal coefficient influence and gives a first-order approximation to a full deletion.

The distinction matters. A derivative is a local quantity, evaluated at the fit with all observations present. Deleting a case changes its weight by an entire unit, so the derivative-based estimate of that change can be inaccurate when the fitting path is curved, or when the deleted fit is very different from the original one.

Likelihood Weights and the Score

Suppose \(Y_1,\ldots,Y_n\) are independent observations in a GLM, with coefficient vector \(\beta\), and let \(\ell_j(\beta)\) denote the log-likelihood contribution from observation \(j\). Any fixed offsets or exposure terms are included in that observation’s model contribution. Define its score contribution by \(u_j(\beta)=\partial\ell_j(\beta)/\partial\beta\). The ordinary log-likelihood is the sum of these contributions.

\ell(\beta)=\sum_{j=1}^n \ell_j(\beta), \qquad U(\beta)=\sum_{j=1}^n u_j(\beta).

Give case \(i\) a likelihood weight \(a_i\), leaving all other weights at one. A weight of zero omits the case; a weight between zero and one downweights it. Here the weights multiply the observation-level log-likelihood contributions, not the response values.

\ell(\beta;a_i)=\sum_{j\ne i}\ell_j(\beta)+a_i\ell_i(\beta), \qquad U(\beta;a_i)=\sum_{j\ne i}u_j(\beta)+a_i u_i(\beta).

Let \(\hat\beta\) be an interior maximum-likelihood estimate at \(a_i=1\), so \(U(\hat\beta;1)=0\). Define the observed information matrix at that fit as the negative derivative of the full score:

J(\hat\beta) = -\left.\frac{\partial U(\beta;1)}{\partial\beta^{\mathsf T}}\right|_{\beta=\hat\beta}.

The derivative of the score with respect to \(\beta\) is a matrix. Assume it exists and that \(J(\hat\beta)\) is nonsingular. Nonsingularity ensures that a small change in the score equation identifies a unique first-order change in the fitted coefficients.

Definition: The case-weight influence derivative for observation \(i\) is the derivative of the fitted coefficient vector with respect to that observation’s likelihood weight, evaluated at the full-data fit: \( \left.\partial\hat\beta(a_i)/\partial a_i\right|_{a_i=1} \). It measures the local coefficient change per unit increase in that case’s likelihood weight.

Deriving the Influence Derivative

For a local perturbation, let \(a_i=1+t\), and write \(\hat\beta(t)\) for the solution of the weighted score equation near \(t=0\). The sign of \(t\) indicates the direction of the perturbation: positive \(t\) increases the case’s weight; negative \(t\) decreases it.

Theorem: Suppose the observation log-likelihood contributions are twice continuously differentiable in a neighbourhood of \(\hat\beta\), \(U(\hat\beta;1)=0\), and \(J(\hat\beta)\) is nonsingular. Then a differentiable local solution \(\hat\beta(t)\) to \(U(\hat\beta(t);1+t)=0\) exists near \(t=0\), and \( \left.\frac{d\hat\beta(t)}{dt}\right|_{t=0}=J(\hat\beta)^{-1}u_i(\hat\beta) \).

The existence and differentiability of the local solution follow from the implicit function theorem applied to the score equation. Differentiating that equation gives the derivative explicitly.

Set up the score equation.
Let \(F(\beta,t)=U(\beta;1+t)=U(\beta;1)+t u_i(\beta)\). At the full-data fit, \(F(\hat\beta,0)=0\).

Check the derivative with respect to coefficients.
\(\partial F/\partial\beta^{\mathsf T}\) at \((\hat\beta,0)\) is \(-J(\hat\beta)\), which is nonsingular by assumption. The implicit function theorem therefore gives a differentiable local root \(\hat\beta(t)\).

Differentiate the root equation.
By the chain rule, \(0=-J(\hat\beta)\hat\beta'(0)+u_i(\hat\beta)\). Multiplying by \(J(\hat\beta)^{-1}\) gives \(\hat\beta'(0)=J(\hat\beta)^{-1}u_i(\hat\beta)\). \(\square\)

Thus, a case with a score contribution pointing strongly in a direction of relatively weak information can have a large coefficient derivative. To approximate deletion, decrease the weight from one toward zero. The sign reverses because deletion moves the weight in the negative direction.

First-order case-deletion approximation: If the local solution is twice continuously differentiable in \(t\), then for small \(t\), \( \hat\beta(t)-\hat\beta(0)=tJ(\hat\beta)^{-1}u_i(\hat\beta)+O(t^2) \). Taking \(t=-1\) suggests the approximate deletion change \( \hat\beta_{(-i)}-\hat\beta\ \approx\ -J(\hat\beta)^{-1}u_i(\hat\beta) \), where \(\hat\beta_{(-i)}\) is the estimate after removing case \(i\).

The \(O(t^2)\) statement is local: it holds as \(t\) approaches zero when the second derivative is bounded in a neighbourhood of zero. It does not guarantee that the approximation is accurate at \(t=-1\). The path of roots must also remain well-defined for the deleted fit to exist.

Observed and Expected Information in a GLM

For a GLM with fixed dispersion \(\phi\), let \(\eta_j=x_j^{\mathsf T}\beta\), \(\mu_j=E[Y_j]\), and \(\operatorname{Var}(Y_j)=\phi V(\mu_j)\). For a differentiable link with \(d\mu_j/d\eta_j\) defined, the score contribution is

u_j(\beta) = x_j\frac{(Y_j-\mu_j)(d\mu_j/d\eta_j)} {\phi V(\mu_j)}.

The expected Fisher information is \(I(\beta)=E[J(\beta)]\). In the usual GLM formulation with independent observations and fixed \(\phi\), it has the weighted-design form below. Replacing the observed information in the exact derivative by this expected information gives a common Fisher-scoring approximation; it is an approximation unless the two information matrices coincide.

I(\beta)=X^{\mathsf T}WX, \qquad W_{jj}=\frac{1}{\phi} \frac{(d\mu_j/d\eta_j)^2}{V(\mu_j)}.

Consequently, one may use \(-I(\hat\beta)^{-1}u_i(\hat\beta)\) as a deletion approximation. It is important not to call this the exact derivative in general: differentiating the observed score equation gives \(J^{-1}u_i\), whereas the Fisher-scoring version substitutes \(I\) for \(J\).

Worked Examples

Worked Example: Decreasing the Weight of a Poisson Count

Consider an intercept-only Poisson model with four observed counts \(1,2,3,10\). Write \(\beta\) for the intercept, so \(\mu=e^\beta\). Calculate the derivative and the first-order deletion approximation for the count \(10\).

The full-data maximum-likelihood estimate of the mean is the sample average: \( \hat\mu=(1+2+3+10)/4=4 \), so \( \hat\beta=\log 4 \). For a Poisson observation, \(u_i(\beta)=Y_i-\mu\). Thus the count \(10\) contributes \(u_i(\hat\beta)=10-4=6\).

The full-data observed information is \(J=4\hat\mu=16\). Therefore the derivative for increasing the weight of the count \(10\) is \( \hat\beta'(0)=6/16=0.375 \). Reducing its weight has the opposite local direction, so the first-order deletion approximation is \( \hat\beta_{(-i)}-\hat\beta\approx-0.375 \).

The exact deletion estimate uses the remaining counts \(1,2,3\), whose mean is \(2\). Its intercept is \(\log 2\), so the exact change is \(\log 2-\log 4=-\log 2\approx-0.693\). The approximation has the correct direction but understates the change, illustrating that a one-unit weight reduction can be far beyond the range where the local slope is accurate.

Worked Example: A Bernoulli Observation in Logistic Regression

Fit an intercept-only logistic model to four binary observations \(1,1,0,0\). Find the weight derivative for one of the successes and its approximate deletion change.

The fitted probability is \(\hat p=2/4=0.5\), and the fitted intercept is \(\hat\beta=\log\{\hat p/(1-\hat p)\}=0\). In an intercept-only logistic model, an observation’s score is \(Y_i-p\). For the selected success, \(u_i(\hat\beta)=1-0.5=0.5\). The observed information is \(J=4\hat p(1-\hat p)=4(0.5)(0.5)=1\).

The derivative for increasing this case’s weight is \(J^{-1}u_i=0.5\). The deletion approximation is therefore \(-0.5\) on the logit scale. After deleting one success, the remaining data contain one success and two failures, giving \(\hat p_{(-i)}=1/3\) and \(\hat\beta_{(-i)}=\log(1/2)\approx-0.693\). The exact change is about \(-0.693\), again showing that the derivative is a local slope, not an exact refit.

Worked Example: A Gaussian Regression Case

For a Gaussian linear model with known variance \(1\), let the design rows be \(x_j^{\mathsf T}=(1,x_j)\) at \(x=0,1,2\), with responses \(0,1,4\). Calculate the local deletion approximation for the observation at \(x=2\), then compare it with exact deletion.

The least-squares fit has intercept \(-1/3\) and slope \(2\): the fitted values are \(-1/3,5/3,11/3\), and the residuals are \(1/3,-2/3,1/3\). For a Gaussian model with known variance \(1\), the score contribution is \(u_i=x_i(Y_i-x_i^{\mathsf T}\hat\beta)\), and \(J=X^{\mathsf T}X\).

For the observation at \(x=2\), its design vector is \((1,2)^{\mathsf T}\), so \(u_i=(1,2)^{\mathsf T}(1/3)=(1/3,2/3)^{\mathsf T}\). Also, \(J=\begin{pmatrix}3&3\\3&5\end{pmatrix}\), with \(J^{-1}=\begin{pmatrix}5/6&-1/2\\-1/2&1/2\end{pmatrix}\). Multiplication gives \(J^{-1}u_i=(-1/18,1/6)^{\mathsf T}\). Hence the approximate deletion change is \((1/18,-1/6)^{\mathsf T}\).

Deleting the observation at \(x=2\) leaves the two points \((0,0)\) and \((1,1)\), which are fitted exactly by intercept \(0\), slope \(1\). The exact change from \((-1/3,2)\) is therefore \((1/3,-1)^{\mathsf T}\). This is much larger than the derivative-based approximation, consistent with the deleted observation having substantial influence on this small data set.

Interpretation and Common Pitfalls

The derivative combines two features: the case’s score contribution and the model’s information. A large score contribution does not by itself guarantee a large coefficient change; the inverse information transforms that contribution into coefficient units and accounts for directions in which the model is more or less well determined. The result is a vector, so different coefficients can change in different directions.

Reversing the sign. The derivative \(J^{-1}u_i\) corresponds to increasing the case’s weight. Deletion decreases the weight, so the first-order deletion change has a minus sign.
Treating the approximation as an exact deletion. The derivative is evaluated at the full-data fit. Moving a weight all the way from one to zero may encounter substantial curvature or a very different fit.
Substituting expected information without saying so. The exact implicit derivative uses observed information \(J\). Using Fisher information \(I\) is a stated approximation unless \(J=I\) at the fit.
Assuming a deleted estimate always exists. The local root guaranteed near weight one need not continue to weight zero. For example, deleting cases can produce separation in logistic regression, in which a finite maximum-likelihood estimate may not exist.

Key takeaway: For a regular full-data GLM fit, the local derivative with respect to case \(i\)’s likelihood weight is \(J^{-1}u_i\). The first-order deletion approximation is its negative, but a full deletion is a finite perturbation and should be checked against an exact refit when accuracy matters.

Check Your Understanding

Use the score-equation argument and the examples above to answer these questions.

Why does the coefficient derivative for increasing a case’s weight have the opposite sign from the first-order deletion change?
What assumptions are needed for the implicit-function argument to give a local differentiable coefficient path?
In a GLM, how does replacing observed information with expected Fisher information change the interpretation of the calculation?
Why can an accurate derivative at weight one still give a poor approximation to deletion at weight zero?
In the Poisson example, what was the approximate change in the intercept, and what was the exact change?