Deriving the Case-Weight Influence Direction

Why Differentiate the Fitted Parameters?

A subject’s score contribution describes how that subject’s likelihood pulls on the parameter estimate. But the fitted model responds to that pull through the curvature of the whole likelihood. The case-weight influence direction combines both pieces: it is the first-order change in the fitted parameter vector when one subject’s contribution is given an infinitesimally different weight.

This direction is a local derivative, not a measure of the subject’s total effect under deletion. It is useful for identifying how the estimate initially responds to a small weight change, and for understanding why the same subject score can produce very different responses in differently curved or poorly conditioned likelihoods. The earlier material on weighted likelihood perturbations introduces subject-specific weights; here the score equations are differentiated to obtain the parameter response.

Weighted Score Equations

Let $\theta$ be a $q$-dimensional vector containing the parameters of a selection model. For subject $i$, let $\ell_i(\theta)$ denote that subject’s observed-data log-likelihood contribution, including the likelihood for the observed outcomes and missingness or selection information specified by the model. Write the weighted log likelihood as

\ell(\theta;\omega)=\sum_{i=1}^n \omega_i\ell_i(\theta),

where $\omega=(\omega_1,\ldots,\omega_n)$ is the vector of case weights. The unperturbed fit has $\omega_i=1$ for every subject. Define the subject score and the weighted total score by

u_i(\theta)=\frac{\partial \ell_i(\theta)}{\partial\theta}, \qquad U(\theta;\omega)=\frac{\partial\ell(\theta;\omega)}{\partial\theta} =\sum_{i=1}^n\omega_i u_i(\theta).

The score is a $q$-vector. At an interior maximum-likelihood estimate $\hat\theta$ under the unperturbed weights, the score equation is $U(\hat\theta;\mathbf{1})=0$, where $\mathbf{1}$ is the vector of all ones. Let $H$ be the Hessian matrix of the unweighted total log likelihood at that fit:

H= \left. \frac{\partial U(\theta;\mathbf{1})}{\partial\theta^{\mathsf T}} \right|_{\theta=\hat\theta} = \left. \sum_{i=1}^n \frac{\partial^2\ell_i(\theta)} {\partial\theta\,\partial\theta^{\mathsf T}} \right|_{\theta=\hat\theta}.

Thus $H$ is the observed log-likelihood Hessian, not the positive observed information. When it is useful to use positive curvature notation, write $J=-H$, the observed information matrix at the fit.

Definition: The case-weight influence direction for subject $i$ is the derivative of the locally continued solution to the weighted score equations with respect to $\omega_i$, evaluated at $\omega=\mathbf{1}$. Under the regularity conditions below, it is $$ \left.\frac{\partial\hat\theta(\omega)}{\partial\omega_i}\right|_{\omega=\mathbf{1}} =-H^{-1}u_i(\hat\theta) =J^{-1}u_i(\hat\theta). $$

Derivation and Conditions

The formula follows by implicit differentiation. It requires a differentiable local solution of the weighted score equations, rather than merely a score value at the original fit. In particular, assume that the $\ell_i(\theta)$ are twice continuously differentiable near $\hat\theta$, that $\hat\theta$ is an interior solution of the score equations, and that $H$ is nonsingular. The implicit function theorem then gives a differentiable local solution $\hat\theta(\omega)$ near $\mathbf{1}$.

To find the derivative for subject $i$, vary only $\omega_i$ and differentiate $U(\hat\theta(\omega);\omega)=0$ with respect to that weight. The chain rule gives

\frac{\partial U}{\partial\theta^{\mathsf T}} \frac{\partial\hat\theta}{\partial\omega_i} + \frac{\partial U}{\partial\omega_i} =0.

At the unperturbed fit, the first factor is $H$. Since $U(\theta;\omega)=\sum_j\omega_j u_j(\theta)$, its partial derivative with respect to $\omega_i$, holding $\theta$ fixed, is $u_i(\theta)$. Therefore

H\left. \frac{\partial\hat\theta}{\partial\omega_i} \right|_{\omega=\mathbf{1}} +u_i(\hat\theta)=0, \qquad \left. \frac{\partial\hat\theta}{\partial\omega_i} \right|_{\omega=\mathbf{1}} =-H^{-1}u_i(\hat\theta).

This also explains the sign convention. At a strict local maximum, $H$ is negative definite, so $J=-H$ is positive definite and the direction can be written $J^{-1}u_i$. A positive subject score does not by itself determine the sign of every parameter response: the inverse information matrix mixes score components across parameters.

Nonsingularity of $H$ is the condition that permits this local derivative. Negative definiteness is a stronger condition: it confirms strict local maximum curvature and, by continuity, the nearby stationary solution remains a local maximum for sufficiently small weight changes. If $H$ is nonsingular but not negative definite, the derivative still describes a local stationary solution, but it need not describe a local maximum. If parameters are on a boundary, constraints may require differentiating the constrained optimality conditions instead.

Worked Example: A Scalar Parameter Response

Suppose a hypothetical fitted selection model has one parameter $\theta$, measured on the log-odds scale. At the fit, subject $i$’s score is $u_i(\hat\theta)=3.00$ log-odds units per unit weight, and the total log-likelihood Hessian is $H=-20.0$ per squared log-odds unit. Assume the fitted parameter is interior, the contribution is twice continuously differentiable, and the nonzero Hessian means the local derivative exists.

The influence direction is

\left.\frac{\partial\hat\theta}{\partial\omega_i}\right|_{\mathbf{1}} =-H^{-1}u_i =-\frac{1}{-20.0}(3.00) =0.150.

Thus, for a small increase of $0.10$ in this subject’s weight, the first-order approximation to the fitted parameter change is $0.150(0.10)=0.0150$ log-odds units. This is a local approximation, not an exact refit. Here $J=20.0>0$, consistent with strict local maximum curvature in this one-parameter example.

Worked Example: A Two-Parameter Direction

Consider a hypothetical fit with two parameters: $\theta_1$, an outcome-model coefficient in log-odds units, and $\theta_2$, a selection-model coefficient in log-odds units. Suppose subject $i$’s score is $u_i=(2.00,-1.00)^{\mathsf T}$, and

H= \begin{pmatrix} -10 & 2\\ 2 & -6 \end{pmatrix}, \qquad J=-H= \begin{pmatrix} 10 & -2\\ -2 & 6 \end{pmatrix}.

The matrix is symmetric and negative definite because its first leading principal minor is $-10<0$ and its determinant is $60-4=56>0$. Hence $J$ is positive definite, and the fit has strict local maximum curvature. Its inverse is

J^{-1} =\frac{1}{56} \begin{pmatrix} 6 & 2\\ 2 & 10 \end{pmatrix}.

Therefore the response is

J^{-1}u_i =\frac{1}{56} \begin{pmatrix} 6 & 2\\ 2 & 10 \end{pmatrix} \begin{pmatrix} 2\\ -1 \end{pmatrix} = \begin{pmatrix} 10/56\\ -6/56 \end{pmatrix} \approx \begin{pmatrix} 0.1786\\ -0.1071 \end{pmatrix}.

An infinitesimal increase in this subject’s weight therefore increases $\theta_1$ and decreases $\theta_2$, locally, at the displayed rates per unit weight. The second parameter moves even though the subject’s score for it is negative; the off-diagonal curvature couples the parameters. Reporting only the two raw score entries would miss that coupling.

Changing Several Weights Together

The same differentiation applies when weights change along a specified direction. Let $v=(v_1,\ldots,v_n)^{\mathsf T}$ be a fixed vector and set $\omega(t)=\mathbf{1}+tv$. For sufficiently small $t$, the derivative of the fitted parameter vector at $t=0$ is the sum of the subject-specific directions weighted by the components of $v$:

\left.\frac{d\hat\theta(\omega(t))}{dt}\right|_{t=0} =-H^{-1}\sum_{i=1}^n v_i u_i(\hat\theta).

This follows because $\partial U/\partial t=\sum_i v_i u_i(\theta)$, while the derivative through $\theta$ remains $H\,d\hat\theta/dt$. The formula is a linear combination of the individual directions, under the same differentiability and nonsingularity conditions.

Worked Example: Two Subjects’ Weights Change Together

Use the two-parameter fit in the preceding example. Suppose a hypothetical perturbation increases subject 1’s weight at rate $1$ and decreases subject 2’s weight at rate $1$, so $v_1=1$, $v_2=-1$, and all other components of $v$ are zero. Let the two scores be $u_1=(2,-1)^{\mathsf T}$ and $u_2=(1,1)^{\mathsf T}$. The combined score change is

\sum_i v_i u_i=u_1-u_2= \begin{pmatrix}1\\-2\end{pmatrix}.

Using the same $J^{-1}$ as before, the parameter response per unit $t$ is

J^{-1} \begin{pmatrix}1\\-2\end{pmatrix} = \frac{1}{56} \begin{pmatrix}6&2\\2&10\end{pmatrix} \begin{pmatrix}1\\-2\end{pmatrix} = \frac{1}{56} \begin{pmatrix}2\\-18\end{pmatrix} \approx \begin{pmatrix}0.03571\\-0.3214\end{pmatrix}.

For example, a sufficiently small $t=0.05$ gives the first-order change $(0.001786,-0.01607)^{\mathsf T}$ in the two log-odds coefficients. The weight of subject 2 decreases in this perturbation, which is why its score enters with a negative sign in the combined score change.

Diagnostics, Alternatives, and Limits

The derivative is only as trustworthy as the fitted score and Hessian used to calculate it. In a selection model, observed-data contributions can involve integration over unobserved outcomes or latent quantities. Numerical error in those contributions can contaminate both scores and second derivatives, so the fit and derivative should be checked using the likelihood-accuracy and optimization diagnostics established earlier in the course.

Check the score equation. The unweighted total score at $\hat\theta$ should be close to zero. A substantial residual means the fitted point may not solve the equation being differentiated.
Check curvature and conditioning. Confirm that $H$ is nonsingular and, when interpreting the solution as a local maximum, negative definite. A nearly singular or poorly conditioned Hessian can make $H^{-1}u_i$ very large and sensitive to small numerical errors.
Check parameter scaling. Rescaling parameters can improve numerical calculations, but directions must be transformed back to the original parameter units before interpretation. Each component of the derivative is change in that parameter per unit change in weight.
Check local agreement. For a small positive and negative change in a subject’s weight, refit the model and compare the finite-difference response with the derivative. Disagreement can indicate that the weight change is no longer small, the likelihood is strongly nonlinear nearby, or numerical optimization is unstable.
Check the local branch. If small perturbations lead to different modes or cause a constrained parameter to reach a boundary, a single smooth derivative may not describe the fitted solution across those changes.

The main alternative for judging a subject’s effect is exact case deletion: remove the subject’s contribution and refit. That comparison gives the actual deleted-data fit, but requires refitting and can be affected by changes in the optimization solution. The case-weight direction is cheaper once the Hessian and subject scores are available, but extending it to deletion means extrapolating a local derivative across a full weight change of $-1$; that extrapolation need not be accurate. A score-only ranking is cheaper still, but ignores total-likelihood curvature. Bootstrap procedures answer a different question about sampling variability and do not replace this local derivative.

Common pitfall: It is incorrect to call $u_i(\hat\theta)$ itself the parameter influence direction. The score is a force in score-equation space, not a parameter change. The correction is to solve the differentiated score equation, giving $-H^{-1}u_i(\hat\theta)$. It is also incorrect to use $H^{-1}u_i$ without the leading minus sign when $H$ denotes the log-likelihood Hessian; the equivalent positive-information form is $J^{-1}u_i$, where $J=-H$.

Check Your Understanding

Answer each question using the score-equation derivative, then compare your reasoning with the solutions.

In a one-parameter fit, $H=-8$ and a subject’s score is $u_i=-2$. What is the case-weight influence direction?
State the main conditions that permit implicit differentiation of the weighted score equations, and distinguish nonsingularity from negative definiteness of $H$.
Why can a subject have a negative score for one parameter while its influence direction for that parameter is positive?
Suppose two subjects’ weights change at rates $v_1=1$ and $v_2=2$, with scores $u_1=(1,0)^{\mathsf T}$ and $u_2=(0,1)^{\mathsf T}$. What score vector enters the joint-direction formula?
Why does a very large computed direction warrant a Hessian diagnostic rather than an immediate conclusion that the subject is extraordinarily influential?

Solutions. (1) The direction is $-H^{-1}u_i=-(-1/8)(-2)=-0.250$ parameter units per unit weight. (2) The contributions should be twice continuously differentiable near an interior score solution, and the Hessian must be nonsingular for the implicit function theorem to provide a differentiable local solution. Negative definiteness is additionally sufficient for strict local maximum curvature; nonsingularity alone does not establish a maximum. (3) The inverse information matrix can mix score components through off-diagonal entries, so the response for one parameter need not have the same sign as that parameter’s raw score. (4) The combined score vector is $v_1u_1+v_2u_2=(1,2)^{\mathsf T}$. The parameter response is $-H^{-1}(1,2)^{\mathsf T}$. (5) A nearly singular or ill-conditioned Hessian can amplify modest scores and numerical error. Check conditioning, score accuracy, and local refits before interpreting the direction.