Influence Functions for Estimators Defined by Equations

Why Differentiate an Estimating Equation?

An estimator defined by an equation responds to a change in the distribution of the data. To describe that response locally, replace a small fraction of the population distribution by observations concentrated at one value $z$, and track how the equation's solution moves. The derivative of that movement is the influence function at $z$.

This calculation turns the estimating equation into a direct description of sensitivity at each possible observation. It also makes the sign and matrix orientation in the formula consequential: both depend on how the equation is written. Here we use the convention that the estimating function is a column vector and its Jacobian is the derivative with respect to the parameter, with the parameter indexing columns.

Definition: Let $F$ be a distribution on the observation space, and let $T(F)=\theta_0$ be a parameter defined as a root of a population estimating equation $$ \Psi(\theta,F)=E_F[\psi(X,\theta)]=0, $$ where $\theta\in\mathbb{R}^p$ and $\psi(x,\theta)\in\mathbb{R}^p$. For a possible observation $z$, define the point-mass contaminated distribution $$ F_\varepsilon=(1-\varepsilon)F+\varepsilon\Delta_z,\qquad 0\leq\varepsilon<1, $$ where $\Delta_z$ places probability one at $z$. If the locally selected root $T(F_\varepsilon)$ is differentiable at $\varepsilon=0$, the influence function is $$ \operatorname{IF}(z;T,F)=\left.\frac{d}{d\varepsilon}T(F_\varepsilon)\right|_{\varepsilon=0+}. $$

The derivative is one-sided because contamination is defined for nonnegative $\varepsilon$. It measures the first-order change per unit of contamination, not the change caused by replacing a particular observation in a finite sample.

The Influence-Function Formula

Write $\theta_0=T(F)$, and define the Jacobian $$ A=E_F\left[\frac{\partial\psi(X,\theta_0)}{\partial\theta^T}\right]. $$ The matrix $A$ describes how the population equation changes when its parameter changes. The contaminated equation adds the contribution of the point $z$, while reducing the contribution from $F$ by the same fraction.

Theorem: Suppose $\theta_0$ is an interior point of an open parameter space in $\mathbb{R}^p$, $E_F[\psi(X,\theta_0)]=0$, and $\psi(z,\theta_0)$ is finite. Suppose also that the map $$ H(\theta,\varepsilon)=(1-\varepsilon)E_F[\psi(X,\theta)]+\varepsilon\psi(z,\theta) $$ is continuously differentiable in a neighborhood of $(\theta_0,0)$, with differentiation under the expectation justified there, and that $A$ is nonsingular. Then the locally unique root $\theta(\varepsilon)$ of $H(\theta,\varepsilon)=0$ is differentiable at zero, and $$ \operatorname{IF}(z;T,F)=-A^{-1}\psi(z,\theta_0). $$

The differentiability and domination conditions in this result are substantive. For example, a sufficient way to justify differentiation under the expectation is to have a neighborhood of $\theta_0$ on which the relevant first derivatives exist and are bounded in norm by an integrable function of $X$. Continuity of the expected derivative is also needed for the stated implicit-function argument.

Proof. At $(\theta_0,0)$, $H(\theta_0,0)=E_F[\psi(X,\theta_0)]=0$. Its derivative with respect to $\theta$ at that point is $A$. Since $A$ is nonsingular, the implicit function theorem gives a differentiable local root $\theta(\varepsilon)$ with $\theta(0)=\theta_0$. The derivative of $H$ with respect to $\varepsilon$, holding $\theta=\theta_0$ fixed, is $$ \left.\frac{\partial H(\theta_0,\varepsilon)}{\partial\varepsilon}\right|_{\varepsilon=0} =\psi(z,\theta_0)-E_F[\psi(X,\theta_0)] =\psi(z,\theta_0). $$ Differentiate $H(\theta(\varepsilon),\varepsilon)=0$ at zero. The chain rule gives $$ A\theta'(0)+\psi(z,\theta_0)=0. $$ Multiplying by $A^{-1}$ yields $\theta'(0)=-A^{-1}\psi(z,\theta_0)$, which is the claimed influence function. $\square$

Worked Example: The Influence Function of the Mean

Let $F$ have a finite mean $\mu=E_F[X]$, and define it by the equation $\psi(x,\theta)=x-\theta$. The population equation is $E_F[X-\theta]=0$, whose root is $\theta_0=\mu$.

The derivative of the estimating function with respect to $\theta$ is $-1$, so $A=-1$. At a contamination point $z$, $\psi(z,\mu)=z-\mu$. Therefore $$ \operatorname{IF}(z;T,F)=-(-1)^{-1}(z-\mu)=z-\mu. $$

This also follows directly from the contaminated mean: $$ T(F_\varepsilon)=(1-\varepsilon)\mu+\varepsilon z =\mu+\varepsilon(z-\mu). $$ Differentiating at zero gives $z-\mu$, confirming both the formula and its sign.

Two Further Derivations

For a vector parameter, each component of the influence function is obtained from the same matrix equation. The next example shows why the Jacobian must include the dependence of every estimating-function component on every parameter.

Worked Example: The Influence Function of Mean and Variance

Let $F$ have mean $\mu$ and finite variance $v=E_F[(X-\mu)^2]$. Define $\theta=(m,s)^T$ through the two estimating equations $$ \psi(x,\theta)= \begin{pmatrix} x-m\\ (x-m)^2-s \end{pmatrix}, \qquad E_F[\psi(X,\theta)]=0. $$ The root is $\theta_0=(\mu,v)^T$.

The Jacobian of $\psi$ is $$ \frac{\partial\psi(x,\theta)}{\partial\theta^T} = \begin{pmatrix} -1 & 0\\ -2(x-m) & -1 \end{pmatrix}. $$ Taking expectations at $\theta_0$ and using $E_F[X-\mu]=0$ gives $A=-I_2$, where $I_2$ is the $2$-by-$2$ identity matrix. At $z$, the estimating function is $$ \psi(z,\theta_0)= \begin{pmatrix} z-\mu\\ (z-\mu)^2-v \end{pmatrix}. $$ Thus $$ \operatorname{IF}(z;T,F) =-(-I_2)^{-1} \begin{pmatrix} z-\mu\\ (z-\mu)^2-v \end{pmatrix} = \begin{pmatrix} z-\mu\\ (z-\mu)^2-v \end{pmatrix}. $$ The first component is the influence function for the mean; the second is the influence function for the variance. This derivation requires only a finite second moment for the defining equations and their expected derivatives.

As a direct check, the contaminated mean is $\mu_\varepsilon=\mu+\varepsilon(z-\mu)$. The contaminated variance is $$ v_\varepsilon=(1-\varepsilon)v+\varepsilon(1-\varepsilon)(z-\mu)^2. $$ The second identity follows by writing the contaminated second moment about $\mu$ as $(1-\varepsilon)v+\varepsilon(z-\mu)^2$, then subtracting $(\mu_\varepsilon-\mu)^2=\varepsilon^2(z-\mu)^2$. Differentiating at zero gives $v_\varepsilon'|_0=(z-\mu)^2-v$, as obtained from the estimating equations.

Worked Example: A Logistic Intercept

Suppose $Y$ is Bernoulli with success probability $q$, where $0<q<1$. Estimate an intercept $\beta$ from the population score equation $$ E_F[Y-\operatorname{expit}(\beta)]=0,\qquad \operatorname{expit}(\beta)=\frac{1}{1+e^{-\beta}}. $$ Its root is $\beta_0=\log\{q/(1-q)\}$, since $\operatorname{expit}(\beta_0)=q$.

The estimating function is $\psi(y,\beta)=y-\operatorname{expit}(\beta)$. Its derivative is $-\operatorname{expit}(\beta)\{1-\operatorname{expit}(\beta)\}$, so $A=-q(1-q)$. At contamination value $y\in\{0,1\}$, the formula gives $$ \operatorname{IF}(y;T,F) =-\frac{1}{-q(1-q)}(y-q) =\frac{y-q}{q(1-q)}. $$ For example, if $q=0.2$, then a success has influence $(1-0.2)/(0.2\cdot0.8)=5$, and a failure has influence $(0-0.2)/(0.2\cdot0.8)=-1.25$.

The calculation can be checked using the contaminated success probability $q_\varepsilon=(1-\varepsilon)q+\varepsilon y$. The contaminated intercept is $\beta_\varepsilon=\log\{q_\varepsilon/(1-q_\varepsilon)\}$, and differentiation gives $$ \left.\frac{d\beta_\varepsilon}{d\varepsilon}\right|_0 =\frac{q_\varepsilon'|_0}{q(1-q)} =\frac{y-q}{q(1-q)}. $$

Connection to Asymptotic Covariance

The influence function also identifies the first-order contribution of an observation to an estimator. Under the usual independent, identically distributed sampling setup, suppose an estimator has the asymptotic linear representation $$ \sqrt{n}\{\hat{\theta}-\theta_0\} =\frac{1}{\sqrt{n}}\sum_{i=1}^n \operatorname{IF}(X_i;T,F)+o_p(1), $$ and suppose $E_F[\|\psi(X,\theta_0)\|^2]<\infty$. Define $B=E_F[\psi(X,\theta_0)\psi(X,\theta_0)^T]$. Since $E_F[\psi(X,\theta_0)]=0$, the influence function has mean zero. Its covariance is $$ \operatorname{Var}_F\{\operatorname{IF}(X;T,F)\} =A^{-1}BA^{-T}. $$

Result: Under the influence-function formula and the finite second-moment condition above, the covariance of the influence function is $A^{-1}BA^{-T}$. If the stated asymptotic linear representation holds, the multivariate central limit theorem gives this same matrix as the asymptotic covariance of $\sqrt{n}(\hat{\theta}-\theta_0)$.

Derivation. Substitute $\operatorname{IF}(X;T,F)=-A^{-1}\psi(X,\theta_0)$. Because $A$ is a fixed nonsingular matrix and $E_F[\psi]=0$, $$ \begin{aligned} \operatorname{Var}_F\{\operatorname{IF}(X;T,F)\} &=E_F\!\left[A^{-1}\psi(X,\theta_0)\psi(X,\theta_0)^T A^{-T}\right]\\ &=A^{-1}BA^{-T}. \end{aligned} $$ The multivariate central limit theorem applies to the independent influence-function terms when they have finite second moments. Adding a term that is $o_p(1)$ does not change the limiting distribution. $\square$

This is the same matrix form that arises in sandwich covariance calculations for estimating equations. The influence-function derivation gives it an observation-level interpretation: each term in the linear approximation is the contribution of one observation, and its covariance determines the first-order uncertainty.

Common Sign and Interpretation Pitfalls

Dropping the minus sign. Differentiation gives $A\theta'(0)+\psi(z,\theta_0)=0$, so the result is $-A^{-1}\psi(z,\theta_0)$, not $A^{-1}\psi(z,\theta_0)$.
Changing the Jacobian convention without changing the formula. Here $\psi$ is a column vector and $A=E_F[\partial\psi/\partial\theta^T]$. If an equation is instead written with the negative score, its Jacobian and formula change accordingly.
Treating the derivative as an exact finite-sample change. The influence function is a derivative at zero contamination. It describes the first-order approximation $T(F_\varepsilon)=T(F)+\varepsilon\operatorname{IF}(z;T,F)+o(\varepsilon)$, not an exact change for arbitrary $\varepsilon$.
Assuming a root automatically has a derivative. A singular Jacobian, an irregular estimating function, or an unjustified exchange of differentiation and expectation can invalidate the formula. The local differentiability and nonsingularity conditions are part of the result.

Key takeaway: To obtain the influence function, differentiate the contaminated population equation, evaluate its Jacobian at the original root, and solve the resulting linear system. With the convention $A=E_F[\partial\psi/\partial\theta^T]$, the answer is $-A^{-1}\psi(z,\theta_0)$.

Check Your Understanding

Use the definitions and sign convention from this tutorial.

For a scalar estimating equation, what do $A$ and $\psi(z,\theta_0)$ represent in the influence-function formula?
Why does differentiating the contaminated equation produce a minus sign in front of $A^{-1}$?
For the variance equation in the worked example, why is the expected derivative of the second component with respect to the mean parameter zero?
In the logistic-intercept example, calculate the influence function of a failure when $q=0.1$.
State one regularity condition needed to justify the implicit-function derivation, and explain what role it plays.