Why Compare Estimating Functions?
An estimating function turns data into an equation for an unknown parameter. Different equations can target the same parameter, but their solutions need not have the same asymptotic precision. In a correctly specified parametric model, the likelihood score provides a benchmark: among regular unbiased estimating functions based on one observation, it has the greatest asymptotic information. The comparison rests on a covariance identity and the Cauchy–Schwarz inequality.
We work with a scalar parameter \(\theta\) and one observation \(X\) with density or mass function \(f_\theta\). For an independent sample, the estimating equation is typically the sum of the individual estimating functions. The asymptotic variance formula for its regular root is the standard estimating-equation result: sensitivity and variability together determine precision. Here the main question is how large that precision can be under the parametric model.
Definitions and Regularity Conditions
For an independent sample \(X_1,\ldots,X_n\), the estimating equation is \(\sum_{i=1}^n\psi(X_i;\hat\theta)=0\). Under the usual conditions for a consistent, regular root—differentiability near the true parameter, a nonzero limiting sensitivity, suitable laws of large numbers and a central limit theorem—the asymptotic variance of \(\sqrt n(\hat\theta-\theta)\) is \(B(\theta)/A(\theta)^2=1/G(\theta)\). Thus a larger \(G(\theta)\) corresponds to a smaller first-order asymptotic variance.
Let \(S_\theta(X)=\partial\log f_\theta(X)/\partial\theta\) be the likelihood score for one observation, and let \(I(\theta)=E_\theta[S_\theta(X)^2]\) be the Fisher information. Assume the model has a common support, the density or mass function is differentiable in \(\theta\), and the regularity conditions justify differentiation under the integral or sum. In particular, assume \(E_\theta[S_\theta(X)]=0\), \(0<I(\theta)<\infty\), and that the products and derivatives below are integrable. For \(\psi\), assume differentiability and enough domination to differentiate \(E_\theta[\psi(X;\theta)]\) under the integral sign. These conditions are what make the following identity valid.
The Covariance Identity and the Information Bound
The key fact is that an unbiased estimating function cannot vary independently of the score: its covariance with the score is fixed by its sensitivity.
Proof of the covariance identity. Differentiate the unbiasedness equation. Since both the dependence of \(\psi\) on \(\theta\) and the dependence of \(f_\theta\) on \(\theta\) must be differentiated, the product rule gives
The same calculation applies with a sum for a discrete model. Both \(\psi\) and \(S_\theta\) have mean zero, so their product expectation is their covariance. Rearranging gives \(\operatorname{Cov}_\theta(\psi,S_\theta)=-E_\theta[\partial\psi/\partial\theta]=A(\theta)\).
Proof of the information bound and equality condition. The Cauchy–Schwarz inequality for square-integrable random variables gives
Dividing by \(B(\theta)>0\) proves \(G(\theta)\le I(\theta)\). Equality in Cauchy–Schwarz holds exactly when the centered variables are linearly dependent with probability one. They are already centered, so equality requires \(\psi=cS_\theta\) almost surely for a constant \(c\). Taking covariance with \(S_\theta\) shows \(A(\theta)=cI(\theta)\), hence \(c=A(\theta)/I(\theta)\). Conversely, this proportionality gives equality directly. \(\square\)
For the score itself, the sensitivity is \(I(\theta)\) under the regularity conditions, and its variability is also \(I(\theta)\). Thus its Godambe information is \(I(\theta)\), attaining the bound. Multiplying an estimating function by a nonzero constant changes \(A\) and \(B\) but not their ratio \(A^2/B\); the equality condition therefore identifies a proportionality class, not a unique normalization.
Worked Comparisons
Worked Example: Bernoulli Estimation of a Success Probability
Let \(X\sim\operatorname{Bernoulli}(p)\), where \(0<p<1\). Its probability mass function is \(f_p(x)=p^x(1-p)^{1-x}\) for \(x\in\{0,1\}\). The score is
Choose \(\psi(X;p)=X-p\). It is unbiased because \(E_p[X-p]=p-p=0\). Its derivative with respect to \(p\) is \(-1\), so \(A(p)=1\). Its variability is \(B(p)=\operatorname{Var}_p(X)=p(1-p)\), and therefore \(G(p)=1/[p(1-p)]\). The score has Fisher information \(I(p)=E_p[S_p^2]=p(1-p)/[p^2(1-p)^2]=1/[p(1-p)]\). In fact, \(\psi=p(1-p)S_p\), so equality holds. The root of the sample estimating equation is the sample proportion, and this calculation confirms that its first-order variance reaches the information bound.
Worked Example: Adding an Unbiased Component in a Normal Model
Suppose \(X\sim N(\mu,\sigma^2)\), with known \(\sigma^2>0\) and unknown \(\mu\). Consider \(\psi(X;\mu)=(X-\mu)+c[(X-\mu)^2-\sigma^2]\), where \(c\) is a fixed real constant. The two terms each have expectation zero, so this estimating function is unbiased. Its derivative is \(-1-2c(X-\mu)\), giving \(A(\mu)=1\).
For a normal variable, \(E[(X-\mu)^2]=\sigma^2\), \(E[(X-\mu)^3]=0\), and \(E[(X-\mu)^4]=3\sigma^4\). Hence the covariance between the two terms in \(\psi\) is zero, while the variance of the second bracket is \(3\sigma^4-\sigma^4=2\sigma^4\). It follows that
The score is \(S_\mu(X)=(X-\mu)/\sigma^2\), so \(I(\mu)=1/\sigma^2\). If \(c=0\), the estimating function is proportional to the score and \(G=I\). If \(c\ne0\), then \(B>\sigma^2\), so \(G<I\). The extra term has mean zero but adds variability without increasing sensitivity; unbiasedness alone does not make an estimating function efficient.
Worked Example: A Perturbed Estimating Function for a Poisson Rate
Let \(X\sim\operatorname{Poisson}(\lambda)\), with \(\lambda>0\). The score is \(S_\lambda(X)=(X-\lambda)/\lambda\), so \(I(\lambda)=1/\lambda\). For a fixed constant \(c\ne-1\), consider \(\psi(X;\lambda)=(X-\lambda)+c[(X-\lambda)^2-\lambda]\). Both terms have mean zero. Differentiating with respect to \(\lambda\) gives \[ \frac{\partial\psi}{\partial\lambda}=-1+c[-2(X-\lambda)-1], \] so \(A(\lambda)=1+c\).
The Poisson central moments needed here are \(E[(X-\lambda)^2]=\lambda\), \(E[(X-\lambda)^3]=\lambda\), and \(E[(X-\lambda)^4]=\lambda+3\lambda^2\). Therefore the first term has variance \(\lambda\), the bracket has variance \((\lambda+3\lambda^2)-\lambda^2=\lambda+2\lambda^2\), and their covariance is \(\lambda\). Thus
For \(c\ne-1\), the resulting information is \[ G(\lambda)=\frac{B(\lambda)}{(1+c)^2}\,^{-1} =\frac{(1+c)^2}{\lambda(1+c)^2+2c^2\lambda^2} \leq \frac{1}{\lambda}=I(\lambda). \] For example, if \(c=1\), then \(A=2\), \(B=4\lambda+2\lambda^2\), and the asymptotic variance per observation is \(B/A^2=\lambda+\lambda^2/2\), strictly above the score bound \(\lambda\). Equality occurs at \(c=0\), when \(\psi=X-\lambda=\lambda S_\lambda\).
Interpretation and a Common Pitfall
The identity \(\operatorname{Cov}(\psi,S_\theta)=A\) explains why the score is optimal. A regular unbiased estimating function has a prescribed covariance with the score: its sensitivity fixes that covariance. Cauchy–Schwarz says that, for a fixed covariance, the smallest possible variability is achieved when the estimating function is proportional to the score. Any component orthogonal to the score adds variability without contributing to that covariance.
Common mistake: “This estimating function is unbiased, so its root must be as efficient as the maximum likelihood estimator.” Unbiasedness only supplies \(E_\theta[\psi]=0\); it does not control \(B(\theta)\) relative to \(A(\theta)\). The correction is to calculate both quantities and compare \(A^2/B\) with \(I(\theta)\). In the normal example, the added quadratic term preserves unbiasedness but increases variability while leaving sensitivity unchanged.
The conclusion is specifically for regular estimating functions under a correctly specified parametric model and the stated differentiability and integrability conditions. It is a first-order asymptotic comparison, not a claim that every finite-sample likelihood estimator has the smallest variance among all possible estimators. The central criterion is exact: an estimating function matches the score’s information if and only if it is proportional to the score almost surely at the parameter value under consideration.
Check Your Understanding
Use the definitions and covariance identity to answer the following.
- For an unbiased estimating function, what is the relationship between its sensitivity and its covariance with the score?
- State the inequality that bounds Godambe information by Fisher information.
- What precise condition on \(\psi(X;\theta)\) gives equality in that inequality?
- In the normal example, why does the quadratic component reduce information when \(c\ne0\)?
- For the Poisson example with \(c=1\), verify the asymptotic variance per observation and compare it with the information bound.