Asymptotic Variance Of Estimating-Equation Estimators

Why Estimating Equations Have a Sandwich Covariance

An estimator can be defined as a solution to an equation built from the data, rather than as a maximizer of a likelihood. Examples include moment equations, score equations, and equations chosen to encode a scientific estimating principle. To understand the estimator's first-order uncertainty, two features matter: how rapidly the equation changes when the parameter changes, and how variable the equation is across observations.

The first feature is its sensitivity; the second is the variability of the individual estimating contributions. Their combination produces a covariance with the variability matrix in the middle and the inverse sensitivity on both sides. We derive this result under independent, identically distributed sampling and a correctly specified estimating equation.

Setup and Definitions

Let $X_1,\ldots,X_n$ be independent observations with the same distribution as $X$, and let the true parameter be $\theta_0\in\mathbb{R}^p$. For each observation, let $\psi(X,\theta)\in\mathbb{R}^p$ be an estimating function. Define its sample average by $\Psi_n(\theta)=n^{-1}\sum_{i=1}^n\psi(X_i,\theta)$. An estimator $\hat{\theta}$ is a root when $\Psi_n(\hat{\theta})=0$.

Definition: The sensitivity matrix and score-variability matrix at $\theta_0$ are $$ A=E\!\left[\frac{\partial\psi(X,\theta_0)}{\partial\theta^T}\right], \qquad B=\operatorname{Var}\!\left(\psi(X,\theta_0)\right). $$ The derivative matrix has entry $j,k$ equal to $\partial\psi_j(X,\theta)/\partial\theta_k$. If $E[\psi(X,\theta_0)]=0$, then $B=E[\psi(X,\theta_0)\psi(X,\theta_0)^T]$.

The matrix $A$ describes the local change in the mean estimating equation as the parameter moves. The matrix $B$ describes the random fluctuation of one observation's contribution. In the scalar case, these are numbers; in the vector case, their order and transposes matter.

The Sandwich Limit

Theorem: Suppose $E[\psi(X,\theta_0)]=0$, $E[\|\psi(X,\theta_0)\|^2]<\infty$, and $A$ exists and is nonsingular. Suppose $\hat{\theta}$ is a consistent root of $\Psi_n(\theta)=0$, and the estimating function is differentiable in a neighborhood of $\theta_0$. Assume the sample-average derivative converges uniformly in probability to its population value in a neighborhood shrinking to $\theta_0$, so that the derivative averaged along the segment from $\theta_0$ to $\hat{\theta}$ converges in probability to $A$. Then $$ \sqrt{n}(\hat{\theta}-\theta_0)\xrightarrow{d} N_p\!\left(0,A^{-1}B A^{-T}\right), $$ where $A^{-T}=(A^{-1})^T$.

These conditions state the ingredients needed in the argument: a centered estimating contribution with finite variance, a local linear approximation with stable derivative, and a consistent root. The derivative condition is often verified by a uniform law of large numbers for the sample-average derivative together with continuity of its expectation near $\theta_0$.

Proof. Write $\dot{\Psi}_n(\theta)=\partial\Psi_n(\theta)/\partial\theta^T$. By the fundamental theorem of calculus applied along the line segment between $\theta_0$ and $\hat{\theta}$, $$ 0=\Psi_n(\hat{\theta}) =\Psi_n(\theta_0)+\left\{\int_0^1\dot{\Psi}_n\!\left(\theta_0+t(\hat{\theta}-\theta_0)\right)\,dt\right\}(\hat{\theta}-\theta_0). $$ Call the matrix in braces $A_n^*$. Consistency and the derivative condition give $A_n^*\xrightarrow{p}A$. Since $A$ is nonsingular, $A_n^*$ is nonsingular with probability tending to one, and $(A_n^*)^{-1}\xrightarrow{p}A^{-1}$. Rearranging on that event gives $$ \sqrt{n}(\hat{\theta}-\theta_0) =-(A_n^*)^{-1}\frac{1}{\sqrt{n}}\sum_{i=1}^n\psi(X_i,\theta_0). $$ The multivariate central limit theorem gives $n^{-1/2}\sum_i\psi(X_i,\theta_0)\xrightarrow{d}N_p(0,B)$. Slutsky's theorem now gives the stated limit: multiplication by $-A^{-1}$ yields covariance $(-A^{-1})B(-A^{-1})^T=A^{-1}BA^{-T}$. $\square$

The minus sign in the linear expansion does not change the covariance. If the estimating function is scaled by a nonzero constant, its sensitivity and variability both change, but the resulting sandwich covariance does not.

Worked Covariance Calculations

Worked Example: A Vector Normal Mean

Suppose $X_i\sim N_p(\mu,\Sigma)$, independently, where $\Sigma$ is known and positive definite, and estimate $\mu$ using $\psi(X,\theta)=X-\theta$. The root is the sample mean $\hat{\mu}=\bar{X}$. At the true value $\mu$, $E[\psi(X,\mu)]=0$, and $$ A=E[-I_p]=-I_p,\qquad B=\operatorname{Var}(X-\mu)=\Sigma. $$

The sandwich covariance is therefore $$ A^{-1}BA^{-T}=(-I_p)\Sigma(-I_p)=\Sigma. $$ Thus $\sqrt{n}(\bar{X}-\mu)$ has limiting covariance $\Sigma$. Equivalently, the covariance scale for $\bar{X}$ itself is approximately $\Sigma/n$; in this normal example that covariance is exact.

Worked Example: An Exponential Rate Equation

Let $X_i$ have an exponential distribution with rate $\lambda>0$, so $E[X]=1/\lambda$ and $\operatorname{Var}(X)=1/\lambda^2$. Use the estimating function $\psi(X,\theta)=1-\theta X$. Its population mean is zero at $\theta_0=\lambda$, and the root is $\hat{\lambda}=1/\bar{X}$.

The derivative is $-X$, so $A=-1/\lambda$. At the true parameter, $\psi(X,\lambda)=1-\lambda X$ has mean zero and variance $1$, because $\lambda X$ has an exponential distribution with rate 1. Hence $B=1$, and $$ A^{-1}BA^{-T} =(-\lambda)(1)(-\lambda)=\lambda^2. $$ Therefore $\sqrt{n}(\hat{\lambda}-\lambda)\xrightarrow{d}N(0,\lambda^2)$, and the approximate covariance of $\hat{\lambda}$ is $\lambda^2/n$.

Worked Example: A Weighted Estimating Equation for a Normal Mean

Suppose $X_i\sim N(\mu,1)$, and consider the estimating equation with $\psi(X,\theta)=(1+X^2)(X-\theta)$. Evaluate its asymptotic covariance at the true value $\mu=0$. The population equation is centered there because the integrand $(1+X^2)X$ is odd under $N(0,1)$. Its population mean at a general $\theta$ is $-2\theta$, since $E[1+X^2]=2$; thus zero is the unique population root.

The sample root is the weighted mean $$ \hat{\theta}=\frac{\sum_{i=1}^n(1+X_i^2)X_i}{\sum_{i=1}^n(1+X_i^2)}. $$ At $\theta_0=0$, the derivative is $-(1+X^2)$, giving $A=-2$. The estimating contribution has mean zero, and its variance is $$ B=E[(1+X^2)^2X^2] =E[X^2]+2E[X^4]+E[X^6] =1+2(3)+15=22, $$ using the standard even moments of $N(0,1)$. The sandwich variance is $22/(-2)^2=11/2$. Consequently, the limiting covariance of $\sqrt{n}\hat{\theta}$ is $11/2$, and the approximate variance of $\hat{\theta}$ is $11/(2n)$. The equation is valid at the stated true value, but its variability is larger than that of the unweighted sample mean, whose limiting variance is 1.

Estimating the Sandwich from Data

In applications, $A$ and $B$ are usually unknown. A direct estimate of the limiting covariance is obtained by evaluating the derivative and the individual estimating contributions at the fitted value: $$ \widehat{A}=\frac{1}{n}\sum_{i=1}^n\frac{\partial\psi(X_i,\hat{\theta})}{\partial\theta^T}, \qquad \widehat{B}=\frac{1}{n}\sum_{i=1}^n\psi(X_i,\hat{\theta})\psi(X_i,\hat{\theta})^T. $$ The estimated sandwich for the covariance of $\sqrt{n}(\hat{\theta}-\theta_0)$ is $\widehat{A}^{-1}\widehat{B}\widehat{A}^{-T}$. The corresponding estimated covariance for $\hat{\theta}$ is this matrix divided by $n$.

Result: If the fitted derivative and estimating contributions obey laws of large numbers giving $\widehat{A}\xrightarrow{p}A$ and $\widehat{B}\xrightarrow{p}B$, and $A$ is nonsingular, then $$ \widehat{A}^{-1}\widehat{B}\widehat{A}^{-T}\xrightarrow{p}A^{-1}BA^{-T}. $$

To justify this, matrix inversion is continuous at every nonsingular matrix, so $\widehat{A}^{-1}\xrightarrow{p}A^{-1}$ and $\widehat{A}^{-T}\xrightarrow{p}A^{-T}$. Combining these convergences with $\widehat{B}\xrightarrow{p}B$ by Slutsky's theorem proves the claim. $\square$

Common Pitfall: Reversing the Sandwich

A frequent error is to put the variability matrix on the outside, or to omit one inverse sensitivity factor. The linearization shows why the correct order is $A^{-1}BA^{-T}$: the random fluctuation is first multiplied on the left by $-A^{-1}$, and a covariance transforms as $C B C^T$ when a random vector is multiplied by $C$. In a scalar problem the formula becomes $B/A^2$, which can hide matrix-order errors.

Another important distinction is scale. The displayed sandwich is the limiting covariance of $\sqrt{n}(\hat{\theta}-\theta_0)$; the approximate covariance of the unscaled estimator is the sandwich divided by $n$. Finally, the sensitivity matrix is the expected derivative of the estimating function, not its negative unless that sign convention is explicitly chosen. A sign error in $A$ does not alter the scalar variance, but using the wrong matrix order can alter a vector covariance.

Key takeaway: A consistent estimating-equation root fluctuates, to first order, like the inverse sensitivity applied to the average estimating contribution. Its limiting covariance is $A^{-1}BA^{-T}$, with finite-sample covariance approximately $A^{-1}BA^{-T}/n$.

Check Your Understanding

Use the definitions and derivation above to answer the following.

For a scalar estimating equation, what do $A$ and $B$ represent, and how do they combine in the limiting variance?
Why does the sandwich covariance contain a transpose on its right-hand inverse factor?
In the exponential-rate example, calculate the approximate variance of the estimator when $n=100$ and $\lambda=2$.
For the weighted normal-mean equation, identify the sensitivity and score-variability values that produce the limiting variance $11/2$.
What additional scaling converts the limiting covariance of $\sqrt{n}(\hat{\theta}-\theta_0)$ to an approximate covariance for $\hat{\theta}$?