Asymptotic Normality of Z-Estimators

From an Estimating Equation to a Normal Limit

An estimating equation turns data into a parameter estimate by requiring a sample average to equal zero. Consistency says that a chosen root approaches the parameter value targeted by the population equation. Asymptotic normality asks a more precise question: after multiplying its error by $\sqrt{n}$, what random variation remains? The key is that a differentiable equation changes approximately linearly near its target, while the empirical equation at the target fluctuates on the $n^{-1/2}$ scale.

This tutorial develops that argument for a finite-dimensional parameter and a just-identified equation: the number of equations equals the number of unknown parameters. It does not require the equation to be a likelihood score. Its limiting covariance is determined by two ingredients: the variability of the equation at the target and the sensitivity of its population mean to changes in the parameter.

Definition: Let $X_1,\ldots,X_n$ be independent and identically distributed observations, let $\theta\in\Theta\subset\mathbb{R}^p$, and let $\psi(X,\theta)\in\mathbb{R}^p$ be a measurable estimating function. Define its empirical and population means by $$ \Psi_n(\theta)=\frac{1}{n}\sum_{i=1}^n\psi(X_i,\theta), \qquad \Psi(\theta)=E[\psi(X,\theta)]. $$ A value $\hat{\theta}$ is a Z-estimator if it solves $\Psi_n(\hat{\theta})=0$. The target $\theta_0$ solves $\Psi(\theta_0)=0$. The sensitivity matrix and score covariance are, respectively, $$ A=E\!\left[\frac{\partial\psi(X,\theta_0)}{\partial\theta^T}\right], \qquad B=\operatorname{Var}(\psi(X,\theta_0)). $$

The derivative is a $p\times p$ matrix, with row $j$, column $k$ equal to the derivative of component $j$ of $\psi$ with respect to parameter component $k$. The empirical equation at $\theta_0$ is an average of mean-zero random vectors. Under a finite second-moment condition, the multivariate central limit theorem gives its $\sqrt{n}$-scaled fluctuation a normal limit with covariance $B$. The remaining task is to transfer that fluctuation from the equation to its root.

Asymptotic Normality Theorem

Here is a set of sufficient conditions that makes the transfer precise. Directly assuming the needed local uniform convergence of the empirical Jacobian avoids hiding a uniformity condition inside a Taylor expansion.

$\theta_0$ is an interior point of $\Theta$, $\Psi(\theta_0)=0$, and $E\|\psi(X,\theta_0)\|^2<\infty$.
$\psi(X,\theta)$ is differentiable in $\theta$ in a neighborhood of $\theta_0$ with probability one. The expectation defining $A$ exists, and $A$ is nonsingular.
For some fixed closed ball $K$ centered at $\theta_0$ and contained in that neighborhood, the empirical Jacobian converges uniformly in probability: $$ \sup_{\theta\in K}\left\| \frac{1}{n}\sum_{i=1}^n\frac{\partial\psi(X_i,\theta)}{\partial\theta^T} - E\!\left[\frac{\partial\psi(X,\theta)}{\partial\theta^T}\right] \right\|\xrightarrow{p}0. $$
The population Jacobian $E[\partial\psi(X,\theta)/\partial\theta^T]$ is continuous at $\theta_0$. The estimator is a consistent root: $\Psi_n(\hat{\theta})=0$ with probability tending to one and $\hat{\theta}\xrightarrow{p}\theta_0$.

Theorem: Under these conditions, $$ \sqrt{n}(\hat{\theta}-\theta_0) = -A^{-1}\frac{1}{\sqrt{n}}\sum_{i=1}^n\psi(X_i,\theta_0)+o_p(1) \xrightarrow{d} N_p\!\left(0,A^{-1}B(A^{-1})^T\right). $$ The first equality is the asymptotic linear representation. The normal limit is allowed to be degenerate if $B$ is singular.

The uniform Jacobian condition is a sufficient condition, not the only way to establish the result. For example, suitable continuity and integrable-envelope assumptions can imply it through a uniform law of large numbers. Whatever route is used, pointwise convergence of the Jacobian at $\theta_0$ alone does not automatically justify evaluating it at the random intermediate values between $\theta_0$ and $\hat{\theta}$.

Proof: Linearize the Root

Write $D\psi(X,\theta)=\partial\psi(X,\theta)/\partial\theta^T$. On the event that $\hat{\theta}$ lies in $K$, define the integrated empirical Jacobian along the segment from $\theta_0$ to $\hat{\theta}$: $$ J_n=\int_0^1\frac{1}{n}\sum_{i=1}^n D\psi\!\left(X_i,\theta_0+t(\hat{\theta}-\theta_0)\right)\,dt. $$ The fundamental theorem of calculus, applied to each component along this segment, gives $$ \Psi_n(\hat{\theta})-\Psi_n(\theta_0)=J_n(\hat{\theta}-\theta_0). $$ Since $\Psi_n(\hat{\theta})=0$, it follows that $J_n(\hat{\theta}-\theta_0)=-\Psi_n(\theta_0)$.

We first show $J_n\xrightarrow{p}A$. Consistency implies that the entire segment lies within any fixed-radius ball around $\theta_0$ with probability tending to one. The uniform Jacobian condition makes the difference between the empirical and population Jacobians uniformly small on $K$. Continuity of the population Jacobian at $\theta_0$ makes its values uniformly close to $A$ on a sufficiently small ball around $\theta_0$. Combining these facts in the integral gives $J_n\xrightarrow{p}A$. Because $A$ is nonsingular, $J_n$ is nonsingular with probability tending to one and $J_n^{-1}\xrightarrow{p}A^{-1}$.

The multivariate central limit theorem applies to the independent, identically distributed vectors $\psi(X_i,\theta_0)$, which have mean zero and finite covariance $B$: $$ \frac{1}{\sqrt{n}}\sum_{i=1}^n\psi(X_i,\theta_0)\xrightarrow{d}N_p(0,B). $$ In particular, this sequence is $O_p(1)$. Using the root identity and multiplying by $\sqrt n$, $$ \sqrt{n}(\hat{\theta}-\theta_0) =-J_n^{-1}\frac{1}{\sqrt{n}}\sum_{i=1}^n\psi(X_i,\theta_0). $$ Replacing $J_n^{-1}$ by its probability limit $A^{-1}$ changes the right-hand side by $o_p(1)$: the matrix difference is $o_p(1)$, and the score sum is $O_p(1)$. This proves the linear representation. Slutsky’s theorem then gives a centered normal limit with covariance $(-A^{-1})B(-A^{-1})^T=A^{-1}B(A^{-1})^T$, as claimed. $\square$

Worked Examples

Worked Example: The Sample Mean

Let $X_i$ be independent observations with $E[X_i]=\mu$ and $\operatorname{Var}(X_i)=\sigma^2<\infty$. Use $\psi(X,\theta)=X-\theta$. Then $\Psi(\theta)=\mu-\theta$, so the unique population root is $\theta_0=\mu$. The empirical root is $\hat{\theta}=\bar{X}$.

The derivative is $-1$, giving $A=-1$, and $B=\operatorname{Var}(X)=\sigma^2$. The theorem yields $$ \sqrt{n}(\bar{X}-\mu) =-(-1)^{-1}\frac{1}{\sqrt{n}}\sum_{i=1}^n(X_i-\mu)+o_p(1) = \frac{1}{\sqrt{n}}\sum_{i=1}^n(X_i-\mu)+o_p(1) \xrightarrow{d}N(0,\sigma^2). $$ In this case the equality before the limit is exact without the $o_p(1)$ term. The calculation also checks the sign: the minus sign in the theorem cancels the negative sensitivity.

Worked Example: A Bivariate Mean

Suppose $X_i\in\mathbb{R}^2$ are independent bivariate normal observations with mean $\theta_0=(2,-1)^T$ and covariance $$ \Sigma=\begin{pmatrix}4&1\\1&9\end{pmatrix}. $$ This covariance is positive definite because its first diagonal entry is positive and its determinant is $4(9)-1(1)=35>0$. Set $\psi(X,\theta)=X-\theta$. The population root is $\theta_0$, the empirical root is $\bar{X}$, and the Jacobian is $-I_2$, so $A=-I_2$ and $B=\Sigma$.

Since $-A^{-1}=I_2$, the theorem gives $$ \sqrt{n}(\bar{X}-\theta_0)\xrightarrow{d} N_2\!\left(0,\begin{pmatrix}4&1\\1&9\end{pmatrix}\right). $$ Thus the limiting variances of the two coordinates are 4 and 9, and their limiting covariance is 1. The off-diagonal term remains: treating the two coordinates as independent would give the wrong joint limit.

Worked Example: An Exponential Rate Equation

Let $X_i$ be independent exponential observations with rate $\lambda>0$, so $E[X_i]=1/\lambda$ and $\operatorname{Var}(X_i)=1/\lambda^2$. Consider the estimating function $\psi(X,\theta)=1/\theta-X$ for $\theta>0$. Its population equation is $1/\theta-1/\lambda=0$, whose root is $\theta_0=\lambda$. The empirical equation has root $\hat{\theta}=1/\bar{X}$.

At the target, the derivative is $-1/\lambda^2$, so $A=-1/\lambda^2$. Also $\psi(X,\lambda)=1/\lambda-X$ has mean zero and variance $B=1/\lambda^2$. Therefore $$ A^{-1}=-\lambda^2, \qquad A^{-1}B(A^{-1})^T =(-\lambda^2)\frac{1}{\lambda^2}(-\lambda^2)=\lambda^2. $$ The asymptotic result is $$ \sqrt{n}(\hat{\theta}-\lambda)\xrightarrow{d}N(0,\lambda^2). $$ Here $E[X^2]<\infty$, the equation is differentiable around $\lambda$, and $\bar{X}\xrightarrow{p}1/\lambda$, so $1/\bar{X}\xrightarrow{p}\lambda$. These facts supply the moment, local smoothness, and consistency requirements for this example.

What the Result Depends On

The limiting covariance combines random fluctuation and equation sensitivity. The matrix $B$ measures how variable the empirical equation is at the target. The matrix $A$ describes how strongly the population equation responds to a change in the parameter. If the equation is only weakly sensitive in some direction, its root can vary substantially in that direction even when the equation itself has moderate variability. Nonsingularity ensures there is a local linear response in every parameter direction.

An important pitfall is to omit the sensitivity matrix and report $B$ as the covariance of the estimator. That is generally incorrect: $B$ is the covariance of the estimating function, not of the root. For the exponential example, $B=1/\lambda^2$, whereas the estimator’s limiting variance is $\lambda^2$. The conversion uses both inverse sensitivity factors, $A^{-1}B(A^{-1})^T$.

Another error is to Taylor-expand at $\theta_0$ and treat the empirical Jacobian at an intermediate random value as though it were already equal to $A$. The estimator’s consistency places that value near $\theta_0$, but a uniformity or equivalent stochastic argument is still required to control the Jacobian there. Finally, the theorem is about a consistent root. A normal approximation around $\theta_0$ cannot be concluded from differentiability and a central limit theorem alone if the selected root does not converge to $\theta_0$.

Key takeaway: A consistent root of a smooth, locally well-behaved estimating equation is asymptotically the inverse sensitivity applied to the empirical equation’s fluctuation. Its normal limit follows by combining that linear representation with the central limit theorem.

Check Your Understanding

Use the theorem and its assumptions to answer the following questions.

For a $p$-dimensional estimating equation, what are the dimensions of $A$ and $B$, and which one must be nonsingular for the theorem?
In the sample-mean example, why is the limiting variance $\sigma^2$, rather than $-\sigma^2$?
Explain why the empirical Jacobian must be controlled near the random estimator, not only evaluated at the fixed target.
If $B$ is singular but $A$ is nonsingular, does the displayed covariance formula still make sense? What kind of normal limit may result?
For the exponential rate equation, identify separately the score covariance $B$ and the estimator’s limiting variance.