From an Empirical Optimum to a Limiting Distribution
An M-estimator is defined by optimizing a criterion computed from data. Consistency says that the optimizer approaches the target parameter, but it does not by itself describe the estimator’s fluctuations around that target. For a smooth criterion, those fluctuations are governed by two local features: how variable the empirical gradient is at the target, and how sharply the population criterion bends there.
The derivation has a direct structure. Near the target, an interior optimizer has zero empirical gradient. A Taylor expansion expresses that gradient as its value at the target plus local curvature times the estimation error. A central limit theorem controls the gradient at the target; convergence of the curvature then converts the gradient limit into the estimator’s limit.
The theorem below treats a maximizer. Its local argument also applies to a minimizer after reversing the sign of the criterion. All vectors are column vectors; \(\nabla m\) is the \(d\)-dimensional gradient, and \(\nabla^2m\) is its \(d\) by \(d\) Hessian matrix.
Asymptotic Normality Theorem
- \(\hat{\theta}_n\xrightarrow{p}\theta_0\), and with probability tending to one, \(\hat{\theta}_n\) is an interior local maximizer where the first-order condition \(\nabla M_n(\hat{\theta}_n)=0\) holds.
- The individual gradient \(s(X,\theta_0)=\nabla m(X,\theta_0)\) has mean zero and finite covariance matrix \(B=\operatorname{Var}(s(X,\theta_0))\), and $$ \sqrt{n}\,\nabla M_n(\theta_0)\xrightarrow{d}N(0,B). $$
- Writing \(A=E[\nabla^2m(X,\theta_0)]\), the integrated empirical Hessian along the segment from \(\theta_0\) to \(\hat{\theta}_n\) satisfies $$ \int_0^1 \nabla^2 M_n\!\left(\theta_0+t(\hat{\theta}_n-\theta_0)\right)\,dt \xrightarrow{p} A, $$ where \(A\) is nonsingular.
The gradient central limit condition follows, for example, from the multivariate central limit theorem when the observations are independent and identically distributed and the stated score has finite covariance. The integrated-Hessian condition is a local condition: one common sufficient route is a uniform law of large numbers for the Hessians in a neighborhood of \(\theta_0\), together with continuity of the population Hessian there. The theorem states the condition directly so that the needed convergence along the random segment is explicit.
On the event that the optimizer is interior and differentiable, its empirical gradient is zero.
Integrate the Hessian along the segment between the target and estimator to express the gradient difference exactly.
The gradient has a central limit, the integrated Hessian converges to \(A\), and matrix inversion and Slutsky’s theorem give the estimator’s limit.
Proof of the Theorem
Set \(\Delta_n=\hat{\theta}_n-\theta_0\), and define the integrated empirical Hessian $$ H_n=\int_0^1 \nabla^2 M_n(\theta_0+t\Delta_n)\,dt. $$ On the event where the first-order condition holds, the fundamental theorem of calculus applied to the gradient gives $$ 0=\nabla M_n(\hat{\theta}_n) =\nabla M_n(\theta_0)+H_n\Delta_n. $$ By assumption \(H_n\xrightarrow{p}A\), and \(A\) is nonsingular. The set of nonsingular matrices is open, and matrix inversion is continuous on that set. Consequently, \(H_n\) is nonsingular with probability tending to one and \(H_n^{-1}\xrightarrow{p}A^{-1}\). Rearranging the expansion on this event yields $$ \sqrt{n}\Delta_n=-H_n^{-1}\sqrt{n}\,\nabla M_n(\theta_0). $$ The first-order condition may fail on an event whose probability tends to zero; changing a random vector on such an event does not change its limiting distribution. By the gradient central limit condition, convergence in probability of \(H_n^{-1}\), and Slutsky’s theorem, the right-hand side converges in distribution to the negative of \(A^{-1}Z\), where \(Z\sim N(0,B)\). Its covariance is $$ A^{-1}B(A^{-1})^{T}, $$ because multiplying a mean-zero normal vector by \(-A^{-1}\) gives that covariance. This proves the result. \(\square\)
For a population maximum, \(A\) is often negative definite, expressing local concavity. The minus sign in the linear expansion is still important, but it does not change the covariance. The matrix \(B\) records the variability of the empirical gradient, while \(A\) describes how strongly the criterion’s gradient changes with the parameter.
Worked Example: Normal Mean with Known Variance
Let \(X_1,\ldots,X_n\) be independent with \(X_i\sim N(\mu_0,\sigma^2)\), where \(\sigma^2\) is known. Use the criterion \(m(x,\theta)=-(x-\theta)^2/2\), with \(\theta\in\mathbb{R}\). Its empirical maximizer is the sample mean: differentiating gives \(\nabla M_n(\theta)=\bar X-\theta\), so \(\hat{\theta}_n=\bar X\).
At \(\theta_0=\mu_0\), the individual gradient is \(s(X,\mu_0)=X-\mu_0\). Thus \(E[s(X,\mu_0)]=0\), \(B=\sigma^2\), and the central limit theorem gives \(\sqrt{n}\nabla M_n(\mu_0)\xrightarrow{d}N(0,\sigma^2)\). The Hessian is always \(-1\), so \(A=-1\). The theorem therefore gives $$ \sqrt{n}(\bar X-\mu_0)\xrightarrow{d}N(0,\sigma^2). $$ In this case the result also agrees with the exact distribution: \(\bar X\sim N(\mu_0,\sigma^2/n)\).
Worked Example: Bernoulli Likelihood and Boundary Maximizers
Let \(Y_1,\ldots,Y_n\) be independent Bernoulli random variables with \(P(Y_i=1)=p_0\), where \(0<p_0<1\). On the closed parameter space \(0\leq p\leq1\), maximize the average log-likelihood $$ M_n(p)=\frac{1}{n}\sum_{i=1}^n\{Y_i\log p+(1-Y_i)\log(1-p)\}. $$ At the boundary, use the continuous extension of each observation’s likelihood contribution: a term with coefficient zero contributes zero, including \(0\log 0=0\); a term with positive coefficient multiplying \(\log 0\) is \(-\infty\).
For \(0<p<1\), writing \(\bar Y=n^{-1}\sum_iY_i\), differentiation gives $$ M_n'(p)=\frac{\bar Y-p}{p(1-p)}. $$ If \(0<\bar Y<1\), this derivative is positive for \(p<\bar Y\) and negative for \(p>\bar Y\), so the maximizer is \(p=\bar Y\). If \(\bar Y=0\), then \(M_n(p)=\log(1-p)\) for \(0<p<1\), and its maximum on the closed parameter space is at \(p=0\). If \(\bar Y=1\), the maximum is at \(p=1\). Thus \(\hat p=\bar Y\) is a maximizer in every sample, including samples in which all observations agree.
For the asymptotic calculation, the individual score at an interior \(p\) is $$ s(Y,p)=\frac{Y}{p}-\frac{1-Y}{1-p}=\frac{Y-p}{p(1-p)}. $$ At \(p_0\), its mean is zero and its variance is \(B=1/[p_0(1-p_0)]\). The expected second derivative is $$ A=E\!\left[-\frac{Y}{p_0^2}-\frac{1-Y}{(1-p_0)^2}\right] =-\frac{1}{p_0}-\frac{1}{1-p_0} =-\frac{1}{p_0(1-p_0)}. $$ Since \(p_0\) is interior and \(\bar Y\xrightarrow{p}p_0\), the estimator is interior with probability tending to one: the probability of a boundary estimate is \(p_0^n+(1-p_0)^n\to0\). The theorem applies and gives $$ \sqrt{n}(\hat p-p_0)\xrightarrow{d}N\!\left(0,\frac{1}{A^2}B\right) =N\!\left(0,p_0(1-p_0)\right). $$ The closed parameter space makes the finite-sample maximizer well-defined, while the interior first-order condition needed for the asymptotic argument holds with probability tending to one.
Worked Example: A Weighted Criterion with Different Curvature and Score Variability
Let \(X_i\) be independent with an exponential distribution of rate 1, and maximize $$ M_n(\theta)=-\frac{1}{2n}\sum_{i=1}^n(1+X_i)(X_i-\theta)^2, \qquad \theta\in\mathbb{R}. $$ The weights \(1+X_i\) are positive, so the criterion is strictly concave and its unique maximizer is $$ \hat{\theta}_n=\frac{\sum_{i=1}^n(1+X_i)X_i}{\sum_{i=1}^n(1+X_i)}. $$ Since \(E[X]=1\), \(E[X^2]=2\), the population maximizer is \(\theta_0=E[(1+X)X]/E[1+X]=3/2\). The law of large numbers applied to numerator and denominator shows \(\hat{\theta}_n\xrightarrow{p}3/2\).
The individual score is \(s(X,\theta)=(1+X)(X-\theta)\), and the Hessian is \(-(1+X)\). Hence \(A=-E[1+X]=-2\). At \(\theta_0=3/2\), the score has mean zero. To calculate \(B\), use \(E[X^k]=k!\) for an exponential random variable of rate 1: $$ \begin{aligned} B &=E[(1+X)^2(X-3/2)^2]\\ &=E[X^4-X^3-\tfrac{11}{4}X^2+\tfrac32X+\tfrac94]\\ &=24-6-\tfrac{11}{4}(2)+\tfrac32+\tfrac94 =\frac{65}{4}. \end{aligned} $$ The exponential distribution has finite moments of every order, so the score has finite variance and the required central limit theorem applies. The theorem gives $$ \sqrt{n}(\hat{\theta}_n-\tfrac32) \xrightarrow{d}N\!\left(0,\frac{B}{A^2}\right) =N\!\left(0,\frac{65}{16}\right). $$ Here curvature alone would not determine the limiting variance: the score variability \(65/4\) is also needed.
Why Both Ingredients Matter
The asymptotic covariance combines the two sources of information as \(A^{-1}B(A^{-1})^T\). If the criterion is steep near its optimum, a given gradient fluctuation produces a relatively small parameter displacement. If the empirical gradient fluctuates substantially, the estimator fluctuates more. Under additional likelihood conditions, curvature and score variability may be related; for a general M-estimator they need not be equal.
A common mistake is to use the inverse curvature as the entire asymptotic covariance. That discards \(B\), which measures the actual variation of the empirical gradient. Another is to write the first-order condition for an optimizer that can lie on the boundary. At a boundary optimum, the gradient need not be zero, so the Taylor argument cannot simply be invoked. One must establish an interior first-order condition, at least with probability tending to one, or use a method designed for boundary behavior.
Check Your Understanding
Use the theorem and its assumptions to answer the following.
- In the theorem, what does the matrix \(B\) measure, and what does \(A\) measure?
- Why does nonsingularity of \(A\), together with \(H_n\xrightarrow{p}A\), allow the proof to use \(H_n^{-1}\) with probability tending to one?
- For the Bernoulli example, why is it important to specify a closed parameter space or otherwise define the estimator on samples with all observations equal?
- If the population criterion is minimized rather than maximized, how can the theorem be applied?
- In the weighted exponential example, identify the separate numerical contributions of curvature and score variability to the limiting variance.