Asymptotic Normality of the Maximum-Likelihood Estimator

Why Maximum-Likelihood Estimators Have a Normal Limit

For a regular model, the maximum-likelihood estimator (MLE) is determined by a balance: the total score is zero at the fitted parameter. Near the true parameter, the score changes approximately linearly, with its slope governed by information. Meanwhile, the score at the true parameter fluctuates on the order of $\sqrt{n}$. Together, these facts explain why the estimator’s error is typically on the order of $1/\sqrt{n}$ and has an approximately normal distribution.

This conclusion is not automatic. A likelihood can have multiple competing maxima, an MLE can fail to exist, or the information matrix can be singular. The result below makes the main requirements explicit: global identification for consistency, and local smoothness and nonsingular information for the normal approximation.

Model, Score, and Information

Let $X_1,\ldots,X_n$ be independent and identically distributed observations from a model with density or mass function $f_\theta$, where $\theta$ belongs to a subset $\Theta$ of an open set in $\mathbb{R}^p$. Suppose the true parameter is $\theta_0$. Write the log-likelihood as

\ell_n(\theta)=\sum_{i=1}^n \log f_\theta(X_i).

The score for one observation is $s_\theta(x)=\nabla_\theta\log f_\theta(x)$. The total score is $U_n(\theta)=\nabla_\theta\ell_n(\theta)$, and the Hessian is $H_n(\theta)=\nabla_\theta^2\ell_n(\theta)$. Gradients are column vectors; information matrices are $p\times p$.

Definition: Under regularity conditions that allow differentiation under the integral or sum defining the model, the Fisher information in one observation at $\theta_0$ is $$ I(\theta_0)=E_{\theta_0}\!\left[s_{\theta_0}(X)s_{\theta_0}(X)^{\mathsf T}\right] =-E_{\theta_0}\!\left[\nabla_\theta^2\log f_\theta(X)\big|_{\theta=\theta_0}\right]. $$ It is nonsingular when it is positive definite. The information in $n$ independent observations is $nI(\theta_0)$.

The equality between the two expressions is the information identity. One sufficient set of conditions is that the support does not depend on $\theta$, the density is twice differentiable near $\theta_0$, and its first and second derivatives can be passed under the integral, with the resulting score and Hessian expectations finite.

Result: Information identity. Under these conditions, $E_{\theta_0}[s_{\theta_0}(X)]=0$ and $E_{\theta_0}[\nabla_\theta^2\log f_\theta(X)|_{\theta=\theta_0}]=-I(\theta_0)$.

To verify the result, differentiate $\int f_\theta(x)\,dx=1$ once. Since $\nabla_\theta f_\theta=f_\theta s_\theta$, this gives $\int f_{\theta_0}s_{\theta_0}\,dx=0$. Differentiate twice and use

\nabla_\theta^2 f_\theta =f_\theta\left\{\nabla_\theta^2\log f_\theta+ s_\theta s_\theta^{\mathsf T}\right\}.

The second derivative of the integral of $f_\theta$ is zero. Dividing the resulting integral by its interpretation as expectation gives $E_{\theta_0}[\nabla_\theta^2\log f_\theta+s_{\theta_0}s_{\theta_0}^{\mathsf T}]=0$, which proves the identity. The interchange conditions are what justify these differentiations. $\square$

Conditions for Consistency and Asymptotic Normality

Here is a standard version of the regular MLE result. It states conditions for a global MLE rather than assuming that a local solution of the score equation is automatically the desired estimator.

Theorem: Asymptotic normality of a regular MLE. Suppose:

$\Theta$ is compact, $\theta_0$ is an interior point, and $X_1,\ldots,X_n$ are iid under $\theta_0$.
The expected log-likelihood $M(\theta)=E_{\theta_0}[\log f_\theta(X)]$ is finite and continuous on $\Theta$, and uniquely maximized at $\theta_0$. This is the expected-log-likelihood form of identifiability.
A uniform law of large numbers holds: $\sup_{\theta\in\Theta}|n^{-1}\ell_n(\theta)-M(\theta)|\xrightarrow{p}0$.
The log density is twice continuously differentiable in a neighborhood of $\theta_0$. The score has mean zero, finite covariance $I(\theta_0)$, and the information identity holds with $I(\theta_0)$ positive definite.
For some neighborhood $B$ of $\theta_0$, the Hessian satisfies $\sup_{\theta\in B}\|n^{-1}H_n(\theta)-E_{\theta_0}[\nabla^2\log f_\theta(X)]\|\xrightarrow{p}0$, and the expected Hessian is continuous at $\theta_0$.
A measurable global maximizer $\hat\theta$ exists, and it is an interior point with probability tending to one.

Then $\hat\theta\xrightarrow{p}\theta_0$, and $$ \sqrt{n}(\hat\theta-\theta_0)\xrightarrow{d}N\!\left(0,I(\theta_0)^{-1}\right). $$

The uniform laws are explicit assumptions here. In applications they can be established using suitable integrable bounds and regularity of the log-likelihood and its derivatives. The final interiority condition matters because a differentiable likelihood has zero score at an interior maximum, but not necessarily at a boundary maximum.

Proof: Consistency and the Score Expansion

First consider consistency. Fix $\varepsilon>0$. The set $C_\varepsilon=\{\theta\in\Theta:\|\theta-\theta_0\|\geq\varepsilon\}$ is compact. By continuity and the unique maximum of $M$, there is a positive gap

\delta=M(\theta_0)-\sup_{\theta\in C_\varepsilon}M(\theta)>0.

On the event $\sup_{\Theta}|n^{-1}\ell_n-M|<\delta/3$, every $\theta\in C_\varepsilon$ has empirical log-likelihood at least $\delta/3$ below the value at $\theta_0$. Indeed, $n^{-1}\ell_n(\theta)\leq M(\theta_0)-2\delta/3$ there, while $n^{-1}\ell_n(\theta_0)>M(\theta_0)-\delta/3$. A global maximizer cannot lie in $C_\varepsilon$. The uniform law makes this event’s probability tend to one, so $P(\|\hat\theta-\theta_0\|\geq\varepsilon)\to0$. Thus $\hat\theta\xrightarrow{p}\theta_0$.

For the distributional limit, with probability tending to one both $\hat\theta$ and the line segment from $\theta_0$ to $\hat\theta$ lie in $B$. Interiority gives $U_n(\hat\theta)=0$. Applying the fundamental theorem of calculus to the score along that segment yields the exact identity

0=U_n(\theta_0)+ \left\{\int_0^1 H_n\!\left(\theta_0+t(\hat\theta-\theta_0)\right)\,dt\right\}(\hat\theta-\theta_0).

This integral form is useful for a vector parameter: it does not assume that one intermediate point represents every component of the score change. Define

A_n=-\int_0^1 n^{-1}H_n\!\left(\theta_0+t(\hat\theta-\theta_0)\right)\,dt.

Consistency, the local uniform Hessian condition, and continuity of the expected Hessian imply $A_n\xrightarrow{p}I(\theta_0)$. To see this, the difference between $A_n$ and $I(\theta_0)$ is bounded by the uniform Hessian error on $B$, plus the largest difference between the expected Hessian at a point on the segment and its value at $\theta_0$. The first term tends to zero in probability; the second does too because the segment endpoint converges in probability to $\theta_0$ and the expected Hessian is continuous there.

The multivariate central limit theorem applied to the iid scores gives $n^{-1/2}U_n(\theta_0)\xrightarrow{d}N(0,I(\theta_0))$. Since $I(\theta_0)$ is positive definite, matrix inversion is continuous near it, so $A_n^{-1}\xrightarrow{p}I(\theta_0)^{-1}$. Rearranging the score identity gives

\sqrt{n}(\hat\theta-\theta_0)=A_n^{-1}\frac{U_n(\theta_0)}{\sqrt{n}}.

Slutsky’s theorem now gives a centered normal limit with covariance $I(\theta_0)^{-1}I(\theta_0)I(\theta_0)^{-1}=I(\theta_0)^{-1}$. This proves both conclusions. $\square$

Worked Examples

Worked Example: Bernoulli Success Probability

Suppose $X_i$ is Bernoulli with success probability $p$, and the true value is $p_0=0.4$. For $0<p<1$, the log-likelihood is $\ell_n(p)=\sum_i\{X_i\log p+(1-X_i)\log(1-p)\}$. Let $\bar X=n^{-1}\sum_iX_i$. The score and its derivative are

U_n(p)=\frac{n(\bar X-p)}{p(1-p)},\qquad H_n(p)=-\frac{\sum_iX_i}{p^2}-\frac{\sum_i(1-X_i)}{(1-p)^2}.

When $0<\bar X<1$, setting the score to zero gives $\hat p=\bar X$. The expected log-likelihood is $p_0\log p+(1-p_0)\log(1-p)$; its derivative vanishes only at $p=p_0$, and its second derivative is strictly negative. Thus it has a unique maximum. The score for one observation is $(X-p)/[p(1-p)]$, so its variance, and hence the information, is $I(p)=1/[p(1-p)]$. At $p_0=0.4$, $I(p_0)=1/0.24=25/6$, giving limiting variance $I(p_0)^{-1}=0.24$.

Therefore $\sqrt{n}(\hat p-0.4)\xrightarrow{d}N(0,0.24)$. For $n=100$, the approximation for $\hat p$ is $N(0.4,0.24/100)=N(0.4,0.0024)$. If every observation is a success or every observation is a failure, the maximizer over the closed parameter space is at a boundary. The probability of that event is $0.4^n+0.6^n\to0$, so it does not change this limit; it does show why finite-sample interiority should not be assumed without checking.

Worked Example: Exponential Rate

Let $X_i$ have density $f_\lambda(x)=\lambda e^{-\lambda x}$ for $x>0$, with $\lambda>0$, and suppose $\lambda_0=2$. The log-likelihood is $n\log\lambda-\lambda\sum_iX_i$. Its score is $n/\lambda-\sum_iX_i$, and its second derivative is $-n/\lambda^2<0$. Setting the score to zero gives the unique MLE $\hat\lambda=1/\bar X$.

The one-observation score is $1/\lambda-X$, which has mean zero and variance $\operatorname{Var}(X)=1/\lambda^2$. Thus $I(\lambda)=1/\lambda^2$, and $I(2)^{-1}=4$. The expected log-likelihood is $\log\lambda-\lambda/\lambda_0$, uniquely maximized at $\lambda=\lambda_0$. The sample mean converges to $1/\lambda_0>0$, so the MLE is consistent. The theorem gives

\sqrt{n}(\hat\lambda-2)\xrightarrow{d}N(0,4).

Equivalently, for large $n$, the estimator has approximate variance $4/n$. This is a local approximation around the true rate; it does not say that the finite-sample distribution of $1/\bar X$ is exactly normal.

Worked Example: Bivariate Normal Mean with Known Covariance

Suppose $X_i\sim N_2(\mu,\Sigma)$, with known $\Sigma=\begin{pmatrix}4&1\\1&2\end{pmatrix}$ and unknown $\mu\in\mathbb{R}^2$. The log-likelihood, up to terms not involving $\mu$, is $-\tfrac12\sum_i(X_i-\mu)^{\mathsf T}\Sigma^{-1}(X_i-\mu)$. Differentiating gives $U_n(\mu)=\Sigma^{-1}\sum_i(X_i-\mu)$; setting it to zero yields $\hat\mu=\bar X$. The Hessian is $-n\Sigma^{-1}$, so it is negative definite.

The score for one observation is $\Sigma^{-1}(X-\mu)$. Its covariance is $\Sigma^{-1}\Sigma\Sigma^{-1}=\Sigma^{-1}$, so $I(\mu)=\Sigma^{-1}$ and the asymptotic covariance of $\sqrt n(\hat\mu-\mu)$ is $I(\mu)^{-1}=\Sigma$. In fact, the sample mean is exactly normal in this model. Since $\det(\Sigma)=8-1=7$,

\Sigma^{-1}=\frac{1}{7}\begin{pmatrix}2&-1\\-1&4\end{pmatrix}, \qquad \operatorname{Cov}(\hat\mu)=\frac{1}{n}\begin{pmatrix}4&1\\1&2\end{pmatrix}.

For $n=50$, this covariance is $\begin{pmatrix}0.08&0.02\\0.02&0.04\end{pmatrix}$. The off-diagonal entry is not an error: the two coordinates of the estimated mean remain correlated because the observations’ coordinates are correlated.

Interpretation and Common Pitfall

The information matrix quantifies local curvature of the expected log-likelihood and the variability of the score. More information means a smaller asymptotic covariance, but the covariance is a matrix: it describes both marginal uncertainty and dependence among parameter estimates. For a regular MLE, the approximate covariance of $\hat\theta$ is $I(\theta_0)^{-1}/n$, not $I(\theta_0)^{-1}$ without the sample-size factor.

Pitfall: using information instead of inverse information. The score’s limiting covariance is $I(\theta_0)$, but the estimator’s limiting covariance is $I(\theta_0)^{-1}$. The score equation divides its fluctuations by the likelihood curvature, which produces the inverse.
Pitfall: treating a stationary point as a consistent MLE. A root of the score equation could be a local minimum, a different local maximum, or a solution far from $\theta_0$. The global expected-log-likelihood maximum and uniform convergence are what establish consistency of a global maximizer.
Pitfall: ignoring boundaries or singular information. Boundary estimates need not satisfy the zero-score equation, and a singular information matrix cannot be inverted to give the stated normal limit. Such cases require different conditions and are not covered by this theorem.

The result is useful because it turns likelihood curvature into a practical large-sample uncertainty calculation. In applications, $I(\theta_0)$ is unknown and is commonly replaced by an estimate, often based on the observed Hessian or estimated score covariance. The justification for that replacement depends on consistency and regularity, not merely on having computed a Hessian.

Check Your Understanding

Use the assumptions and derivation above to answer these questions.

Why does a unique maximum of the expected log-likelihood help prove consistency of a global MLE?
What two quantities are equated by the information identity, and what type of regularity permits the identity?
In the score expansion, why is the integrated Hessian used instead of assuming a single intermediate point works for a vector parameter?
If $I(\theta_0)=\begin{pmatrix}4&0\\0&2\end{pmatrix}$, what is the limiting covariance of $\sqrt n(\hat\theta-\theta_0)$?
Why does the theorem require positive-definite information and an interior MLE with probability tending to one?