Asymptotic Normality of Exponential-Family MLEs

Why Large-Sample Behavior Has a Simple Form

For a finite sample, a maximum likelihood estimator (MLE) can be irregular: it may be far from the true parameter, or it may lie near a boundary. Under suitable conditions, however, its large-sample behavior has a stable description. The estimator converges to the true parameter, and its error, multiplied by the square root of the sample size, approaches a multivariate normal distribution. This result justifies approximate standard errors and is a foundation for likelihood-based inference.

Here the setting is an independent, identically distributed sample from a regular exponential family. We use the natural parameterization, so the likelihood score is a centered sum of sufficient statistics and the Fisher information is the covariance matrix of one observation’s sufficient statistic. The main argument has two parts: first establish consistency, then expand the score around the true parameter.

Model and Regularity Conditions

Definition: A $d$-dimensional full exponential family in natural coordinates has density or mass function $$ f_\theta(x)=h(x)\exp\{\theta^{\mathsf T}T(x)-A(\theta)\}, \qquad \theta\in\Theta, $$ with respect to a common measure. Here $\Theta$ is an open subset of $\mathbb{R}^d$, $T(x)$ is a $d$-vector of sufficient statistics, and $A(\theta)$ is the log-partition function. For observations $X_1,\ldots,X_n$, the log-likelihood, up to terms not involving $\theta$, is $$ \ell_n(\theta)=\theta^{\mathsf T}\sum_{i=1}^n T(X_i)-nA(\theta). $$

The following conditions make the consistency and normality statements precise. Let $K$ be a compact subset of $\Theta$, and suppose the true parameter $\theta_0$ is in the interior of $K$. The observations are iid from $f_{\theta_0}$; the model is identifiable on $K$; $E_{\theta_0}\|T(X_1)\|<\infty$; and $A$ is continuous on $K$. For the normality result, also assume $E_{\theta_0}\|T(X_1)\|^2<\infty$, that $A$ has continuous second derivatives in a neighborhood of $\theta_0$, and that the Fisher information matrix at $\theta_0$ is positive definite. The MLE is taken over $K$. Compactness ensures that a maximizer exists, and consistency will imply it is in the interior with probability tending to one.

In a regular exponential family, differentiation of the normalizing identity gives $E_\theta[T(X_1)]=\nabla A(\theta)$ and $\operatorname{Cov}_\theta(T(X_1))=\nabla^2 A(\theta)$, whenever the stated differentiation conditions hold. Consequently, the score and Fisher information for one observation are $$ s_\theta(X_1)=T(X_1)-\nabla A(\theta), \qquad I(\theta)=\nabla^2 A(\theta). $$ For the sample, $U_n(\theta)=\nabla\ell_n(\theta)=\sum_{i=1}^n s_\theta(X_i)$. The information matrix $I(\theta)$ is per observation; the sample information is $nI(\theta)$.

Consistency of the MLE

Theorem: Under the consistency conditions above, any maximizer $\hat{\theta}_n$ of $\ell_n(\theta)$ over $K$ satisfies $\hat{\theta}_n\xrightarrow{p}\theta_0$.

Proof. Divide the log-likelihood by $n$, and omit the term $n^{-1}\sum_i\log h(X_i)$, which does not depend on $\theta$. The resulting criterion is $Q_n(\theta)=\theta^{\mathsf T}\bar T_n-A(\theta)$, where $\bar T_n=n^{-1}\sum_iT(X_i)$. Its expectation-level counterpart is $Q(\theta)=\theta^{\mathsf T}E_{\theta_0}[T(X_1)]-A(\theta)$. The law of large numbers gives $\bar T_n\xrightarrow{p}E_{\theta_0}[T(X_1)]$. Since $K$ is bounded, $$ \sup_{\theta\in K}|Q_n(\theta)-Q(\theta)| \leq \sup_{\theta\in K}\|\theta\|\, \|\bar T_n-E_{\theta_0}[T(X_1)]\| \xrightarrow{p}0. $$ Thus the criteria converge uniformly in probability on $K$.

For any $\theta\in K$, the difference in expected criteria is the negative Kullback–Leibler divergence: $$ Q(\theta)-Q(\theta_0) =E_{\theta_0}\!\left[\log\frac{f_\theta(X_1)}{f_{\theta_0}(X_1)}\right] =-D_{\mathrm{KL}}(f_{\theta_0}\Vert f_\theta)\leq 0. $$ The divergence is zero only when the two distributions agree almost surely; identifiability then implies $\theta=\theta_0$. Thus $Q$ has a unique maximizer at $\theta_0$. For any fixed $\varepsilon>0$, the set $K_\varepsilon=\{\theta\in K:\|\theta-\theta_0\|\geq\varepsilon\}$ is compact. If it is nonempty, continuity and uniqueness imply that for some $\delta>0$, $\sup_{\theta\in K_\varepsilon}Q(\theta)\leq Q(\theta_0)-\delta$. Uniform convergence in probability implies that, with probability tending to one, $Q_n$ is closer than $\delta/3$ to $Q$ everywhere on $K$. On that event, no maximizer of $Q_n$ can lie in $K_\varepsilon$, because its criterion value would be strictly smaller than at $\theta_0$. Hence $P(\|\hat{\theta}_n-\theta_0\|\geq\varepsilon)\to0$. If $K_\varepsilon$ is empty, the conclusion is immediate. Therefore $\hat{\theta}_n\xrightarrow{p}\theta_0$. $\square$

The proof uses more than a pointwise law of large numbers: it needs uniform control of the likelihood criterion over the parameter set. In this exponential-family setting, boundedness of $K$ makes that step follow directly from convergence of the sample mean of $T(X_i)$.

Asymptotic Normality from the Score Expansion

Theorem: Under the conditions above, if $I(\theta_0)$ is positive definite, then $$ \sqrt{n}(\hat{\theta}_n-\theta_0) \xrightarrow{d} N\!\left(0,I(\theta_0)^{-1}\right). $$ Equivalently, the large-sample covariance of $\hat{\theta}_n$ is approximately $I(\theta_0)^{-1}/n$.

Proof. Consistency and the fact that $\theta_0$ is interior to $K$ imply that the MLE is interior with probability tending to one. On that event its score is zero: $U_n(\hat{\theta}_n)=0$. The derivative of the score is $\nabla U_n(\theta)=-n\nabla^2 A(\theta)$. Applying the fundamental theorem of calculus along the line segment from $\theta_0$ to $\hat{\theta}_n$ gives the exact identity $$ 0=U_n(\theta_0) -n\left\{\int_0^1\nabla^2 A\!\left(\theta_0+t(\hat{\theta}_n-\theta_0)\right)\,dt\right\}(\hat{\theta}_n-\theta_0). $$ Write the matrix in braces as $J_n$. Consistency and continuity of $\nabla^2A$ at $\theta_0$ imply $J_n\xrightarrow{p}I(\theta_0)$. Since $I(\theta_0)$ is positive definite, it is invertible, and $J_n^{-1}\xrightarrow{p}I(\theta_0)^{-1}$ on events whose probability tends to one.

The individual scores at $\theta_0$ have mean zero and covariance $I(\theta_0)$. The multivariate central limit theorem therefore gives $n^{-1/2}U_n(\theta_0)\xrightarrow{d}N(0,I(\theta_0))$. Rearranging the score identity yields $$ \sqrt{n}(\hat{\theta}_n-\theta_0)=J_n^{-1}\frac{U_n(\theta_0)}{\sqrt{n}}. $$ Slutsky’s theorem now gives a centered normal limit with covariance $I(\theta_0)^{-1}I(\theta_0)I(\theta_0)^{-1}=I(\theta_0)^{-1}$. The event on which the score equation is valid has probability tending to one, so it does not change the limiting distribution. $\square$

This derivation also explains the role of information. The score at the truth fluctuates on the order of $\sqrt n$; the derivative of the sample score is on the order of $n$. Their ratio produces an estimation error on the order of $n^{-1/2}$, with curvature $I(\theta_0)$ converting score variability into parameter variability.

Worked Examples

Worked Example: Bernoulli Observations

Let $X_i\sim\operatorname{Bernoulli}(p)$, with $p\in(0,1)$, and use natural parameter $\theta=\log\{p/(1-p)\}$. Then $A(\theta)=\log(1+e^\theta)$, $T(X_i)=X_i$, and $p=e^\theta/(1+e^\theta)$. The score for one observation is $X_i-p$, and the information is $I(\theta)=p(1-p)$.

If the sample contains $s$ successes, then $\ell_n(\theta)=s\theta-n\log(1+e^\theta)$. Setting its derivative $s-np$ to zero gives $\hat p=s/n$, and hence $\hat\theta=\log\{\hat p/(1-\hat p)\}$ when $0<s<n$. For $n=100$ and $s=43$, $\hat p=0.43$ and $\hat\theta=\log(0.43/0.57)\approx-0.282$. If the true $p=0.4$, then $\theta_0=\log(2/3)\approx-0.405$, and $I(\theta_0)=0.4(0.6)=0.24$. The asymptotic standard error of $\hat\theta$ is therefore $1/\sqrt{100(0.24)}\approx0.204$. This is a large-sample approximation, not an exact finite-sample standard error.

Worked Example: Poisson Observations

Let $X_i\sim\operatorname{Poisson}(\lambda)$, with $\lambda>0$, and write $\theta=\log\lambda$. The natural form has $T(X_i)=X_i$ and $A(\theta)=e^\theta$. Thus the score is $X_i-\lambda$, while $I(\theta)=e^\theta=\lambda$.

The sample log-likelihood, up to a constant, is $\ell_n(\theta)=\theta\sum_iX_i-ne^\theta$. Its score equation is $\sum_iX_i-ne^\theta=0$, so $e^{\hat\theta}=\bar X$ and $\hat\theta=\log\bar X$ when $\bar X>0$. Suppose $n=100$ and the observed sample mean is $4.2$. Then $\hat\lambda=4.2$ and $\hat\theta=\log(4.2)\approx1.435$. At a true $\lambda=4$, $I(\theta_0)=4$, so the asymptotic standard error of $\hat\theta$ is $1/\sqrt{100(4)}=0.05$. For the estimator $\hat\lambda=e^{\hat\theta}=\bar X$, the corresponding first-order standard error is $\sqrt{4/100}=0.2$.

Worked Example: Normal Mean with Known Variance

Let $X_i\sim N(\mu,9)$, with known variance $9$ and unknown mean $\mu$. As a one-parameter exponential family, its log-partition function in the mean parameter is $A(\mu)=\mu^2/18$, up to a term independent of $\mu$, and its sufficient statistic is $T(X_i)=X_i/9$. The score is $(X_i-\mu)/9$, and $I(\mu)=1/9$.

Solving $\sum_i(X_i-\mu)/9=0$ gives $\hat\mu=\bar X$. If $n=36$ and $\bar X=12.3$, the estimate is $12.3$. At the true value $\mu_0=12$, the asymptotic variance is $I(\mu_0)^{-1}/n=9/36=0.25$, so the standard error is $0.5$. In this particular model, $\bar X\sim N(\mu,9/36)$ exactly for every $n$, so the normal approximation is exact rather than merely asymptotic.

Conditions, Interpretation, and a Common Pitfall

The normal limit is a local statement around an interior, identifiable true parameter. Positive-definite information rules out a direction in which the likelihood has no quadratic curvature. If information is singular, the inverse in the covariance formula does not exist, and a different parameterization or a separate analysis may be needed. If the true parameter is on a boundary, the MLE may not satisfy an interior score equation; the normal limit derived here need not apply.

A common mistake is to write “the score is zero at the MLE, so the MLE is asymptotically normal” without checking the other steps. The score equation alone does not ensure that an MLE exists, is consistent, or is interior. Nor does it identify the limiting covariance. The argument requires consistency, a valid score expansion, a central limit theorem for the score, and nonsingular information. Under those conditions, the inverse information gives the covariance on the $\sqrt n$ scale—not the covariance of the estimator without the factor $1/n$.

In practice, $I(\theta_0)$ is unknown and is commonly replaced by information evaluated at the MLE. The resulting standard error is an estimate of the large-sample uncertainty, not a guarantee that a particular finite sample is close to its limiting regime. For sufficiently large samples under the stated conditions, the theorem gives the approximation $\hat{\theta}_n\approx N(\theta_0,I(\theta_0)^{-1}/n)$.

Check Your Understanding

Use the definitions and derivations above to answer these questions.

Why does compactness of $K$ help turn convergence of $\bar T_n$ into uniform convergence of the likelihood criterion?
For a Bernoulli model with true success probability $p$, what is the Fisher information in the natural parameter, and what is the asymptotic variance of the natural-parameter MLE on the $\sqrt n$ scale?
In the score expansion, why does the matrix multiplying $\hat{\theta}_n-\theta_0$ converge to $I(\theta_0)$?
What role does positive definiteness of $I(\theta_0)$ play in the asymptotic normality result?
Why is it not enough to know that the score vanishes at an interior MLE when claiming asymptotic normality?