Why the Score Expansion Matters
The maximum likelihood estimator is often said to be asymptotically efficient: its large-sample variance reaches the information bound. That statement is not automatic. It depends on the likelihood having enough smoothness, the estimator settling near the true parameter, and the curvature of the log-likelihood behaving predictably there. The score equation supplies the key link: near the truth, a small score is approximately the curvature multiplied by the estimation error.
We consider a scalar parameter. Write \(X_1,\ldots,X_n\) for independent observations from a model with density or mass function \(f(x;\theta)\), and let the true parameter be \(\theta_0\). The parameter space is an open interval containing \(\theta_0\). Define the log-likelihood \(\ell_n(\theta)=\sum_{i=1}^n \log f(X_i;\theta)\), the individual score \(u(X_i;\theta)=\partial \log f(X_i;\theta)/\partial\theta\), and the Fisher information per observation \(I(\theta_0)=E_{\theta_0}[u(X_1;\theta_0)^2]\).
The First-Order Limit
Here are sufficient conditions for the curvature and information statements to be valid. Assume the observations are identically distributed and that, on a neighborhood of \(\theta_0\), the density or mass function has common support and is twice differentiable in \(\theta\). Assume differentiation under the integral or sum is justified twice by integrable dominating functions. Also assume that the log-density has the derivatives used below, that \(0<I(\theta_0)<\infty\), and that the score has mean zero. For the curvature, assume a local uniform law of large numbers: on some neighborhood \(U\) of \(\theta_0\),
and assume the expectation inside the absolute value is continuous at \(\theta_0\). In addition, suppose \(\hat{\theta}\xrightarrow{p}\theta_0\), and that with probability tending to one it is an interior maximizer where \(\ell_n'(\hat{\theta})=0\). These are explicit requirements, not consequences of the phrase “maximum likelihood estimator” alone.
Proof. First, the information identity follows from the regularity conditions. Let \(u(x;\theta)=\partial\log f(x;\theta)/\partial\theta\). Differentiating the normalization \(\int f(x;\theta)\,dx=1\) at \(\theta_0\) gives
A second differentiation, using \(\partial^2 f/\partial\theta^2 =f\{(\partial\log f/\partial\theta)^2+\partial^2\log f/\partial\theta^2\}\), gives
Therefore the expected log-likelihood curvature per observation is \(-I(\theta_0)\).
By the mean value theorem, on the event that the score equation holds, there is a value \(\tilde{\theta}\) between \(\hat{\theta}\) and \(\theta_0\) such that
Consistency implies \(\tilde{\theta}\xrightarrow{p}\theta_0\). The local uniform law of large numbers and continuity of the expected curvature then imply \(\ell_n''(\tilde{\theta})/n\xrightarrow{p}-I(\theta_0)\). Meanwhile, the individual scores at \(\theta_0\) are independent and identically distributed, have mean zero, and have variance \(I(\theta_0)\). The central limit theorem therefore gives
Rearranging the score expansion yields
The first factor converges in probability to \(1/I(\theta_0)\). Slutsky’s theorem gives the stated normal limit, whose variance is \(I(\theta_0)/I(\theta_0)^2=1/I(\theta_0)\). Hence the estimator reaches the first-order information bound. \(\square\)
Worked Models
Worked Example: Normal Mean With Known Variance
Suppose \(X_i\sim N(\mu,\sigma^2)\), independently, with known \(\sigma^2>0\) and unknown \(\mu\). The log-likelihood, up to a term not depending on \(\mu\), is \(\ell_n(\mu)=-\sum_{i=1}^n(X_i-\mu)^2/(2\sigma^2)\).
Differentiating gives \(\ell_n'(\mu)=\sum_i(X_i-\mu)/\sigma^2\), so setting the score to zero gives \(\hat{\mu}=\bar X\). The individual score is \(u(X_i;\mu)=(X_i-\mu)/\sigma^2\), and its variance is \(I(\mu)=\sigma^2/\sigma^4=1/\sigma^2\). Thus the theorem predicts asymptotic variance \(1/I(\mu)=\sigma^2\) for \(\sqrt n(\hat\mu-\mu)\).
In this example the result is exact: \(\sqrt n(\bar X-\mu)\sim N(0,\sigma^2)\) for every \(n\). The exact finite-sample variance of \(\bar X\) is \(\sigma^2/n\), agreeing with the bound on the estimator scale as well.
Worked Example: Bernoulli Success Probability
Let \(X_i\) be independent Bernoulli observations with success probability \(p\), where \(0<p<1\). The log-likelihood is \(\ell_n(p)=\sum_i[X_i\log p+(1-X_i)\log(1-p)]\). Its score is \(\ell_n'(p)=\sum_i[X_i/p-(1-X_i)/(1-p)]\), and solving the score equation gives \(\hat p=\bar X\) when \(0<\bar X<1\). For fixed interior \(p\), the probability of \(\bar X=0\) or \(1\) is \( (1-p)^n+p^n\), which tends to zero.
The individual score can be written as \((X_i-p)/[p(1-p)]\). Its mean is zero, and since \(\operatorname{Var}(X_i)=p(1-p)\), its variance is \(I(p)=1/[p(1-p)]\). The information identity also follows directly: the second derivative of the individual log-likelihood is \(-X_i/p^2-(1-X_i)/(1-p)^2\), whose expectation is \(-1/p-1/(1-p)=-1/[p(1-p)]\).
The theorem therefore gives \(\sqrt n(\hat p-p)\xrightarrow{d}N(0,p(1-p))\). The estimator’s first-order variance is \(p(1-p)/n\), the inverse of the total information \(nI(p)\).
Worked Example: Exponential Rate
Suppose \(X_i\) are independent exponential observations with rate \(\lambda>0\), so \(f(x;\lambda)=\lambda e^{-\lambda x}\) for \(x\geq0\). The support does not depend on \(\lambda\). The log-likelihood is \(\ell_n(\lambda)=n\log\lambda-\lambda\sum_iX_i\). The score equation \(n/\lambda-\sum_iX_i=0\) gives \(\hat\lambda=1/\bar X\).
The individual score is \(1/\lambda-X_i\). Since \(E[X_i]=1/\lambda\) and \(\operatorname{Var}(X_i)=1/\lambda^2\), it has mean zero and information \(I(\lambda)=1/\lambda^2\). Also, \(\ell_n''(\lambda)=-n/\lambda^2\), so the normalized curvature at any intermediate value \(\tilde\lambda\xrightarrow{p}\lambda\) tends to \(-1/\lambda^2\).
The estimator is consistent because \(\bar X\xrightarrow{p}1/\lambda\), and taking reciprocals gives \(\hat\lambda\xrightarrow{p}\lambda\). The theorem now yields \(\sqrt n(\hat\lambda-\lambda)\xrightarrow{d}N(0,\lambda^2)\). Its first-order variance is \(\lambda^2/n\), equal to \(1/[nI(\lambda)]\).
What the Efficiency Claim Does—and Does Not—Say
The conclusion concerns the limiting distribution after scaling by \(\sqrt n\). It does not say that the finite-sample variance of every maximum likelihood estimator is exactly \(1/[nI(\theta_0)]\), nor that every likelihood has a well-behaved maximizer. In the normal example, exact equality happens because the sample mean is exactly normal; in the Bernoulli and exponential examples, the inverse-information statement is a first-order asymptotic conclusion.
A common mistake is to write “the score is zero at the MLE, so the MLE has variance inverse information” and stop there. The score equation alone does not justify replacing the random curvature by \(-nI(\theta_0)\). The replacement needs the estimator to be consistent, the score equation to apply at an interior point, and the normalized curvature to converge locally. If any of these fail—for example, because the true parameter lies on a boundary or the model’s support changes with the parameter—the Taylor argument above may not apply. Such cases require separate analysis.
Check Your Understanding
Use the definitions and derivation above to answer these questions.
- Why is consistency needed to conclude that the intermediate value \(\tilde{\theta}\) in the score expansion approaches \(\theta_0\)?
- For \(X_i\sim N(\mu,\sigma^2)\) with known \(\sigma^2\), calculate the Fisher information per observation for \(\mu\) and its inverse.
- What property of the score allows the central limit theorem to produce a centered normal limit?
- Why does the score equation not, by itself, prove that an MLE reaches the information bound?
- In the Bernoulli example, why is it enough for the score equation to hold with probability tending to one?