Why Maximum Likelihood Estimates Become Approximately Normal
A maximum likelihood estimate is usually found by maximizing a likelihood or, equivalently, setting its score equal to zero. For a large independent sample, that score is a sum of many contributions. Its fluctuations are therefore often approximately normal, and the score equation transfers those fluctuations to the estimate. The resulting approximation is useful for describing estimation uncertainty and is aligned with the SOA Exam ASTAM (Advanced Short-Term Actuarial Mathematics) syllabus.
The approximation requires regularity: it is not a guarantee for every likelihood or every parameter value. The parameter must be identifiable and in the interior of its parameter space, the likelihood must be sufficiently smooth near the true value, and the estimator must converge to that value. Boundary estimates, nonidentifiable models, or likelihoods that are not differentiable can require different methods.
Here \(N(\mu,\sigma^2)\) uses the second argument as the variance. The information in the definition is per observation; the information in \(n\) independent observations is \(nI(\theta_0)\). Keeping those two quantities distinct prevents a common factor-of-\(n\) error.
The Score Expansion: Why the Result Holds
For a differentiable log-likelihood, an interior maximum has sample score \(U_n(\hat{\theta})=0\). Expand that score around the true parameter \(\theta_0\). For some value \(\tilde{\theta}\) between \(\hat{\theta}\) and \(\theta_0\), Taylor’s theorem gives
Rearranging and multiplying by \(\sqrt{n}\) yields
The two factors have familiar limits. First, the central limit theorem applies to the sum of independent scores. Under the regular model, \(E_{\theta_0}[u(X_i;\theta_0)]=0\) and \(\operatorname{Var}_{\theta_0}(u(X_i;\theta_0))=I(\theta_0)\), so
Second, the average derivative of the score converges to its expectation. Consistency makes \(\tilde{\theta}\) approach \(\theta_0\); the regularity conditions allow the law of large numbers to give
The last equality is the Fisher information identity for a regular model. Substituting these limits into the rearranged score equation gives a positive factor \(I(\theta_0)^{-1}\) multiplying the limiting score. Scaling a zero-mean normal variable by this factor gives variance \(I(\theta_0)/I(\theta_0)^2=1/I(\theta_0)\). This establishes the stated limit.
The argument depends on more than differentiability at a single point. The model must be correctly specified, the true parameter must be an interior point, the information must be finite and positive, and the MLE must be consistent. The score must have a central limit theorem and the average score derivative must converge as used above. These conditions justify both limit steps; without them, the stated normal approximation need not hold.
Applying the Result to Common Likelihoods
Worked Example: Estimating an Exponential Mean
A hypothetical portfolio records \(n=64\) independent claim amounts \(X_i\), measured in days of business interruption. Assume the amounts have an exponential distribution with mean parameter \(\theta>0\), so \(f(x;\theta)=\theta^{-1}e^{-x/\theta}\) for \(x\geq 0\). For this calculation, take the true parameter to be \(\theta_0=5\) days, and suppose the observed total is \(\sum X_i=345.6\) days. Find the MLE and its large-sample distribution under the true parameter.
Method. The log-likelihood, apart from no omitted parameter-dependent terms, is \(\ell(\theta)=-n\log\theta-\sum X_i/\theta\). Its score is \(U_n(\theta)=-n/\theta+\sum X_i/\theta^2\). Setting it equal to zero gives \(\hat{\theta}=\sum X_i/n\).
Calculate the estimate. The sample mean is \(345.6/64=5.4\) days, so \(\hat{\theta}=5.4\) days.
Calculate the information. The score from one observation is \(u(X;\theta)=-1/\theta+X/\theta^2\). Since an exponential variable with mean \(\theta\) has variance \(\theta^2\), this score has variance \(\theta^2/\theta^4=1/\theta^2\). Thus \(I(\theta)=1/\theta^2\), and at \(\theta_0=5\), \(I(\theta_0)=1/25\).
Apply the limit. The variance of the large-sample approximation is \(1/[64(1/25)]=25/64=0.390625\) square days. Therefore \(\hat{\theta}\) is approximately \(N(5,0.390625)\) under the stated true parameter, with approximate standard deviation \(\sqrt{0.390625}=0.625\) days. The approximation describes repeated-sample behavior around 5, not a claim that the observed estimate 5.4 is the true parameter.
Check and trap avoided. The estimate is the observed sample mean, while the approximation is centered at the specified true value. Using \(25\), the per-observation inverse information, as the variance would omit the factor \(n=64\); the correct variance is \(25/64\).
Worked Example: A Bernoulli Probability Near the Interior
In a hypothetical review of \(n=75\) independently sampled policies, let \(X_i=1\) if policy \(i\) has a specified feature and \(X_i=0\) otherwise. Assume \(X_i\) is Bernoulli with probability \(p\), and take \(p_0=0.25\) as the true value for the sampling calculation. If 20 policies have the feature, find the MLE and its asymptotic distribution.
Method. The likelihood is \(p^{\sum X_i}(1-p)^{n-\sum X_i}\). The score is \(U_n(p)=\sum X_i/p-(n-\sum X_i)/(1-p)\). Solving \(U_n(p)=0\) gives \(\hat{p}=\sum X_i/n\), provided the maximum is in the parameter space.
Calculate the estimate. The observed proportion is \(20/75=4/15=0.2667\) to four decimal places.
Calculate the information. The one-observation score can be written \((X-p)/[p(1-p)]\). A Bernoulli variable has variance \(p(1-p)\), so \(I(p)=1/[p(1-p)]\). At \(p_0=0.25\), \(I(p_0)=1/(0.25\cdot0.75)=16/3\).
Apply the limit. The approximate variance of \(\hat{p}\) is \(1/[75(16/3)]=3/1200=0.0025\). Thus the sampling distribution is approximately \(N(0.25,0.0025)\), with approximate standard deviation \(0.05\). The observed value 0.2667 is one realized estimate; it does not change the theorem’s center from the assumed \(p_0=0.25\).
Check and trap avoided. The variance also equals \(p_0(1-p_0)/n=0.1875/75=0.0025\). Using \(1/I(p_0)=0.1875\) without dividing by \(n\) confuses single-observation information with sample information.
Worked Example: A Poisson Rate from Independent Counts
A hypothetical administrator observes \(n=40\) independent monthly counts, each modeled as Poisson with the same rate \(\lambda\). For the sampling-distribution calculation, assume \(\lambda_0=2.5\) events per month. The observed sum of the 40 counts is 103. Find the MLE and its large-sample distribution.
Method. For observations \(X_i\sim\operatorname{Poisson}(\lambda)\), the log-likelihood is \(\ell(\lambda)=\sum X_i\log\lambda-n\lambda-\sum\log(X_i!)\). Its score is \(U_n(\lambda)=\sum X_i/\lambda-n\). Setting the score to zero gives \(\hat{\lambda}=\sum X_i/n\).
Calculate the estimate. The MLE is \(103/40=2.575\) events per month.
Calculate the information. The score from one observation is \(X/\lambda-1\). Since a Poisson variable has variance \(\lambda\), its score variance is \(\lambda/\lambda^2=1/\lambda\), giving \(I(\lambda)=1/\lambda\). At \(\lambda_0=2.5\), the total information is \(40/2.5=16\).
Apply the limit. The asymptotic variance is the reciprocal of total information, \(1/16=0.0625\), and the approximate standard deviation is \(0.25\). Therefore \(\hat{\lambda}\) is approximately \(N(2.5,0.0625)\) under the assumed true rate.
Check and trap avoided. The formula \(I(\lambda_0)=1/2.5=0.4\) is per count; multiplying by 40 gives 16. Centering the sampling approximation at the observed estimate 2.575 instead of the assumed true value 2.5 would describe a different, plug-in approximation rather than the theorem as stated.
Interpreting the Approximation and Its Limits
In a scalar regular model, the leading large-sample variance is the reciprocal of total Fisher information, \(1/[nI(\theta_0)]\). More observations therefore reduce the variance at rate \(1/n\), while the standard deviation decreases at rate \(1/\sqrt{n}\). This is a limiting result: at a particular finite sample size, the distribution can still be skewed or otherwise differ noticeably from a normal distribution.
For a vector parameter \(\boldsymbol{\theta}\) with positive-definite per-observation information matrix \(\mathcal{I}(\boldsymbol{\theta}_0)\), the same score-expansion argument gives \(\sqrt{n}(\hat{\boldsymbol{\theta}}-\boldsymbol{\theta}_0)\) approximately multivariate normal with mean vector zero and covariance matrix \(\mathcal{I}(\boldsymbol{\theta}_0)^{-1}\). The off-diagonal entries matter: they describe dependence between the components of the estimator. If the information matrix is singular, this regular multivariate conclusion does not apply as written.
A particularly important exception occurs at a boundary. For example, if the true Bernoulli probability is \(p_0=0\), the usual interior score argument fails: the information expression \(1/[p(1-p)]\) is not finite there, and a sample of zeros produces an estimate exactly at the boundary. One should not mechanically apply the interior normal formula in that case. Similar care is needed when the model is misspecified, the parameter is not identifiable, or the likelihood has irregular behavior.
Check Your Understanding
For each problem, use the stated true parameter for the asymptotic sampling distribution and show how the sample size enters the variance.
- Suppose \(n=25\) independent observations come from an exponential distribution with rate \(\lambda_0=2\), whose density is \(\lambda e^{-\lambda x}\) for \(x\geq0\). If their total is 13, find the MLE and the asymptotic distribution of that MLE. (Answer: \(\hat{\lambda}=13/25=0.52\); approximately \(N(2,0.16)\).)
- For \(n=100\) independent Bernoulli observations, let the true probability be \(p_0=0.4\). Find the asymptotic variance and standard deviation of the MLE of \(p\). (Answer: variance \(0.4(0.6)/100=0.0024\); standard deviation approximately \(0.0490\).)
- Suppose \(n=36\) independent Poisson counts have true mean \(\lambda_0=3\), and their observed sum is 118. Find the MLE and the asymptotic distribution of the MLE. (Answer: \(\hat{\lambda}=118/36=59/18\approx3.2778\); approximately \(N(3,1/12)\), with variance approximately \(0.0833\).)
- Let \(X_i\sim N(\mu,9)\) independently, with the variance known to be 9, and suppose \(n=25\), \(\mu_0=12\), and the observed sample mean is 12.4. Find the MLE and its asymptotic distribution. (Answer: \(\hat{\mu}=12.4\); approximately \(N(12,9/25)=N(12,0.36)\), with standard deviation \(0.6\).)
- A Bernoulli model is used with true probability \(p_0=0\). Is the usual interior-parameter asymptotic normal formula applicable? State the relevant issue. (Answer: No; \(p_0=0\) is a boundary point, and \(1/[p(1-p)]\) is not finite there, so the regular interior score argument does not apply.)