Consistency and Asymptotic Normality of the MLE

Why Study the Large-Sample Behavior of an MLE?

A maximum-likelihood estimator is defined by optimizing a sample likelihood, but its practical value depends on what happens as the sample grows. Consistency says that the estimator approaches the true parameter in probability. Asymptotic normality describes its approximate sampling distribution at large sample sizes, making it possible to quantify its precision.

The argument has two distinct parts. First, the sample log likelihood must be controlled over the whole parameter space well enough to ensure that its maximizer is near the population maximizer. Then, near the true parameter, a score expansion and the central limit theorem give the estimator’s limiting distribution. The first part is global; the second is local. A local expansion alone cannot establish that a global maximizer is close to the truth.

Consistency: The Global Likelihood Argument

Suppose \(X_1,\ldots,X_n\) are independent and identically distributed with density or probability mass function \(f(x;\theta)\), where the true parameter is \(\theta_0\). Write the per-observation log likelihood and its population counterpart as

Q_n(\theta)=\frac{1}{n}\sum_{i=1}^n \log f(X_i;\theta), \qquad Q(\theta)=E_{\theta_0}\!\left[\log f(X_i;\theta)\right].

The MLE \(\hat{\theta}_n\) maximizes \(Q_n(\theta)\) over the parameter space \(\Theta\). A useful sufficient set of conditions for consistency is: the sample criterion converges uniformly in probability to \(Q\) over \(\Theta\); \(Q\) has a unique maximizer at \(\theta_0\); and that maximum is well separated, meaning that for every \(\varepsilon>0\), there is an \(\eta_\varepsilon>0\) such that

\sup_{\theta:\,|\theta-\theta_0|\geq\varepsilon} Q(\theta) \leq Q(\theta_0)-\eta_\varepsilon.

Under these conditions, and assuming a measurable global maximizer exists, \(\hat{\theta}_n\xrightarrow{p}\theta_0\). Compactness of \(\Theta\), continuity, and a unique maximizer can help ensure separation and existence, but compactness alone does not replace the other conditions.

Theorem: If \(\sup_{\theta\in\Theta}|Q_n(\theta)-Q(\theta)|\xrightarrow{p}0\), \(Q\) has a unique, well-separated maximizer at \(\theta_0\), and \(\hat{\theta}_n\) is a global maximizer of \(Q_n\), then \(\hat{\theta}_n\xrightarrow{p}\theta_0\). This is a global conclusion: convergence only on a neighborhood of \(\theta_0\) is not enough unless one separately shows that the global sample maximizer enters that neighborhood with probability approaching one.

Asymptotic Normality from the Score

For a scalar parameter, let \(s_i(\theta)=\partial\log f(X_i;\theta)/\partial\theta\) denote the individual score, and let \(H_i(\theta)=\partial^2\log f(X_i;\theta)/\partial\theta^2\) denote the individual second derivative. The total score is \(S_n(\theta)=\sum_{i=1}^n s_i(\theta)\), and the total log likelihood is \(\ell_n(\theta)=\sum_{i=1}^n\log f(X_i;\theta)\).

Assume the true parameter is interior, the log likelihood is sufficiently differentiable near \(\theta_0\), the MLE is consistent and interior with probability approaching one, and the regularity conditions permit differentiation under the expectation. Suppose also that the score has mean zero, finite positive variance, and that the information identity holds:

E_{\theta_0}[s_i(\theta_0)]=0, \qquad I(\theta_0)=E_{\theta_0}[s_i(\theta_0)^2] =-E_{\theta_0}[H_i(\theta_0)]>0.

The information \(I(\theta_0)\) is per observation. The central limit theorem gives \(S_n(\theta_0)/\sqrt n\xrightarrow{d}N(0,I(\theta_0))\). Since an interior MLE satisfies \(S_n(\hat{\theta}_n)=0\), a Taylor expansion around \(\theta_0\) yields, for an intermediate value \(\tilde{\theta}_n\) between \(\hat{\theta}_n\) and \(\theta_0\),

0=S_n(\hat{\theta}_n) =S_n(\theta_0)+\ell_n''(\tilde{\theta}_n)(\hat{\theta}_n-\theta_0).

If the average second derivative at this intermediate value converges in probability to \(-I(\theta_0)\), consistency and the law of large numbers give

\sqrt n(\hat{\theta}_n-\theta_0) = \left[-\frac{1}{n}\ell_n''(\tilde{\theta}_n)\right]^{-1} \frac{S_n(\theta_0)}{\sqrt n} \xrightarrow{d} N\!\left(0,I(\theta_0)^{-1}\right).

The inverse-curvature factor scales the random score into the estimator’s error. For a vector parameter, the corresponding result replaces the scalar score and information with a score vector and a nonsingular information matrix: \(\sqrt n(\hat{\theta}_n-\theta_0)\xrightarrow{d}N(0,I(\theta_0)^{-1})\). The scalar derivation above is the same argument with matrices reduced to numbers.

Estimated Standard Error

The asymptotic variance of \(\hat{\theta}_n\) is approximately \(1/[nI(\theta_0)]\), so its standard error is approximately \(1/\sqrt{nI(\theta_0)}\). Since \(\theta_0\) is unknown, replace it with the MLE and estimate information from the curvature of the observed log likelihood. The observed information is \(J_n(\theta)=-\ell_n''(\theta)\), giving

\widehat{SE}(\hat{\theta}_n) = \left[-\ell_n''(\hat{\theta}_n)\right]^{-1/2}.

Alternatively, if \(I(\theta)\) is known as a function of the parameter, use \([nI(\hat{\theta}_n)]^{-1/2}\). These estimates are useful when the observed information is positive and the regular large-sample approximation is appropriate. The standard error describes estimated sampling variability; it is not the MLE’s distance from the truth in a particular sample.

Worked Example: Bernoulli Probability Including Endpoint Samples

Suppose \(Y_1,\ldots,Y_n\) are independent Bernoulli observations with success probability \(p_0\), and let the parameter space be the closed interval \([0,1]\). In a hypothetical sample of \(n=400\), there are 128 successes. Find the MLE and its estimated standard error, and describe its large-sample behavior.

The likelihood is \(L(p)=\prod_i p^{Y_i}(1-p)^{1-Y_i}\). Maximizing it over the closed parameter space gives \(\hat{p}_n=\bar{Y}\); this also holds if all observations are successes or all are failures, in which cases the MLE is respectively 1 or 0. For the stated sample,

\hat{p}_n=\bar{Y}=\frac{128}{400}=0.3200.

The sample proportion converges in probability to \(p_0\) by the law of large numbers. When \(0<p_0<1\), the Bernoulli score has variance \(I(p_0)=1/[p_0(1-p_0)]\), so the asymptotic variance of \(\hat{p}_n\) is \(p_0(1-p_0)/n\). Substituting the estimate gives

\widehat{SE}(\hat{p}_n) =\sqrt{\frac{\hat{p}_n(1-\hat{p}_n)}{n}} =\sqrt{\frac{0.3200(0.6800)}{400}} =\sqrt{0.000544} \approx 0.02332.

Thus, under repeated sampling from the stated Bernoulli model, the estimated standard deviation of the sample proportion is about 0.02332, or 2.332 percentage points. Using \([0,1]\) matters: with parameter space \((0,1)\), a finite sample containing only successes or only failures has no maximizing parameter in that space.

Worked Example: Normal Mean with Known Variance

Let \(X_i\) be independent \(N(\mu,4)\) observations, so the variance is known to be \(4\) and the unknown parameter is the mean \(\mu\in\mathbb{R}\). Suppose a hypothetical sample has \(n=25\) and \(\bar{x}=10.40\). Derive the MLE and its standard error.

Ignoring terms that do not depend on \(\mu\), the log likelihood is \(\ell_n(\mu)=-\frac{1}{8}\sum_i(X_i-\mu)^2+C\). Its score is \(S_n(\mu)=\frac{1}{4}\sum_i(X_i-\mu)\). Setting the score equal to zero gives \(\hat{\mu}_n=\bar{X}\), and therefore \(\hat{\mu}_n=10.40\) for this sample. The individual score variance is \(I(\mu)=1/4\); equivalently, the total observed information is \(n/4\). Hence

\widehat{SE}(\hat{\mu}_n) =\left(\frac{n}{4}\right)^{-1/2} =\frac{2}{\sqrt{25}} =0.4000.

The model assumptions are satisfied by construction for this hypothetical example: independent observations follow the specified normal distribution, the parameter is interior, and the log likelihood is differentiable. In fact, \(\bar{X}\sim N(\mu,4/n)\) exactly, not merely asymptotically. Thus the estimator is consistent and has standard error 0.4000 measurement units for this sample size.

Worked Example: Poisson Rate

Suppose independent counts \(X_i\) follow a Poisson distribution with rate \(\lambda_0>0\). Use the parameter space \([0,\infty)\), and consider a hypothetical sample of \(n=100\) counts with total \(\sum_iX_i=230\). Find the MLE and its estimated standard error.

For \(\lambda>0\), the log likelihood is \(\ell_n(\lambda)=\sum_i[X_i\log\lambda-\lambda-\log(X_i!)]\). Its score is \(\sum_iX_i/\lambda-n\). Solving the score equation gives \(\hat{\lambda}_n=\bar{X}=230/100=2.300\). If all counts are zero, the likelihood is maximized at \(\lambda=0\) in the stated parameter space; that endpoint possibility does not affect the interior large-sample calculation when the true rate is positive.

For a Poisson observation, the score is \(X_i/\lambda-1\), whose variance is \(1/\lambda\). The observed information is \(\sum_iX_i/\lambda^2\), which at the MLE equals \(n/\hat{\lambda}_n\). Therefore,

\widehat{SE}(\hat{\lambda}_n) =\sqrt{\frac{\hat{\lambda}_n}{n}} =\sqrt{\frac{2.300}{100}} =\sqrt{0.02300} \approx 0.1517.

The counts are independent, the true rate is positive and interior, and the score has finite variance, so the central limit theorem and information-based approximation apply. The estimated standard error is about 0.1517 counts per observation in rate units. The sample mean is also consistent by the law of large numbers.

Why Global Control Matters

A common mistake is to argue that uniform convergence near \(\theta_0\) forces a global sample maximizer to be near \(\theta_0\). That conclusion does not follow: the sample criterion could agree closely with the population criterion in a fixed neighborhood yet have a larger remote peak elsewhere. Consistency requires global uniform control together with separation, or a separate argument that localizes the global maximizer with probability approaching one.

A second pitfall is to treat the score equation as proof of consistency. The equation \(S_n(\hat{\theta}_n)=0\) identifies a stationary point when the MLE is interior; by itself, it neither shows that the point is a global maximum nor that it is near \(\theta_0\). The likelihood convergence argument establishes consistency; the score expansion then establishes the limiting distribution.

Key takeaway: Establish global consistency before using a local score expansion. Under regularity, the score contributes asymptotic variation, while expected likelihood curvature determines how that variation translates into estimation error. Estimated information supplies the corresponding standard error.

Check Your Understanding

Answer each question using the definitions and arguments in this tutorial.

Why does uniform convergence of the sample criterion only on a neighborhood of \(\theta_0\) fail, by itself, to ensure consistency of a global MLE?
State the scalar score expansion used to derive the MLE’s asymptotic distribution.
If the per-observation information is \(I(\theta_0)\), what is the asymptotic variance of \(\hat{\theta}_n\) to first order in sample size?
Why is the closed parameter space \([0,1]\) useful when defining the Bernoulli MLE?
For the Poisson example, calculate the estimated standard error if the sample size is 400 and the sample mean is 1.60.