The MLE Score Expansion And Asymptotic Normality

Why the Score Expansion Matters

A maximum-likelihood estimator is defined by optimizing a likelihood, but its large-sample behavior is understood through the score equation. Near the true parameter, the score changes approximately linearly, with the Hessian governing that change. Combining this local approximation with the sampling variation of the score yields the estimator’s asymptotic distribution.

The key is to use a valid multivariate expansion. For a vector score, one cannot generally replace the integrated Hessian by the Hessian at a single intermediate parameter shared by all score components. The exact integral expansion avoids that problem and gives a direct route to the first-order representation.

Setup and Regularity Conditions

Suppose $Y_1,\ldots,Y_n$ are independent and identically distributed observations from a correctly specified parametric model with parameter $\theta\in\mathbb{R}^k$. Write the log-likelihood contribution of observation $i$ as $\ell_i(\theta)$, so the sample log-likelihood is $\ell_n(\theta)=\sum_{i=1}^n\ell_i(\theta)$. Define the individual score $s_i(\theta)=\partial\ell_i(\theta)/\partial\theta$, the sample score $U_n(\theta)=\sum_{i=1}^n s_i(\theta)$, and the Hessian $H_n(\theta)=\partial U_n(\theta)/\partial\theta'$.

The true parameter is $\theta_0$, and $\hat\theta$ denotes a maximum-likelihood estimator. The relevant regularity conditions are not merely computational conveniences. They ensure that the estimator is near the true value, the score has a central limit theorem, and the curvature of the likelihood is stable near that value.

Definition: The per-observation expected Fisher information at $\theta_0$ is $$ \mathcal I(\theta_0)=E_{\theta_0}\!\left[s_i(\theta_0)s_i(\theta_0)'\right] =-E_{\theta_0}\!\left[\frac{\partial s_i(\theta_0)}{\partial\theta'}\right], $$ where the equality is the information equality under the correctly specified, regular likelihood. The matrix is required to be finite and positive definite.

A standard regular-MLE result assumes that the parameter space contains an open neighborhood of $\theta_0$, that $\theta_0$ is an interior point, and that the model identifies it: distinct parameter values do not generate the same probability distribution, and the population likelihood is uniquely maximized at $\theta_0$. The log-likelihood contributions must be sufficiently differentiable near $\theta_0$, with derivatives that permit differentiation under the expectation and have suitable finite moments. In addition, a uniform law of large numbers for the Hessian in a neighborhood of $\theta_0$ is needed. Finally, the MLE must exist, be consistent, and with probability approaching one be an interior maximizer, so that $U_n(\hat\theta)=0$.

Under these conditions, the score has mean zero and finite covariance $\mathcal I(\theta_0)$, and the score central limit theorem gives $n^{-1/2}U_n(\theta_0)$ an asymptotic normal distribution. Correct specification and information equality make the covariance of this score equal to the expected negative Hessian. This tutorial concerns that regular, correctly specified case.

The Exact Integrated Score Expansion

Let $\Delta_n=\hat\theta-\theta_0$. Because the score is differentiable, apply the fundamental theorem of calculus to the vector-valued function $t\mapsto U_n(\theta_0+t\Delta_n)$, component by component. This yields an exact expansion, not an approximation:

0=U_n(\hat\theta) =U_n(\theta_0)+ \left[\int_0^1 H_n\!\left(\theta_0+t(\hat\theta-\theta_0)\right)\,dt\right](\hat\theta-\theta_0).

The integral is a matrix, and its multiplication by $\Delta_n$ is the remainder in the expansion. There is generally no single intermediate parameter at which the entire Hessian matrix can replace this integral. Define the average integrated Hessian $A_n=\int_0^1 n^{-1}H_n(\theta_0+t\Delta_n)\,dt$. Consistency makes the segment from $\theta_0$ to $\hat\theta$ shrink toward $\theta_0$; the uniform law of large numbers and smoothness then imply $A_n\overset{p}{\longrightarrow}-\mathcal I(\theta_0)$.

Since $\mathcal I(\theta_0)$ is nonsingular, $A_n$ is nonsingular with probability approaching one. Rearranging the exact equation on that event gives the first-order representation:

\sqrt n(\hat\theta-\theta_0) =-A_n^{-1}\frac{U_n(\theta_0)}{\sqrt n} =\mathcal I(\theta_0)^{-1}\frac{U_n(\theta_0)}{\sqrt n}+o_p(1).

For the second equality, $A_n^{-1}$ converges in probability to $-\mathcal I(\theta_0)^{-1}$; the minus sign in front of $A_n^{-1}$ therefore produces a positive inverse-information matrix. The remainder is negligible because $A_n^{-1}$ converges and the scaled score is bounded in probability by its central limit theorem.

Theorem: Under the stated identification, differentiability, moment, consistency, and nonsingular-information conditions, $$ \sqrt n(\hat\theta-\theta_0)\ \overset{d}{\longrightarrow}\ N\!\left(0,\mathcal I(\theta_0)^{-1}\right). $$ Equivalently, the large-sample covariance of $\hat\theta$ is approximately $\mathcal I(\theta_0)^{-1}/n$. Here $\mathcal I(\theta_0)$ is information per observation; the expected information in the full sample is $n\mathcal I(\theta_0)$.

Worked Examples

Worked Example: Normal Mean With Known Variance

Suppose four hypothetical independent observations are $2,3,4,5$ from $N(\mu,4)$, with known variance $4$. The parameter is $\mu$, and the log-likelihood, apart from a constant, is $\ell_n(\mu)=-\sum_i(y_i-\mu)^2/8$. The model is identifiable, $\mu$ is an interior real-valued parameter, and the likelihood is smooth.

The score and Hessian are $$ U_n(\mu)=\frac{1}{4}\sum_{i=1}^4(y_i-\mu),\qquad H_n(\mu)=-\frac{4}{4}=-1. $$ Setting the score to zero gives $\hat\mu=\bar y=(2+3+4+5)/4=3.5$. Evaluate the expansion at the reference value $\mu_0=3$. The score there is $U_n(3)=(14-12)/4=0.5$, while the integrated Hessian is $-1$, since it is constant. Thus $$ 0=0.5+(-1)(3.5-3)=0. $$

For one observation, the score variance and expected negative Hessian are both $1/4$, so $\mathcal I(\mu_0)=1/4$. The asymptotic variance of $\sqrt n(\hat\mu-\mu_0)$ is therefore $4$; the corresponding large-sample variance of $\hat\mu$ is $4/n$, or $1$ when $n=4$. In this normal model that variance formula is also exact at finite sample sizes. The four-observation calculation illustrates the score equation; by itself, it does not establish a large-sample approximation.

Worked Example: A Poisson Rate and the Integrated Hessian

Consider hypothetical independent Poisson counts $1,0,2,1,3$, with rate parameter $\lambda>0$. Their sum is $S=7$, and the log-likelihood, omitting terms that do not depend on $\lambda$, is $\ell_n(\lambda)=S\log\lambda-n\lambda$, where $n=5$. This model is identifiable, and its likelihood is smooth for every interior value $\lambda>0$.

The score is $U_n(\lambda)=S/\lambda-n$, so $U_n(\hat\lambda)=0$ gives $\hat\lambda=S/n=7/5=1.4$. To show the integrated expansion explicitly, evaluate it from $\lambda_0=1$ to $\hat\lambda=1.4$. At $\lambda_0$, $U_n(1)=7-5=2$. The Hessian is $H_n(\lambda)=-S/\lambda^2$, and its integral along this segment is $$ \int_0^1 H_n(1+0.4t)\,dt =-7\int_0^1\frac{1}{(1+0.4t)^2}\,dt =-\frac{7}{1.4}=-5. $$ Consequently, $0=2+(-5)(1.4-1)=2-2$, as required. This uses the integral Hessian rather than asserting that one intermediate Hessian gives the full vector remainder.

For one Poisson observation, the score variance and expected negative Hessian at $\lambda_0=1$ are both $1$, so $\mathcal I(1)=1$. The asymptotic variance of $\hat\lambda$ is approximately $1/n$ at that parameter value. More generally, the information per observation at rate $\lambda$ is $1/\lambda$, giving variance approximately $\lambda/n$. Substituting the estimate gives estimated standard error $\sqrt{1.4/5}=\sqrt{0.28}\approx0.5292$ counts per observation. This is a large-sample approximation, not an exact finite-sample standard error claim.

Worked Example: A Vector Score and Covariance Matrix

Suppose a hypothetical normal regression has independent errors with known variance $1$, and the three observations are $(x_i,y_i)=(-1,1),(0,2),(1,4)$. Let $\theta=(\beta_0,\beta_1)'$, with conditional mean $\beta_0+\beta_1x_i$. The design matrix has full column rank, the parameter is unrestricted, and the normal likelihood is smooth, so the finite-sample score equation has a unique solution.

Writing $X$ for the matrix with rows $(1,x_i)$, the score and Hessian are $U_n(\theta)=X'(y-X\theta)$ and $H_n(\theta)=-X'X$. Direct calculation gives $$ X'X= \begin{pmatrix}3&0\\0&2\end{pmatrix}. $$ The score equation gives $\hat\beta_0=(1+2+4)/3=7/3$ and $\hat\beta_1=[(-1)(1)+(0)(2)+(1)(4)]/2=3/2$. Evaluate the expansion at $\theta_0=(2,1)'$. Its fitted values are $1,2,3$, so the residual vector is $(0,0,1)'$ and $$ U_n(\theta_0)=X'(y-X\theta_0)= \begin{pmatrix}1\\1\end{pmatrix},\qquad \hat\theta-\theta_0= \begin{pmatrix}1/3\\1/2\end{pmatrix}. $$

The integrated Hessian is constant and equals $-X'X$. Hence the expansion checks componentwise as $$ \begin{pmatrix}1\\1\end{pmatrix} - \begin{pmatrix}3&0\\0&2\end{pmatrix} \begin{pmatrix}1/3\\1/2\end{pmatrix} = \begin{pmatrix}0\\0\end{pmatrix}. $$ For this model, the finite-sample covariance matrix of the estimator is $(X'X)^{-1}=\begin{pmatrix}1/3&0\\0&1/2\end{pmatrix}$. In a sequence of such designs where $X'X/n$ converges to a positive-definite matrix $Q$, the covariance of $\sqrt n(\hat\theta-\theta_0)$ converges to $Q^{-1}$. The positive-definiteness condition matters: without sufficient variation in the regressors, the two coefficients cannot both be identified.

Interpretation and Common Pitfalls

The expansion separates two sources of the estimator’s behavior. The scaled score $U_n(\theta_0)/\sqrt n$ supplies the random sampling variation; the inverse information transforms that variation into parameter units and accounts for the likelihood’s curvature. More information per observation means less asymptotic variance, while a nearly singular information matrix signals that some parameter combinations are weakly distinguished by the data.

Using a single intermediate Hessian for a vector score. The multivariate mean-value theorem does not generally provide one common intermediate parameter whose Hessian gives the exact remainder. Use the integrated Hessian along the segment, as above.
Dropping the minus sign. The Hessian converges to negative information at a maximum. Inverting the expansion gives $-A_n^{-1}$, which converges to $+\mathcal I(\theta_0)^{-1}$, not its negative.
Treating an asymptotic result as exact. The normal approximation describes the estimator’s limiting distribution as $n$ grows; it does not imply a normal finite-sample distribution in every likelihood.
Ignoring identification or nonsingularity. If the information matrix is singular, the inverse-information covariance formula is unavailable, and the regular result does not apply in its stated form.

Key takeaway: A regular MLE is asymptotically the inverse-information transformation of the score evaluated at the true parameter. The exact starting point is the integrated-Hessian expansion; consistency, smoothness, identification, and a nonsingular information matrix justify its normal limit.

Check Your Understanding

Answer each question using the score expansion and its conditions.

Why is an integrated Hessian the appropriate exact remainder for a vector-valued score?
If the expected information per observation is $\mathcal I(\theta_0)$, what is the asymptotic covariance matrix of $\sqrt n(\hat\theta-\theta_0)$?
In the Poisson example, verify the score at $\lambda=1$ using the five observed counts.
What role does positive definiteness of the information matrix play in the derivation?
Why does the regular asymptotic normality result not by itself guarantee an exactly normal finite-sample MLE?