Likelihood Taylor Expansions and Local Asymptotic Normality

Why Expand the Likelihood at a Local Scale?

The maximum-likelihood estimator is often close to the true parameter when the sample is large. To understand the likelihood near that parameter, consider perturbations that shrink at the same rate: a fixed vector $h$ changes the parameter from $\theta_0$ to $\theta_0+h/\sqrt n$. This is the root-sample-size scale. At this scale, random score fluctuations and the deterministic curvature of the log likelihood both remain visible.

The resulting approximation is called local asymptotic normality (LAN). It says that, for a regular model, the log likelihood ratio for a local perturbation is approximately a linear random term minus a quadratic term. The random term is governed by a central sequence, and the quadratic term is governed by Fisher information. This tutorial derives that form directly from Taylor’s theorem.

Setup and Regularity Conditions

Let $X_1,\ldots,X_n$ be independent and identically distributed observations with density $f_\theta$ relative to a common measure, where $\theta$ belongs to an open subset of $\mathbb R^p$. Suppose the true parameter $\theta_0$ is an interior point. Write the sample log likelihood as

\ell_n(\theta)=\sum_{i=1}^n \log f_\theta(X_i).

For one observation, the score is the column vector $s_\theta(x)=\nabla_\theta \log f_\theta(x)$, and the Hessian is $H_\theta(x)=\nabla_\theta^2\log f_\theta(x)$. We use the following regularity conditions for the expansion below:

In a neighborhood of $\theta_0$, the model has a common support, and $f_\theta(X_i)>0$ almost surely under $P_{\theta_0}$.
The log density is three times continuously differentiable in that neighborhood. Its third derivatives are bounded there, in absolute value, by an envelope $M(X_i)$ with $E_{\theta_0}[M(X_i)]<\infty$.
The score has mean zero and finite covariance $I(\theta_0)$, and the Hessian has finite entries with $E_{\theta_0}[H_{\theta_0}(X_i)]=-I(\theta_0)$. The matrix $I(\theta_0)$ is positive definite.
The differentiations of the density under its integral are valid through second order. In particular, the common support and integrability conditions allow the derivatives to be interchanged with integration.

Definition: The per-observation Fisher information at $\theta_0$ is $I(\theta_0)=E_{\theta_0}[s_{\theta_0}(X)s_{\theta_0}(X)^\mathsf{T}]$. The central sequence is $$ \Delta_n=\frac{1}{\sqrt n}\sum_{i=1}^n s_{\theta_0}(X_i). $$ It is a random vector that summarizes the first-order score fluctuation at the true parameter.

Under the stated conditions, the information also equals $-E_{\theta_0}[H_{\theta_0}(X)]$. To see why, differentiate $\int f_\theta(x)\,d\mu(x)=1$. The first derivative gives $\int f_\theta s_\theta\,d\mu=0$, so the score has mean zero. Since $\nabla_\theta^2 f_\theta=f_\theta(H_\theta+s_\theta s_\theta^\mathsf{T})$, the second derivative gives

0=\int \nabla_\theta^2 f_\theta(x)\,d\mu(x) =E_\theta[H_\theta(X)]+E_\theta[s_\theta(X)s_\theta(X)^\mathsf{T}].

This identity relies on the stated ability to differentiate under the integral; it is not automatic for every statistical model.

The Local Likelihood Expansion

Theorem (Local asymptotic normality): Under the regularity conditions above, for every fixed $h\in\mathbb R^p$, $$ \ell_n\left(\theta_0+\frac{h}{\sqrt n}\right)-\ell_n(\theta_0) =h^\mathsf{T}\Delta_n-\frac{1}{2}h^\mathsf{T}I(\theta_0)h+o_{P_{\theta_0}}(1). $$ Moreover, $\Delta_n\xrightarrow{d}N_p(0,I(\theta_0))$. The expansion holds uniformly when $h$ ranges over any fixed compact subset of $\mathbb R^p$.

Proof. Put $\delta=h/\sqrt n$. Apply the multivariate Taylor expansion to each log density at $\theta_0$, then sum over observations:

\ell_n(\theta_0+\delta)-\ell_n(\theta_0) =\delta^\mathsf{T}\sum_{i=1}^n s_{\theta_0}(X_i) +\frac{1}{2}\delta^\mathsf{T} \left\{\sum_{i=1}^n H_{\theta_0}(X_i)\right\}\delta +R_n.

The first term is exactly $h^\mathsf{T}\Delta_n$. For the second term, the law of large numbers applied entrywise to the Hessian gives $n^{-1}\sum_iH_{\theta_0}(X_i)\xrightarrow{p}-I(\theta_0)$. Substituting $\delta=h/\sqrt n$, that term is therefore $-\tfrac12h^\mathsf{T}I(\theta_0)h+o_p(1)$.

For fixed $h$, the third-derivative bound gives $|R_n|\leq C\|h\|^3 n^{-3/2}\sum_{i=1}^n M(X_i)$, for a finite constant $C$. By the law of large numbers, $n^{-1}\sum_iM(X_i)=O_p(1)$, so $R_n=O_p(n^{-1/2})=o_p(1)$. The same bound holds uniformly for $h$ in a fixed compact set because $\|h\|$ is then bounded. Finally, the multivariate central limit theorem applies to the independent score vectors, which have mean zero and covariance $I(\theta_0)$. It yields $\Delta_n\xrightarrow{d}N_p(0,I(\theta_0))$. Combining these steps proves the expansion and its stated normal limit. $\square$

The theorem separates two contributions. The linear term $h^\mathsf{T}\Delta_n$ is random and remains of order one. The quadratic term is nonrandom and measures the local curvature, with larger information producing a larger likelihood penalty for the same displacement $h$.

Worked Examples

Worked Example: Normal Location with Known Variance

Suppose $X_i\sim N(\theta,4)$, independently, and the true value is $\theta_0=1$. The one-observation log density, up to a constant that does not depend on $\theta$, is $-\tfrac18(X_i-\theta)^2$. Thus

Score and information. Differentiating gives $s_\theta(X_i)=(X_i-\theta)/4$, so at $\theta_0$, $E[s_{\theta_0}(X_i)]=0$ and $I(\theta_0)=\operatorname{Var}(X_i)/16=4/16=1/4$. Hence $\Delta_n=\sum_i(X_i-1)/(4\sqrt n)$.

Likelihood ratio. For a fixed scalar $h$, the local parameter is $1+h/\sqrt n$. Direct subtraction of the two squared terms gives

\ell_n\left(1+\frac{h}{\sqrt n}\right)-\ell_n(1) =\frac{h}{4\sqrt n}\sum_{i=1}^n(X_i-1)-\frac{h^2}{8} =h\Delta_n-\frac{1}{2}h^2\left(\frac14\right).

This is an exact equality, not merely an asymptotic approximation. For $h=1$, the quadratic penalty is $1/8$, and $\Delta_n$ has distribution $N(0,1/4)$ for every $n$.

Worked Example: A Two-Dimensional Normal Mean

Suppose $X_i\sim N_2(\theta,\Sigma)$, independently, with known covariance $\Sigma=\begin{pmatrix}1&1/2\\1/2&2\end{pmatrix}$ and true mean $\theta_0=(0,0)^\mathsf{T}$. Its determinant is $2-1/4=7/4>0$, and

\Sigma^{-1}= \begin{pmatrix}8/7&-2/7\\-2/7&4/7\end{pmatrix}.

The score is $s_\theta(X_i)=\Sigma^{-1}(X_i-\theta)$, and its covariance, the Fisher information for the mean, is $I(\theta_0)=\Sigma^{-1}$. The central sequence is $\Delta_n=n^{-1/2}\sum_i\Sigma^{-1}X_i$. Because the normal log likelihood is quadratic in $\theta$, the local likelihood ratio is exactly

\ell_n\left(\theta_0+\frac{h}{\sqrt n}\right)-\ell_n(\theta_0) =h^\mathsf{T}\Delta_n-\frac12h^\mathsf{T}\Sigma^{-1}h.

For $h=(1,-1)^\mathsf{T}$, multiplication gives $\Sigma^{-1}h=(10/7,-6/7)^\mathsf{T}$, and therefore $h^\mathsf{T}\Sigma^{-1}h=16/7$. The expansion becomes $h^\mathsf{T}\Delta_n-8/7$. Also $\Delta_n\sim N_2(0,\Sigma^{-1})$, so its variance in direction $h$ is $h^\mathsf{T}\Sigma^{-1}h=16/7$. This checks both the matrix ordering and the quadratic penalty.

Worked Example: Bernoulli Probability at an Interior Value

Let $X_i\sim\operatorname{Bernoulli}(\theta)$, independently, with $\theta_0=0.4$. For one observation, the log likelihood is $X_i\log\theta+(1-X_i)\log(1-\theta)$. Its score and second derivative are

s_\theta(X_i)=\frac{X_i-\theta}{\theta(1-\theta)},\qquad H_\theta(X_i)=-\frac{X_i}{\theta^2}-\frac{1-X_i}{(1-\theta)^2}.

At $\theta_0=0.4$, the score has mean zero and variance $1/[0.4(0.6)]=25/6$. The expected second derivative is $-0.4/0.4^2-0.6/0.6^2=-2.5-1.666\ldots=-25/6$, confirming the information identity. For $h=0.5$, the local probability is $0.4+0.5/\sqrt n$, and the LAN form is

\ell_n\left(0.4+\frac{0.5}{\sqrt n}\right)-\ell_n(0.4) =0.5\Delta_n-\frac12(0.5)^2\left(\frac{25}{6}\right)+o_p(1) =0.5\Delta_n-\frac{25}{48}+o_p(1),

where $\Delta_n=\sum_i(X_i-0.4)/[0.4(0.6)\sqrt n]$, and $\Delta_n\xrightarrow{d}N(0,25/6)$. The local probability is in $(0,1)$ for all sufficiently large $n$, as required for the expansion around this interior parameter.

What the LAN Form Tells Us

For each fixed local displacement $h$, the log likelihood ratio is approximated by a Gaussian linear term and a negative quadratic term. The central sequence records the sample’s random evidence in different parameter directions. The information matrix describes how sharply the likelihood curves in those directions. If $h$ is fixed, the quadratic penalty is $\tfrac12h^\mathsf{T}I(\theta_0)h$, which is positive for nonzero $h$ when the information is positive definite.

The approximation is local: it describes changes of size $1/\sqrt n$, not arbitrary fixed changes to $\theta_0$. Its importance is that it gives a common quadratic description of likelihood behavior in a neighborhood that becomes narrower as the sample grows. The asymptotic normality of $\Delta_n$ follows from the score central limit theorem; the likelihood expansion itself also needs control of curvature and the Taylor remainder.

Common pitfall: Replacing $\theta_0+h/\sqrt n$ by $\theta_0+h$ and using the same expansion changes the scale. A fixed nonzero displacement is not generally small as $n$ grows: its score contribution is typically of order $n$, not order one. The LAN formula concerns fixed $h$ multiplying $1/\sqrt n$, so that both the random linear contribution and the quadratic curvature contribution remain of order one.

Another frequent error is to use the observed Hessian at a random fitted parameter without justification. The derivation here expands at the fixed true parameter, then applies a law of large numbers to the Hessian evaluated there. Replacing that Hessian by another quantity requires a separate argument.

Check Your Understanding

Use the expansion and its assumptions to answer the following.

What is the central sequence, and why does it have an asymptotic normal distribution under the stated conditions?
In the LAN expansion, which term represents random score fluctuation and which term represents curvature?
For the normal location model with known variance $4$, calculate the information and the quadratic penalty when $h=2$.
Why is it important that $\theta_0$ be an interior point when considering $\theta_0+h/\sqrt n$?
What role does the integrable third-derivative envelope play in controlling the Taylor remainder?