Local Likelihood Expansion and Quadratic Approximation

Why Expand the Log-Likelihood Locally?

A likelihood can be complicated over the full parameter space yet have a simple shape close to the parameter that generated the data. A Taylor expansion makes that local shape explicit: the score determines the first-order change, while the curvature determines the second-order change. In large samples, this distinction is especially useful because parameter changes of order \(1/\sqrt{n}\) can produce log-likelihood changes of order one.

Here, \(\beta\) is the finite-dimensional GLM regression parameter and \(\beta_0\) is its true value. Write \(\ell_n(\beta)\) for the log-likelihood based on \(n\) observations. The discussion concerns the likelihood as a function of \(\beta\); any dispersion parameter is treated as known and fixed. The expansion does not itself establish a distributional limit for an estimator.

Definition: The score at \(\beta\) is the gradient \(U_n(\beta)=\nabla\ell_n(\beta)\). The observed negative Hessian is \(J_n(\beta)=-\nabla^2\ell_n(\beta)\). When the log-likelihood is twice differentiable, the Hessian describes local curvature; positive \(J_n(\beta)\) means the log-likelihood curves downward in the corresponding directions.

For independent observations, the log-likelihood is a sum of individual log-likelihood contributions. Consequently, its score and Hessian are also sums. Under usual smoothness and moment conditions, their magnitudes are typically of orders \(\sqrt{n}\) and \(n\), respectively. The exact expansion below states explicit conditions sufficient for using those orders.

The Taylor Expansion at the True Parameter

For a displacement \(d\) such that the line segment from \(\beta_0\) to \(\beta_0+d\) lies in the parameter space, Taylor's theorem gives

\ell_n(\beta_0+d) = \ell_n(\beta_0) + U_n(\beta_0)^\mathsf{T}d - \frac{1}{2}d^\mathsf{T}J_n(\beta_0)d + R_n(d).

The remainder \(R_n(d)\) collects terms beyond second order. The first-order term is a signed change in log-likelihood: it can be positive or negative depending on the observed score and the direction \(d\). The quadratic term is nonpositive when \(J_n(\beta_0)\) is positive semidefinite. Thus, the curvature term describes the local tendency for the likelihood to fall away from \(\beta_0\), while the score term can tilt that local shape.

Theorem: Suppose \(\beta\) has fixed dimension \(p\), \(\beta_0\) is an interior point of the parameter space, and, with probability tending to one, \(\ell_n\) is three times continuously differentiable on a fixed neighborhood of \(\beta_0\). Suppose also that \(U_n(\beta_0)=O_p(\sqrt{n})\), \(J_n(\beta_0)/n\xrightarrow{p} I\), where \(I\) is finite and positive definite, and the supremum on that neighborhood of the operator norm of the third derivative of \(\ell_n\) is \(O_p(n)\). Then, for every fixed \(K<\infty\), uniformly over \(\|h\|\leq K\),

\(\ell_n(\beta_0+h/\sqrt{n})-\ell_n(\beta_0) = h^\mathsf{T}U_n(\beta_0)/\sqrt{n} -\frac{1}{2}h^\mathsf{T}Ih+o_p(1).\)

The conditions are stated directly in terms of the log-likelihood and are sufficient conditions for this conclusion. In regular fixed-dimensional GLMs, the score and curvature conditions arise from summing observation-level contributions, with suitable moment and smoothness assumptions. Positive definiteness of \(I\) excludes directions with asymptotically no curvature.

Proof. Apply the multivariable Taylor theorem along the line \(\beta(t)=\beta_0+td\), \(0\leq t\leq1\). The one-variable third-order remainder is bounded in absolute value by \(M_n\|d\|^3/6\), where \(M_n\) is the supremum of the third-derivative operator norm on the neighborhood. Set \(d=h/\sqrt{n}\). For \(\|h\|\leq K\), the bound is at most \(M_nK^3/(6n^{3/2})=O_p(n^{-1/2})\), uniformly in \(h\), and so it is \(o_p(1)\).

The second-order term after this substitution is \(-\frac12h^\mathsf{T}\{J_n(\beta_0)/n\}h\). Because \(J_n(\beta_0)/n\xrightarrow{p}I\), the difference between this expression and \(-\frac12h^\mathsf{T}Ih\) converges to zero in probability uniformly for \(\|h\|\leq K\): its absolute value is at most \(\frac12 K^2\|J_n(\beta_0)/n-I\|\). Combining this with the Taylor remainder proves the stated expansion. \(\square\)

The score term remains random in this result. Its assumed order makes \(h^\mathsf{T}U_n(\beta_0)/\sqrt{n}\) an order-one term for bounded \(h\); the theorem does not set it to zero. The quadratic term is also order one because \(J_n(\beta_0)\) is order \(n\) and the displacement is order \(1/\sqrt{n}\).

Worked Examples

Worked Example: Logistic Intercept with Four Observations

Consider four independent Bernoulli observations with common success probability \(p(\beta)=e^\beta/(1+e^\beta)\), the logistic model with an intercept only. Suppose the observed outcomes are \(1,1,0,1\), and expand at \(\beta_0=\log(1.5)\). The log-likelihood, omitting no parameter-dependent terms, is

\ell_4(\beta)=3\beta-4\log(1+e^\beta).

At \(\beta_0\), \(e^{\beta_0}=1.5\), so \(p(\beta_0)=1.5/2.5=0.6\). Differentiating gives the score and curvature:

U_4(\beta)=3-4p(\beta),\qquad \ell_4''(\beta)=-4p(\beta)\{1-p(\beta)\}.

Thus \(U_4(\beta_0)=3-4(0.6)=0.6\), and \(J_4(\beta_0)=4(0.6)(0.4)=0.96\). For a displacement \(d\), the second-order Taylor approximation to the log-likelihood change is therefore \(0.6d-0.48d^2\). At \(d=0.1\), this is \(0.06-0.0048=0.0552\). The nonzero score makes the linear term visible; the curvature subtracts \(0.0048\) from that predicted increase. This is a finite-sample Taylor calculation, not a claim that the remainder is negligible for this sample.

Worked Example: Exact Remainder for a Poisson Intercept

Let \(Y_1,Y_2,Y_3\) be independent Poisson variables with mean \(e^\beta\), and suppose the observed counts are \(0,1,2\). Their sum is \(3\), and the log-likelihood is \(\ell_3(\beta)=3\beta-3e^\beta-\log(2)\). Expand at \(\beta_0=0\). The score is \(U_3(\beta)=3-3e^\beta\), so \(U_3(0)=0\), and the second derivative is \(\ell_3''(\beta)=-3e^\beta\), so \(J_3(0)=3\).

For a displacement \(d\), subtracting the log-likelihood at zero gives the exact change

\ell_3(d)-\ell_3(0)=3d-3(e^d-1).

The quadratic expansion is \(0\cdot d-\frac12(3)d^2=-1.5d^2\). At \(d=0.2\), it predicts \(-1.5(0.2)^2=-0.06\). The exact change is \(0.6-3(e^{0.2}-1)\). Using \(e^{0.2}\approx1.2214028\), this equals \(0.6-3(0.2214028)\approx-0.0642084\). The exact remainder after the quadratic term is therefore approximately \(-0.0042084\). It is not zero: the exponential log-likelihood has nonzero third and higher derivatives.

Worked Example: Gaussian Intercept and an Exact Quadratic

Suppose \(Y_1,Y_2,Y_3\) are independent \(N(\beta,1)\) observations, with observed values \(1,2,3\). Treat the variance \(1\) as known. The log-likelihood, up to its constant term, is \(\ell_3(\beta)=-\frac12\sum_{i=1}^3(y_i-\beta)^2\). At \(\beta_0=2\), the score is \(\sum_i(y_i-\beta_0)=(1-2)+(2-2)+(3-2)=0\), and the second derivative is \(-3\), giving \(J_3(\beta_0)=3\).

For any displacement \(d\), expanding each squared residual gives

\ell_3(2+d)-\ell_3(2) = d\sum_{i=1}^3(y_i-2)-\frac{3}{2}d^2 = -\frac{3}{2}d^2.

For example, at \(d=0.1\) the change is exactly \(-3(0.1)^2/2=-0.015\). Here the quadratic approximation is exact because the Gaussian log-likelihood with known variance is itself quadratic in the mean parameter. This exactness is special; the Poisson example shows why a remainder must usually be considered.

What Makes the Approximation Local?

The scale of the displacement matters. With \(d=h/\sqrt{n}\) and bounded \(h\), the second-order term is typically order one, while the third-order remainder is typically order \(n^{-1/2}\). This is why a quadratic approximation can be accurate in a shrinking neighborhood of the true parameter even when the full log-likelihood is not quadratic.

By contrast, for a fixed nonzero \(d\), a third-derivative bound of order \(n\) gives a remainder bound of order \(n\|d\|^3\). That bound does not generally become small as \(n\) increases. A second-order expansion at the true parameter should therefore not be treated as a global approximation without additional justification.

Key takeaway: Near \(\beta_0\), the score supplies the linear tilt and the observed negative Hessian supplies the local curvature. On the \(1/\sqrt{n}\) scale, both terms can remain order one, while regular third-order remainder conditions make the quadratic approximation accurate for bounded local displacements.

Common Pitfall

A frequent error is to argue that the score term vanishes because the expansion is centered at the true parameter. Centering means that the Taylor expansion is evaluated at \(\beta_0\); it does not mean the observed score \(U_n(\beta_0)\) equals zero. The score is random, and the expansion retains \(h^\mathsf{T}U_n(\beta_0)/\sqrt{n}\). A zero score is a likelihood stationarity condition at a fitted parameter when such a stationary point exists, not an identity at the true parameter for every sample.

A second related error is to drop the remainder for arbitrary displacements. The theorem controls it for bounded \(h\) when \(d=h/\sqrt{n}\), under its stated conditions. It does not claim that the same remainder is negligible for a fixed displacement, or that every GLM has a globally quadratic log-likelihood.

Check Your Understanding

Use the score, curvature, and remainder ideas to answer these questions.

In the Taylor expansion, which term describes the first-order change, and which term describes the second-order curvature?
Why does a displacement \(h/\sqrt{n}\), with bounded \(h\), make an order-\(n\) third-derivative bound compatible with a negligible remainder?
In the Poisson example, why is the quadratic approximation not exact even though the score at the expansion point is zero?
Does centering an expansion at the true parameter force the observed score to be zero? Explain.
What additional feature of the Gaussian example makes its quadratic approximation exact for every displacement?