How Nonlinearity Changes Laplace Curvature

Why Local Curvature Matters

A Laplace approximation replaces an integral over a random effect with the contribution near a mode of the integrand. The height of the integrand at that mode is only part of the approximation: its local width matters too. A sharper mode contributes less integrated mass than a broader one with the same height. For a nonlinear prediction, the curvature determining that width contains more than the squared first derivative of the prediction. Its second derivative also matters, and its contribution can either increase or decrease the Hessian.

Consider one observation $y$, one scalar random effect $\eta$, and a prediction $f(\eta)$ that is twice differentiable in a neighborhood of the point where the Hessian is evaluated. Assume an additive normal observation model with constant residual standard deviation $\sigma>0$, and a normal random effect with standard deviation $\omega>0$. Thus $Y\mid\eta\sim N(f(\eta),\sigma^2)$ and $\eta\sim N(0,\omega^2)$. In this scalar example, $\eta$ and $\omega$ are dimensionless, while $y$, $f(\eta)$, and $\sigma$ have the outcome’s units.

Definition: Let $q(\eta)$ be the negative log of the joint density of the observation and random effect, with constants independent of $\eta$ omitted. For this model, $$ q(\eta)=\frac{[y-f(\eta)]^2}{2\sigma^2}+\frac{\eta^2}{2\omega^2}. $$ At an interior conditional mode $\hat{\eta}$, the scalar Laplace Hessian is $H=q''(\hat{\eta})$. A nondegenerate local minimum requires $q'(\hat{\eta})=0$ and $H>0$.

Deriving the Nonlinear Hessian

Write the residual as $r(\eta)=y-f(\eta)$. Differentiating $q$ once gives $q'(\eta)=-r(\eta)f'(\eta)/\sigma^2+\eta/\omega^2$, since $r'(\eta)=-f'(\eta)$. Differentiating again gives the exact curvature:

q''(\eta) = \frac{1}{\omega^2} +\frac{[f'(\eta)]^2}{\sigma^2} -\frac{[y-f(\eta)]f''(\eta)}{\sigma^2}.

The prior contributes its precision $1/\omega^2$. The squared-slope term $[f'(\eta)]^2/\sigma^2$ is the curvature obtained when the prediction is locally linear. The additional term, $-[y-f(\eta)]f''(\eta)/\sigma^2$, is the nonlinear curvature contribution. It depends jointly on the residual and the prediction’s second derivative. Therefore, nonlinearity alone does not determine the direction of the change: the residual’s sign matters too.

The distinction from local-linear curvature can be made explicit. At a specified point, define $H_{\mathrm{lin}}=1/\omega^2+[f'(\eta)]^2/\sigma^2$. Then $H=H_{\mathrm{lin}}-[y-f(\eta)]f''(\eta)/\sigma^2$. This comparison isolates the second-derivative term at the same point. In a full analysis, changing the approximation may also change the point at which curvature is evaluated; the comparison here does not assume those modes coincide.

For a one-dimensional integral with a single relevant, nondegenerate mode, the Laplace approximation follows by expanding $q$ to second order around $\hat{\eta}$: $q(\eta)\approx q(\hat{\eta})+\tfrac12H(\eta-\hat{\eta})^2$. The integral of this local exponential is $\exp\{-q(\hat{\eta})\}\sqrt{2\pi/H}$. Restoring the normalizing constants of both normal densities gives the approximate marginal likelihood:

L = \frac{1}{2\pi\sigma\omega}\int_{-\infty}^{\infty}\exp\{-q(\eta)\}\,d\eta \quad\Longrightarrow\quad L_{\mathrm{Lap}} \approx \frac{1}{2\pi\sigma\omega}\exp\{-q(\hat{\eta})\}\sqrt{\frac{2\pi}{H}} = \frac{\exp\{-q(\hat{\eta})\}}{\sqrt{2\pi}\,\sigma\omega\sqrt{H}}.

The constants matter: the joint density contributes $1/(2\pi\sigma\omega)$, and the Gaussian integral contributes $\sqrt{2\pi/H}$. Holding the mode and its objective value fixed, a change from $H_{\mathrm{lin}}$ to $H$ changes the curvature factor in the likelihood by $\sqrt{H_{\mathrm{lin}}/H}$. On the log-likelihood scale, the change is $-\tfrac12\log(H/H_{\mathrm{lin}})$. This isolates curvature’s effect; it is not a claim that other parts of two likelihood approximations are identical.

Worked Example: A Quadratic Prediction Reduces Curvature

Suppose $f(\eta)=\eta+\eta^2$, $y=7/3$, and $\omega=\sigma=1$. At $\hat{\eta}=1$, the prediction is $f(1)=2$, the residual is $r=1/3$, the slope is $f'(1)=3$, and the second derivative is $f''(1)=2$. First check that this point is stationary: $$ q'(1)=-\frac{(1/3)(3)}{1^2}+\frac{1}{1^2}=0. $$

The objective value is $q(1)=\tfrac12(1/3)^2+\tfrac12(1)^2=5/9$. The exact Hessian is $$ H=1+3^2-(1/3)(2)=\frac{28}{3}\approx9.333. $$ It is positive, so this stationary point is a nondegenerate local minimum. The locally linear curvature at the same point would be $H_{\mathrm{lin}}=1+3^2=10$; the second-derivative term lowers the Hessian by $2/3$.

Using the normalized Laplace formula, $$ L_{\mathrm{Lap}}\approx\frac{\exp(-5/9)}{\sqrt{2\pi}\sqrt{28/3}}\approx0.07492. $$ If only the locally linear curvature were used at this same point, the corresponding curvature factor would give $0.5738/[\sqrt{2\pi}\sqrt{10}]\approx0.07238$. Their ratio is $\sqrt{10/(28/3)}=\sqrt{15/14}\approx1.035$. Thus, in this calculation, the exact curvature yields an approximate likelihood about 3.51% larger, with the mode and objective value held fixed.

When the Change Is Material

The relative importance of the nonlinear contribution is more informative than the size of $f''$ by itself. At a given point, compare $|[y-f(\eta)]f''(\eta)|/\sigma^2$ with $H_{\mathrm{lin}}$. A large ratio means that omitting the term can substantially misstate the local width. If the exact $H$ remains positive but is much smaller than $H_{\mathrm{lin}}$, the Laplace likelihood’s curvature factor is larger. If the exact $H$ is much larger, the factor is smaller. If the nonlinear term makes $H\leq0$ at a proposed stationary point, that point is not a nondegenerate local minimum and the ordinary local Laplace calculation there is invalid.

Worked Example: A Large Curvature and Likelihood Change

Let $f(\eta)=\eta+0.4(\eta-1)^2$, $y=2$, and again set $\omega=\sigma=1$. At $\hat{\eta}=1$, $f(1)=1$, so $r=1$; also $f'(1)=1$ and $f''(1)=0.8$. The mode condition holds because $q'(1)=-(1)(1)+1=0$, and $q(1)=\tfrac12(1)^2+\tfrac12(1)^2=1$.

The exact Hessian and the curvature from a local linearization at this point are $$ H=1+1^2-(1)(0.8)=1.2,\qquad H_{\mathrm{lin}}=1+1^2=2. $$ Both are positive, but the exact Hessian is 40% smaller than the locally linear value. The two curvature-only Laplace calculations are $$ L_{\mathrm{Lap}}\approx\frac{\exp(-1)}{\sqrt{2\pi}\sqrt{1.2}}\approx0.1340,\qquad L_{\mathrm{lin\ curvature}}\approx\frac{\exp(-1)}{\sqrt{2\pi}\sqrt{2}}\approx0.1038. $$ Their ratio is $\sqrt{2/1.2}\approx1.291$: holding the mode and objective value fixed, the smaller exact Hessian raises the curvature factor by about 29.1%. This example shows why a second-derivative contribution can matter even when the mode objective is unchanged.

Worked Example: The Nonlinear Term Can Increase Curvature

Take $f(\eta)=\eta-0.4(\eta-1)^2$, $y=2$, and $\omega=\sigma=1$. At $\hat{\eta}=1$, $f(1)=1$, $r=1$, $f'(1)=1$, and $f''(1)=-0.8$. As before, $q'(1)=-(1)(1)+1=0$ and $q(1)=1$. Now the nonlinear term is positive: $$ H=1+1^2-(1)(-0.8)=2.8, \qquad H_{\mathrm{lin}}=2. $$

The point is a nondegenerate local minimum because $H>0$. The exact-curvature Laplace value is $\exp(-1)/[\sqrt{2\pi}\sqrt{2.8}]\approx0.08771$, whereas the same-point locally linear curvature value is $\exp(-1)/[\sqrt{2\pi}\sqrt{2}]\approx0.1038$. Their ratio, $\sqrt{2/2.8}\approx0.8452$, shows that the exact curvature factor is about 15.5% smaller. The sign has reversed relative to the preceding example because $f''$ has reversed sign while the residual remains positive.

Assumptions, Diagnostics, and Alternatives

The calculation assumes a smooth scalar prediction with the derivatives used above, a normal random-effect density and a normal conditional observation density with constant variance. For the usual Laplace interpretation, the relevant mode should be interior and nondegenerate, and the integrand’s mass should be concentrated in a neighborhood where the quadratic expansion of $q$ is adequate. The condition $H>0$ is necessary for a one-dimensional local Gaussian integral, but is not by itself evidence that the approximation is accurate: a positive Hessian can coexist with substantial changes in curvature across the neighborhood that contributes appreciable mass.

Useful checks include verifying $q'(\hat{\eta})$ is close to zero, calculating $H$ from the full second-derivative formula, and confirming $H>0$. Compare $H$ with $H_{\mathrm{lin}}$, and inspect how $q$ changes on either side of the mode over a scale roughly $1/\sqrt{H}$. If that neighborhood is strongly nonquadratic, the local Gaussian approximation may be poor even if the Hessian is positive. A computed negative or nearly zero Hessian can indicate a nonminimum, a poorly located mode, or an unstable curvature calculation; it should not be repaired by taking an absolute value inside the Laplace factor.

The first-order and FOCE approximations use different linearization strategies for constructing an approximate likelihood; the Laplace approximation instead uses the local curvature of the negative log joint density at its conditional mode. The comparison with $H_{\mathrm{lin}}$ in these examples is deliberately narrower: it isolates the Hessian’s nonlinear term at a fixed point and does not equate that calculation with an entire FOCE likelihood. Numerical integration of the one-dimensional integral provides a direct alternative for checking a Laplace result when the scalar model is tractable. It avoids the local quadratic replacement, though it has its own numerical integration error.

A scalar example clarifies the mechanism but does not settle the accuracy of a general NLME likelihood. Laplace accuracy can be affected by departures from quadratic shape over the influential region, and the Hessian alone cannot quantify all such error. Also, comparing two approximations’ full likelihoods requires tracking changes in mode and objective value as well as curvature. The formula here identifies exactly how nonlinearity enters the local Hessian; it does not guarantee that a large or small curvature correction alone predicts overall approximation quality.

Common Pitfall

A common mistake is to say, “The prediction is nonlinear, so the Hessian must be larger than the linearized Hessian.” That reasoning ignores the signed product of residual and second derivative. The correct difference is $H-H_{\mathrm{lin}}=-[y-f(\eta)]f''(\eta)/\sigma^2$. It can be positive, negative, or zero. Another mistake is to treat $H>0$ as proof that the Laplace approximation is reliable; positivity permits the local Gaussian integral, while adequacy also requires the quadratic approximation to represent the important region of the integrand.

Key takeaway: In a scalar normal-observation model, the Laplace Hessian is the prior precision plus squared prediction slope, adjusted by a residual-times-second-derivative term. That term controls whether nonlinear curvature narrows or widens the local approximation and can materially change the approximate likelihood.

Check Your Understanding

Answer each question, then compare with the brief solutions.

For $f(\eta)=a+b\eta+c\eta^2$, what is the nonlinear term in $q''(\eta)$?
If $y-f(\hat{\eta})<0$ and $f''(\hat{\eta})>0$, does the nonlinear term raise or lower $H$?
At fixed mode and objective value, what happens to the Laplace curvature factor if $H$ is multiplied by four?
Why does a positive Hessian not establish that the Laplace approximation is accurate?

Solutions. (1) Since $f''(\eta)=2c$, the term is $-2c[y-f(\eta)]/\sigma^2$. (2) The product of residual and second derivative is negative, so its negative is positive; $H$ is raised. (3) The factor $1/\sqrt{H}$ is divided by $\sqrt{4}=2$. (4) Positivity ensures a local minimum with a real Gaussian-width factor, but does not ensure that $q$ is well represented by a quadratic over the region contributing substantial integral mass.