Envelope Theorem for Derivatives at the Conditional Mode

Why the Conditional Mode Has Two Different Derivative Roles

A Laplace approximation uses the conditional mode of the random effects to summarize the height and local curvature of an integrand. When a model parameter changes, that mode generally changes too. But its movement does not contribute to the derivative of the integrand's value evaluated at the mode: stationarity makes that part of the chain rule vanish. The mode's movement can still matter when differentiating the curvature evaluated there.

This distinction is useful in likelihood calculations. It avoids unnecessary terms in the derivative of the maximized log integrand, while preventing the opposite error of ignoring the mode's movement everywhere in a Laplace approximation. All examples below are hypothetical.

Setup and Assumptions

Let $b\in\mathbb{R}^q$ denote a vector of random effects for one cluster, and let $\theta$ be a scalar model parameter. Write $h(b;\theta)$ for the cluster's conditional log integrand: it includes the conditional data log likelihood and the random-effects log density, with any terms relevant to the integral. The corresponding marginal contribution is an integral of $\exp\{h(b;\theta)\}$ over $b$. Let $\hat b(\theta)$ be an interior conditional mode.

Definition: The conditional mode $\hat b(\theta)$ satisfies $h_b(\hat b(\theta);\theta)=0$, where $h_b$ is the vector of first derivatives with respect to $b$. Define the negative Hessian at the mode as $H(\theta)=-h_{bb}(\hat b(\theta);\theta)$. A nondegenerate local maximum has $H$ positive definite.

For the envelope derivative and mode-sensitivity formula, assume $h$ has continuous second derivatives in a neighbourhood of the mode, the mode is interior and locally unique, and $H$ is nonsingular. These conditions allow the implicit function theorem to give a differentiable $\hat b(\theta)$. For a Laplace approximation to represent the integral well, the relevant mode should also be a dominant mode, not merely an arbitrary stationary point. If the parameter is vector-valued, the same statements apply one coordinate at a time.

The local Laplace approximation to the log integral has the form

\ell_{\mathrm{Lap}}(\theta) \approx h(\hat b(\theta);\theta) +\frac{q}{2}\log(2\pi)-\frac{1}{2}\log|H(\theta)|.

The focus here is the derivative of the first term and its contrast with the final, curvature-dependent term. The determinant term is evaluated at the mode, so it is a different function of $\theta$ from the maximized height.

The Envelope Theorem at a Conditional Mode

Apply the ordinary chain rule to the maximized log integrand. A prime denotes differentiation with respect to $\theta$, and $h_\theta$ means a partial derivative holding $b$ fixed:

\frac{d}{d\theta}h(\hat b(\theta);\theta) =h_b(\hat b(\theta);\theta)^{T}\frac{d\hat b}{d\theta} +h_\theta(\hat b(\theta);\theta).

At the conditional mode, $h_b(\hat b(\theta);\theta)=0$. Therefore the first term is zero, regardless of the value of $d\hat b/d\theta$.

Theorem: Under the differentiability and interior-stationarity conditions above, $$ \frac{d}{d\theta}h(\hat b(\theta);\theta) =h_\theta(\hat b(\theta);\theta). $$ The derivative of the optimized height is the partial derivative with respect to the model parameter, evaluated at the mode. It is not necessary to calculate the mode's derivative to obtain this particular derivative.

This result is often called the envelope theorem. It is a chain-rule cancellation, not a claim that the mode is constant as $\theta$ changes. In particular, the partial derivative must hold $b$ fixed; replacing it with a derivative that already follows the mode would obscure the distinction.

Worked Example: A Scalar Mode That Moves with the Parameter

Consider the hypothetical log integrand $h(b;\theta)=-(b-\theta)^2/2+\theta b$. Find the derivative of its value at the mode at $\theta=0.4$, and verify the envelope calculation against direct maximization.

The stationarity equation is $h_b=-(b-\theta)+\theta=0$, so $\hat b(\theta)=2\theta$ and $d\hat b/d\theta=2$. At $\theta=0.4$, the mode is $\hat b=0.8$. The second derivative is $h_{bb}=-1$, so this stationary point is a strict maximum.

The partial derivative holding $b$ fixed is $h_\theta=(b-\theta)+b=2b-\theta$. At the mode it is $2(0.8)-0.4=1.2$. The envelope theorem therefore gives $d h(\hat b;\theta)/d\theta=1.2$.

For a direct check, substituting $b=2\theta$ gives $h(\hat b(\theta);\theta)=-(\theta)^2/2+2\theta^2=3\theta^2/2$. Its derivative is $3\theta$, which equals $1.2$ at $\theta=0.4$. In the chain rule, the mode-motion contribution is $h_b(d\hat b/d\theta)=0$, not a generally zero mode derivative.

How the Mode Itself Changes

Although its derivative cancels from the derivative of the maximized height, the mode's sensitivity can be obtained by differentiating the stationarity equation. Differentiate $h_b(\hat b(\theta);\theta)=0$:

h_{bb}(\hat b;\theta)\frac{d\hat b}{d\theta} +h_{b\theta}(\hat b;\theta)=0.

Since $h_{bb}=-H$, solving this equation gives

\frac{d\hat b}{d\theta} =-h_{bb}^{-1}h_{b\theta} =H^{-1}h_{b\theta}.

The sign in the last expression follows from the definition $H=-h_{bb}$. For example, in one dimension, if $h_{bb}=-4$ and $h_{b\theta}=2$, then $d\hat b/d\theta=2/4=0.5$. This sensitivity formula is useful when a downstream quantity depends on the mode; it is not needed for the envelope derivative itself.

Worked Example: A Poisson Random-Intercept Cluster

Suppose a hypothetical cluster has one Poisson count $y=4$, with log mean $\theta+b$, and a random intercept $b\sim N(0,1)$. Ignoring terms constant in $b$ and $\theta$, its conditional log integrand is $h(b;\theta)=4(\theta+b)-\exp(\theta+b)-b^2/2$. Find the envelope derivative at $\theta=0$, and check that the mode is a nondegenerate maximum.

The stationarity equation is $h_b=4-\exp(\theta+b)-b=0$. At $\theta=0$, solve $4-\exp(\hat b)-\hat b=0$, giving $\hat b\approx1.073729$ and $\exp(\hat b)\approx2.926271$. Substitution gives $4-2.926271-1.073729\approx0$.

The second derivative is $h_{bb}=-\exp(\theta+b)-1$. Thus $H=\exp(\hat b)+1\approx3.926271>0$, confirming a nondegenerate local maximum. The partial derivative is $h_\theta=4-\exp(\theta+b)$; evaluated at the mode, it is $4-2.926271=1.073729$. Hence $d h(\hat b(\theta);\theta)/d\theta\approx1.073729$ at $\theta=0$.

For comparison, differentiating stationarity gives $h_{b\theta}=-\exp(\theta+b)$, so $d\hat b/d\theta=H^{-1}h_{b\theta}\approx-2.926271/3.926271\approx-0.7453$. The mode moves as $\theta$ changes, but that movement contributes zero to the derivative of the log integrand's value at the mode. Here $\theta$ is a log-mean parameter and is dimensionless; the derivative is in log-integrand units per unit change in $\theta$.

Vector Example and a Direct Check

Worked Example: Two Random Effects and a Quadratic Log Integrand

Consider a hypothetical two-dimensional log integrand $h(b;\theta)=c(\theta)+a(\theta)^Tb-b^TQb/2$, where $Q=\operatorname{diag}(2,1)$, $a(\theta)=(1+2\theta,\,2-\theta)^T$, and $c(\theta)=-\theta^2/2$. Evaluate the derivative of the maximized height at $\theta=0.5$.

The gradient in $b$ is $a-Qb$, so the mode is $Q^{-1}a$. At $\theta=0.5$, $a=(2,1.5)^T$ and $\hat b=(1,1.5)^T$. The Hessian is $-Q$, which is negative definite, so this is the unique global maximum.

Here $c'(\theta)=-\theta$ and $a'(\theta)=(2,-1)^T$. The partial derivative at fixed $b$ is $h_\theta=c'(\theta)+a'(\theta)^Tb$. At the mode it equals $-0.5+2(1)-1.5=0$. By the envelope theorem, the derivative of the maximized height is $0$.

For a direct check, the optimized value is $c(\theta)+a(\theta)^TQ^{-1}a(\theta)/2$. Its derivative is $c'+a'^TQ^{-1}a$, because $Q$ is constant. At $\theta=0.5$, this is $-0.5+(2,-1)(1,1.5)^T=0$, as required. The mode sensitivity is $Q^{-1}a'=(1,-1)^T$, but it is not needed to compute this derivative.

Why the Mode Derivative Still Matters

The envelope cancellation applies specifically to $h(\hat b(\theta);\theta)$. It does not remove the mode's movement from every function evaluated at $\hat b$. In a Laplace approximation, the negative Hessian is itself evaluated at that moving point: $H(\theta)=-h_{bb}(\hat b(\theta);\theta)$. For the derivative formula below and differentiation of the log-determinant term, additionally assume $h$ has continuous third derivatives near the mode. Its total derivative generally contains both direct parameter dependence and mode dependence:

\frac{dH}{d\theta} =-h_{bb\theta}(\hat b;\theta) -h_{bbb}(\hat b;\theta)\left[\frac{d\hat b}{d\theta}\right].

For a scalar random effect, this is an ordinary product of derivatives; for vector random effects, the last term denotes the appropriate contraction of the third-derivative array with $d\hat b/d\theta$. Thus the derivative of the log-determinant term, $-\tfrac12\log|H|$, generally requires the mode sensitivity. The additive $\tfrac q2\log(2\pi)$ has zero derivative when $q$ is fixed. These facts explain why the envelope theorem simplifies one part of the Laplace derivative, not the whole expression.

In the Poisson example, $H=\exp(\theta+\hat b)+1$. Its derivative is $\exp(\theta+\hat b)(1+d\hat b/d\theta)$, which explicitly depends on the mode sensitivity. The distinction is practical: use the envelope result for the optimized height, but account for the moving evaluation point in curvature-dependent terms.

Diagnostics, Alternatives, and Failure Modes

Before using the result, verify that the computed conditional mode actually satisfies $h_b(\hat b;\theta)\approx0$, and inspect the Hessian. A positive definite $H$ supports a strict local maximum; a nearly singular $H$ signals weak curvature and can make both the mode and its sensitivity numerically unstable. In a GLMM calculation, also check that the mode is the relevant dominant mode for the integral. A stationary point that is a minimum or saddle point is not a valid Laplace expansion point.

A useful diagnostic is to compare the analytic derivative $h_\theta(\hat b;\theta)$ with a centered finite difference of the optimized value:

\frac{h(\hat b(\theta+\varepsilon);\theta+\varepsilon) -h(\hat b(\theta-\varepsilon);\theta-\varepsilon)}{2\varepsilon}.

For small, suitably chosen $\varepsilon$, this should be close to the envelope derivative. Each perturbed value must use a re-optimized mode. A discrepancy may indicate an inaccurate mode, an incorrect partial derivative, a parameter-dependent term omitted from $h$, or finite-difference step-size error.

There are two common computational alternatives. One can differentiate the full chain rule and explicitly calculate $d\hat b/d\theta$; this is valid, but the term involving $h_b$ should cancel at a correctly computed mode. Or one can use finite differences of re-optimized heights as a check; these are approximate and can be noisy. The envelope calculation is usually simpler for the maximized-height term, while implicit differentiation is necessary for quantities such as $H(\theta)$.

The result may fail in its ordinary differentiable form if the optimum is constrained to a boundary, if $h$ is not differentiable, or if the Hessian is singular. With multiple competing modes, the selected global maximum can switch as $\theta$ changes; the optimized value may then fail to have a single derivative at the switching point. In these cases, verify the optimization structure rather than applying the interior-mode formula mechanically.

Common Mistakes

Including a mode-motion term in the envelope derivative. The chain rule has such a term, but it is $h_b^T(d\hat b/d\theta)=0$ at an interior mode.
Concluding that the mode derivative is zero. Stationarity sets $h_b$ to zero, not $d\hat b/d\theta$. The mode can move substantially with the model parameter.
Using the envelope formula for the entire Laplace approximation. It simplifies the maximized log integrand, not the curvature term evaluated at the moving mode.
Getting the sensitivity sign wrong. With $H=-h_{bb}$, the correct expression is $d\hat b/d\theta=H^{-1}h_{b\theta}$.
Treating any stationary point as a mode. Check the Hessian and whether the point is the relevant mode for the integral.

Key takeaway: At a differentiable interior conditional mode, the derivative of the maximized log integrand is its partial derivative with respect to the model parameter, evaluated at that mode. The mode's derivative is generally nonzero and remains relevant whenever another term, such as the Laplace curvature, is evaluated at the moving mode.

Check Your Understanding

Answer each question, then compare with the solutions below.

If $h_b(\hat b;\theta)=0$, what happens to the mode-motion term in the derivative of $h(\hat b(\theta);\theta)$?
In one dimension, suppose $h_{bb}=-5$ and $h_{b\theta}=-1.5$ at the mode. Find $d\hat b/d\theta$.
Why does differentiating $-\tfrac12\log|H(\theta)|$ generally require more than the envelope derivative?
Name two conditions that can prevent direct use of the ordinary interior-mode formula.

Solutions. (1) The term is $h_b(d\hat b/d\theta)=0$, so the derivative is $h_\theta$ evaluated at the mode. (2) Since $H=-h_{bb}=5$, the sensitivity is $H^{-1}h_{b\theta}=(-1.5)/5=-0.3000$. (3) $H$ is evaluated at $\hat b(\theta)$, so its total derivative generally includes the mode sensitivity as well as direct parameter dependence. (4) Examples include a boundary optimum, a singular Hessian, non-differentiability, or switching between competing modes.