Bayesian Information For Nonlinear Models

Why Add Prior Information?

In a nonlinear model, the information supplied by an observation depends on the parameter values at which the model is evaluated. A prior distribution can contribute information about those parameters as well. Combining the two gives a local approximation to posterior uncertainty: useful for understanding how data and prior knowledge jointly constrain parameters, and for comparing the uncertainty implied by different data scenarios.

The word local matters. The combined matrix is based on curvature near a specified parameter value, usually the posterior mode. It approximates the posterior distribution by a multivariate normal distribution in that neighbourhood; it does not make a nonlinear posterior exactly normal.

Definition: Let $\theta$ be a vector of model parameters, let $I(\theta)$ denote the likelihood Fisher information matrix, and suppose the prior is Gaussian, $\theta\sim N(\mu_0,\Sigma_0)$, with positive-definite covariance matrix $\Sigma_0$. The prior precision matrix is $\Lambda_0=\Sigma_0^{-1}$. The local Bayesian information matrix at $\theta^\ast$ is $$ I_B(\theta^\ast)=I(\theta^\ast)+\Lambda_0. $$ When this matrix is positive definite, its inverse, $I_B(\theta^\ast)^{-1}$, is a local Gaussian approximation to posterior covariance if the likelihood is represented by its Fisher information at $\theta^\ast$.

For a Laplace approximation, the corresponding curvature is the negative Hessian of the log posterior evaluated at its mode $\hat{\theta}_{MAP}$. Its inverse is the Laplace posterior covariance approximation. Replacing the likelihood's observed curvature by Fisher information gives the local Bayesian information approximation above. These coincide when the observed likelihood curvature equals the Fisher information at the mode, but need not coincide in a particular data set.

Model, Assumptions, and Curvature

For independent observations $Y_i$ with model means $f_i(\theta)$ and known variances $\sigma_i^2$, suppose $Y_i\mid\theta\sim N(f_i(\theta),\sigma_i^2)$. Write $J(\theta)$ for the matrix whose $i$th row is the gradient of $f_i(\theta)$, and $W=\operatorname{diag}(1/\sigma_i^2)$. The expected Fisher information is

I(\theta)=J(\theta)^\mathsf{T}WJ(\theta).

This expression assumes independent Gaussian measurement errors with fixed variances and differentiable model means. For other likelihoods, use their appropriate expected Fisher information instead; the sensitivity formula is not universal.

If $r_i=y_i-f_i(\theta)$, the observed negative Hessian of the Gaussian log likelihood is

J(\theta)^\mathsf{T}WJ(\theta) -\sum_i \frac{r_i}{\sigma_i^2}\nabla^2 f_i(\theta).

The second term records curvature of the nonlinear response and depends on the observed residuals. Its expectation is zero under the stated model, which explains why the expected information is $J^\mathsf{T}WJ$. A Gaussian prior contributes the constant negative-Hessian term $\Lambda_0$. Thus the exact Laplace curvature at the posterior mode is the observed negative likelihood Hessian plus $\Lambda_0$; the Fisher-based approximation substitutes $I(\hat{\theta}_{MAP})+\Lambda_0$.

Why the Covariance Is the Inverse Information

The approximation follows from a second-order expansion. Let $\ell(\theta)$ be the log likelihood and let $\pi(\theta)$ be the prior density. The log posterior, up to an additive constant, is $\ell(\theta)+\log\pi(\theta)$. At an interior posterior mode $\hat{\theta}_{MAP}$, its gradient is zero. Expanding around that mode gives

\ell(\theta)+\log\pi(\theta) \approx \ell(\hat{\theta}_{MAP})+\log\pi(\hat{\theta}_{MAP}) -\frac{1}{2}(\theta-\hat{\theta}_{MAP})^\mathsf{T} H_B(\theta-\hat{\theta}_{MAP}),

where $H_B$ is the negative Hessian of the log posterior at the mode. Exponentiating the quadratic expression gives the kernel of a multivariate normal distribution with covariance $H_B^{-1}$, provided $H_B$ is positive definite. Under the Fisher approximation, $H_B$ is replaced by $I(\hat{\theta}_{MAP})+\Lambda_0$. This derivation explains both the inverse and the local nature of the result.

Worked Example: A Nonlinear Scalar Response

Suppose one hypothetical observation follows $Y\mid\theta\sim N(f(\theta),1)$, where $f(\theta)=\theta^2$. The observed value is $y=4$, and the prior is $\theta\sim N(2,0.25)$. Calculate the local posterior variance approximation.

The prior precision is $\Lambda_0=1/0.25=4$. The likelihood sensitivity is $f'(\theta)=2\theta$, so the Fisher information at $\theta=2$ is

I(2)=\frac{[f'(2)]^2}{1}=(4)^2=16.

At $\theta=2$, the predicted value is $4$, exactly the observation, and the prior mean is also $2$. Both likelihood and prior scores are therefore zero there, so this is a posterior mode. The Fisher-based local posterior precision is $16+4=20$, giving

\operatorname{Var}(\theta\mid y)\approx \frac{1}{20}=0.0500, \qquad \operatorname{SD}(\theta\mid y)\approx\sqrt{0.0500}=0.224.

Here the observed likelihood curvature also equals 16: with residual zero, the residual-curvature term vanishes. Thus the Laplace and Fisher-based local variances agree. This agreement is specific to this evaluation point; nonlinear models need not have matching observed and expected curvature for other data.

Worked Example: Combining a Prior With Nonlinear Model Sensitivities

Consider a hypothetical two-parameter nonlinear model at $\theta^\ast=(1,2)^\mathsf{T}$. Three observations have independent unit-variance errors. Their model sensitivities at $\theta^\ast$ are the rows of

J(\theta^\ast)= \begin{pmatrix} 1&1\\ \sqrt{3}&0\\ 0&1 \end{pmatrix}.

For concreteness, these are the sensitivities at $(1,2)$ of $f_1=\theta_1+\theta_2+0.1(\theta_1-1)^2$, $f_2=\sqrt{3}\theta_1+0.1(\theta_2-2)^2$, and $f_3=\theta_2+0.1(\theta_1-1)^2$. Suppose the hypothetical observations equal the model predictions at $\theta^\ast$, and the prior is $N(\theta^\ast,\Sigma_0)$, with $\Sigma_0=\operatorname{diag}(0.25,1)$. The zero residuals and prior-centred value make $\theta^\ast$ a posterior mode.

Because $W$ is the identity matrix, multiply $J^\mathsf{T}J$ to obtain the likelihood information:

I(\theta^\ast)= \begin{pmatrix} 1&\sqrt{3}&0\\ 1&0&1 \end{pmatrix} \begin{pmatrix} 1&1\\ \sqrt{3}&0\\ 0&1 \end{pmatrix} = \begin{pmatrix} 4&1\\ 1&2 \end{pmatrix}.

The prior precision is $\Lambda_0=\operatorname{diag}(4,1)$. Therefore

I_B(\theta^\ast)= \begin{pmatrix} 8&1\\ 1&3 \end{pmatrix}, \qquad I_B(\theta^\ast)^{-1} =\frac{1}{23} \begin{pmatrix} 3&-1\\ -1&8 \end{pmatrix}.

The approximate marginal posterior variances are $3/23=0.1304$ for $\theta_1$ and $8/23=0.3478$ for $\theta_2$, in squared parameter units. The corresponding standard deviations are about $0.361$ and $0.590$ parameter units. The off-diagonal covariance is $-1/23$, so the local approximation implies a negative association between parameter deviations. These are approximate posterior summaries, not exact moments of the nonlinear posterior.

Worked Example: Prior Precision for One Nonlinear Measurement

A hypothetical measurement has model mean $f(t;\theta)=\theta/(1+t\theta)$, error standard deviation $0.2$, and is taken at $t=1$. Let the prior be $N(1,0.25)$, and suppose the observed value is $0.5=f(1;1)$. Find the local posterior variance approximation at the mode.

The derivative with respect to $\theta$ is $1/(1+t\theta)^2$, which at $t=1,\theta=1$ equals $1/4$. The measurement information is the squared sensitivity divided by the error variance:

I(1)=\frac{(1/4)^2}{0.2^2}=\frac{0.0625}{0.04}=1.5625.

The prior mean equals the parameter value and the residual is zero, so $\theta=1$ is a posterior mode. The prior precision is $4$, making the combined local precision $1.5625+4=5.5625$. Hence

\operatorname{Var}(\theta\mid y)\approx\frac{1}{5.5625}=0.1798.

The prior supplies more local precision than this single measurement. That fact describes the curvature contributions at the mode; it does not by itself establish that the prior is scientifically appropriate.

Assumptions, Diagnostics, and Failure Modes

A local covariance approximation is most defensible when the posterior has a well-defined interior mode, is concentrated in a neighbourhood where a quadratic expansion is adequate, and has positive-definite curvature there. The model must be differentiable at the evaluation point, and the likelihood and prior must be combined on the same parameter scale. A Gaussian prior with positive-definite covariance has a proper, finite precision matrix. A non-Gaussian prior can still be used in a Laplace approximation, but its negative log-density curvature at the mode replaces $\Lambda_0$; it is not generally a constant precision matrix.

Useful checks include verifying that the information matrix is symmetric, that the combined curvature is positive definite, and that sensitivities and variance weights were evaluated at the stated parameter value. Compare likelihood and prior contributions to identify which directions are informed mainly by data and which mainly by the prior. If a direction remains nearly flat, the inverse can be unstable or very large. A positive-definite prior can make the sum invertible, but that does not mean the data themselves identify every parameter direction.

The approximation can fail when the posterior is strongly skewed, has multiple modes, is truncated by parameter bounds, or changes curvature substantially across its high-probability region. A nonlinear model may also have important residual-curvature terms, so Fisher-based and observed-Hessian approximations can differ. In those cases, inspect posterior shape using an appropriate computational or graphical analysis rather than treating one inverse matrix as a complete description.

How to Interpret the Result

The likelihood-only covariance approximation $I(\theta^\ast)^{-1}$ describes local uncertainty from the likelihood when the information matrix is invertible. Adding prior precision generally increases curvature and reduces local variance in the directions the prior constrains. The prior covariance alone describes prior uncertainty, not posterior uncertainty. The combined inverse is a local approximation incorporating both sources, conditional on the model and the chosen prior.

This is distinct from adding an arbitrary numerical stabilizer to a matrix: $\Lambda_0$ has a probability-model interpretation only when it comes from a specified prior (or a stated local prior curvature). It is also distinct from exact Bayesian computation. The approximation summarizes posterior curvature near a mode; it does not account fully for non-Gaussian shape or distant modes.

Common pitfall: “The prior precision is added, so every posterior variance must be smaller and the approximation must be reliable.” The correction is that the variance comparison is direction-dependent and requires valid positive-definite matrices on a consistent parameter scale. Even when the local covariance shrinks, the normal approximation can still be poor if the posterior is skewed, multimodal, boundary-constrained, or strongly nonlinear near its mode.

Check Your Understanding

Answer each question, then compare with the short solutions below.

A scalar likelihood has local Fisher information 6 and a Gaussian prior variance of 0.5. What is the Fisher-based local posterior variance?
Why might the observed-Hessian and Fisher-based covariance approximations differ for a nonlinear Gaussian mean model?
A two-parameter combined information matrix is not positive definite. What does this imply for simply interpreting its inverse as a local covariance?
What does it mean if prior precision is much larger than likelihood information in a particular parameter direction?

Solutions. (1) The prior precision is $1/0.5=2$, so the combined precision is $6+2=8$, and the local variance is $1/8=0.125$ squared parameter units. (2) The observed Hessian includes a residual-weighted second derivative of the nonlinear mean, whereas expected Fisher information does not; these terms need not agree for observed data with nonzero residuals. (3) The quadratic curvature is not positive definite, so it cannot define a proper Gaussian covariance by inversion; investigate the model, mode, and curvature rather than reporting that inverse as a covariance. (4) Local posterior uncertainty in that direction is strongly affected by the prior; it should not be described as being determined primarily by the likelihood.