Observed Information From the Likelihood Hessian

Why the Likelihood Hessian Matters

A fitted parameter value tells us where a likelihood is maximized, but not how sharply it is maximized. If the log-likelihood changes steeply near its maximum, nearby parameter values have substantially lower likelihood; if it is relatively flat, a wider range of values may fit almost as well. The observed information measures this local curvature using the data actually observed.

This tutorial focuses on calculating that curvature from a likelihood Hessian and using it to describe local uncertainty in a fitted parameter vector. The calculations are aligned with the SOA Exam ASTAM (Advanced Short-Term Actuarial Mathematics) syllabus. The inverse-information interpretation is a local approximation, not a claim that every finite-sample estimator has an exact normal distribution.

Definition: Let $\ell(\boldsymbol{\theta})$ be the log-likelihood for parameter vector $\boldsymbol{\theta}=(\theta_1,\ldots,\theta_p)^\mathsf{T}$. Its Hessian is the $p\times p$ matrix of second partial derivatives, $$ H(\boldsymbol{\theta})= \left[ \frac{\partial^2\ell(\boldsymbol{\theta})} {\partial\theta_j\,\partial\theta_k} \right]_{j,k=1}^{p}. $$ The observed information matrix at estimate $\hat{\boldsymbol{\theta}}$ is $$ J(\hat{\boldsymbol{\theta}})=-H(\hat{\boldsymbol{\theta}}). $$ For a scalar parameter, this reduces to $J(\hat{\theta})=-\ell''(\hat{\theta})$.

The evaluation point matters: first take the second derivatives of the log-likelihood as functions of the parameters, then substitute the fitted values. The negative sign also matters. At a regular interior maximum, the Hessian is usually negative definite, so the observed information is positive definite. If the Hessian is evaluated away from a maximum, or the likelihood has a boundary maximum or a flat direction, that positivity need not hold.

From Curvature to Local Uncertainty

The reason the inverse observed information is used as a local measure of uncertainty follows from a second-order Taylor expansion. Suppose $\hat{\boldsymbol{\theta}}$ is an interior maximum, so the score (the vector of first derivatives) is zero there. For a small displacement $\boldsymbol{h}$, the log-likelihood is approximately

\ell(\hat{\boldsymbol{\theta}}+\boldsymbol{h}) \approx \ell(\hat{\boldsymbol{\theta}}) +\frac{1}{2}\boldsymbol{h}^{\mathsf T} H(\hat{\boldsymbol{\theta}})\boldsymbol{h} = \ell(\hat{\boldsymbol{\theta}}) -\frac{1}{2}\boldsymbol{h}^{\mathsf T} J(\hat{\boldsymbol{\theta}})\boldsymbol{h}.

The linear term is absent because the score is zero at the fitted maximum. A large curvature in a direction makes a displacement in that direction reduce the log-likelihood quickly. In a regular model, matching this local quadratic likelihood shape to a multivariate normal shape gives the approximation

\operatorname{Cov}(\hat{\boldsymbol{\theta}}) \ \approx\ J(\hat{\boldsymbol{\theta}})^{-1}.

This approximation requires an invertible observed information matrix and a well-behaved interior fit. Its diagonal entries approximate the variances of the parameter estimates, and its off-diagonal entries approximate their covariances. It is not an exact finite-sample identity. If the matrix is singular, nearly singular, or not positive definite, its inverse does not provide a reliable local uncertainty summary.

Calculation Strategy

For a vector parameter, every entry matters: differentiate twice with respect to each parameter pair, preserve the mixed-derivative terms, negate the complete Hessian, and only then invert the observed information if an uncertainty approximation is required.

Write the log-likelihood.
State the model and parameterization, and remove only terms that are constant in the parameters.

Differentiate before substituting estimates.
Calculate the full Hessian, including cross-partial derivatives for a vector parameter.

Evaluate and negate.
Substitute $\hat{\boldsymbol{\theta}}$ into the Hessian to obtain $H(\hat{\boldsymbol{\theta}})$, then set $J(\hat{\boldsymbol{\theta}})=-H(\hat{\boldsymbol{\theta}})$.

Check curvature and interpret.
Check that the information is invertible and appropriate for a local maximum. If justified, use its inverse as the approximate covariance matrix.

Worked Applications

Worked Example: Observed Information for an Exponential Rate

A hypothetical portfolio has four independent, identically distributed loss amounts $X_1,\ldots,X_4$, each exponential with rate parameter $\lambda>0$. The observed losses, in thousands of dollars, are $1,2,3,4$. Calculate the maximum likelihood estimate, the observed information at that estimate, and the local inverse-information variance approximation. State the method and check the sign.

Method. For an exponential distribution with rate $\lambda$, the density is $f_X(x)=\lambda e^{-\lambda x}$ for $x\geq 0$. Independence gives

\ell(\lambda)=4\log\lambda-\lambda(1+2+3+4) =4\log\lambda-10\lambda.

Differentiate to find the interior maximum:

\ell'(\lambda)=\frac{4}{\lambda}-10, \qquad \frac{4}{\hat{\lambda}}-10=0, \qquad \hat{\lambda}=\frac{4}{10}=0.4.

The second derivative, evaluated at the estimate, is

\ell''(\hat{\lambda}) =-\frac{4}{(0.4)^2} =-\frac{4}{0.16} =-25.

Therefore the observed information is the negative of this Hessian value:

J(\hat{\lambda})=-\ell''(\hat{\lambda})=25, \qquad J(\hat{\lambda})^{-1}=\frac{1}{25}=0.04.

Thus the local inverse-information approximation to the variance of $\hat{\lambda}$ is $0.04$, in squared rate units. Check: the log-likelihood is concave because $\ell''(\lambda)=-4/\lambda^2<0$ for $\lambda>0$; at the fitted value its Hessian is $-25$, and its observed information is $+25$. A common mistake is to report the negative Hessian itself as the information. The definition requires negating the Hessian.

Worked Example: A Two-Parameter Normal Likelihood

Suppose three independent losses are normally distributed as $X_i\sim N(\mu,\tau)$, where the second parameter $\tau>0$ is the variance. The observed losses are $1,2,3$. Calculate the fitted vector, observed information matrix, and inverse-information covariance approximation. Name the parameterization because using standard deviation in place of variance would change the derivatives.

Method. The log-likelihood, up to its full parameter-dependent form, is

\ell(\mu,\tau) =-\frac{3}{2}\log(2\pi\tau) -\frac{(1-\mu)^2+(2-\mu)^2+(3-\mu)^2}{2\tau}.

The maximum likelihood estimate of $\mu$ is the sample mean: $\hat{\mu}=(1+2+3)/3=2$. The sum of squared deviations from 2 is $Q=(1-2)^2+(2-2)^2+(3-2)^2=1+0+1=2$, so the maximum likelihood estimate of the variance is $\hat{\tau}=Q/3=2/3$.

For the Hessian, write $Q(\mu)=\sum_{i=1}^{3}(x_i-\mu)^2$. The second derivatives are

\ell_{\mu\mu}=-\frac{3}{\tau}, \qquad \ell_{\mu\tau}=\ell_{\tau\mu} =-\frac{\sum_{i=1}^3(x_i-\mu)}{\tau^2}, \qquad \ell_{\tau\tau} =\frac{3}{2\tau^2}-\frac{Q(\mu)}{\tau^3}.

At $\hat{\mu}=2$, the residuals sum to zero, so the mixed derivatives are zero. Substituting $\hat{\tau}=2/3$ gives

H(\hat{\mu},\hat{\tau}) = \begin{pmatrix} -3/(2/3) & 0\\ 0 & 3/[2(2/3)^2]-2/(2/3)^3 \end{pmatrix} = \begin{pmatrix} -4.5 & 0\\ 0 & -3.375 \end{pmatrix}.

Negating the Hessian gives the observed information, and inversion gives the local covariance approximation:

J(\hat{\mu},\hat{\tau})= \begin{pmatrix} 4.5 & 0\\ 0 & 3.375 \end{pmatrix}, \qquad J(\hat{\mu},\hat{\tau})^{-1} = \begin{pmatrix} 2/9 & 0\\ 0 & 8/27 \end{pmatrix}.

The approximate variances are $2/9$ for the fitted mean and $8/27$ for the fitted variance; the estimated covariance between them is zero for this observed information calculation. Check: both Hessian diagonal entries are negative at the fitted maximum, while both observed information diagonal entries are positive. In particular, the $(\tau,\tau)$ entry of the Hessian is $-3.375$, so the observed information entry, after negation, is $+3.375$.

Worked Example: Observed Information in a Poisson Regression Fit

A hypothetical claims study records independent Poisson counts at two exposure levels. At predictor values $z=0$, the counts are $1$ and $3$; at $z=1$, they are $2$ and $6$. Model the count $Y_i$ as Poisson with mean $\mu_i=\exp(\alpha+\beta z_i)$. Find the maximum likelihood estimates, the full observed information matrix, and its inverse. The cross-partial is important because the two fitted parameters need not be independent.

Method. Ignoring the terms $-\log(Y_i!)$, which do not depend on $\alpha,\beta$, the log-likelihood is

\ell(\alpha,\beta) =\sum_i \left[Y_i(\alpha+\beta z_i) -\exp(\alpha+\beta z_i)\right]+\text{constant}.

Within the $z=0$ group, the fitted mean is the group average $(1+3)/2=2$, so $\hat{\alpha}=\log 2$. Within the $z=1$ group, the fitted mean is $(2+6)/2=4$, giving $\hat{\alpha}+\hat{\beta}=\log 4$. Therefore $\hat{\beta}=\log 4-\log 2=\log 2$.

For each observation, the Hessian contribution is $-\mu_i\begin{pmatrix}1&z_i\\z_i&z_i^2\end{pmatrix}$. At the estimates, the two observations at $z=0$ each have fitted mean 2 and contribute a total of $-4$ to the $(\alpha,\alpha)$ entry. The two at $z=1$ each have fitted mean 4 and contribute a total of $-8$ to each of the four entries. Thus

H(\hat{\alpha},\hat{\beta}) = -\begin{pmatrix} 12 & 8\\ 8 & 8 \end{pmatrix}, \qquad J(\hat{\alpha},\hat{\beta}) = \begin{pmatrix} 12 & 8\\ 8 & 8 \end{pmatrix}.

The determinant of the observed information is $12(8)-8(8)=96-64=32$, so it is invertible. Its inverse is

J(\hat{\alpha},\hat{\beta})^{-1} = \frac{1}{32} \begin{pmatrix} 8 & -8\\ -8 & 12 \end{pmatrix} = \begin{pmatrix} 0.25 & -0.25\\ -0.25 & 0.375 \end{pmatrix}.

The diagonal entries are the local approximate variances of $\hat{\alpha}$ and $\hat{\beta}$; the negative off-diagonal entry records their local approximate covariance. Check: multiplying the information matrix by the displayed inverse gives the identity matrix. For example, the upper-left product entry is $12(0.25)+8(-0.25)=1$, and the upper-right is $12(-0.25)+8(0.375)=0$. Omitting the mixed derivatives would incorrectly discard the dependence between the parameter estimates.

Common Errors and Interpretation

Observed information summarizes curvature at the fitted parameter vector for the realized data. It is distinct from the Hessian: $H$ contains the second derivatives of the log-likelihood, while $J=-H$ is the observed information. Keeping these matrices distinct prevents a frequent sign error.

Using the wrong sign. A negative diagonal Hessian entry at a maximum becomes a positive observed information entry after negation. Do not call the Hessian itself the information matrix.
Substituting too early. Differentiate the log-likelihood first, then evaluate the derivatives at the estimates. Premature substitution can erase parameter dependence needed for second derivatives.
Dropping cross-partials. For multiple parameters, include every mixed second derivative. A diagonal-only calculation is not generally the Hessian and can miss fitted-parameter dependence.
Confusing curvature with exact variance. The inverse observed information is a local approximation under regularity conditions, not an exact finite-sample variance formula.
Inverting without checking. A singular or poorly behaved information matrix cannot support a stable inverse-based uncertainty summary.

The key exam habit is to label the Hessian and observed information separately at every stage. Display the Hessian at the estimate, apply the negative sign explicitly, and then show the matrix inversion. This makes the sign, evaluation point, and uncertainty interpretation auditable.

Check Your Understanding

For each problem, calculate the fitted information at the estimate; use the inverse-information approximation only when the matrix is invertible.

Five independent exponential losses have rate $\lambda$, and their sum is 20. Find $\hat{\lambda}$, the observed information, and its inverse. (Answer: $\hat{\lambda}=0.25$, $J=80$, $J^{-1}=0.0125$.)
Four independent observations from $N(\mu,\tau)$, with $\tau$ the variance, are $1,1,3,3$. Find the fitted vector and the inverse observed information matrix. (Answer: $(\hat{\mu},\hat{\tau})=(2,1)$, $J^{-1}=\begin{pmatrix}0.25&0\\0&0.5\end{pmatrix}$.)
In a Poisson regression with mean $\exp(\alpha+\beta z)$, three observations at $z=0$ have counts $1,2,3$, and two observations at $z=1$ have counts $3,5$. Find the fitted parameters and observed information matrix. (Answer: $\hat{\alpha}=\log 2$, $\hat{\beta}=\log 2$, $J=\begin{pmatrix}14&8\\8&8\end{pmatrix}$.)
An exponential-rate log-likelihood is $\ell(\lambda)=6\log\lambda-9\lambda$, for $\lambda>0$. Find the MLE, the Hessian at the MLE, and the observed information there. (Answer: $\hat{\lambda}=2/3$, $\ell''(\hat{\lambda})=-13.5$, $J(\hat{\lambda})=13.5$.)
For five independent normal observations with unknown mean $\mu$ and variance $\tau$, suppose the fitted mean is $\hat{\mu}=4$, the fitted variance is $\hat{\tau}=2$, and the residual sum of squares is 10. Calculate the observed information and inverse-information covariance approximation in the parameter order $(\mu,\tau)$. (Answer: $J=\begin{pmatrix}2.5&0\\0&0.625\end{pmatrix}$, $J^{-1}=\begin{pmatrix}0.4&0\\0&1.6\end{pmatrix}$.)

What You'll Learn