Likelihood Curvature and Parameter Uncertainty

Why Curvature Measures Uncertainty

An estimate identifies a location in parameter space, but it does not by itself indicate how precisely the data determine that location. The shape of the log likelihood near its maximum supplies that information: a sharp peak means that small changes in the parameter reduce the likelihood quickly, while a broad peak means that a wider range of values remains plausible. This tutorial develops that relationship for a single scalar parameter.

Let \(Y\) denote the observed data, let \(\theta\) be a scalar parameter, and let \(\ell(\theta)\) be the log likelihood. Suppose \(\hat{\theta}\) is an interior maximizer and that \(\ell\) has a continuous second derivative near \(\hat{\theta}\). At an interior maximum, \(\ell'(\hat{\theta})=0\). The second derivative therefore describes the leading change in log likelihood near the estimate.

The distinction is important. Observed information describes the curvature for the data actually collected; expected information averages that curvature over repeated data generated at \(\theta\). The earlier material on Fisher information established the score-variance and expected-information identities under their regularity conditions. Here the focus is how either kind of information translates into an uncertainty scale.

The Local Quadratic Approximation

For a twice continuously differentiable log likelihood, Taylor expansion around the interior maximum gives, for \(\theta\) sufficiently close to \(\hat{\theta}\),

Because the score \(\ell'(\hat{\theta})\) is zero and \(\ell''(\hat{\theta})=-j(\hat{\theta})\), this becomes

Exponentiating the approximation gives a local likelihood proportional to \(\exp\{-j(\hat{\theta})(\theta-\hat{\theta})^2/2\}\). This has the shape of a normal density, centered at \(\hat{\theta}\), with variance \(1/j(\hat{\theta})\). It motivates the curvature-based standard error \(\{j(\hat{\theta})\}^{-1/2}\). This is a local approximation, not an assertion that the likelihood is itself a probability distribution for \(\theta\).

The same calculation can be expressed as a local likelihood-ratio comparison:

Thus, values of \(\theta\) a few curvature-based standard errors from the estimate produce a noticeable decrease in log likelihood. This local picture gives a practical way to interpret the information number: greater information means greater curvature and a smaller uncertainty scale.

From Curvature to Sampling Uncertainty

A quadratic shape alone does not prove that an estimator has a normal sampling distribution. That conclusion needs regularity conditions on the data-generating model and a large-sample argument. The following standard result makes the connection precise in an independent, identically distributed setting.

Proof. On the event that the score root exists, the mean value theorem applied to \(U_n\) gives a value \(\tilde{\theta}\) between \(\hat{\theta}\) and \(\theta_0\) such that

Rearranging and multiplying by \(\sqrt n\),

By assumption, the first factor converges in probability to \(1/I(\theta_0)\), and the second converges in distribution to \(N(0,I(\theta_0))\). Slutsky's theorem gives the stated limit, with variance \(I(\theta_0)^{-2}I(\theta_0)=1/I(\theta_0)\). The curvature condition also applies at the consistent estimate \(\hat{\theta}\), so \(j_n(\hat{\theta})/n\xrightarrow{p}I(\theta_0)\). Taking reciprocals gives \(j_n(\hat{\theta})^{-1}\) asymptotic to \(1/\{nI(\theta_0)\}\), as claimed. \(\square\)

This result explains why the inverse observed information at the estimate is commonly used as an approximate variance estimate. Replacing observed information with expected information gives the related approximation \(1/I_n(\hat{\theta})\), where \(I_n\) is the total expected information for the sample. The two can differ in a particular sample, even when both are useful under regular large-sample conditions.

Using and Interpreting Approximate Standard Errors

When the normal approximation for \(\hat{\theta}\) is adequate, a common approximate confidence interval is

where \(z_{1-\alpha/2}\) is the corresponding standard normal quantile. The formula combines the estimate with its local curvature-based standard error. It does not guarantee good coverage in a small sample, at a boundary, or when the likelihood is strongly asymmetric.

A frequent mistake is to interpret a large value of \(j(\hat{\theta})\) as a large standard error because it looks like a measure of “uncertainty.” The direction is the opposite: standard error is proportional to the inverse square root of information. Another mistake is to treat \(1/j(\hat{\theta})\) as the exact variance of \(\hat{\theta}\). The quadratic approximation concerns the local likelihood shape; the theorem shows a sampling-distribution connection only under its stated large-sample assumptions.

Curvature is less informative when the maximum is not interior, the second derivative is zero or positive, or the likelihood is notably non-quadratic around its maximum. In those cases, the local symmetric normal approximation may be poor. A parameter constraint can also make a symmetric interval extend outside the permissible parameter space. These are reasons to inspect the likelihood shape and the model conditions rather than applying the standard-error formula mechanically.