Common Misreadings of Fisher Information

What Fisher Information Does—and Does Not—Measure

Fisher information is often summarized as “more information means more precision.” That is useful only with qualifications. Information describes how sharply the likelihood changes with a parameter near a specified value, and its numerical value depends on how that parameter is expressed. It does not, on its own, establish that uncertainty is small over the full parameter space, that the likelihood has only one plausible region, or that two parameters with different units can be ranked by their raw information values.

These qualifications matter in QSP calibration. A local calculation can be informative when the likelihood is smooth and well behaved around an interior estimate. But weakly identified combinations, parameter bounds, strong curvature changes, or distant likelihood regions can make a local summary incomplete. The focus here is how to interpret Fisher information without giving it more reach than it has.

The equality between the two expected-information expressions requires regularity conditions, including a differentiable likelihood, an appropriate parameter-independent support, and permission to interchange differentiation and expectation. The score must have finite second moments. In applications, these conditions and the assumed observation model matter: a curvature calculation from a misspecified likelihood is not automatically a trustworthy summary of uncertainty under the real data-generating process.

For a regular model with an interior maximum-likelihood estimate \(\hat{\theta}\), a common large-sample approximation to the covariance matrix is \(I(\hat{\theta})^{-1}\), or \(J(\hat{\theta})^{-1}\) using observed information. This approximation depends on local likelihood shape and regularity; it is not a general identity for the exact sampling covariance. The inverse must also exist. A singular or nearly singular information matrix indicates that some parameter direction is not locally distinguished, or is estimated very imprecisely.

Why the Approximation Is Local

The local covariance approximation follows from expanding the log likelihood around an interior maximum. The score is zero there, and the second derivative describes the leading change for small parameter displacements. In one dimension, writing \(\delta=\theta-\hat{\theta}\), the Taylor expansion is

If \(J(\hat{\theta})>0\), this quadratic has the shape of a normal log density with variance \(1/J(\hat{\theta})\). In several dimensions, the quadratic term is \(-\frac{1}{2}\delta^\mathsf{T}J(\hat{\theta})\delta\); its inverse-curvature matrix is \(J(\hat{\theta})^{-1}\). This reasoning explains why local standard errors are commonly obtained from the square roots of the inverse-information diagonal. It also shows the limitation: the approximation replaces the actual likelihood by a quadratic near its maximum. It says nothing by itself about whether that quadratic remains accurate far away.

A Wald interval built from this approximation can therefore differ from an interval based on the likelihood over a wider range. Such differences can be substantial with asymmetry, changing curvature, multiple peaks, or a boundary near the estimate. The local calculation is a diagnostic and approximation, not a guarantee of global uncertainty coverage.

Information Changes When Parameter Units Change

Information is parameterization-dependent. If \(\phi\) is a smooth re-expression of a scalar parameter \(\theta\), then the score transforms by the chain rule, and the expected information transforms as

The corresponding local variance transforms in the opposite direction: \(1/I_{\phi}=(d\phi/d\theta)^2/I_{\theta}\). The uncertainty in physical terms is consistent after converting units, but the numerical information is not invariant. In a vector model, if \(\theta=g(\phi)\) and \(G=\partial\theta/\partial\phi\), then \(I_{\phi}=G^\mathsf{T}I_{\theta}G\). In particular, information entries have units reciprocal to the product of the corresponding parameter units; a diagonal entry has units reciprocal to the square of that parameter’s units.

Diagonal Entries Do Not Tell the Whole Joint Story

Another common mistake is to interpret \(1/\sqrt{I_{jj}}\) as the standard error for parameter \(j\) even when other parameters are estimated jointly. In a multivariate model, the local covariance approximation is the inverse of the full information matrix, not the reciprocal of each diagonal entry separately. The off-diagonal terms describe coupled directions of curvature.

Checks, Alternatives, and Limitations

Before interpreting an information-based standard error, check what quantity was calculated and whether the assumptions that support its use are plausible. Expected information is a model-based average; observed information is data-specific. Neither should be confused with a global description of the likelihood. A useful interpretation states the parameterization, units, and whether the result comes from the full matrix or an isolated element.

• Check local adequacy. A likelihood with sharp asymmetry, a nearby boundary, changing curvature, or multiple peaks is poorly summarized by one quadratic approximation. A finite inverse matrix does not rule out these features.
• Check joint directions. Inspect the eigenvalues and eigenvectors, or otherwise assess whether parameter combinations are weakly determined. A small eigenvalue can signal near-singularity even when the diagonal entries appear large.
• Check scaling. Convert parameters to interpretable scales before comparing numerical entries. Rescaling can change their magnitudes without changing the fitted model or adding data.
• Check model assumptions. The calculation reflects the assumed likelihood and error model. Incorrect observation variances, dependence, or a misspecified mean structure can make the resulting information misleading.

Local Fisher diagnostics and likelihood-based approaches answer related but distinct questions. Fisher information summarizes local curvature economically and is useful for approximate covariance calculations and design comparisons under a stated parameterization. A likelihood profile examines changes in fit across a wider range while allowing other parameters to adjust; it can reveal asymmetry or a distant weakly constrained region that a local matrix misses. Neither calculation repairs model misspecification, and the local matrix remains useful as a diagnostic rather than a substitute for examining the likelihood when local assumptions are doubtful.

A practical pitfall is to say, “Parameter A has information 100 and parameter B has information 4, so A is 25 times better estimated.” If the parameters have different units, the comparison is not meaningful; even with common units, correlations and nonlinear likelihood shape can alter marginal uncertainty. The correction is to compare uncertainty for clearly defined, suitably scaled quantities using the full joint calculation, and to verify whether the local approximation describes the relevant likelihood region.