Common Errors In Fisher Information Calculations

Why Small Calculation Errors Matter

Fisher information calculations connect a model and a design to the precision that can be expected when estimating parameters. A missing variance weight, an incorrect derivative, or a matrix copied in the wrong parameter order can change estimated standard errors and even reverse a design comparison. These are not cosmetic errors: the final matrix is used to assess uncertainty and to evaluate design criteria.

This tutorial focuses on ways to detect and correct such errors. The calculations are local when the model is nonlinear: sensitivities are evaluated at a specified parameter value. The information matrix is not itself a guarantee that an estimator will attain the uncertainty it suggests; the usual interpretation relies on an appropriate model, an identifiable parameterization, and large-sample regularity conditions.

Definition: For a parameter vector \(\theta\) and design \(\xi\), the Fisher information matrix is the expected outer product of the score vector, \(I(\theta;\xi)=E\{U(\theta)U(\theta)^T\}\), where \(U(\theta)\) is the derivative of the log-likelihood with respect to \(\theta\). Under regularity conditions it also equals the negative expected Hessian of the log-likelihood. It is symmetric and positive semidefinite; when it is nonsingular, its inverse is the usual large-sample covariance approximation for an efficient estimator.

The Variance Weight Is Part of the Information

For independent observations \(Y_i\sim N(f_i(\theta),\sigma_i^2)\), assume the variances \(\sigma_i^2\) are known and do not depend on \(\theta\), and each mean function \(f_i\) is differentiable. Write \(g_i(\theta)=\partial f_i(\theta)/\partial\theta\) as a column vector. The information is

I(\theta;\xi)=\sum_i \frac{1}{\sigma_i^2}g_i(\theta)g_i(\theta)^T.

Thus each sensitivity outer product is weighted by the inverse observation variance. A precise observation contributes more information than an otherwise identical noisy observation. Omitting the weights implicitly treats all observations as having equal variance. If variances depend on parameters, this formula is incomplete: the likelihood's variance terms also contribute to the score and information.

A Short Derivation and a Variance-Weight Counterexample

For the stated normal model, the log-likelihood, apart from constants not depending on \(\theta\), is

\ell(\theta)=-\frac{1}{2}\sum_i\left\{\log(\sigma_i^2)+\frac{[Y_i-f_i(\theta)]^2}{\sigma_i^2}\right\}.

Differentiating with respect to \(\theta\) gives \(U(\theta)=\sum_i [Y_i-f_i(\theta)]g_i(\theta)/\sigma_i^2\). Independence and the zero mean of each residual make cross-observation terms vanish in \(E\{UU^T\}\). Since \(E\{[Y_i-f_i(\theta)]^2\}=\sigma_i^2\), the contribution from observation \(i\) is \(g_i g_i^T/\sigma_i^2\), yielding the formula above. This derivation also shows why a variance weight cannot be dropped merely because the sensitivity was computed correctly.

Worked Example: Restoring a Missing Variance Weight

Suppose two hypothetical independent observations have mean functions \(f_1(\theta)=\theta_1+\theta_2\) and \(f_2(\theta)=\theta_1-\theta_2\), with variances \(1\) and \(4\), respectively. Calculate the information and show what goes wrong if the second observation is treated as equally precise.

The gradients are \(g_1=(1,1)^T\) and \(g_2=(1,-1)^T\). Applying the inverse-variance weights,

I=g_1g_1^T+\frac{1}{4}g_2g_2^T =\begin{pmatrix}1&1\\1&1\end{pmatrix} +\frac14\begin{pmatrix}1&-1\\-1&1\end{pmatrix} =\begin{pmatrix}1.25&0.75\\0.75&1.25\end{pmatrix}.

The determinant is \(1.25(1.25)-0.75^2=1.5625-0.5625=1\), so

I^{-1}=\begin{pmatrix}1.25&-0.75\\-0.75&1.25\end{pmatrix}.

The approximate marginal standard errors are \(\sqrt{1.25}=1.118\) for each parameter, in the parameter units. If the variance weight is omitted, the erroneous calculation gives \(g_1g_1^T+g_2g_2^T=\begin{pmatrix}2&0\\0&2\end{pmatrix}\), with inverse \(\begin{pmatrix}0.5&0\\0&0.5\end{pmatrix}\) and standard errors \(0.7071\). That calculation falsely treats the noisy second observation as equally informative and substantially understates uncertainty in this example.

Check Derivatives Against the Model

A derivative error often looks plausible in the final matrix: the result may still be symmetric and positive semidefinite. Differentiate the stated mean function directly, keeping track of signs, factors such as time, and the parameter with respect to which each derivative is taken. For nonlinear models, also state the parameter value at which the sensitivities are evaluated.

Worked Example: A Missing Time Factor in an Exponential Model

Consider the hypothetical mean function \(f(t;A,k)=A\exp(-kt)\), with parameters \(A\) and \(k\), and two independent observations at \(t=1\) and \(t=2\). Let \(A=10\), \(k=0.5\), and let both observation variances equal \(1\). Compute the local information, then compare it with the result from omitting the factor \(t\) in the derivative with respect to \(k\).

The derivatives are \(\partial f/\partial A=\exp(-kt)\) and \(\partial f/\partial k=-At\exp(-kt)\). At the specified parameter values, the sensitivity vectors, in the order \((A,k)\), are approximately \(g_1=(0.606531,-6.06531)^T\) and \(g_2=(0.367879,-7.35759)^T\). With unit variances, the information is their sum of outer products:

I\approx \begin{pmatrix} 0.606531^2+0.367879^2 & 0.606531(-6.06531)+0.367879(-7.35759)\\ 0.606531(-6.06531)+0.367879(-7.35759) & (-6.06531)^2+(-7.35759)^2 \end{pmatrix} = \begin{pmatrix} 0.503215&-6.38550\\ -6.38550&90.9221 \end{pmatrix}.

If the factor \(t\) is omitted, the incorrect derivative becomes \(-A\exp(-kt)\). The resulting \(k\)-sensitivities are \(-6.06531\) and \(-3.67879\), instead of \(-6.06531\) and \(-7.35759\). The erroneous information is approximately \(\begin{pmatrix}0.503215&-5.03215\\-5.03215&50.3215\end{pmatrix}\). Both matrices are symmetric and have nonnegative diagonal entries, so those checks alone do not catch the mistake. Comparing the derivative with the original function does: changing \(k\) affects the exponent by \(-t\), so the time factor must appear.

Keep Parameter Order Consistent

An information matrix is tied to a particular ordering of the parameter vector. If the order changes, both rows and columns must be permuted. The covariance approximation \(I^{-1}\) uses that same order, so its diagonal entries and off-diagonal covariances must be interpreted against the corresponding parameter labels. A matrix with correct numerical entries but incorrect labels gives incorrect parameter-specific conclusions.

Worked Example: Reordering Parameters and Checking the Inverse

Suppose the information matrix for parameters ordered as \((\alpha,\beta)\) is \(I_{\alpha,\beta}=\begin{pmatrix}2&1\\1&1\end{pmatrix}\). Find its inverse and then reorder the parameters as \((\beta,\alpha)\).

For a two-by-two matrix, the determinant is the product of the diagonal entries minus the product of the off-diagonal entries. Here, \(\det(I_{\alpha,\beta})=2(1)-1(1)=1\), so

I_{\alpha,\beta}^{-1}= \frac{1}{1}\begin{pmatrix}1&-1\\-1&2\end{pmatrix} =\begin{pmatrix}1&-1\\-1&2\end{pmatrix}.

Swapping the parameter order swaps both rows and columns, giving \(I_{\beta,\alpha}=\begin{pmatrix}1&1\\1&2\end{pmatrix}\). Its determinant is \(1(2)-1(1)=1\), and its inverse is

I_{\beta,\alpha}^{-1}=\frac{1}{1}\begin{pmatrix}2&-1\\-1&1\end{pmatrix} =\begin{pmatrix}2&-1\\-1&1\end{pmatrix}.

The new inverse is the original inverse with rows and columns swapped, as required. In particular, the approximate variance for \(\beta\) is now the first diagonal entry, \(2\), not the second. A useful arithmetic check is to multiply each matrix by its claimed inverse; the product must be the identity matrix.

Interpret the Criterion You Actually Calculated

An optimality criterion is a defined summary of information or uncertainty, not a general statement that one design is “better in every way.” For example, D-optimality compares determinants of information matrices, equivalently generalized variance under the usual covariance approximation. It does not require every individual parameter variance to be smaller. A-optimality instead compares the trace of the inverse information matrix, the sum of approximate marginal variances in the chosen parameterization and scale.

Worked Example: A D-Optimal Choice Can Sacrifice One Parameter

Compare hypothetical designs C and D with \(I_C=\begin{pmatrix}25&0\\0&0.8\end{pmatrix}\) and \(I_D=\begin{pmatrix}4&0\\0&4\end{pmatrix}\), for the same ordered parameter vector. Which design is D-optimal, and what does that conclusion not imply?

Their determinants are \(\det(I_C)=25(0.8)=20\) and \(\det(I_D)=4(4)=16\), so D-optimality selects C because it has the larger information determinant. The corresponding covariance approximations are \(I_C^{-1}=\begin{pmatrix}0.04&0\\0&1.25\end{pmatrix}\) and \(I_D^{-1}=\begin{pmatrix}0.25&0\\0&0.25\end{pmatrix}\). Design C has a lower approximate variance for the first parameter but a higher one for the second. Its determinant of covariance is \(0.04(1.25)=0.05\), compared with \(0.25(0.25)=0.0625\) for D. The D-optimal conclusion concerns the joint uncertainty summary; it does not mean C has lower uncertainty for each parameter. If the scientific goal prioritizes the second parameter, a targeted criterion may be more relevant.

Diagnostics and Common Pitfalls

Before interpreting a calculated matrix, use checks that can expose errors without assuming that a plausible-looking result is correct. For the normal model above, verify that each observation contributes its inverse-variance-weighted sensitivity outer product. Check that derivatives match the written model, and that the parameter order is recorded and used consistently in both \(I\) and \(I^{-1}\). Recompute determinants and inverses independently or verify \(II^{-1}\) is the identity to numerical tolerance.

The information should be symmetric. For any vector \(v\), \(v^TIv\) should be nonnegative; a materially negative value indicates an error or numerical problem. A singular matrix means at least one parameter direction is not informed by the design, so the ordinary inverse and its standard-error interpretation are unavailable. Near-singularity can make inverse-based summaries unstable. These checks diagnose problems; they do not establish that the assumed model, variance structure, or local parameter values are scientifically adequate.

Wrong reasoning: “The model has two observations and two parameters, so I can add two unweighted outer products.” Correction: each observation is weighted by its inverse variance under the stated model; the number of observations does not remove that factor.
Wrong reasoning: “The matrix is symmetric, so my derivatives must be right.” Correction: outer products produce symmetric matrices even when the sensitivity vectors are wrong. Differentiate the model and check each component.
Wrong reasoning: “The larger determinant guarantees smaller uncertainty for every parameter.” Correction: the determinant summarizes joint uncertainty. Inspect the relevant diagonal or contrast variance when an individual parameter or combination is the target.

The main alternative to hand calculation is numerical differentiation or software-generated information from a likelihood. These can reduce arithmetic burden, but they do not remove the need to check parameter order, variance assumptions, and derivative behavior. Numerical derivatives can be sensitive to step size; compare them with analytic derivatives where possible, or test several step sizes. For complicated likelihoods, also check that the software's reported quantity is expected Fisher information, observed information, or another matrix, since these are not interchangeable in every setting.

Key takeaway: Trace every information contribution from the likelihood: identify its sensitivity, variance weight, and parameter order; verify the inverse; then interpret the criterion according to its definition and the scientific target.

Check Your Understanding

Answer each question, then compare with the solutions below.

For one normal observation with mean \(f(\theta)\), gradient \(g\), and known variance \(9\), what is its information contribution?
For \(f(t;A,k)=A\exp(-kt)\), state the derivative with respect to \(k\) and identify the factor commonly missed.
For \(M=\begin{pmatrix}3&1\\1&1\end{pmatrix}\), calculate \(\det(M)\) and \(M^{-1}\).
In one sentence, explain why selecting the larger-determinant information matrix does not guarantee a smaller variance for every parameter.

Solutions. (1) The contribution is \(gg^T/9\), not \(gg^T\). (2) \(\partial f/\partial k=-At\exp(-kt)\); the factor \(t\) comes from differentiating the exponent \(-kt\). (3) \(\det(M)=3(1)-1(1)=2\), so \(M^{-1}=\frac12\begin{pmatrix}1&-1\\-1&3\end{pmatrix}=\begin{pmatrix}0.5&-0.5\\-0.5&1.5\end{pmatrix}\); multiplication by \(M\) returns the identity. (4) The determinant summarizes joint uncertainty, while the variance of an individual parameter is a particular diagonal entry of the inverse, which can be larger even when the determinant is more favorable.