Propagating Parameter Uncertainty Through the Fisher Matrix

Why Propagate Uncertainty Through the Information Matrix?

A Fisher information matrix is local: it describes the information a design is expected to provide at a specified parameter value. When plausible parameter values differ, the same sampling schedule can have different sensitivities and therefore different information matrices. A single matrix evaluated at a nominal value may not represent that variation.

Propagating parameter uncertainty means evaluating the conditional Fisher information across a distribution or set of plausible parameter values, then summarizing those matrices in a way that matches the question. In particular, averaging information matrices is not the same operation as applying a nonlinear criterion to each matrix and then averaging the results. The order of operations matters.

Definition: For a design \(d\), parameter vector \(\theta\), and data \(Y\), the conditional Fisher information is \(I(d;\theta)=E_{Y\mid\theta,d}[s(Y;\theta,d)s(Y;\theta,d)^\mathsf{T}]\), where \(s=\partial\log p(Y\mid\theta,d)/\partial\theta\) is the score. If parameter uncertainty is described by a distribution \(\pi(\theta)\), the expected Fisher information is \(\bar I(d)=E_{\theta\sim\pi}[I(d;\theta)]=\int I(d;\theta)\pi(\theta)\,d\theta\), when the expectation exists.

The matrix \(I(d;\theta)\) is conditional on a fixed parameter value; the expectation defining it is over the data that could arise at that value. The additional expectation in \(\bar I(d)\) is over uncertainty in the parameter value. These are distinct sources of averaging. The distribution \(\pi(\theta)\) may represent a prior distribution or a specified uncertainty distribution for design evaluation. In a finite scenario analysis, the integral is replaced by a weighted sum.

All matrices being averaged must use the same parameter vector, ordering, units, and parameterization. For a regular model, the information matrix is symmetric and positive semidefinite. It is positive definite when the data and design locally identify all components of \(\theta\). In examples involving inverses, assume positive definiteness; otherwise an inverse-based variance bound may not be defined.

Calculating and Interpreting an Average Information Matrix

Because expectation is linear, each entry of the expected matrix is the corresponding expected entry of the conditional matrices. For scenarios \(\theta_1,\ldots,\theta_K\), with probabilities \(w_k\geq 0\) and \(\sum_{k=1}^K w_k=1\), the calculation is

\bar I(d)=\sum_{k=1}^{K}w_k I(d;\theta_k),\qquad \bar I_{ij}(d)=\sum_{k=1}^{K}w_k I_{ij}(d;\theta_k).

The diagonal entries describe expected conditional information for individual parameter directions, but they do not generally give the precision available for each parameter when other parameters must also be estimated. Under the usual regularity conditions, the covariance matrix of an unbiased estimator is bounded below by \(I(d;\theta)^{-1}\) at a fixed \(\theta\). Correlations and nuisance parameters are therefore reflected in the full matrix inverse, not just in its diagonal.

Averaging matrices is also different from evaluating the information at the mean parameter value. In general, \(\bar I(d)\ne I(d;E[\theta])\), because the model sensitivities that determine information may be nonlinear in \(\theta\). The average matrix summarizes conditional information across the uncertainty distribution; it is not itself the information matrix at a particular parameter value.

Worked Example: Averaging Scalar Information Across Two Parameter Values

Suppose one concentration observation is taken at \(t=1\) h under the hypothetical model \(Y=Ae^{-kt}+\varepsilon\), where \(A=10\) mg/L is known, \(k\) is the unknown elimination rate in \(\text{h}^{-1}\), and \(\varepsilon\sim N(0,\sigma^2)\) with known \(\sigma=1\) mg/L. Assume the normal model and fixed variance are correct. The parameter uncertainty distribution assigns probability \(0.5\) to each of \(k=0.2\) and \(k=0.8\ \text{h}^{-1}\).

For a normal observation with known variance, the scalar Fisher information for \(k\) is the squared derivative of its mean divided by \(\sigma^2\). Here \(\partial E[Y]/\partial k=-At e^{-kt}\), so

I_k(t;k)=\frac{(At e^{-kt})^2}{\sigma^2}.

At \(k=0.2\ \text{h}^{-1}\), the sensitivity magnitude is \(10(1)e^{-0.2}=8.187\) mg·h/L, giving \(I_k=8.187^2/1^2=67.03\ \text{h}^2\). At \(k=0.8\ \text{h}^{-1}\), it is \(10(1)e^{-0.8}=4.493\) mg·h/L, giving \(I_k=4.493^2/1^2=20.19\ \text{h}^2\). Therefore,

\bar I_k=0.5(67.03)+0.5(20.19)=43.61\ \text{h}^2.

The observation is less informative about \(k\) at the larger rate in this example, because the predicted concentration and its sensitivity to \(k\) are smaller at the sampling time. The average is the expected information under the stated two-point uncertainty distribution—not the information at the average rate \(0.5\ \text{h}^{-1}\).

Nonlinear Summaries: The Order of Averaging

A design may be summarized by a nonlinear function \(g(I)\), such as an inverse-based variance bound or a matrix criterion. There are then two distinct summaries: \(g(E[I])\), the function evaluated at the average matrix, and \(E[g(I)]\), the average of the function evaluated at each matrix. There is no general identity equating them. Which one is relevant depends on the intended design objective; neither should be substituted for the other without justification.

A short Taylor expansion explains the issue. Let \(Z\) denote a scalar information quantity with mean \(\mu_Z\), and suppose \(g\) is twice differentiable near that mean. Expanding \(g(Z)\) around \(\mu_Z\) gives

g(Z)\approx g(\mu_Z)+g'(\mu_Z)(Z-\mu_Z) +\frac{1}{2}g''(\mu_Z)(Z-\mu_Z)^2.

Taking expectations eliminates the first-order term because \(E[Z-\mu_Z]=0\). Thus, when the approximation is adequate, \(E[g(Z)]\) differs from \(g(E[Z])\) by approximately \(\tfrac{1}{2}g''(\mu_Z)\operatorname{Var}(Z)\). Unless the curvature or the variation is negligible, the difference need not be small. The same principle applies to nonlinear functions of matrices, although the relevant curvature is then multidimensional.

Worked Example: Average Information Versus an Inverse-Information Summary

Use the two information values from the preceding hypothetical example. At each fixed \(k\), the scalar Cramér–Rao lower bound for the variance of an unbiased estimator of \(k\) is \(1/I_k\), under the regularity conditions for that bound. Compare the reciprocal of expected information with the average of the two conditional bounds.

The reciprocal of expected information is

\frac{1}{\bar I_k}=\frac{1}{43.61}=0.02293\ (\text{h}^{-1})^2.

At the two parameter values, the conditional bounds are \(1/67.03=0.01492\ (\text{h}^{-1})^2\) and \(1/20.19=0.04953\ (\text{h}^{-1})^2\). Their equally weighted average is

0.5(0.01492)+0.5(0.04953)=0.03223\ (\text{h}^{-1})^2.

These are different calculations: \(1/E[I_k]\) is not \(E[1/I_k]\). The second number averages the conditional lower bounds across the two scenarios; the first inverts the averaged information. Neither calculation, by itself, is a complete description of estimator performance across uncertainty. In particular, an average of conditional bounds is not automatically a bound on an estimator's unconditional mean squared error.

Worked Example: A Nonlinear Matrix Summary Can Change After Averaging

Consider a hypothetical two-parameter design with two equally likely parameter scenarios. Suppose the conditional information matrices are \(I_1=\begin{pmatrix}9&0\\0&1\end{pmatrix}\) and \(I_2=\begin{pmatrix}1&0\\0&9\end{pmatrix}\). Each is positive definite. For illustration, use the determinant as a nonlinear scalar summary; this example compares two calculations rather than choosing a design criterion.

The average matrix is

\bar I=0.5I_1+0.5I_2 =\begin{pmatrix}5&0\\0&5\end{pmatrix}, \qquad \det(\bar I)=5(5)=25.

The determinant in each scenario is \(9(1)=9\), so the average of the scenario-specific determinants is

0.5\det(I_1)+0.5\det(I_2)=0.5(9)+0.5(9)=9.

Thus \(\det(E[I])=25\), whereas \(E[\det(I)]=9\). In each scenario, one parameter direction is relatively informative and the other is less informative; the average matrix combines those directions. The calculation demonstrates why applying a nonlinear summary after matrix averaging can give a different result from averaging that summary across scenarios. It does not establish a universal ordering for other matrices or other nonlinear functions.

Matrix Entries, Nuisance Parameters, and Diagnostics

The off-diagonal entries must be propagated along with the diagonal entries. They represent information coupling between parameter directions and affect matrix-based uncertainty calculations. For example, even if every conditional matrix is well behaved, the uncertainty in a parameter-specific variance bound cannot usually be assessed by averaging just the corresponding diagonal element.

For a finite set of scenarios, check that the weights are nonnegative and sum to one, that every matrix uses the same parameter ordering and units, and that each matrix is symmetric to numerical tolerance. The weighted average of positive semidefinite matrices is positive semidefinite. If the average is unexpectedly indefinite, investigate implementation errors, inconsistent coordinates, or numerical instability. If it is singular, the design may not identify all parameter directions in the average; averaging does not guarantee identifiability.

For a continuous distribution, numerical integration or Monte Carlo sampling may approximate the expectation. A practical check is to increase quadrature resolution or the number of draws and confirm that important entries and the final summary are stable. Also check that the sampled parameter values lie in the model's valid domain and that the information is finite over the uncertainty distribution. A few extreme values can dominate an average, while a narrow or poorly chosen uncertainty distribution can conceal important low-information regions.

Worked Example: Averaging Off-Diagonal Information and Inverting

Suppose two equally likely hypothetical scenarios give \(I_1=\begin{pmatrix}4&2\\2&4\end{pmatrix}\) and \(I_2=\begin{pmatrix}1&0\\0&9\end{pmatrix}\). Both are positive definite: their determinants are \(12\) and \(9\), respectively. Average every entry, including the off-diagonal term:

\bar I=\frac{1}{2} \begin{pmatrix}4&2\\2&4\end{pmatrix} +\frac{1}{2} \begin{pmatrix}1&0\\0&9\end{pmatrix} = \begin{pmatrix}2.5&1\\1&6.5\end{pmatrix}.

Its determinant is \(2.5(6.5)-1^2=15.25\). Therefore,

\bar I^{-1}=\frac{1}{15.25} \begin{pmatrix}6.5&-1\\-1&2.5\end{pmatrix},

whose diagonal entries are approximately \(0.4262\) and \(0.1639\). By contrast, the average of the two conditional inverse matrices has diagonal entries \(\tfrac12(4/12+1)=0.6667\) and \(\tfrac12(4/12+1/9)=0.2222\). This confirms that inverting the average matrix and averaging the scenario-specific inverses are different operations. These matrices illustrate calculations of local inverse-information bounds; they do not imply that either average is automatically the appropriate unconditional estimator-variance summary.

Choosing the Summary and Avoiding Common Errors

The appropriate summary follows from the question being asked. If the goal is to report expected information in each parameter direction, average the conditional matrices entry by entry. If the goal is to assess a nonlinear utility or loss at each uncertain parameter value, evaluate that quantity at each value and average those results. These summaries answer different questions. Scenario-based analysis can retain the separate results when a single average would hide meaningful variation; a nominal calculation instead describes performance at one chosen parameter value.

Plugging in the mean parameter. The incorrect reasoning is that the average parameter value represents all plausible values. The correction is to calculate conditional information across the uncertainty distribution; nonlinear model sensitivities generally make \(I(d;E[\theta])\) differ from \(E[I(d;\theta)]\).
Applying a nonlinear criterion in the wrong order. The incorrect reasoning is that averaging information and then applying a criterion must match averaging criterion values. The correction is to state which quantity is intended and calculate that order explicitly.
Keeping only diagonal entries. The incorrect reasoning is that each parameter's information is just its diagonal entry. The correction is to retain the full matrix when parameters are jointly estimated, since off-diagonal entries affect the inverse and other matrix summaries.
Interpreting expected information as observed information. The incorrect reasoning is that \(\bar I\) is the information realized in a study. The correction is that it is an expectation over the specified parameter uncertainty and the conditional data model, not the information calculated from actual observations.

This calculation is a building block for robust design, not a design objective by itself. Its limitations include dependence on the chosen uncertainty distribution, local information approximations, and the adequacy of the likelihood and design assumptions. Reporting scenario-specific matrices alongside a summary helps reveal whether the average obscures weak-information regions.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions.

Two equally likely scenarios have scalar information values 4 and 16. Calculate expected information and the average reciprocal information.
Why can the information at the mean parameter value differ from expected information?
Two equally likely matrices are \(I_1=\begin{pmatrix}4&0\\0&1\end{pmatrix}\) and \(I_2=\begin{pmatrix}1&0\\0&4\end{pmatrix}\). Calculate the determinant of their average and the average of their determinants.
What checks should be made before averaging scenario-specific information matrices?

Solutions. 1. Expected information is \(0.5(4)+0.5(16)=10\). Average reciprocal information is \(0.5(1/4)+0.5(1/16)=0.15625\), whereas the reciprocal of expected information is \(1/10=0.1\); the operations differ. 2. The information depends on model sensitivities, which can be nonlinear in the parameter, so evaluating at \(E[\theta]\) generally does not equal integrating over \(\theta\). 3. The average matrix is \(\begin{pmatrix}2.5&0\\0&2.5\end{pmatrix}\), with determinant \(6.25\). Each scenario determinant is \(4\), so their average is \(4\). 4. Confirm that weights are nonnegative and sum to one, matrices share parameter ordering and units, matrices are symmetric and positive semidefinite to numerical tolerance, and the resulting calculation is stable and finite over the specified uncertainty distribution.