Fisher Information With Heteroscedastic Observation Error

Why unequal observation precision matters

A QSP model may predict several observables or measurements at different times, while the corresponding assays have different error magnitudes. A precise measurement can constrain parameters more strongly than a noisy one, even when both measurements have the same model sensitivity. Fisher information makes this distinction explicit by weighting each observation's sensitivity according to its error variance.

The focus here is the local information supplied by independent Gaussian observations whose variances are known and unequal. The result is useful for understanding parameter uncertainty near a specified parameter value or fitted estimate. It does not, by itself, establish global identifiability or guarantee that a nonlinear model's likelihood is well behaved.

Definition: Let $y_i$ be observation $i$, with model-predicted mean $\mu_i(\theta)$, where $\theta$ is a vector of $p$ parameters. Assume $y_i$ are independent and $y_i\sim N(\mu_i(\theta),\sigma_i^2)$, with known, positive variances $\sigma_i^2$ that do not depend on $\theta$. The weighted Fisher information matrix is $$ \mathcal I(\theta)=\sum_{i=1}^{n}\frac{1}{\sigma_i^2} \left(\frac{\partial \mu_i(\theta)}{\partial\theta}\right) \left(\frac{\partial \mu_i(\theta)}{\partial\theta}\right)^{T}. $$

The derivative in each summand is a $p$-component column vector. It describes how the predicted observation changes locally as each parameter changes. The factor $1/\sigma_i^2$ is the precision weight: a smaller variance gives more weight. If $J$ is the $n\times p$ sensitivity matrix, whose $i$th row is $(\partial\mu_i/\partial\theta)^T$, and $W$ is diagonal with entries $1/\sigma_i^2$, the same expression is $\mathcal I(\theta)=J^T WJ$.

Derivation from the Gaussian likelihood

For the stated assumptions, the log likelihood, omitting a constant that does not depend on $\theta$, is

\ell(\theta)=-\frac{1}{2}\sum_{i=1}^{n}\frac{[y_i-\mu_i(\theta)]^2}{\sigma_i^2}.

Write $g_i=\partial\mu_i(\theta)/\partial\theta$. Differentiating the log likelihood gives the score vector

U(\theta)=\frac{\partial\ell}{\partial\theta} =\sum_{i=1}^{n}\frac{y_i-\mu_i(\theta)}{\sigma_i^2}g_i.

The Fisher information is the expected outer product of the score, or equivalently the negative expected Hessian under regularity conditions. Independence makes cross-observation score products have expectation zero. For each observation, $E[y_i-\mu_i(\theta)]=0$ and $E[(y_i-\mu_i(\theta))^2]=\sigma_i^2$. Therefore its contribution to $E[UU^T]$ is $g_i g_i^T/\sigma_i^2$, and summing gives the definition above.

The same result follows from the Hessian: terms involving second derivatives of $\mu_i(\theta)$ are multiplied by $y_i-\mu_i(\theta)$, whose expectation is zero. Thus the expected information depends on the first derivatives of the predicted means and the known variances, not on the realized residuals.

Interpretation: A sensitivity contribution is large when a predicted observation changes substantially in a parameter direction and its measurement variance is small. The matrix combines these contributions across observations; its eigenstructure reflects which local parameter combinations are informed well or poorly.

When $\mathcal I(\theta)$ is nonsingular and the model is locally regular, the inverse information gives the usual local, large-sample approximation to the covariance matrix of an efficient estimator:

\operatorname{Cov}(\hat{\theta})\approx \mathcal I(\theta)^{-1}.

In practice, this matrix is often evaluated at a fitted estimate, producing a local approximation rather than an exact finite-sample uncertainty statement. A diagonal entry estimates a parameter's variance on the chosen parameter scale; off-diagonal entries describe local covariance. If the information is singular, there is at least one parameter direction with no local information in the sensitivity matrix, and the ordinary inverse does not exist.

Worked Example: A Scalar Parameter With Three Unequal Variances

Suppose a hypothetical QSP output depends on one dimensionless parameter $\theta$. At a specified nominal value, the predicted means have sensitivities $2$, $1$, and $3$ output units per unit of $\theta$ at three observation occasions. Assume independent Gaussian errors with known variances $1$, $4$, and $9$ output-units squared, respectively. Calculate the information and the approximate standard error for $\theta$.

For one parameter, each sensitivity is a scalar, so the information is the sum of squared sensitivities divided by their observation-specific variances:

\mathcal I(\theta)=\frac{2^2}{1}+\frac{1^2}{4}+\frac{3^2}{9} =4+0.25+1=5.25.

The local variance approximation is $1/5.25=0.1905$ squared parameter units. The corresponding approximate standard error is

SE(\hat{\theta})\approx\sqrt{\frac{1}{5.25}}=0.4364

parameter units. The first observation contributes $4$ information units, the second contributes $0.25$, and the third contributes $1$. Although the third sensitivity is larger than the first, its larger variance reduces its information contribution. These are local uncertainty calculations, not a claim that the exact sampling distribution of the fitted parameter is Gaussian.

Multivariate information and parameter combinations

For multiple parameters, information is a matrix, not a collection of independent scalar contributions. One observation can inform several parameters at once, and its contribution may link them through off-diagonal entries. The matrix must be assembled using the full sensitivity vector for each observation before its inverse is interpreted.

Worked Example: Two Parameters and Three Observation Types

Consider two dimensionless parameters $\theta_1$ and $\theta_2$ and three hypothetical independent Gaussian observations. Suppose the local sensitivity vectors, in output units per parameter unit, are $g_1=(1,0)^T$, $g_2=(0,2)^T$, and $g_3=(1,1)^T$. Their known variances are $1$, $4$, and $1$ output-units squared. Calculate the information matrix and the local covariance approximation.

The first observation contributes $g_1g_1^T/1$, the second contributes $g_2g_2^T/4$, and the third contributes $g_3g_3^T/1$. Thus

\mathcal I = \begin{pmatrix}1&0\\0&0\end{pmatrix} + \begin{pmatrix}0&0\\0&1\end{pmatrix} + \begin{pmatrix}1&1\\1&1\end{pmatrix} = \begin{pmatrix}2&1\\1&2\end{pmatrix}.

The determinant is $2(2)-1(1)=3$, so this matrix is invertible. Its inverse is

\mathcal I^{-1} =\frac{1}{3}\begin{pmatrix}2&-1\\-1&2\end{pmatrix}.

The approximate standard errors are $\sqrt{2/3}=0.8165$ parameter units for each parameter. The approximate covariance is $-1/3$ squared parameter units, and the implied local correlation is $(-1/3)/\sqrt{(2/3)(2/3)}=-0.5$. This negative correlation indicates that, locally, changes in one parameter can partly be offset by changes in the other while producing similar predicted observations. The calculation does not imply that the parameters are exactly correlated in the full nonlinear likelihood.

How measurement precision changes uncertainty

Because the information contribution is inversely proportional to variance, doubling an observation's variance halves that observation's contribution, provided the mean sensitivity is unchanged. The effect on a parameter's standard error is not generally a simple reciprocal rescaling: information from all observations combines, and the inverse of a multivariate matrix also reflects overlaps among sensitivity directions.

Worked Example: The Cost of Ignoring Unequal Variances

Suppose three independent measurements inform a single dimensionless parameter with sensitivity $1$ output unit per parameter unit at each measurement. Their known variances are $1$, $4$, and $16$ output-units squared. Compare the correct weighted information with the result from incorrectly treating all observations as equally precise.

The correct information is

\mathcal I_{\mathrm{weighted}}=\frac{1}{1}+\frac{1}{4}+\frac{1}{16} =1+0.25+0.0625=1.3125.

The approximate standard error is $\sqrt{1/1.3125}=\sqrt{16/21}=0.8729$ parameter units. If the variances were incorrectly ignored and each observation assigned weight one, the computed information would be $1+1+1=3$, giving a standard error of $1/\sqrt{3}=0.5774$ parameter units. That calculation overstates the information because the noisiest measurement is treated as if it were as precise as the first.

For comparison, if the third measurement's variance were $4$ rather than $16$, with the sensitivity unchanged, its contribution would rise from $1/16$ to $1/4$. Total information would become $1+1/4+1/4=1.5$, and the approximate standard error would be $\sqrt{1/1.5}=0.8165$ parameter units. Greater precision in that measurement improves local precision, but the size of the improvement depends on its contribution relative to the information already present.

Assumptions, diagnostics, and alternatives

Before using this expression for a QSP observation model, check that its assumptions match the measurement process and the likelihood actually fitted:

Independent errors: after conditioning on the model, observation errors have no cross-observation covariance. If errors are correlated, a diagonal weight matrix is not appropriate.
Gaussian errors with known variances: the variance for each observation is specified and positive. The formula above does not account for uncertainty from estimating an error model.
Parameter-independent variances: $\sigma_i^2$ is fixed as $\theta$ changes. If variance depends on parameters, the likelihood contains additional information terms and this mean-sensitivity-only expression is incomplete.
Differentiable mean predictions: the sensitivities exist at the parameter value where information is evaluated. For an ODE observation model, these are local derivatives of the predicted measurements with respect to parameters.
Local regularity and adequate rank: interpreting the inverse as a covariance approximation requires a locally informative, nonsingular matrix and conditions under which the estimator has an approximately normal sampling distribution.

Useful checks include verifying that each variance uses the square of the observation's units, that weights are $1/\sigma_i^2$ rather than $1/\sigma_i$, and that the computed matrix is symmetric and positive semidefinite. Inspect its eigenvalues or condition: a zero or very small eigenvalue signals an uninformative or weakly informed local parameter combination. Also check the sensitivity and weight for each observation; an implausibly large contribution may arise from a unit conversion error or an unrealistically small variance. These diagnostics assess the calculation and local geometry; they do not replace assessment of the fitted likelihood's broader shape.

The weighted expression is the appropriate alternative to ordinary unweighted least-squares information when Gaussian observations have unequal known variances. With equal variances, the matrix reduces to a common scalar multiple of $J^T J$. With correlated Gaussian errors having a known covariance matrix $\Sigma$, the corresponding mean information is $J^T\Sigma^{-1}J$, not $J^TWJ$ for diagonal $W$. If the observation distribution or its variance model differs, information should be derived from that likelihood rather than borrowed unchanged from the independent, fixed-variance Gaussian case.

Common pitfalls and limitations

Using equal weights by habit. Equal weighting is not justified merely because measurements share a named assay or output. Use the variance model that matches the assumed observation likelihood.
Confusing variance and standard deviation. The denominator is $\sigma_i^2$. If a standard deviation is supplied, square it before calculating the weight.
Reading diagonal entries of information as variances. The local covariance approximation is $\mathcal I^{-1}$, not $\mathcal I$. For multiple parameters, off-diagonal terms affect the inverse and the resulting standard errors.
Treating local information as a global guarantee. A well-conditioned matrix at one parameter point does not prove global identifiability, exclude distant likelihood modes, or ensure reliable uncertainty near parameter bounds.

Weighted Fisher information is therefore a principled local summary under a specific likelihood, not a substitute for examining the model and data assumptions. The earlier material on practical identifiability and sensitivity matrices gives the broader context: small information in a direction means weak local constraint, while the weighting here specifies how observation precision shapes that constraint.

Check Your Understanding

Answer each question using the independent Gaussian, known-variance assumptions unless stated otherwise. Short solutions follow.

A scalar parameter has sensitivities $1$ and $2$, and the corresponding known variances are $1$ and $4$. Calculate weighted information and the approximate standard error.
If one observation's variance doubles while its sensitivity is unchanged, what happens to that observation's information contribution?
Two parameters have sensitivity vectors $g_1=(1,0)^T$ and $g_2=(1,1)^T$, with variances $1$ and $4$. Calculate the information matrix.
Why does an observation with zero sensitivity to every parameter contribute no mean information, regardless of its variance?

Solutions. (1) $\mathcal I=1^2/1+2^2/4=2$, so $SE\approx\sqrt{1/2}=0.7071$ parameter units. (2) Its contribution is halved because it is proportional to $1/\sigma_i^2$. (3) $\mathcal I=g_1g_1^T+g_2g_2^T/4=\begin{pmatrix}1.25&0.25\\0.25&0.25\end{pmatrix}$. (4) If $g_i=0$, then $g_ig_i^T/\sigma_i^2=0$ for every positive variance, so changing precision cannot create information about parameters that do not affect the predicted mean.