Information From Heteroscedastic Observations

Why Variance Changes Carry Information

A measurement can respond to a parameter in two ways: its expected value can change, and its variability can change. If a parameter affects only the spread of observations, a calculation based only on derivatives of the mean can report zero information even when the distribution changes substantially with that parameter. In model-informed design, distinguishing these contributions matters whenever the observation variance depends on the model parameters.

We consider observations \(Y_1,\ldots,Y_n\) at fixed design settings \(x_1,\ldots,x_n\). Conditional on those settings, assume the observations are independent and \(Y_i\sim N(\mu_i(\theta),v_i(\theta))\), where \(\theta=(\theta_1,\ldots,\theta_p)^T\) is the parameter vector and \(v_i\) is a variance, not a standard deviation. The functions \(\mu_i\) and \(v_i\) are differentiable in the parameters, and \(v_i(\theta)>0\) in the parameter region considered. The design settings are treated as fixed, and the Gaussian support does not depend on \(\theta\).

All functions and derivatives in this expression are evaluated at the parameter value \(\theta\). For a parameter component that does not affect a particular observation's mean or variance, the corresponding derivative is zero. If a variance is constant with respect to every parameter, that observation contributes no variance-sensitivity term, though it can still contribute information through its mean.

Deriving the Two Contributions

For one observation, write \(y_i\) for its realized value and \(r_i=y_i-\mu_i(\theta)\) for its residual. Ignoring no parameter-dependent terms, its log likelihood is \(\ell_i=-\frac{1}{2}\log(2\pi v_i)-\frac{r_i^2}{2v_i}\). The score for component \(\theta_a\) is the derivative of this log likelihood. Applying the product and chain rules gives

Here \(\mu_{i,a}\) and \(v_{i,a}\) abbreviate derivatives with respect to \(\theta_a\). The first score term reflects a shifted mean; the second reflects a changed variance. Under the assumed model, \(Z_i=r_i/\sqrt{v_i}\sim N(0,1)\). Thus the score has mean zero because \(E(Z_i)=0\) and \(E(Z_i^2-1)=0\).

The expected Fisher information contribution for components \(a,b\) is the expected product of their scores. The standard normal moments \(E(Z_i^2)=1\), \(E(Z_i^3)=0\), and \(E(Z_i^4)=3\) show why the two contributions separate. The product of a mean term and a variance term has expectation proportional to \(E[Z_i(Z_i^2-1)]=0\). For the variance terms, \(E[(Z_i^2-1)^2]=E(Z_i^4)-2E(Z_i^2)+1=3-2+1=2\). Consequently, multiplying the two scores and taking expectations yields the stated expression for one observation. Independence makes the joint log likelihood a sum of individual log likelihoods, so their expected information contributions add.

This also justifies the formula directly from the score definition of expected Fisher information. The expected negative Hessian gives the same result under the stated regularity conditions. Observed information, in contrast, depends on the realized data; the formula here is expected information and does not depend on the observed residual values.

Worked Calculations

Assumptions, Diagnostics, and Alternatives

The formula is useful only to the extent that its model assumptions are defensible. Independence is needed for information contributions to add without covariance terms. Gaussianity supplies the particular second-moment factors in the variance term; a different error distribution can yield a different information expression. Variances must be positive, and the mean and variance functions must have the required derivatives. The regularity conditions must permit differentiation of the likelihood and expectation to be interchanged when using the expected-negative-Hessian definition.

In applications, inspect residual patterns across design settings and parameter-relevant covariates. Systematic changes in residual spread can indicate that a constant variance is inadequate; residual dependence can signal that the independence assumption is inadequate. Check that the specified variance remains positive throughout the parameter region of interest and that its derivatives are calculated for the variance itself. If the model instead specifies a standard deviation \(s_i(\theta)\), then \(v_i=s_i^2\), so \(\partial v_i/\partial\theta_a=2s_i\,\partial s_i/\partial\theta_a\). Substituting a standard-deviation derivative directly for a variance derivative gives the wrong result.

A common alternative is to assume constant variance and use only mean sensitivity. That is appropriate when variance is truly parameter-independent under the model, as in the second example. It is not generally appropriate when a parameter changes the variance: the first example shows that omitting the second term can materially understate expected information. Conversely, introducing a variance model without evidence or a defensible scientific basis can add poorly estimated parameters and make conclusions sensitive to misspecification. Modeling variance is not automatically better; the model should reflect the measurement process and be checked against the data.

A further limitation is that the formula describes local expected information at a specified parameter value. It does not guarantee that the parameter can be estimated well from a particular finite data set, nor does it diagnose misspecification by itself. If observations are correlated, their joint Gaussian covariance structure must be handled rather than applying this independent-observation sum. If the variance approaches zero, the displayed expression may become very large or undefined; such behavior calls for scrutiny of the model and parameter region, not automatic interpretation as practically unlimited information.