Deriving the Gaussian Population Fisher Matrix

From a Gaussian Likelihood to Information

In population design, a subject contributes a vector of repeated observations, not just one measurement. Under the Gaussian approximation to the subject-level likelihood, that vector is described by a parameter-dependent mean and covariance. Fisher information measures how much the probability distribution of the vector changes as the parameter changes. Consequently, changes in the mean are not the only source of information: changes in the covariance can also make parameter values distinguishable.

Let \(Y\) be a \(d\)-dimensional observation vector from one subject, and let \(\theta=(\theta_1,\ldots,\theta_p)^T\) be the parameter vector. For a fixed design, write the subject-level model as \(Y\sim N(m(\theta),V(\theta))\), where \(m(\theta)\) is a \(d\)-vector and \(V(\theta)\) is a \(d\times d\) covariance matrix. The design is held fixed when differentiating with respect to \(\theta\).

For the formula derived here, assume that \(V(\theta)\) is symmetric and positive definite in a neighborhood of the parameter value of interest; \(m(\theta)\) and \(V(\theta)\) have continuous first derivatives there; and the normal likelihood is correctly specified for the observation vector. These conditions ensure that the density is well-defined, its derivatives exist, and expectations can be taken through the relevant derivatives. A singular covariance matrix, a parameter on a nonregular boundary, or a discontinuous model requires separate treatment.

The Score and Its Expected Outer Product

Set \(r=Y-m(\theta)\). The log-likelihood, omitting no terms that depend on \(\theta\), is

For parameter component \(\theta_a\), define \(m_a=\partial m/\partial\theta_a\) and \(V_a=\partial V/\partial\theta_a\). Differentiating uses \(\partial\log|V|/\partial\theta_a=\operatorname{tr}(V^{-1}V_a)\), \(\partial V^{-1}/\partial\theta_a=-V^{-1}V_aV^{-1}\), and \(\partial r/\partial\theta_a=-m_a\). The resulting score component is

The score has a linear term in the residual \(r\) and a centered quadratic term. To evaluate its expected outer product, standardize the residual: let \(Z=V^{-1/2}r\), so that \(Z\sim N(0,I_d)\). Define \(A_a=V^{-1/2}V_aV^{-1/2}\) and \(b_a=V^{-1/2}m_a\). Because \(V_a\) is symmetric, \(A_a\) is symmetric. The score can then be written as

The standardized representation makes the expectations transparent. The linear term has mean zero. The quadratic term is centered because \(E[Z^TA_aZ]=\operatorname{tr}(A_a)\), so \(E[U_a]=0\). For the product of two score components, the product of linear terms has expectation \(b_a^Tb_b\). A linear term times a centered quadratic term has expectation zero: its expectation is a sum of third moments of centered normal components, and all such third moments are zero.

For the remaining quadratic terms, the standard normal fourth-moment identity gives

Thus the expected product of the quadratic score terms, each carrying a factor of one half, is \(\frac{1}{2}\operatorname{tr}(A_aA_b)\). Substituting the definitions of \(A_a\) and \(b_a\), and using the cyclic property of the trace, yields the subject-level information formula:

The first term arises from the parameter’s effect on the mean; the second arises from its effect on the covariance. The derivation also justifies their separation: their cross-expectation is zero under the multivariate normal model. This formula concerns expected information, not the information calculated from one particular observed residual vector.

Worked Calculations

Assumptions, Diagnostics, and Alternatives

In an application, \(m\), \(V\), and their derivatives are usually computed from the chosen design and nominal parameter values. Before trusting the matrix, check that \(V\) is symmetric positive definite and that the derivatives match the actual parameterization. A useful numerical check is to compare analytic derivatives with finite-difference derivatives at several small step sizes. Material disagreement can reveal an indexing error, a missing covariance dependence, or an unstable finite-difference step.

The resulting information matrix should be symmetric, up to numerical rounding. Its diagonal entries should be nonnegative, and the whole matrix should be positive semidefinite; a materially negative eigenvalue is a warning to investigate calculation errors or violations of the assumed model. A zero or very small eigenvalue can indicate that some parameter direction has no or little local identification under this design. It is not, by itself, proof of a coding error.

The derivation depends on the Gaussian distribution, not merely on knowing the first two moments. Under a non-Gaussian likelihood, third- and fourth-moment relationships may differ, so the same formula need not give the Fisher information. Likewise, if the Gaussian model is only an approximation to a marginal subject-level likelihood, the result is information under that approximation. It may not capture skewness, multimodality, or other features of the exact likelihood.

Expected Fisher information averages over all possible observations under the model. Observed information instead uses the negative Hessian evaluated at the actual data; it can depend on the realized residuals and need not equal the expected information in a finite sample. Another alternative is to calculate information from the exact marginal likelihood, often by numerical integration or simulation when it is tractable. That may better reflect a non-Gaussian likelihood but can be computationally more demanding. These approaches answer related, but not interchangeable, questions.

Common pitfall: A tempting shortcut is to use only \(m_a^TV^{-1}m_b\), on the grounds that Fisher information is about a change in the expected response. That reasoning omits information carried by changes in variability. In the variance-only example, the shortcut gives zero, although \(I(\theta)=1/(2\theta^2)>0\). The correction is to include the covariance-derivative term whenever \(V\) depends on the parameter. Conversely, if \(V\) is parameter-independent, that term is zero and should not be added artificially.

The formula is local: it is evaluated at a specified parameter value and design. It does not guarantee that an estimator will be unbiased, that the likelihood is well behaved globally, or that the Gaussian approximation is adequate. Near a boundary or a point where the covariance becomes nearly singular, the regularity assumptions and numerical stability deserve particular scrutiny.