Cross-Information Between Population Parameter Groups

What Cross-Information Means

A population Fisher information matrix is often partitioned into blocks for typical values, random-effect variability, and residual-error parameters. Its diagonal blocks describe information within a parameter group; its off-diagonal blocks describe how score contributions from two groups vary together. These cross-blocks matter because a parameter group is not always informed independently of the others. A design may, for example, produce similar changes in the likelihood when a random-effect variance increases or when residual variance increases.

Cross-information is a property of the model, parameterization, nominal parameter values, and design. A nonzero block does not say that one parameter causes another to change, nor is it directly the correlation between their estimates. It indicates local statistical coupling in the score or likelihood curvature.

Definition: Let \(\theta=(\theta_1,\ldots,\theta_p)\) be a population parameter vector and let \(U(\theta)=\partial \ell(\theta)/\partial\theta\) be its score. Under regularity conditions, the Fisher information is \(I(\theta)=E[U(\theta)U(\theta)^T]\), equivalently the expected negative Hessian of the log likelihood. For parameter groups \(a\) and \(b\), the cross-information block is \(I_{ab}=E[U_aU_b^T]\), where \(U_a\) and \(U_b\) are the score subvectors for those groups.

The expectation is taken under the model at the parameter values being evaluated. The usual equivalence to expected negative curvature requires a correctly specified, differentiable likelihood, parameter-independent support, and conditions allowing differentiation under the integral. In design calculations, the information is typically evaluated at nominal parameter values.

Gaussian Cross-Information

For a subject-level Gaussian approximation, write \(Y\sim N(m(\theta),V(\theta))\), where \(m\) is the mean vector and \(V\) is a positive-definite covariance matrix. For a scalar parameter \(\theta_a\), let \(m_a=\partial m/\partial\theta_a\) and \(V_a=\partial V/\partial\theta_a\). The Gaussian Fisher information between two scalar parameters \(\theta_a\) and \(\theta_b\) is

I_{ab}=m_a^T V^{-1}m_b+\frac{1}{2}\operatorname{tr}\!\left(V^{-1}V_aV^{-1}V_b\right).

For groups, apply this expression to every pair of components to form the block. The first term is the contribution from changes in the mean; the second is the contribution from changes in covariance. In a general approximate mixed-effects model, a typical-value parameter can affect both \(m\) and \(V\), while random-effect and residual parameters often affect \(V\). Thus cross-information may come from either term.

The formula follows from the Gaussian score. Set \(z=Y-m\). For scalar \(\theta_a\),

U_a=m_a^TV^{-1}z+\frac{1}{2}\left[z^TV^{-1}V_aV^{-1}z-\operatorname{tr}(V^{-1}V_a)\right].

Because \(E[z]=0\), the product of a linear term in \(z\) and a centered quadratic term in \(z\) has expectation zero under a Gaussian distribution. The linear terms have expected product \(m_a^TV^{-1}VV^{-1}m_b=m_a^TV^{-1}m_b\). For centered Gaussian \(z\), the covariance of quadratic forms \(z^TAz\) and \(z^TBz\) is \(2\operatorname{tr}(AVBV)\). Applying this identity to the two centered quadratic score terms gives \(\frac{1}{2}\operatorname{tr}(V^{-1}V_aV^{-1}V_b)\). Adding the two contributions gives the stated information. This also explains why there is no mean-covariance cross term in the expected Gaussian information.

Worked Example: Cross-Information in a Nonlinear Random-Effect Approximation

Suppose an individual response is generated by \(Y=f(\beta+\eta)+\epsilon\), with \(f(x)=x^2\), \(\eta\sim N(0,\omega)\), and independent \(\epsilon\sim N(0,r)\). Here \(\beta\) is a typical value, \(\omega\) is a random-effect variance, and \(r\) is residual variance. Use a second-order Taylor approximation for the mean and a first-order approximation for the random-effect contribution to variance. The resulting Gaussian approximation is \(Y\sim N(m,v)\), where \(m=\beta^2+\omega\) and \(v=4\beta^2\omega+r\). These are approximate moments, not the exact moments of the nonlinear response.

At the nominal values \(\beta=1\), \(\omega=0.25\), and \(r=1\), the variance is positive and the local Taylor approximation is being used at a nonzero \(\beta\). We have \(m=1^2+0.25=1.25\) and \(v=4(1)^2(0.25)+1=2\). The derivatives are \(m_\beta=2\), \(m_\omega=1\), \(m_r=0\), and \(v_\beta=8\beta\omega=2\), \(v_\omega=4\beta^2=4\), \(v_r=1\).

For example, the cross-information between \(\beta\) and \(\omega\) is

I_{\beta\omega} =\frac{m_\beta m_\omega}{v} +\frac{1}{2}\frac{v_\beta v_\omega}{v^2} =\frac{(2)(1)}{2}+\frac{1}{2}\frac{(2)(4)}{2^2} =1+1=2.

The other entries follow from the same formula. For instance, the typical-value/residual cross-block is \(I_{\beta r}=0+\frac{1}{2}(2)(1)/2^2=0.25\), while the variability/residual cross-information is \(I_{\omega r}=0+\frac{1}{2}(4)(1)/2^2=0.5\). The full scalar-parameter information matrix, in the order \((\beta,\omega,r)\), is

I= \begin{pmatrix} 2.5&2&0.25\\ 2&2.5&0.5\\ 0.25&0.5&0.125 \end{pmatrix}.

For example, \(I_{\omega\omega}=1^2/2+\frac{1}{2}(4^2)/2^2=0.5+2=2.5\), and \(I_{rr}=0+\frac{1}{2}(1^2)/2^2=0.125\). All three parameter groups have cross-information in this approximation. The mean depends on \(\beta\) and \(\omega\), and the variance depends on all three parameters, which produces cross-terms. These values describe the stated approximate Gaussian likelihood at the stated nominal point; they should not be treated as exact information for the nonlinear model.

When Cross-Blocks Vanish, and When They Do Not

A useful contrast is the linear random-intercept model \(Y_j=\beta+\eta+\epsilon_j\), for \(j=1,2\), with \(\eta\sim N(0,\omega)\) and independent \(\epsilon_j\sim N(0,r)\). Its marginal mean is \(m=\beta\mathbf{1}\), and its covariance is \(V=\omega\mathbf{1}\mathbf{1}^T+rI_2\), where \(\mathbf{1}=(1,1)^T\). The mean depends on \(\beta\) alone, whereas the covariance depends on \(\omega\) and \(r\).

For this model, the cross-information between \(\beta\) and either covariance parameter is zero. The mean contribution is zero because \(V_\beta=0\); the covariance contribution is zero because \(m_\omega=m_r=0\). This is an exact expected-information result for this Gaussian model, not a universal property of mixed-effects models.

Worked Example: Random-Intercept Variance and Residual Variance

Continue with two observations per individual in the linear random-intercept model, at \(\omega=0.5\) and \(r=1\). The covariance has eigenvalue \(r+2\omega=2\) in the common-mean direction and eigenvalue \(r=1\) in the contrast direction. In these directions, the covariance derivatives with respect to \(\omega\) are \(2\) and \(0\), and those with respect to \(r\) are \(1\) and \(1\).

The covariance-only information formula therefore gives

I_{\omega\omega}=\frac{1}{2}\frac{2^2}{2^2}=0.5,\qquad I_{\omega r}=\frac{1}{2}\frac{2(1)}{2^2}=0.25,

and

I_{rr}=\frac{1}{2}\left(\frac{1^2}{2^2}+\frac{1^2}{1^2}\right) =\frac{1}{2}(0.25+1)=0.625.

Thus the variance-parameter information block is \(\begin{pmatrix}0.5&0.25\\0.25&0.625\end{pmatrix}\). The positive cross-entry reflects the fact that increasing either variance increases variation in the common-mean direction. In contrast, the contrast direction contains residual variance but not random-intercept variance. The typical-value information is \(I_{\beta\beta}=\mathbf{1}^TV^{-1}\mathbf{1}=2/(r+2\omega)=1\), and \(I_{\beta\omega}=I_{\beta r}=0\), as predicted from the derivative structure.

These examples illustrate a practical diagnostic: inspect which parameter groups affect the mean and covariance before interpreting a numerical block. If one group changes only the mean and another changes only covariance, their expected cross-information is zero in the Gaussian model. If a parameter affects both, or two parameters both change covariance in overlapping directions, nonzero cross-information is possible. A zero can also result from cancellation in the trace term; nonzero derivatives alone do not guarantee a nonzero block.

Parameterization and Interpretation

Numerical cross-information depends on how parameters are represented. If each group is transformed separately by a smooth one-to-one transformation that preserves the groups, the cross-block transforms by the corresponding Jacobians. In particular, a structural zero remains zero under these group-preserving transformations. But an arbitrary transformation that mixes groups can create cross-information where the original matrix had none.

Worked Example: A Transformation That Creates a Cross-Block

Suppose the information matrix for two scalar parameters \((\theta_1,\theta_2)\) is the identity matrix, so its off-diagonal entry is zero. Define new coordinates by \(\theta_1=u+v\) and \(\theta_2=v\). The Jacobian of \((\theta_1,\theta_2)\) with respect to \((u,v)\) is \(J=\begin{pmatrix}1&1\\0&1\end{pmatrix}\). The transformed information is

I_{(u,v)}=J^TI_{(\theta_1,\theta_2)}J =\begin{pmatrix}1&0\\1&1\end{pmatrix} \begin{pmatrix}1&1\\0&1\end{pmatrix} =\begin{pmatrix}1&1\\1&2\end{pmatrix}.

The new off-diagonal entry is \(1\), not zero. The transformation mixes the original parameter groups, so the original zero does not have to be preserved. This is why cross-block comparisons require the same parameterization and group definitions.

A cross-block is a covariance of score components, not the covariance of parameter estimates. Under standard large-sample likelihood conditions, the approximate covariance of the full estimator is related to \(I^{-1}\), and its off-diagonal blocks need not match those of \(I\). A nonzero \(I_{ab}\) is evidence of local score coupling; its sign depends on the coordinate directions and coding. It does not by itself tell whether estimates will be positively or negatively correlated.

Expected Fisher information is also distinct from the observed negative Hessian at one particular data set. The observed curvature may have cross-terms even where an expected cross-block is zero. For numerical work, check that the matrix is symmetric up to rounding, compare analytic derivatives with finite-difference checks, and verify that variances are positive and the evaluated covariance matrix is positive definite. Large discrepancies can indicate derivative errors, a poor local approximation, or a parameterization or scaling problem.

Common Pitfalls and Limits

Calling a cross-block an estimate correlation. Fisher cross-information describes score coupling; correlations between estimates come from the inverse information under suitable asymptotic conditions.
Assuming all fixed-effect/covariance cross-blocks are zero. They are zero in the stated Gaussian model when fixed effects affect only the mean and covariance parameters affect only covariance. Nonlinear mixed-effects approximations can make the covariance depend on typical values.
Treating a zero as parameterization-invariant. A zero is preserved by separate group-preserving transformations, not by general transformations that mix the groups.
Reading the sign as a universal direction of confounding. The sign can change with parameter coding and coordinates; interpret it in the specified parameterization.
Using approximate moments as exact. A Gaussian approximation can be useful for design, but its cross-information inherits the limitations of the approximation and its nominal parameter values.

Cross-blocks provide a focused description of joint sensitivity, but they do not alone quantify the information remaining for one group after accounting for the others. That requires examining the full block matrix, which is a separate calculation. Likewise, a large or small raw entry can be hard to interpret when parameters use different units; comparisons should be made with explicit awareness of scaling.

Check Your Understanding

Answer each question, then compare with the short solutions below.

In a Gaussian model, a parameter \(a\) affects only the mean and parameter \(b\) affects only the covariance. What is \(I_{ab}\), and why?
For a scalar Gaussian observation with \(m_a=1\), \(m_b=0\), \(v=4\), \(v_a=2\), and \(v_b=3\), calculate \(I_{ab}\).
In the two-observation random-intercept example, why is \(I_{\omega r}\) nonzero even though \(\omega\) and \(r\) represent different sources of variation?
Does an off-diagonal Fisher entry of zero have to remain zero after every smooth one-to-one reparameterization?

Solutions. (1) \(I_{ab}=0\): the mean term is zero because \(m_b=0\), and the covariance term is zero because \(V_a=0\). (2) The mean term is zero, and \(I_{ab}=\frac{1}{2}v_av_b/v^2=\frac{1}{2}(2)(3)/16=0.1875\). (3) Both variance parameters affect the common-mean direction, so their covariance derivatives overlap there; the contrast direction contributes residual information but no random-intercept-variance information. (4) No. A zero is preserved by separate group-preserving transformations, but a transformation mixing parameters can create a nonzero cross-entry.