The Independence Error in Population Information

Why the Independence Assumption Matters

In a population design, several measurements may be collected from each subject. Even when residual errors are independent conditional on a subject’s random effects, the measurements are generally not independent after those random effects are integrated out. A shared random effect links the observations: a subject whose latent response level is high tends to have multiple measurements that are high together.

Treating those measurements as independent can therefore change the population Fisher information matrix. It may count repeated evidence about a typical value more than once, misrepresent information about a variance parameter, or erase information about a covariance parameter. The direction and size of the error depend on the model and the parameter; there is no universal correction factor.

The key distinction is between independence of subjects and independence of measurements within a subject. Independent subjects contribute independent subject-level likelihoods. Repeated observations within a subject must ordinarily be modeled jointly.

Definition: The independence error in population information is the discrepancy caused by replacing a subject’s joint likelihood for repeated observations with a product of separate observation-level likelihoods when those observations are dependent marginally. Such a product discards the off-diagonal covariance terms and can produce an incorrect Fisher information matrix.

Joint Information Under a Gaussian Approximation

Let \(y_i\) be the vector of observations from subject \(i\), and suppose subjects are independent and follow the same design. A common first-order approximation linearizes the individual prediction in random effects. Write \(\mu\) for the vector of population mean predictions, \(\eta_i\sim N(0,\Omega)\) for the subject’s random effects, and \(\epsilon_i\sim N(0,R)\) for residual errors. If the local sensitivity of the prediction vector to the random effects is \(J\), then the approximate subject-level model is

y_i \approx \mu + J\eta_i+\epsilon_i,\qquad y_i\sim N(\mu,V),\qquad V=J\Omega J^\mathsf{T}+R.

Here \(R\) is the within-subject residual covariance matrix. The usual independent residual-error assumption makes \(R\) diagonal, but it does not make \(V\) diagonal: the term \(J\Omega J^\mathsf{T}\) can have nonzero off-diagonal entries. This marginal covariance is the relevant covariance for the joint likelihood of \(y_i\).

For a Gaussian vector with mean \(\mu\) and covariance \(V\), the expected Fisher information for parameters \(\psi_a,\psi_b\) is

I_{ab} = \frac{\partial\mu^\mathsf{T}}{\partial\psi_a} V^{-1} \frac{\partial\mu}{\partial\psi_b} + \frac{1}{2}\operatorname{tr} \left( V^{-1}\frac{\partial V}{\partial\psi_a} V^{-1}\frac{\partial V}{\partial\psi_b} \right).

This expression assumes a correctly specified multivariate normal model, a positive-definite \(V\), and differentiable mean and covariance with respect to the parameters. The first term describes information carried by changes in the mean; the second describes information carried by changes in the covariance. If a parameter affects both, both terms matter. For independent subjects, their information matrices add.

Why Dropping Covariance Changes Information

Replacing \(V\) by its diagonal matrix \(D=\operatorname{diag}(V)\) produces an independence model for the observations within a subject. Its model-based information is computed as if the observation variances were \(D\) and all covariances were zero. In particular, it replaces the joint mean contribution by derivatives weighted with \(D^{-1}\), rather than \(V^{-1}\), and it omits the effects of off-diagonal covariance derivatives.

A short derivation shows where the joint formula comes from. For \(e=y_i-\mu\), the Gaussian log likelihood, apart from a constant, is

\ell=-\frac{1}{2}\log|V|-\frac{1}{2}e^\mathsf{T}V^{-1}e.

Differentiating with respect to \(\psi_a\) gives a mean term and a centered covariance term:

\frac{\partial\ell}{\partial\psi_a} = \frac{\partial\mu^\mathsf{T}}{\partial\psi_a}V^{-1}e + \frac{1}{2} \left[ e^\mathsf{T}V^{-1}\frac{\partial V}{\partial\psi_a}V^{-1}e -\operatorname{tr}\left(V^{-1}\frac{\partial V}{\partial\psi_a}\right) \right].

Under the stated model, \(E(e)=0\) and \(E(ee^\mathsf{T})=V\). Taking the expected product of two scores gives the displayed information formula: the linear mean terms yield the first term, the centered quadratic covariance terms yield the trace term, and their cross expectation is zero for a centered Gaussian vector. If one substitutes a diagonal covariance before forming these scores, the resulting information pertains to that different, independent model—not to the original joint distribution.

Worked Example: Information About a Typical Response Level

Suppose each subject has three measurements of a response with a common population mean \(\mu\). Use the hypothetical random-intercept model \(y_{ij}=\mu+b_i+\epsilon_{ij}\), where \(b_i\sim N(0,1)\), \(\epsilon_{ij}\sim N(0,1)\), and the residual errors are independent of one another and of \(b_i\). Variances are known. Calculate the information about \(\mu\) from one subject under the joint model and under the false assumption that the three observations are independent marginally.

The assumptions imply a marginal variance of \(1+1=2\) for each measurement and a covariance of \(1\) between any two measurements from the same subject. Thus

V=I_3+\mathbf{1}\mathbf{1}^\mathsf{T},\qquad \frac{\partial\mu_{\!y}}{\partial\mu}=\mathbf{1},

where \(\mu_{\!y}=\mu\mathbf{1}\) is the three-element mean vector. The joint information for \(\mu\) is \(\mathbf{1}^\mathsf{T}V^{-1}\mathbf{1}\). Since \(V\mathbf{1}=4\mathbf{1}\), it follows that \(V^{-1}\mathbf{1}=\mathbf{1}/4\), so

I_{\mathrm{joint}}(\mu) =\mathbf{1}^\mathsf{T}\frac{\mathbf{1}}{4} =\frac{3}{4} =0.7500\ \text{response}^{-2}.

If the observations are incorrectly treated as independent with their correct marginal variance of \(2\), the calculation becomes

I_{\mathrm{ind}}(\mu)=\frac{3}{2}=1.500\ \text{response}^{-2}.

The independence calculation doubles the information attributed to one subject. The error is not that each measurement has the wrong marginal variance; it is that the shared random intercept makes the three measurements partly redundant for estimating the common population mean.

Variance Parameters Also Depend on Covariance

The covariance part of the information formula is especially important for variance parameters. A random-effect variance changes both marginal variances and, when the effect is shared across observations, their covariances. An independence calculation may notice the changes in the marginal variances while missing the joint pattern that identifies the variance.

Worked Example: Information About Random-Intercept Variance

Use the same hypothetical three-measurement model, but now let the random-intercept variance be \(\tau^2\), with the residual variance fixed at \(1\). Evaluate the information for \(\tau^2\) at \(\tau^2=1\), assuming the mean is known. The model is Gaussian, the variance is positive, and the residual and random-intercept assumptions are as stated above.

For one subject, \(V=I_3+\tau^2\mathbf{1}\mathbf{1}^\mathsf{T}\), so \(\partial V/\partial\tau^2=\mathbf{1}\mathbf{1}^\mathsf{T}\). The mean term is zero because the mean is known. Using the covariance part of the Gaussian information formula and \(V^{-1}\mathbf{1}=\mathbf{1}/(1+3\tau^2)\),

I_{\mathrm{joint}}(\tau^2) = \frac{1}{2} \left(\mathbf{1}^\mathsf{T}V^{-1}\mathbf{1}\right)^2 = \frac{1}{2} \left(\frac{3}{1+3\tau^2}\right)^2.

At \(\tau^2=1\), this is

I_{\mathrm{joint}}(\tau^2) =\frac{1}{2}\left(\frac{3}{4}\right)^2 =\frac{9}{32} =0.2813\ \text{response}^{-4}.

The independence model treats three observations as independent \(N(\mu,1+\tau^2)\) values. Each contributes \(1/[2(1+\tau^2)^2]\) information about the variance, giving

I_{\mathrm{ind}}(\tau^2) =\frac{3}{2(1+\tau^2)^2} =\frac{3}{8} =0.3750\ \text{response}^{-4} \quad\text{at }\tau^2=1.

Here the diagonal calculation overstates information by \(0.09375\ \text{response}^{-4}\). It also has the wrong scaling as the number of measurements grows: the joint information for this shared variance approaches \(1/(2\tau^4)\), whereas the independent calculation grows in proportion to the number of observations. Repeated measurements help characterize a subject’s shared deviation, but they do not become independent subjects.

Ignoring covariance can also erase information rather than inflate it. Consider a covariance parameter that changes dependence while leaving each observation’s marginal variance unchanged.

Worked Example: A Correlation Parameter Hidden by Diagonalization

Suppose each subject contributes a bivariate Gaussian observation with known mean and covariance \(V=v\begin{pmatrix}1&\rho\\ \rho&1\end{pmatrix}\), where \(v>0\) is known and \(-1<\rho<1\). The parameter \(\rho\) changes the within-subject correlation but not either marginal variance. The covariance is positive definite under the stated condition.

The covariance derivative is \(V_\rho=v\begin{pmatrix}0&1\\1&0\end{pmatrix}\). The two orthogonal directions \((1,1)^\mathsf{T}\) and \((1,-1)^\mathsf{T}\) have variances \(v(1+\rho)\) and \(v(1-\rho)\), respectively. In those same directions, the derivative of the variance with respect to \(\rho\) is \(v\) and \(-v\). Therefore the covariance information is

I_{\mathrm{joint}}(\rho) = \frac{1}{2} \left[ \left(\frac{v}{v(1+\rho)}\right)^2 + \left(\frac{-v}{v(1-\rho)}\right)^2 \right] = \frac{1+\rho^2}{(1-\rho^2)^2}.

At \(\rho=0.5\), the information is \(1.25/0.5625=2.222\) per subject. If the covariance is replaced by its diagonal, both marginal variances are simply \(v\), independent of \(\rho\); the diagonal model then gives zero information about \(\rho\). The dependence pattern itself carries information, and diagonalization removes it.

Diagnostics, Alternatives, and Limits

Before interpreting an information matrix, check the covariance structure at the planned observation times and nominal parameter values. In particular, inspect whether the off-diagonal entries of \(V\) are negligible for the questions being asked. Also compare the joint and diagonal calculations for the parameters of interest. A large discrepancy is direct evidence that the independence simplification matters for the proposed design.

Check the model, not just the residuals. Independent residual errors do not justify marginal independence when random effects are shared.
Check parameter-specific information. A diagonal approximation may inflate information for a mean while erasing information for a correlation parameter. Inspecting only a single summary of the matrix can hide this contrast.
Check positive definiteness and sensitivities. The joint formula requires a valid positive-definite covariance matrix and appropriate derivatives. Numerical instability or implausible covariance estimates can make information comparisons unreliable.
Check what the calculation represents. The sum of separate marginal likelihoods is a composite likelihood if the observations are actually dependent. Its ordinary model-based information is not automatically the information for the joint model. Inference based on a composite likelihood generally requires accounting for score variability as well as expected curvature.

The principal alternative is to use the joint subject-level likelihood with its full covariance matrix. That is the appropriate basis for Gaussian first-order information when its approximation is adequate. If the first-order approximation is inadequate for the nonlinear model, a more suitable marginal or conditional approximation may be needed; the independence shortcut does not repair a poor approximation. The exact choice depends on the model and estimation method.

A diagonal calculation can still be useful as a rough comparison or as an explicitly defined working model, provided its assumptions are defensible and its limitations are reported. It should not be presented as the joint population information merely because each observation’s marginal variance is correct. Nor does a correct covariance model ensure that a local information approximation is accurate far from the nominal parameter values.

Common Mistake

Common mistake: “The residual errors are independent, so I can add the information from every measurement as if they were independent.” This confuses conditional residual independence with marginal independence. After integrating over shared random effects, the observation covariance is \(J\Omega J^\mathsf{T}+R\), which generally has off-diagonal entries. Use the joint covariance for population information, or clearly label a diagonal calculation as a working independence model.

Key takeaway: Population information is based on the joint distribution of a subject’s repeated observations. Ignoring their covariance can overcount evidence about typical values, distort information about variance parameters, or discard information carried by dependence itself.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions.

In a random-intercept model with independent residual errors, are repeated observations necessarily independent after integrating out the random intercept? Why?
For three measurements with random-intercept variance \(1\) and residual variance \(1\), what are the joint and independence-model information values for the common mean?
Why can a diagonal covariance approximation give zero information about a correlation parameter even when the full data contain information about it?
What is the main distinction between the joint likelihood and a product of marginal observation-level likelihoods when observations are correlated?

Solutions. (1) No. The shared random intercept induces marginal covariance between every pair of observations, even though the residual errors are conditionally independent. (2) The joint value is \(3/(1+3)=0.7500\ \text{response}^{-2}\); the independence value is \(3/(1+1)=1.500\ \text{response}^{-2}\). (3) If the correlation changes only off-diagonal entries, diagonalization removes all dependence of the working covariance on that parameter. (4) The joint likelihood retains the dependence structure and its covariance contributions to information; the product of marginal likelihoods discards that structure and describes a different working model.