Subject-Level Information Contributions

Why Examine Information One Subject at a Time?

A population design is built from the observations planned for individual subjects. Two subjects can contribute different amounts, or different directions, of information: one may have a covariate value that makes the model response more sensitive to a parameter, while another may have observation times that better distinguish changes in the response. Measurement error also matters. A subject with more precise, less redundant measurements will generally contribute more information than one whose measurements are noisy or strongly correlated.

The earlier material on the Gaussian approximation to the subject-level likelihood and the Gaussian population Fisher matrix provides the ingredients for this calculation. Here, the focus is the subject as the unit of contribution. The resulting information is an expectation under the specified approximate model, not a score or curvature calculated from one realized subject’s data.

Definition: For subject $i$, let $y_i$ be the vector of planned measurements, $d_i$ denote that subject’s design (including covariates and observation schedule), and $\theta$ be the parameter vector of dimension $p$. Under the Gaussian approximation, suppose $$ y_i\mid d_i \sim N\bigl(\mu_i(\theta;d_i),\Sigma_i(\theta;d_i)\bigr), $$ where $\mu_i$ is the subject-specific mean vector and $\Sigma_i$ is the subject-specific covariance matrix. The approximate Fisher information contributed by subject $i$ is the expected outer product of that subject’s score, evaluated at the parameter value of interest.

The covariance matrix must be symmetric and positive definite for the inverse in the information formula to exist. The mean and covariance need to be differentiable with respect to the parameters in the region being considered. As in any local Fisher-information calculation, derivatives are evaluated at a specified nominal parameter value. Consequently, a design’s information can change when that nominal value changes.

The Subject-Level Information Formula

Write $\mu_{i,a}=\partial\mu_i/\partial\theta_a$ and $\Sigma_{i,a}=\partial\Sigma_i/\partial\theta_a$. The $(a,b)$ entry of subject $i$’s information matrix is

\mathcal{I}_{i,ab} = \mu_{i,a}^{\mathsf T}\Sigma_i^{-1}\mu_{i,b} + \frac{1}{2}\operatorname{tr}\!\left( \Sigma_i^{-1}\Sigma_{i,a} \Sigma_i^{-1}\Sigma_{i,b} \right).

The first term is information conveyed through changes in the mean. The second is information conveyed through changes in the covariance. If a parameter changes only the mean, its covariance derivative is zero; if it changes only the covariance, its mean derivative is zero. When both depend on a parameter, both contributions must be included. The covariance term is not an optional adjustment: omitting it discards information whenever the covariance depends on the parameter.

Why the Formula Has These Two Terms

For a derivation, temporarily suppress the subject index and set $z=y-\mu$, $A=\Sigma^{-1}$. The score for parameter $\theta_a$, obtained by differentiating the Gaussian log likelihood, is

s_a = \mu_{,a}^{\mathsf T}Az +\frac{1}{2}z^{\mathsf T}A\Sigma_{,a}Az -\frac{1}{2}\operatorname{tr}(A\Sigma_{,a}).

Under the model, $E[z]=0$ and $E[zz^{\mathsf T}]=\Sigma$. Thus the score has expectation zero. The expected product $E[s_as_b]$ has a product of the two linear terms and a product of the two centered quadratic terms. The mixed products have expectation zero because centered Gaussian variables have zero third moments. For symmetric matrices $B$ and $C$, a centered Gaussian vector also satisfies

E\!\left[ \{z^{\mathsf T}Bz-\operatorname{tr}(B\Sigma)\} \{z^{\mathsf T}Cz-\operatorname{tr}(C\Sigma)\} \right] = 2\operatorname{tr}(B\Sigma C\Sigma).

Applying $E[zz^{\mathsf T}]=\Sigma$ to the linear terms gives $\mu_{,a}^{\mathsf T}\Sigma^{-1}\mu_{,b}$. Applying the centered-quadratic identity with $B=A\Sigma_{,a}A$ and $C=A\Sigma_{,b}A$, and retaining the one-quarter multiplier from the score, gives the trace term with multiplier one-half. Together these yield the stated formula. In particular, each term describes a distinct way the probability distribution can change with the parameters.

Worked Example: Covariate and Schedule Change Mean Information

Suppose a hypothetical subject has observations at two planned times. The approximate mean at time $t$ is $\theta x f(t)$, where $x$ is a measured covariate and $f(t)$ is a known dimensionless time function. The two measurement errors are independent, each with variance $0.04$ concentration-units squared. At the nominal value, take $x=2$ and $f(t_1)=0.5,\ f(t_2)=1$. Calculate the information about the scalar parameter $\theta$, assuming the covariance does not depend on $\theta$.

The mean vector is $\mu=\theta x(0.5,1)^{\mathsf T}$. Its derivative with respect to $\theta$ is $x(0.5,1)^{\mathsf T}=(1,2)^{\mathsf T}$. The covariance matrix is $0.04I$, so its inverse is $25I$. The covariance-derivative term is zero. Therefore,

\mathcal{I}_i(\theta) = (1,2) \begin{pmatrix}25&0\\0&25\end{pmatrix} \begin{pmatrix}1\\2\end{pmatrix} = 25(1^2+2^2) = 125.

The information is $125$ per concentration-unit squared of $\theta$. The two observations contribute $25$ and $100$, respectively: at the given covariate value, the later observation has the larger mean sensitivity. If the schedule included only the first time, information would be $25$. If both times were retained but $x=1$, information would be $31.25$, since the squared sensitivities would be one-quarter as large. These comparisons isolate how schedule and covariate values enter through the derivative of the mean.

Measurement-Error Structure Can Change the Contribution

The inverse covariance in the mean-information term determines how measurement precision and dependence are weighted. With independent errors of equal variance, the inverse covariance simply weights each squared sensitivity by the inverse variance. With correlated errors, the observations cannot be treated as independent pieces of evidence. The covariance term also depends on the measurement-error structure when that covariance changes with a parameter.

Worked Example: Correlated Errors and a Covariance Parameter

For a hypothetical subject with two measurements, suppose $y_i\sim N(a h,vR)$, where $h=(1,1)^{\mathsf T}$, $v=4$ concentration-units squared, and $$ R=\begin{pmatrix}1&0.6\\0.6&1\end{pmatrix}. $$ The parameter vector is $(a,v)$; $R$ is known and does not depend on either parameter. Find the subject’s information matrix at the stated $v$.

The mean derivative for $a$ is $h$, and the covariance derivative for $a$ is zero. Since $\Sigma=vR$, the mean information for $a$ is

h^{\mathsf T}(vR)^{-1}h = \frac{1}{4}\frac{1}{1-0.6^2} (1,1) \begin{pmatrix}1&-0.6\\-0.6&1\end{pmatrix} \begin{pmatrix}1\\1\end{pmatrix} = \frac{1}{4}\frac{0.8}{0.64} = 0.3125.

For $v$, the mean derivative is zero and $\Sigma_{,v}=R$. Thus $\Sigma^{-1}\Sigma_{,v}=I/v$, and the covariance contribution is

\frac{1}{2}\operatorname{tr} (\Sigma^{-1}R\Sigma^{-1}R) = \frac{1}{2}\operatorname{tr}\bigl((I/v)^2\bigr) = \frac{2}{2(4^2)} = 0.0625.

The cross-information between $a$ and $v$ is zero: the mean derivative for $v$ and covariance derivative for $a$ are both zero, so neither term in the formula contributes. The information matrix, with rows and columns ordered as $(a,v)$, is therefore

\mathcal{I}_i = \begin{pmatrix} 0.3125&0\\ 0&0.0625 \end{pmatrix}.

If the same two measurements instead had independent errors with the same marginal variance, then $R=I$ and the information for $a$ would be $h^{\mathsf T}h/v=2/4=0.5$. Here positive correlation reduces information about a common mean shift: the two measurements partly move together and therefore provide less independent evidence about $a$. The information for $v$ remains $0.0625$ in this particular scale-covariance model. That equality is a consequence of this specified model, not a general claim that correlation never affects information.

Combining Contributions Across Independent Subjects

Suppose subjects are independent conditional on their designs and share the population parameter vector $\theta$. Their joint log likelihood is the sum of the subject-level log likelihoods, so the joint score is the sum of their scores. Each subject’s score has expectation zero under the specified model. Independence therefore makes the expected cross-products between different subjects’ scores zero, leaving the total approximate information as the sum of the subject-level matrices:

\mathcal{I}_{\mathrm{total}}(\theta) = \sum_{i=1}^{N}\mathcal{I}_i(\theta;d_i).

This statement is conditional on the chosen covariates and designs. It does not require every subject to have the same schedule or covariate values. Each matrix can be calculated at that subject’s design and then added, provided the subjects’ measurement vectors are independent under the model.

Worked Example: Adding Two Different Subject Designs

Consider two hypothetical independent subjects, each contributing one measurement with variance $0.25$ outcome-units squared. For subject $i$, the mean is $\theta_1 x_i+\theta_2 g_i$, where $x_i$ is a covariate and $g_i$ represents the sensitivity associated with the selected observation time. At the design values, subject A has $(x_A,g_A)=(1,1)$, and subject B has $(x_B,g_B)=(2,0.5)$. The covariance is parameter-independent.

For one measurement, the mean derivative vector is $q_i=(x_i,g_i)^{\mathsf T}$. The subject information is $q_iq_i^{\mathsf T}/0.25=4q_iq_i^{\mathsf T}$. Thus

\mathcal{I}_A = 4\begin{pmatrix}1\\1\end{pmatrix}(1,1) = \begin{pmatrix}4&4\\4&4\end{pmatrix}, \qquad \mathcal{I}_B = 4\begin{pmatrix}2\\0.5\end{pmatrix}(2,0.5) = \begin{pmatrix}16&4\\4&1\end{pmatrix}.

Adding the matrices gives

\mathcal{I}_{\mathrm{total}} = \begin{pmatrix}20&8\\8&5\end{pmatrix}.

Each single-observation contribution has rank one because one sensitivity vector cannot independently identify two parameter directions. The combined matrix has determinant $20(5)-8^2=36>0$, and its leading diagonal entry is positive, so it is positive definite and contains information in both parameter directions in this example. The off-diagonal value $8$ records that the subject-level sensitivities link the two parameters; it is not an additional observation or a separate measure of precision.

Design Checks, Alternatives, and Limitations

Before interpreting a calculated matrix, check the assumptions and the numerical result. A covariance matrix that is not positive definite cannot be inverted as required. A subject’s information matrix should be symmetric and positive semidefinite; small asymmetries in computation may reflect rounding, but substantial ones suggest an implementation error. If the total matrix is singular or nearly singular, some parameter directions are unsupported or weakly supported by the planned observations. Very large information entries can also be a warning if they result from implausibly small assumed measurement error or unstable numerical derivatives.

A direct alternative is to form the joint likelihood for all subjects and calculate its expected information in one step. Under conditional independence, the log likelihood is a sum, and this gives the same result as summing subject matrices. Subject-by-subject calculation is particularly useful when schedules or covariates differ, because it makes the design-specific sources of information visible. The joint calculation can be more convenient when the implementation already represents the full data set, but it can obscure which subjects or design features drive the result.

The sum is not valid without modification if subject data are dependent, for example through a shared latent effect or a common source of measurement error that induces cross-subject covariance. Likewise, a Gaussian approximation is a modeling choice: if it poorly represents the subject-level likelihood, the resulting expected information may not accurately describe the original model. Finally, Fisher information is local and model-based. It depends on the nominal parameter values and assumed covariance structure; it does not guarantee that an estimator will behave well in a small sample or under model misspecification.

Common pitfall: It is tempting to say that every subject contributes the same information because all subjects belong to one study, or to add only the information from mean sensitivities. Both steps can be wrong. The correct contribution is computed using that subject’s own mean, covariance, and their parameter derivatives; covariance derivatives must be included whenever they are nonzero. Contributions add in the stated form only under the assumed independence across subjects.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions.

If a parameter changes only the subject-level mean and the covariance is parameter-independent, which part of the Gaussian information formula remains?
A single observation has mean $\theta c$, variance $0.5$, and known constant variance. What is its information about $\theta$ when $c=3$?
Why can two positively correlated measurements provide less information about a common mean than two independent measurements with the same marginal variances?
Two independent subjects have information matrices $A$ and $B$. What is the total information matrix, and what assumption supports this calculation?

Solutions. (1) Only the mean term remains: $\mu_{,a}^{\mathsf T}\Sigma^{-1}\mu_{,b}$; all covariance derivatives are zero. (2) The mean derivative is $c=3$, so the information is $3^2/0.5=18$ per squared unit of $\theta$. (3) Correlation makes the measurements partly redundant, and the inverse covariance accounts for that dependence rather than counting each measurement as independent evidence. (4) The total is $A+B$, provided the subjects’ data are independent conditional on their designs and share the parameter vector being assessed.