Averaging Information Over a Covariate Distribution

Why Average Over Covariates?

In a population design, subjects may differ in characteristics such as body weight, age, or baseline biomarker values. Even when the sampling schedule is the same for everyone, the information contributed by a subject can depend on those characteristics. If subjects are expected to be sampled from a covariate distribution, a design calculation should reflect information across that distribution—not simply the information for a subject at its average covariate value.

The starting point is the subject-level Fisher information under fixed covariate values. Here the covariates are treated as design inputs for the subject-level likelihood. The population calculation averages that information over the distribution from which those inputs are expected to be drawn.

Definition: Let $Z$ be a subject’s covariate vector, with design distribution $g(z)$, and let $Y\mid Z=z$ have likelihood $f(y\mid z,\theta)$ for parameter vector $\theta$. The conditional Fisher information for one subject is $$ I(z;\theta)=E_{Y\mid Z=z,\theta}\left[s(Y,z;\theta)s(Y,z;\theta)^T\right], $$ where $s(Y,z;\theta)=\partial \log f(Y\mid z,\theta)/\partial\theta$ is the score vector. The population-averaged information per subject is $$ \bar I(\theta)=E_Z[I(Z;\theta)]=\int I(z;\theta)g(z)\,dz. $$ For a discrete covariate distribution, the integral is replaced by the probability-weighted sum over its possible values.

The expectation in $I(z;\theta)$ is over the subject’s outcome, conditional on covariates. The additional expectation defining $\bar I(\theta)$ is over covariates. Keeping those two averaging steps separate is essential: the first describes uncertainty in outcomes at a specified covariate value; the second describes how information varies across the target population.

Deriving the Population Average

Suppose $n$ subjects are sampled independently from the same covariate distribution, and their outcomes are conditionally independent given their covariates. Assume that $g(z)$ is specified by the design and does not depend on $\theta$. Under the usual regularity conditions for Fisher information—such as differentiability of the likelihood and interchange of differentiation and integration—the conditional information for the observed covariates $z_1,\ldots,z_n$ is the sum of the subject-level contributions:

I_{\mathrm{total}}(z_1,\ldots,z_n;\theta)=\sum_{i=1}^{n} I(z_i;\theta).

Taking an expectation over the sampled covariates and using linearity of expectation gives the population result:

E_Z[I_{\mathrm{total}}(Z_1,\ldots,Z_n;\theta)] =\sum_{i=1}^{n} E_Z[I(Z_i;\theta)] =n\bar I(\theta).

This derivation does not require the observed covariate values to be identical. It says that before sampling the covariates, the expected total information is $n$ times the average contribution per sampled subject. Once actual covariate values are known, the conditional total is instead the sum evaluated at those values. For a large independent sample from $g$, the observed average of the contributions will typically approach $\bar I(\theta)$, but a particular sample need not equal its expectation.

The definition uses the score-product form of information. When the regularity conditions also give the information equality, the same conditional information can be calculated as the negative expected Hessian:

I(z;\theta) =-E_{Y\mid z,\theta}\left[ \frac{\partial^2 \log f(Y\mid z,\theta)} {\partial\theta\,\partial\theta^T} \right].

If the covariate distribution is continuous, a numerical integral may be needed. If it is discrete, each information matrix is multiplied by the probability of its covariate value, and the weighted matrices are added element by element.

Worked Examples

Worked Example: A Discrete Covariate in a Poisson Model

Suppose a hypothetical subject has outcome $Y\mid Z=z\sim\operatorname{Poisson}(\exp(\theta z))$. The scalar parameter is $\theta$, and $Z$ is a design covariate taking values 0 and 2 with probability $1/2$ each. Calculate the expected information per subject at the nominal value $\theta=0$, then compare it with information at the mean covariate.

For a Poisson observation with mean $\lambda=\exp(\theta z)$, the log-likelihood, apart from terms not involving $\theta$, is $y\theta z-\exp(\theta z)$. Its score is $z(y-\lambda)$. Since the conditional variance of $Y$ is $\lambda$, the conditional information is

I(z;\theta)=E[z^2(Y-\lambda)^2\mid z,\theta]=z^2\lambda=z^2\exp(\theta z).

At $\theta=0$, the information is $I(0;0)=0^2\exp(0)=0$ and $I(2;0)=2^2\exp(0)=4$. Therefore,

\bar I(0)=\frac{1}{2}(0)+\frac{1}{2}(4)=2 \quad\text{information units per subject.}

The mean covariate is $E[Z]=\tfrac12(0)+\tfrac12(2)=1$. Evaluating information there gives $I(1;0)=1^2\exp(0)=1$, not 2. For $n$ independently sampled subjects, the expected total is $2n$; substituting the mean covariate for every subject would instead give $n$. The difference arises because information depends quadratically on $z$.

Worked Example: A Gaussian Model With Covariate-Dependent Precision

Suppose, hypothetically, $Y\mid Z=z\sim N(\theta,1+z^2)$, where the variance function is known and only the mean parameter $\theta$ is unknown. Let $Z=0$ or $Z=2$, each with probability $1/2$. Find the expected information per subject and compare it with information at $E[Z]$.

For a normal observation with known variance $v(z)=1+z^2$ and mean $\theta$, the score for $\theta$ is $(Y-\theta)/v(z)$. Its conditional variance is $v(z)$, so the information is

I(z;\theta)=E\left[\frac{(Y-\theta)^2}{v(z)^2}\middle|z,\theta\right] =\frac{v(z)}{v(z)^2} =\frac{1}{1+z^2}.

At $z=0$, the information is 1. At $z=2$, it is $1/(1+4)=0.2$. The covariate-averaged information is therefore $\bar I=\tfrac12(1)+\tfrac12(0.2)=0.6$ per subject. The mean covariate is 1, where the information is $1/(1+1)=0.5$ per subject. With $n=50$ subjects, the expected total information is $50(0.6)=30$, whereas evaluating every subject at the mean covariate gives $50(0.5)=25$. The assumptions used here are that the conditional normal model is correct, its variance function is known, and the specified two-point distribution describes the planned covariate sampling.

Worked Example: Integrating Over a Continuous Covariate

For a third hypothetical design, suppose $Y\mid Z=z$ is Bernoulli with success probability $p(z)=1/(1+\exp[-(\theta+z)])$. The covariate is a known offset in the model, $Z$ is uniformly distributed on $[0,2]$, and the nominal value is $\theta=0$. Calculate the average information per subject and compare it with information at the mean covariate.

For a Bernoulli observation with logit probability $\theta+z$, differentiation of the log-likelihood gives score $Y-p(z)$. Since $\operatorname{Var}(Y\mid z)=p(z)[1-p(z)]$, the conditional information is $I(z;0)=p(z)[1-p(z)]$. Writing $p(z)=e^z/(1+e^z)$, this becomes $e^z/(1+e^z)^2$. A uniform distribution on $[0,2]$ has density $1/2$ on that interval, so

\bar I(0)=\frac{1}{2}\int_0^2 \frac{e^z}{(1+e^z)^2}\,dz =\frac{1}{2}\left[\frac{1}{2}-\frac{1}{1+e^2}\right] \approx 0.1904.

The mean covariate is 1. At $z=1$, $p(1)=e/(1+e)\approx0.7311$, giving $I(1;0)=0.7311(1-0.7311)\approx0.1966$. Thus the continuous-distribution average is about 0.1904 information units per subject, not 0.1966. The integral is the relevant population average because it weights information at every covariate value according to the uniform design distribution.

Why the Mean Covariate Is Not a Substitute

In general, information is a nonlinear function of covariates. Consequently,

E_Z[I(Z;\theta)]\ne I(E[Z];\theta).

The three examples show that the difference can be positive or negative: information at the mean was lower than the average in the Poisson and Gaussian examples, but slightly higher in the Bernoulli example. There is no general rule that one quantity must exceed the other. For a matrix-valued information function, the same point applies element by element: averaging each matrix over the covariate distribution is not equivalent to evaluating the matrix at a vector of mean covariates.

The averaging operation is also distinct from taking an inverse or transforming the information to a measure of precision. In general, the inverse of an average information matrix is not the average of the inverse matrices. The target here is specifically the expected Fisher information under the stated covariate distribution. Any later design criterion built from that matrix should be applied after the averaging step appropriate to that criterion.

Specify the target distribution.
State which covariates vary across subjects and give their planned or target distribution, including any dependencies among covariates.

Calculate conditional information.
For each covariate value, obtain $I(z;\theta)$ from the conditional likelihood at the nominal parameter values.

Average over that distribution.
Use a probability-weighted sum for discrete support or an integral (or justified numerical approximation) for continuous support.

Scale to the planned sample.
For $n$ independent subjects drawn from the same distribution, the expected total information is $n\bar I(\theta)$.

Assumptions, Diagnostics, and Limitations

This calculation treats the covariate distribution as part of the design description and assumes that it is correctly specified for the population of interest. The conditional likelihood must also be appropriate, and the information should be evaluated at clearly stated nominal parameter values. If subjects are sampled independently from a common distribution, the expected-total formula follows directly. With fixed quotas or strata, average within each group and combine group-specific information using the planned group proportions; the common-distribution formula should not be used without accounting for that allocation.

The covariate distribution is assumed not to depend on $\theta$. If the sampling mechanism itself depends on the unknown parameters, the likelihood for the covariates may contribute information too; simply averaging conditional outcome information would then not describe all information in the joint data. Likewise, this tutorial concerns the expectation of conditional information when covariates are observed. It does not replace the information calculation for a different model in which covariates are unobserved and outcomes are analyzed after integrating the covariates out.

For numerical calculations, check that the probabilities or density used for weighting are normalized, that the integration covers the relevant covariate support, and that the same nominal parameter values are used throughout. With a matrix, check symmetry and that the averaged matrix is positive semidefinite up to numerical tolerance. For numerical integration, increase the number of quadrature points or refine the integration grid and confirm that the resulting entries are stable. A coarse grid can miss regions where information changes rapidly or where the covariate distribution places substantial weight.

A common mistake is to replace every subject’s covariate with the population mean and call the resulting matrix “population information.” That calculation answers a different question: it describes a hypothetical design in which every subject has the mean covariate. The population calculation instead averages the subject-level information over the distribution. The mean-value shortcut is justified only in special cases where the information function happens to average to its value at the mean, not merely because the covariate mean is known.

Finally, $\bar I(\theta)$ is a local model-based expectation, not a guarantee about the realized information in any one sample. A small sample can have an unusual covariate mix, and a misspecified target distribution can make the average unrepresentative. Report the distributional assumptions alongside the result so that the interpretation remains tied to the population the design is meant to represent.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions below.

If a covariate takes values 1 and 3 with equal probability and $I(z)=z^2$, what is the expected information? What is information at the mean covariate?
For $n$ independent subjects sampled from a common covariate distribution, express the expected total information in terms of the per-subject population average.
A design fixes half the subjects at $z=0$ and half at $z=2$. Is its expected information necessarily calculated using the distribution of covariates in a random sample? Explain.
Why is evaluating $I(z;\theta)$ at $E[Z]$ generally not a valid substitute for averaging $I(Z;\theta)$?

Solutions. (1) The average is $\tfrac12(1^2)+\tfrac12(3^2)=5$. The mean covariate is 2, where $I(2)=4$; the two quantities differ. (2) If the per-subject average is $\bar I(\theta)$, linearity of expectation gives $E[I_{\mathrm{total}}]=n\bar I(\theta)$. (3) No random covariate sampling is required: with fixed equal allocation, calculate $\tfrac12 I(0;\theta)+\tfrac12 I(2;\theta)$ as the per-subject average for that design. (4) The information function may be nonlinear, so averaging its values over the distribution need not equal evaluating it at the mean covariate.