Information for Covariate Effects

Why Covariate Allocation Matters

A covariate effect is estimated by comparing subjects with different covariate values. If all subjects have the same value, their observations may estimate a typical parameter well, but they provide no direct contrast for determining how that parameter changes with the covariate. More generally, a design with a wider range of covariate values can provide more information about the effect, while imbalance can make the effect harder to separate from the baseline parameter.

The key is to account for both pieces of information in the model: how precisely each subject's data inform the relevant typical parameter, and where that subject lies in the covariate distribution. The results below are local Fisher-information approximations, evaluated at specified nominal parameter values and covariate values. They complement the earlier material on information at fixed covariate values and information averaged over a covariate distribution.

From Typical-Parameter Information to Covariate-Effect Information

Consider a positive typical parameter, such as clearance, denoted by $\theta_i$ for subject $i$. Let $x_i$ be a measured covariate and suppose the parameter follows a log-linear relationship:

\log(\theta_i)=\alpha+\beta x_i, \qquad \theta_i=\exp(\alpha+\beta x_i).

Here $\alpha$ is the log typical parameter at $x_i=0$, and $\beta$ is the change in log typical parameter per one-unit increase in $x$. Thus, a one-unit increase multiplies $\theta_i$ by $\exp(\beta)$. Centering $x$ changes the covariate value at which $\alpha$ refers, but not the meaning of $\beta$.

Definition: Let $j_i$ be the subject-level Fisher information for $\eta_i=\log(\theta_i)$, evaluated at the nominal model and design. Under independent subjects and a model in which subject $i$'s data depend on $\alpha,\beta$ through $\eta_i=\alpha+\beta x_i$, the local information for $(\alpha,\beta)$ is $$ \mathcal I(\alpha,\beta)=\sum_{i=1}^n j_i \begin{pmatrix} 1 & x_i\\ x_i & x_i^2 \end{pmatrix}. $$

This expression follows by the chain rule. The derivative of $\eta_i$ with respect to $(\alpha,\beta)$ is $(1,x_i)$. If the subject-level score for $\eta_i$ has expected squared value $j_i$, then its information contribution for $(\alpha,\beta)$ is the outer product of this derivative, multiplied by $j_i$. Independent subjects' information contributions add.

The information entry for $\beta$ alone, $ \sum_i j_i x_i^2$, does not generally describe the precision for estimating $\beta$: $\alpha$ is also estimated. The relevant information is the Schur complement after accounting for $\alpha$.

\mathcal I_{\beta\mid\alpha} = \sum_i j_i x_i^2- \frac{\left(\sum_i j_i x_i\right)^2}{\sum_i j_i} = \sum_i j_i(x_i-\bar{x}_w)^2, \qquad \bar{x}_w=\frac{\sum_i j_i x_i}{\sum_i j_i}.

The final equality follows by expanding the weighted squared deviations and using the definition of $\bar{x}_w$. This is a complete justification of the weighted-spread result: the information for the covariate effect is the total subject-level information multiplied by the weighted variation in covariate values. When all subjects have the same $j_i=j$, this becomes $jn$ times the sample variance of the covariate values when variance is defined with divisor $n$. If all $x_i$ are identical, the weighted spread is zero, so the covariate effect cannot be separated from the baseline parameter in this model.

For a positive parameter, information on the log scale and information on the original scale are related by another chain-rule transformation. If a subject contributes $j_{\theta,i}$ information for $\theta_i$, then its information for $\eta_i=\log(\theta_i)$ is $j_i=\theta_i^2j_{\theta,i}$, since $d\theta_i/d\eta_i=\theta_i$. This scale conversion matters: information values have units determined by the parameterization, while $\beta$ in the log-linear model is a log-scale effect.

Worked Example: Balanced Versus Imbalanced Binary Covariates

Suppose a hypothetical study has 20 subjects, with $x_i=0$ or $1$. Each subject contributes $j_i=4$ units of information for the log typical parameter. Compare an allocation with 10 subjects at each covariate value to one with 16 at $x=0$ and 4 at $x=1$.

For the balanced allocation, $\sum j_i=80$, $\sum j_ix_i=40$, and $\sum j_ix_i^2=40$. Therefore,

\mathcal I_{\beta\mid\alpha} =40-\frac{40^2}{80} =20.

The full information matrix is $\begin{pmatrix}80&40\\40&40\end{pmatrix}$. Under a local normal approximation, the standard error for $\beta$, with $\alpha$ estimated, is approximately $1/\sqrt{20}=0.2236$ log units per covariate unit.

For the imbalanced allocation, $\sum j_i=80$, $\sum j_ix_i=16$, and $\sum j_ix_i^2=16$. Hence,

\mathcal I_{\beta\mid\alpha} =16-\frac{16^2}{80} =12.8.

Its approximate standard error is $1/\sqrt{12.8}=0.2795$ log units per covariate unit. The total subject-level information is unchanged, but the more imbalanced allocation provides less information for distinguishing the covariate effect from the baseline. These standard errors are local approximations, not guaranteed finite-sample sampling errors.

Covariate Spread and Centering

For a continuous covariate, the weighted-spread formula makes the role of range explicit. With equal $j_i=j$, more spread in $x_i$ produces more information for $\beta$, provided the model remains adequate across that range and the subject-level information does not deteriorate. Centering a covariate can improve interpretation and reduce the intercept-effect cross-information, but centering alone does not change the information available for the slope.

Worked Example: Doubling the Covariate Range

Consider 18 hypothetical subjects, six each at $x=-1,0,1$, with $j_i=2$. The covariate is balanced around zero, so $\sum j_ix_i=0$. Since $\sum x_i^2=6(1)+6(0)+6(1)=12$, the information for the effect is

\mathcal I_{\beta\mid\alpha} =\sum_i j_ix_i^2 =2(12)=24.

The intercept information is $\sum_i j_i=36$, and its cross-information with $\beta$ is zero. The approximate standard error for $\beta$ is $1/\sqrt{24}=0.2041$ log units per covariate unit.

Now place six subjects each at $x=-0.5,0,0.5$, keeping the same number of subjects and the same $j_i$. The sum of squared covariates is $6(0.25)+6(0)+6(0.25)=3$, so the effect information is $2(3)=6$. Its approximate standard error is $1/\sqrt{6}=0.4082$. Halving the covariate range has reduced the information by a factor of four and doubled the approximate standard error.

This comparison holds the per-subject information fixed by assumption. In an actual model, changing covariate values may also change $j_i$, for example by changing predicted concentrations or measurement variability. In that case, both the covariate spread and the subject-specific information must be recalculated.

For several covariates, the same logic extends to a weighted covariance matrix. Write $\eta_i=\alpha+\boldsymbol{\beta}^{T}\mathbf{x}_i$, where $\mathbf{x}_i$ is a vector of covariates. With scalar $\eta_i$ and subject-level information $j_i$, eliminating the intercept gives information for the covariate effects equal to

\sum_i j_i(\mathbf{x}_i-\bar{\mathbf{x}}_w) (\mathbf{x}_i-\bar{\mathbf{x}}_w)^T, \qquad \bar{\mathbf{x}}_w=\frac{\sum_i j_i\mathbf{x}_i}{\sum_i j_i}.

The diagonal entries reflect weighted spread in each covariate; off-diagonal entries reflect weighted association between covariates. If two covariates move together closely, their effects may be difficult to distinguish even if each has substantial marginal spread. The information matrix, rather than either covariate's spread considered alone, describes that separation.

Worked Example: Converting Information to a Log-Scale Covariate Effect

Suppose the typical clearance at $x=0$ is $\theta_0=10$ L/h, and each subject contributes $j_{\theta,i}=0.04$ $(\text{L/h})^{-2}$ information for their individual typical clearance. At the nominal value $\theta_i=10$ L/h, the information for $\eta_i=\log(\theta_i)$ is

j_i=\theta_i^2j_{\theta,i} =(10\ \text{L/h})^2 \left[0.04\ (\text{L/h})^{-2}\right]=4.

Suppose 12 subjects are allocated evenly at $x=-1$ and $x=1$. The weighted mean covariate is zero and $\sum j_ix_i^2=12(4)(1)=48$. Thus the information for $\beta$, after estimating the baseline, is 48 and its local approximate standard error is $1/\sqrt{48}=0.1443$ per covariate unit.

If the estimated effect were $\beta=0.10$, a one-unit increase in $x$ would correspond to a clearance ratio of $\exp(0.10)=1.105$, or approximately a 10.5% increase. The information calculation is on the log-effect scale; it does not itself establish that an effect is present or that the assumed relationship is correct.

General Parameter Vectors and Practical Checks

The log-linear scalar example is useful because its effect information has a simple weighted-variance form. In a general model, let $\boldsymbol{\phi}_i=\mathbf{h}(\boldsymbol{\gamma},x_i)$ denote the subject-specific model parameters as a function of population parameters $\boldsymbol{\gamma}$ and covariate $x_i$. Let $\mathcal I_i$ be the subject-level information for $\boldsymbol{\phi}_i$, and let $D_i=\partial\boldsymbol{\phi}_i/\partial\boldsymbol{\gamma}^{T}$ be the Jacobian. The local population information contribution is

\mathcal I_{\gamma}=\sum_i D_i^T\mathcal I_iD_i.

This result applies the chain rule to the subject-level information. It requires differentiability at the nominal values, independent subject contributions, and a valid subject-level information calculation for the model being used. If $\boldsymbol{\gamma}$ includes both covariate effects and other unknown parameters, the effect information after adjusting for those nuisance parameters is obtained by the corresponding Schur complement, not merely by taking the effect-effect block.

A useful design comparison therefore has several steps: calculate or approximate each subject's information at the proposed covariate value; map it through the parameter-to-population-parameter Jacobian; sum contributions; and account for jointly estimated parameters. This is consistent with the general information framework; it is not a replacement for evaluating the full model-specific population information.

Check for covariate overlap and spread. A nearly constant covariate has little information for its effect. For multiple covariates, inspect whether their design values provide distinct contrasts.
Check the weighted, not only raw, distribution. Subjects with different $j_i$ do not contribute equally. A broad range concentrated among subjects with little subject-level information may be less useful than the raw range suggests.
Check the parameterization and units. Rescaling $x$ changes the numerical value and units of $\beta$. Changing from $\theta$ to $\log(\theta)$ also changes information through the Jacobian.
Check model adequacy across the proposed range. The approximation is local. A wider range can improve slope information under the assumed model, but does not protect against a misspecified functional form or unsupported extrapolation.

A common mistake is to conclude that $\sum_i j_ix_i^2$ is the information for the covariate effect, then recommend a design with large covariate values without considering the baseline parameter. That quantity is only the unadjusted effect-effect entry. The correction is to subtract the information shared with the baseline, giving $\sum_i j_i(x_i-\bar{x}_w)^2$. A second mistake is to assume that centering creates information: centering can make the intercept refer to a useful covariate value and can remove intercept-effect cross-information, but it leaves the slope information unchanged.

These approximations have limitations. Fisher information is local to the nominal model and is not a direct guarantee of estimator performance in small samples. The simple formulas assume the subject-level $j_i$ values are appropriate for the design and model at hand. More complex dependence, random-effect structures, or parameter interactions can alter population information and require the corresponding full calculation. Information also measures local curvature, not whether the covariate effect is scientifically important or causally interpretable.

Check Your Understanding

Use the weighted-spread result and the assumptions stated above to answer these questions.

In the log-linear model, all 15 subjects have $x_i=2$. What is the information for $\beta$ after estimating $\alpha$, regardless of the positive values of $j_i$?
With equal $j_i=3$, four subjects have $x=0$ and four have $x=1$. Calculate the information for $\beta$ after accounting for $\alpha$.
Why can $\sum_i j_ix_i^2$ overstate the information available for estimating $\beta$?
If the covariate is recentered by subtracting a constant from every $x_i$, does the slope information change? Explain.

Solutions. (1) The weighted covariate values are all identical, so $\bar{x}_w=2$ and $\sum_i j_i(x_i-\bar{x}_w)^2=0$. The covariate effect is not separable from the baseline. (2) Here $\sum j_i=24$, $\sum j_ix_i=12$, and $\sum j_ix_i^2=12$, so $\mathcal I_{\beta\mid\alpha}=12-12^2/24=6$. (3) It is the unadjusted effect-effect entry; part of it may be shared with the estimated baseline, so the Schur-complement subtraction is needed. (4) No. Recentring changes the intercept reference and the intercept-effect cross-information, but the weighted deviations from their weighted mean, and hence the slope information, remain unchanged.

Information for Covariate Effects

What You'll Learn