Information Matrices For Approximate Designs

From Candidate Settings to a Design Matrix

In a pharmacometric study, a design specifies where observations will be collected: for example, at which doses, times, or regimens. Different settings can provide different amounts and patterns of information about the model parameters. An approximate design represents a plan using weights on candidate settings, so its information matrix can be calculated before committing to integer subject or observation counts.

The key calculation is a weighted sum: evaluate the per-observation Fisher information matrix at each support point, multiply each matrix by that point’s design weight, and add. This representation makes designs with many support points manageable and separates the information contributed by a setting from how much of the study is allocated to that setting.

Definition: An approximate design $\xi$ on a design space $\mathcal{X}$ is a probability distribution over design settings. For a finite design, it is written as support points $x_1,\ldots,x_k$ with weights $w_1,\ldots,w_k$, where $w_i\geq 0$ and $\sum_{i=1}^k w_i=1$. If $I(x_i;\theta_0)$ is the Fisher information matrix from one observation at setting $x_i$, the design information matrix per observation at nominal parameter value $\theta_0$ is $$ M(\xi;\theta_0)=\sum_{i=1}^k w_i I(x_i;\theta_0). $$

The weights are proportions of the design, not information multipliers that replace the information matrices. Each matrix $I(x_i;\theta_0)$ must first be calculated using the observation model, then multiplied by $w_i$. For a continuous design space, the corresponding expression is $M(\xi;\theta_0)=\int_{\mathcal{X}} I(x;\theta_0)\,d\xi(x)$. The finite weighted sum is the special case in which the design places all its weight on finitely many settings.

Here $x_i$ denotes a design setting, which may represent more than one scalar value—for example, a dose together with a sampling time. The matrix dimension equals the number of model parameters. As in local Fisher information, the information generally depends on the nominal parameter value $\theta_0$, particularly in nonlinear models.

Why the Weighted Sum Represents Information

Suppose a study has $n$ independent observations, with settings fixed by the design, and the observation model is correctly specified. For observation $j$ at setting $x_j$, let $I(x_j;\theta_0)$ be its expected Fisher information matrix. Using the expected negative Hessian definition, the total log likelihood is the sum of the observation log likelihoods, so its negative Hessian is also a sum. Taking expectations therefore gives

I_{\mathrm{total}}(\theta_0)=\sum_{j=1}^n I(x_j;\theta_0).

This additivity of expected negative Hessian information follows from the sum of log likelihoods and does not require the score cross-products to vanish. If instead information is calculated as the covariance matrix of the total score, the familiar equality with expected negative Hessian requires suitable regularity conditions. In particular, parameter-independent support and valid differentiation under the integral are standard conditions ensuring each score has mean zero. Under these conditions and independence, cross-products of scores from distinct observations have expectation zero, so score-covariance information is additive as well.

An exact design with $n_i$ observations at setting $x_i$ has $\sum_i n_i=n$. Its empirical weights are $w_i=n_i/n$, and its total information is

I_{\mathrm{total}}(\theta_0) =\sum_{i=1}^k n_i I(x_i;\theta_0) =n\sum_{i=1}^k w_i I(x_i;\theta_0) =nM(\xi;\theta_0).

The approximate design treats the weights as freely chosen proportions, even if $nw_i$ is not an integer. It therefore describes a continuous relaxation of the allocation problem. An actual study must use feasible counts; its empirical weights may differ slightly from a proposed approximate design.

Assumptions and scope: The weighted-sum formula applies when the design weights describe the relative frequency of settings and $I(x;\theta_0)$ is the appropriate per-observation information under the specified model. The total-information scaling $nM(\xi;\theta_0)$ assumes observations contribute independently under that model and that the design proportions apply to the $n$ observations. With correlated repeated observations, information should be calculated for the joint observation unit or joint likelihood; simply summing marginal per-observation matrices can be wrong.

Worked Examples of Design Information

Worked Example: Two-Point Linear Regression Design

Hypothetical example. Suppose each response follows $Y=\beta_0+\beta_1x+\varepsilon$, where independent errors satisfy $\varepsilon\sim N(0,4)$. Consider an approximate design with weight $0.25$ at $x=-1$ and weight $0.75$ at $x=1$. Calculate its information matrix per observation and the total information for $n=20$ observations.

For a normal linear model with known variance $\sigma^2=4$, the per-observation information is $\sigma^{-2}f(x)f(x)^\mathsf{T}$, where $f(x)=(1,x)^\mathsf{T}$. Thus,

I(x)=\frac{1}{4} \begin{pmatrix} 1 & x\\ x & x^2 \end{pmatrix}.

At $x=-1$, this is $I(-1)=\begin{pmatrix}0.25&-0.25\\-0.25&0.25\end{pmatrix}$. At $x=1$, it is $I(1)=\begin{pmatrix}0.25&0.25\\0.25&0.25\end{pmatrix}$. Weight each full matrix before adding:

\begin{aligned} M(\xi) &=0.25I(-1)+0.75I(1)\\ &= \begin{pmatrix}0.0625&-0.0625\\-0.0625&0.0625\end{pmatrix} + \begin{pmatrix}0.1875&0.1875\\0.1875&0.1875\end{pmatrix}\\ &= \begin{pmatrix}0.25&0.125\\0.125&0.25\end{pmatrix}. \end{aligned}

The weights sum to one, and the off-diagonal entry reflects the weighted mean of the settings, scaled by $1/4$. For $n=20$, the corresponding allocation is 5 observations at $-1$ and 15 at $1$, so

I_{\mathrm{total}}=20M(\xi) =\begin{pmatrix}5&2.5\\2.5&5\end{pmatrix}.

The per-observation matrix and total matrix are information matrices for $(\beta_0,\beta_1)$; their entries inherit the units associated with those parameters and the response variance. They should not be interpreted as parameter variances.

Worked Example: Locally Weighted Logistic Regression Design

Hypothetical example. Suppose a binary response at setting $x$ has success probability $p(x)=\{1+\exp[-(\beta_0+\beta_1x)]\}^{-1}$. At nominal values $\beta_0=0$ and $\beta_1=\log(3)$, consider weights $0.4$ at $x=0$ and $0.6$ at $x=1$. Find the design information matrix and total information for $n=40$.

For one Bernoulli observation, the Fisher information is $p(x)\{1-p(x)\}f(x)f(x)^\mathsf{T}$, with $f(x)=(1,x)^\mathsf{T}$. At $x=0$, $p(0)=0.5$, so $p(0)\{1-p(0)\}=0.25$. At $x=1$, $p(1)=3/4=0.75$, so $p(1)\{1-p(1)\}=0.1875$. Therefore,

I(0)= \begin{pmatrix}0.25&0\\0&0\end{pmatrix}, \qquad I(1)= \begin{pmatrix}0.1875&0.1875\\0.1875&0.1875\end{pmatrix}.

The design matrix is

\begin{aligned} M(\xi) &=0.4I(0)+0.6I(1)\\ &=\begin{pmatrix}0.1&0\\0&0\end{pmatrix} +\begin{pmatrix}0.1125&0.1125\\0.1125&0.1125\end{pmatrix}\\ &=\begin{pmatrix}0.2125&0.1125\\0.1125&0.1125\end{pmatrix}. \end{aligned}

For 40 observations, the allocation is 16 at $x=0$ and 24 at $x=1$, giving $I_{\mathrm{total}}=\begin{pmatrix}8.5&4.5\\4.5&4.5\end{pmatrix}$. The matrix is local: changing the nominal parameter values changes the probabilities, the per-setting information, and hence the design matrix. The determinant of $M$ is $0.2125(0.1125)-(0.1125)^2=0.01125>0$, so this two-parameter information matrix is nonsingular at the stated nominal values.

Worked Example: Approximate Weights and Integer Allocations

Hypothetical example. In the linear model $Y=\beta_0+\beta_1x+\varepsilon$ with independent errors of known variance $1$, consider weights $0.25,0.50,0.25$ at $x=-2,0,2$, respectively. Calculate $M(\xi)$ and the approximate total information for $n=7$. Then compare it with a feasible allocation of 2, 3, and 2 observations.

Here $I(x)=\begin{pmatrix}1&x\\x&x^2\end{pmatrix}$. The weighted mean of $x$ is $0.25(-2)+0.50(0)+0.25(2)=0$, and the weighted mean of $x^2$ is $0.25(4)+0.50(0)+0.25(4)=2$. Thus $M(\xi)=\begin{pmatrix}1&0\\0&2\end{pmatrix}$, and the approximate total information is $7M(\xi)=\begin{pmatrix}7&0\\0&14\end{pmatrix}$.

The proposed proportions correspond to nominal counts $7(0.25)=1.75$, $7(0.50)=3.5$, and $7(0.25)=1.75$, which are not feasible integer counts. A feasible allocation of $2,3,2$ has empirical weights $2/7,3/7,2/7$. Its total information is

2I(-2)+3I(0)+2I(2) = \begin{pmatrix}7&0\\0&16\end{pmatrix}.

The intercept information is 7 in both cases, while the slope information differs: 14 under the approximate proportions and 16 under the feasible allocation. The difference arises because the feasible design uses its actual counts, not the noninteger nominal counts. For larger samples, rounding can often bring empirical proportions closer to the intended weights, but the resulting matrix must still be recalculated from the actual allocation.

Derivation for a Randomized Design Point

The same weighted average also follows when each observation’s setting is drawn from a design distribution $\xi$ that does not depend on the unknown model parameter. Conditional on the chosen setting $X=x$, let the observation’s expected information be $I(x;\theta_0)$. Averaging over the design-point distribution gives the expected information for one observation:

\begin{aligned} E_{\xi}\{I(X;\theta_0)\} &=\int_{\mathcal{X}} I(x;\theta_0)\,d\xi(x)\\ &=\sum_{i=1}^k P(X=x_i)I(x_i;\theta_0)\\ &=\sum_{i=1}^k w_i I(x_i;\theta_0) =M(\xi;\theta_0). \end{aligned}

For $n$ observations with the same design-point distribution and independent observation contributions, the expected total information is $nM(\xi;\theta_0)$. In a controlled trial, settings are often fixed by allocation rather than randomly sampled. The same weighted formula then describes the average per-observation information implied by those allocation proportions; it is not necessary to assume that the settings themselves were randomly sampled.

Checks, Alternatives, and Limitations

Several quick checks catch common errors. First, verify that all weights are nonnegative and sum to one. Second, compute each $I(x_i;\theta_0)$ before applying its weight; forgetting a variance factor or other multiplier in a per-observation matrix changes every entry of the design matrix. Third, confirm that the resulting matrix is symmetric. Since Fisher information matrices are positive semidefinite, a substantially negative eigenvalue usually indicates an arithmetic or model-form error. A singular matrix can be valid, however: it indicates that some parameter direction is not locally informed by the chosen settings.

A common mistake is to treat a support point’s matrix as though its weight were already included. For example, if $I(x)$ contains entries of $0.25$, multiplying by weight $0.25$ gives entries of $0.0625$, not $0.25$. The correction is to preserve the distinction between the per-observation matrix and its weighted contribution, then add the weighted contributions entry by entry.

An exact design lists integer counts at settings; an approximate design lists proportions and is convenient for mathematical optimization. For a fixed feasible sample size, converting proportions into counts may require rounding, and constraints such as minimum enrollment at a setting can change the feasible allocation. A direct list of observation settings is another equivalent way to compute information for an exact design: sum $I(x_j;\theta_0)$ over observations. Grouping identical settings and using counts is simply a more compact representation.

The matrix $M(\xi;\theta_0)$ describes local information under the assumed model and nominal parameters. It does not guarantee that the model is correct, that the nominal values are accurate, or that the observations will be independent. In repeated-measures or cluster designs, the joint information for a subject or cluster may depend on the entire schedule and covariance structure; calculate that joint contribution before weighting or summing across independent units. This tutorial establishes how to construct the design information matrix, not how to rank candidate matrices using an optimality criterion.

Check Your Understanding

Answer each question, then compare with the solutions below.

A design uses weights $0.3$ and $0.7$. What two conditions must these weights satisfy to define an approximate design on two support points?
At two settings, suppose $I(x_1)=\begin{pmatrix}2&0\\0&0\end{pmatrix}$, $I(x_2)=\begin{pmatrix}0&0\\0&4\end{pmatrix}$, and the weights are $0.25$ and $0.75$. Find $M(\xi)$.
For $n=12$, a design has empirical proportions $w_1=1/3$ and $w_2=2/3$. How many observations are assigned to each setting, and how is total information related to $M(\xi)$?
Why does independence by itself not justify dropping cross-products when deriving information from the covariance of the total score?
An approximate design for $n=5$ assigns weight $0.4$ to one setting and $0.6$ to another. Are the implied counts feasible integers, and what should be done to evaluate an actual allocation?

Solutions. (1) Each weight must be nonnegative, and the weights must sum to one; here $0.3+0.7=1$. (2) $M=0.25I(x_1)+0.75I(x_2)=\begin{pmatrix}0.5&0\\0&3\end{pmatrix}$. (3) The counts are $12/3=4$ and $24/3=8$, so $I_{\mathrm{total}}=12M(\xi)$. (4) Independence makes distinct scores’ cross-product expectations factor into products of their means. Those terms vanish when each score has mean zero, which requires appropriate regularity conditions; independence alone does not ensure zero-mean scores. (5) The implied counts are 2 and 3, so they are feasible integers. For any actual allocation, calculate total information from the integer counts, $\sum_i n_i I(x_i;\theta_0)$, or equivalently from its empirical weights and $nM(\xi;\theta_0)$.