Why the Parameter Values Used in Design Matter
A design is not informative in the abstract: its information depends on the model parameters that govern the observations. In pharmacometrics, those parameters might include clearance, volume, absorption rate, or residual error. Before data are collected, their values are not known exactly. A design scored at one chosen parameter vector therefore answers a conditional question: how informative would this design be if the chosen values were the true values?
That conditional score is useful, but it does not describe how a design performs when the true values differ from the chosen values. An uncertainty-aware score instead incorporates a distribution or a set of plausible parameter values. The choice between these approaches is not just a computational detail. It changes what “good design” means, and can change which candidate design is preferred.
Local Criteria and Their Assumptions
Here, \(\mathcal{I}(d;\boldsymbol{\theta})\) denotes the Fisher information matrix for the parameters under design \(d\) when their value is \(\boldsymbol{\theta}\). Its calculation depends on the observation model, the sampling plan, and any approximations used to obtain information. The earlier material on population Fisher information develops ways of constructing such matrices for pharmacometric designs.
A local comparison requires that the candidate designs use the same parameterization and target the same estimands. For a determinant-based comparison, the information matrix must also be positive definite for every design being scored; otherwise its determinant is zero and the usual finite-volume interpretation fails. Information-based uncertainty approximations additionally rely on regularity conditions under which the estimator is approximately unbiased and its covariance is approximated by the inverse information matrix. These are approximations, not guarantees for small samples or poorly identified models.
One common local score is D-optimality: maximize \(\det\{\mathcal{I}(d;\boldsymbol{\theta}_0)\}\), or equivalently its logarithm. For \(p\) parameters, the determinant describes the overall information scale across parameter directions. Under the regular large-sample approximation, the covariance matrix is approximately \(\mathcal{I}^{-1}\); a larger information determinant corresponds to a smaller covariance-ellipsoid volume. This does not mean every parameter’s variance is smaller, nor does it directly score prediction or dosing performance.
Why Local Scores Can Change
A local criterion is transparent: it says which parameter vector was used and how each candidate performed there. Its weakness is equally direct. If small plausible changes in \(\boldsymbol{\theta}\) substantially alter the information matrix, the local ranking may be fragile. An uncertainty-aware criterion incorporates more than one plausible value before comparing designs.
There are two broad ways to express that uncertainty. A distribution assigns weights or probabilities to parameter values, and a criterion can average performance with those weights. A set specifies values or ranges considered plausible without necessarily assigning probabilities. A score can then summarize performance across that set. The distribution or set is an input to the design evaluation; it is not automatically supplied by the likelihood or justified simply by calling a design robust.
Worked Example: Choosing a Local D-Optimal Design
Suppose two hypothetical candidate designs target the same two parameters, and their information matrices at the nominal parameter vector are \(\mathcal{I}(d_A;\boldsymbol{\theta}_0)=\begin{pmatrix}8&2\\2&3\end{pmatrix}\) and \(\mathcal{I}(d_B;\boldsymbol{\theta}_0)=\begin{pmatrix}6&0\\0&4\end{pmatrix}\). Assume both matrices are positive definite and both designs have the same resource cost. Compare them by the local D-criterion.
Design A. Its determinant is \(\det(\mathcal{I}_A)=8(3)-2(2)=24-4=20\).
Design B. Its determinant is \(\det(\mathcal{I}_B)=6(4)-0(0)=24\). Thus \(\log\det(\mathcal{I}_A)=\log(20)\approx2.996\), whereas \(\log\det(\mathcal{I}_B)=\log(24)\approx3.178\). The local D-criterion prefers design B at \(\boldsymbol{\theta}_0\).
The matrices are positive definite: their leading diagonal entries are positive and their determinants, 20 and 24, are positive. The conclusion is only about the chosen nominal vector and the stated criterion. It does not establish that B is better at other parameter values or under a different target, such as the precision of one particular parameter.
What Is Aggregated Under Parameter Uncertainty?
An uncertainty-aware criterion needs an aggregation rule as well as plausible parameter values. For example, if a distribution \(q(\boldsymbol{\theta})\) is specified, one could average a local score \(U(d;\boldsymbol{\theta})\):
Alternatively, one could first average information matrices and then apply a matrix criterion: \(\phi\!\left(\mathbb{E}_{q}[\mathcal{I}(d;\boldsymbol{\theta})]\right)\). These are generally different. In particular, averaging information and then taking a logarithmic determinant is not the same as averaging the logarithmic determinant. The first evaluates a criterion on mean information; the second averages local criterion values. Neither is universally correct: the design question determines which quantity is relevant.
For a plausible set \(\Theta\), a score can instead summarize \(U(d;\boldsymbol{\theta})\) over \(\boldsymbol{\theta}\in\Theta\), for example by examining its range or its least favorable value. A set-based summary does not require probability weights, but it does depend on which values are included. A wide set can make a design appear less attractive simply because it includes extreme values; a narrow set can conceal sensitivity outside its boundaries.
Worked Example: Averaging Information Versus Averaging Its Logarithm
Consider one scalar parameter and two hypothetical designs. At two plausible parameter values, design A has information 10 and 2, while design B has information 6 and 5. Suppose the two values receive equal weights. For scalar information \(I\), the local D-style score is \(\log I\).
Average information first. Design A has mean information \((10+2)/2=6\), giving score \(\log(6)\approx1.792\). Design B has mean information \((6+5)/2=5.5\), giving score \(\log(5.5)\approx1.705\). This aggregation prefers A.
Average the local scores. For A, \(\{\log(10)+\log(2)\}/2=(2.303+0.6931)/2=1.498\). For B, \(\{\log(6)+\log(5)\}/2=(1.792+1.609)/2=1.701\). This aggregation prefers B.
The reversal is not an arithmetic inconsistency: the two calculations answer different questions. One scores the mean information; the other scores the mean of the local log-information values. The equal weights here are hypothetical and supplied for illustration. In an application, the uncertainty representation and the intended aggregation must be justified rather than inferred from these numbers.
Worked Example: Comparing Designs Across a Plausible Set
Suppose a scalar parameter is considered plausible at two endpoints, \(\theta_L\) and \(\theta_H\). Two hypothetical designs have information values \(I(d_C;\theta_L)=9\), \(I(d_C;\theta_H)=3\), \(I(d_D;\theta_L)=6\), and \(I(d_D;\theta_H)=5\). The nominal value is \(\theta_L\), but the set-based comparison also reports the smallest information across these two values.
At the nominal value, C has information 9 and D has information 6, so a local comparison at \(\theta_L\) prefers C. Across the stated set, C’s smallest information is \(\min(9,3)=3\), while D’s is \(\min(6,5)=5\). Thus D has the higher least favorable information among the two candidates over this particular set.
This comparison illustrates how a set-based summary can favor a design with less nominal information but more even performance. It does not establish performance between the endpoints, outside the set, or under a probability-weighted objective. If the model’s information is not known to vary monotonically, checking endpoints alone would not establish the minimum over a continuous range.
How to Choose and Diagnose a Criterion
The local and uncertainty-aware approaches are useful for different purposes. A local criterion is appropriate when a nominal parameter vector is well supported and the design decision is explicitly conditional on it, or when a transparent reference comparison is needed. An uncertainty-aware criterion is more relevant when plausible parameter variation is consequential and the design objective should reflect that variation. In either case, the score should match the scientific target: information about all parameters, precision of selected parameters, or another stated objective.
| Approach | What is evaluated | What must be specified | Main caution |
|---|---|---|---|
| Local | A criterion at one nominal parameter vector | The nominal vector and local score | May be sensitive to misspecification of the nominal values |
| Distribution-based | Performance aggregated over weighted parameter values | The uncertainty distribution and aggregation rule | Results depend on both weights and which quantity is averaged |
| Set-based | Performance summarized across plausible values | The plausible set and summary rule | Results depend on set boundaries and the chosen summary |
Useful diagnostics start by evaluating candidate designs at several plausible parameter values, even when the final criterion is local. Compare the resulting scores and rankings; a large spread or a reversal signals sensitivity worth reporting. Check that information matrices remain well-conditioned and that the target parameters remain identifiable. A determinant can hide a weak direction if another direction has very high information, so inspect eigenvalues or parameter-specific uncertainty when those matter to the scientific question.
For uncertainty-aware evaluations, also check the provenance and coverage of the distribution or set, the sensitivity to reasonable changes in it, and whether every design is scored on the same parameter values and scale. A numerical score can look precise while depending heavily on arbitrary weights or an unsupported range. Information calculations may also be approximate; if the approximation is known to fail in relevant parts of parameter space, aggregating those scores does not repair the failure.
Common Pitfalls and Limits
- Treating the nominal value as certain. A local optimum is optimal only for the stated local criterion and nominal vector, among the designs considered. It is not automatically optimal over plausible parameter values.
- Confusing average information with average performance. Applying a nonlinear criterion after averaging matrices generally differs from averaging the criterion itself. State which operation defines the objective.
- Assuming uncertainty weights are objective facts. A distribution-based score depends on its weights; a set-based score depends on its boundaries and aggregation rule. These choices need scientific justification.
- Interpreting a larger determinant as universal superiority. D-optimality summarizes joint information in a particular way. It need not minimize the variance of every parameter or optimize a downstream dosing decision.
- Overlooking model or approximation failure. A robust-looking score computed from inaccurate information matrices may still recommend a poor design. Check the information calculation as well as its sensitivity to parameter values.
No criterion removes the need to specify the scientific objective, model, candidate designs, and uncertainty representation. Local scores give a clear reference point; uncertainty-aware scores make parameter sensitivity part of the objective. Reporting both can be informative, provided each score is labelled and interpreted according to what it actually aggregates.
Check Your Understanding
Answer each question, then compare with the solutions below.
- What parameter values enter a local design criterion, and what does its score claim about performance?
- For scalar information values 4 and 16 with equal weights, calculate the average information and the average log-information.
- Why might a set-based summary prefer a design that is not preferred at the nominal value?
- Name two checks that can reveal fragility or failure in a design comparison.
Solutions. (1) A local criterion evaluates information at one specified nominal vector; its score compares designs conditional on that vector, not across all plausible values. (2) The average information is \((4+16)/2=10\). The average log-information is \(\{\log(4)+\log(16)\}/2=(1.386+2.773)/2=2.080\). For comparison, the log of average information is \(\log(10)=2.303\), so the two aggregations differ. (3) The design may have lower information at the nominal value but perform more evenly across the stated set; a summary emphasizing performance across that set can therefore rank it higher. (4) Examples include checking rankings at multiple plausible parameter values, inspecting information-matrix eigenvalues or conditioning, and assessing sensitivity to the chosen distribution, set, or information approximation.