Comparing Designs With Information Criteria

Why Compare Designs With More Than One Criterion?

A candidate QSP experiment can provide strong information about some parameter combinations and weak information about others. A single summary of its Fisher information matrix cannot describe every precision objective equally well. D-, A-, and E-optimality are three standard criteria for comparing candidate designs: each reduces a matrix to a score, but each emphasizes a different feature of parameter uncertainty.

The comparison is local: the candidate matrices are evaluated for a specified model, parameterization, and nominal parameter vector. The criteria are meaningful only when the matrices refer to the same parameter set and use the same units and parameterization. They do not guarantee that a design will meet every parameter-specific precision target.

The inverse \(I^{-1}\) is the local covariance approximation for parameter estimates under regularity conditions and a sufficiently informative likelihood. Consequently, the A criterion is the sum of the diagonal elements of that covariance approximation: the sum of the marginal variances on the stated parameter scale. E-optimality emphasizes the weakest-informed direction in parameter space. If \(I\) is singular, \(\det(I)=0\), \(\lambda_{\min}(I)=0\), and \(I^{-1}\) does not exist; the standard A criterion is then undefined.

What the Three Criteria Reward

The eigenvalues of a positive-definite \(I\), written \(\lambda_1,\ldots,\lambda_p\), describe information in orthogonal parameter directions. The determinant is their product, \(\det(I)=\prod_{j=1}^{p}\lambda_j\). D-optimality therefore rewards large overall information volume. In the local normal approximation, an uncertainty ellipsoid based on \(I^{-1}\) has volume proportional to \(\sqrt{\det(I^{-1})}=1/\sqrt{\det(I)}\), for a fixed confidence level and parameter dimension. Maximizing \(\det(I)\) reduces that joint volume; it does not necessarily minimize the uncertainty of each individual parameter.

For A-optimality, the eigenvalues of \(I^{-1}\) are \(1/\lambda_j\), so \(\operatorname{tr}(I^{-1})=\sum_{j=1}^{p}1/\lambda_j\). Minimizing this sum favors lower total marginal variance, but the result depends on the scales used for the parameters. E-optimality maximizes the smallest \(\lambda_j\); equivalently, it minimizes the largest eigenvalue of the local covariance approximation. It is especially sensitive to a poorly informed direction.

These interpretations rely on the same local approximation: the likelihood is sufficiently regular near the parameter value of interest, the parameters are locally identifiable, and the information matrix is a useful description of curvature. When these conditions are doubtful, a criterion score should not be treated as a complete description of uncertainty.

Criteria Can Prefer Different Designs

There is no contradiction if the criteria rank candidates differently: they answer different design questions. D-optimality focuses on joint volume, A-optimality on summed marginal variance, and E-optimality on the weakest direction. A design with very high information in one direction can have a large determinant while leaving another direction poorly informed. The next example makes that trade-off explicit.

A Short Justification for the Criteria

The mathematical forms follow from the local covariance approximation. If the covariance is approximated by \(I^{-1}\), then the product of its eigenvalues is \(\det(I^{-1})=1/\det(I)\). The square root of this product determines the volume scaling of a covariance ellipsoid, which justifies maximizing \(\det(I)\) for a small joint uncertainty volume. Also, the sum of the covariance eigenvalues is \(\operatorname{tr}(I^{-1})=\sum_j1/\lambda_j\), giving the A criterion its summed-variance interpretation. Finally, the largest covariance eigenvalue is \(1/\lambda_{\min}(I)\); reducing the worst-direction variance is therefore equivalent to maximizing \(\lambda_{\min}(I)\).

These are local geometric summaries, not guarantees of global precision. In a nonlinear model, the likelihood may be asymmetric or poorly approximated by a quadratic over a relevant parameter range. The information matrix may also depend strongly on the nominal parameter values used to evaluate the candidate design.

Parameter Scaling and Fair Comparisons

Parameter units matter when interpreting these scores. For example, expressing a concentration in millimolar rather than micromolar changes numerical parameter values and hence the numerical entries of the information matrix. Under a common nonsingular linear reparameterization \(\theta=J\phi\), the information matrix transforms as \(I_{\phi}=J^{T}I_{\theta}J\). Its determinant becomes \(\det(I_{\phi})=\det(J)^2\det(I_{\theta})\). Since the same transformation multiplies every candidate's determinant by the same positive factor, D-optimal rankings are unchanged under this common transformation. A- and E-optimal rankings need not be unchanged, because a change of scale can alter the relative weighting of parameter directions in the trace and eigenvalue.

Practical Checks, Alternatives, and Limitations

Before using a criterion to choose a design, verify that the candidate matrices are symmetric, refer to the same parameters and nominal values, and are positive definite if the standard inverse-based A score is to be used. Inspect the eigenvalues as well as the scalar score: a near-zero minimum eigenvalue can reveal a weak direction that a determinant alone may obscure. For determinants that are extremely large or small, compare log determinants to reduce numerical overflow or underflow; this does not change the ranking when all candidates have the same dimension.

A common mistake is to say, “Design \(U\) has the largest determinant, so it gives the best precision for every parameter.” That conclusion is too broad. The determinant summarizes joint volume, not every marginal variance or the weakest direction; the example above shows that D-optimality can favor a design that E-optimality rejects. The correction is to state the objective explicitly and examine the individual parameter or direction of scientific concern when that objective requires it.

A parameter-specific precision target may be better assessed by the corresponding diagonal element of \(I^{-1}\) or by a criterion that weights parameters according to their scientific importance. Such comparisons still require a stated parameterization and a defensible local approximation. D-, A-, and E-optimality are useful summaries, but none replaces checking whether the model and information calculation are adequate for the intended inference.