Bayesian Expected-Utility Design

Why Average Design Performance Over a Prior?

A local design evaluates performance at one specified parameter value. But when parameters are uncertain before data collection, a design that is excellent at one value may perform poorly at another. Bayesian expected-utility design addresses this by averaging a design’s utility over a prior distribution for the unknown parameter. The result is a single score for comparing candidate designs, with the prior making explicit how different parameter values are weighted.

The average is taken over the utility of a design at each parameter value. That detail matters: if utility is a nonlinear function of the information matrix, averaging the matrices first and then applying the utility generally gives a different answer. The two procedures can even rank designs differently.

Definition: Let $\theta$ denote the unknown model parameter, let $\xi$ denote a candidate design, and let $U(\xi,\theta)$ be a utility for that design at parameter value $\theta$, with larger values preferred. Given a prior distribution $\pi(\theta)$, the Bayesian expected utility of $\xi$ is $$ \overline{U}(\xi)=\mathbb{E}_{\theta\sim\pi}\!\left[U(\xi,\theta)\right] =\int U(\xi,\theta)\,\pi(\theta)\,d\theta. $$ For a discrete prior with parameter values $\theta_1,\ldots,\theta_K$ and probabilities $w_1,\ldots,w_K$, this is $\overline{U}(\xi)=\sum_{k=1}^{K}w_k U(\xi,\theta_k)$, where $w_k\geq 0$ and $\sum_k w_k=1$. An expected-utility design maximizes $\overline{U}(\xi)$ over the permitted design set $\Xi$.

The prior represents uncertainty about $\theta$ before the proposed study’s data are observed. It may be continuous or discrete, but it must be a valid probability distribution. The calculation also requires a specified design set, a model that defines performance at each relevant parameter value, and a utility whose prior expectation exists. If the utility is based on an information matrix, that matrix must be computable for the design and parameter values considered.

The convention that larger utility is better is convenient but not essential. If the starting criterion is a loss $L(\xi,\theta)$ to be minimized, use $U(\xi,\theta)=-L(\xi,\theta)$, or minimize the expected loss $\mathbb{E}_{\theta\sim\pi}[L(\xi,\theta)]$ directly. State which convention is being used; otherwise it is easy to reverse the design ranking by mistake.

Expected Criterion Versus Criterion at Expected Information

Suppose the model supplies an information matrix $I(\xi;\theta)$, and a criterion $C$ converts that matrix to a design score. The expected-utility calculation based on that criterion is

\mathbb{E}_{\theta\sim\pi}\!\left[C\{I(\xi;\theta)\}\right].

A different calculation first averages the information matrices and then evaluates the criterion:

C\!\left\{\mathbb{E}_{\theta\sim\pi}[I(\xi;\theta)]\right\}.

These expressions are equal if, for example, $C$ is linear over the relevant matrices. They are not generally equal for nonlinear criteria such as the log-determinant or the negative trace of an inverse. Thus, “expected information” is not a substitute for “expected utility” unless equality is justified for the criterion and model in question.

For D-optimality, a common utility is $U(\xi,\theta)=\log\det I(\xi;\theta)$. It rewards designs that provide information across the parameter directions represented in the matrix. To use this utility as written, the information matrix must be positive definite at the parameter values receiving prior weight, and its log-determinant must have a finite prior expectation. If the matrix is singular, $\log\det I$ is not a finite utility; the design or criterion must be handled explicitly rather than silently treating the value as ordinary.

Worked Example: Averaging a D-Optimal Utility Over Two Parameter Values

Consider two hypothetical candidate designs, A and B, and a discrete prior assigning probability $0.5$ to each of two parameter values, $\theta_1$ and $\theta_2$. For this example, use the scalar information $I(\xi;\theta)$ and the utility $U(\xi,\theta)=\log I(\xi;\theta)$. Suppose the information values are:

Design	$I(\xi;\theta_1)$	$I(\xi;\theta_2)$
A	1	16
B	3	3

All information values are positive, so the log utility is defined. For design A, the expected utility is

\overline{U}(\mathrm{A}) =0.5\log(1)+0.5\log(16) =0+0.5(2.7726) =1.3863.

For design B, the information and utility are the same at both parameter values:

\overline{U}(\mathrm{B}) =0.5\log(3)+0.5\log(3) =\log(3) =1.0986.

Therefore, under this prior and utility, design A has the larger expected utility and is preferred. This does not say that A is better at every parameter value: at $\theta_1$, its information is 1, compared with 3 for B. The preference reflects the prior-weighted average of the log-information values.

Why the Order of Averaging Matters

The difference between the two calculations is not just a matter of terminology. A nonlinear transformation changes the contribution of different information values, and matrices can also differ in which parameter directions they inform. The following comparison makes this concrete for a matrix criterion.

Worked Example: The Two Calculations Can Reverse a Design Ranking

Suppose a hypothetical two-point prior assigns probability $0.5$ to each of two parameter values. Compare designs A and B using $C(I)=\log\det(I)$. Their information matrices are:

I(\mathrm{A};\theta_1)= \begin{pmatrix}9&0\\0&1\end{pmatrix}, \qquad I(\mathrm{A};\theta_2)= \begin{pmatrix}1&0\\0&9\end{pmatrix}, \qquad I(\mathrm{B};\theta_1)=I(\mathrm{B};\theta_2)= \begin{pmatrix}4&0\\0&4\end{pmatrix}.

Every matrix is positive definite, so its log-determinant is defined. For design A, each determinant is $9$, giving

\mathbb{E}_{\theta\sim\pi}[\log\det I(\mathrm{A};\theta)] =0.5\log(9)+0.5\log(9) =\log(9) =2.1972.

For design B, both determinants are $16$, so its expected utility is $\log(16)=2.7726$. Expected utility therefore prefers B.

Now average the matrices first. For A, the expected information matrix is

\mathbb{E}_{\theta\sim\pi}[I(\mathrm{A};\theta)] =0.5 \begin{pmatrix}9&0\\0&1\end{pmatrix} +0.5 \begin{pmatrix}1&0\\0&9\end{pmatrix} = \begin{pmatrix}5&0\\0&5\end{pmatrix}.

Its log-determinant is $\log(25)=3.2189$. For B, the expected matrix is unchanged and its log-determinant is $\log(16)=2.7726$. Applying the criterion to expected information therefore prefers A, the reverse ranking.

The reversal occurs because A’s information is concentrated in different directions at the two parameter values. Averaging its matrices produces a balanced matrix, but the log-determinant of that average is not the average of the log-determinants. If the goal is prior-averaged log-determinant utility, the first calculation answers that question; the second answers a different one.

A Short Justification for the Log-Determinant Difference

For positive-definite matrices, log-determinant is concave. This explains why, when the relevant expectations exist,

\mathbb{E}_{\theta\sim\pi}[\log\det I(\xi;\theta)] \leq \log\det\!\left(\mathbb{E}_{\theta\sim\pi}[I(\xi;\theta)]\right).

A direct justification uses a line between two positive-definite matrices. Let $A_t=A+tH$ remain positive definite for $0\leq t\leq1$, where $H$ is a symmetric matrix. The first derivative of $\log\det A_t$ with respect to $t$ is $\mathrm{tr}(A_t^{-1}H)$. Differentiating again gives

\frac{d^2}{dt^2}\log\det(A_t) =-\mathrm{tr}(A_t^{-1}H A_t^{-1}H)\leq 0.

To see the inequality, write the trace as the sum of squares of the entries of the symmetric matrix $A_t^{-1/2}HA_t^{-1/2}$. The trace is therefore nonnegative before the minus sign. A nonpositive second derivative establishes concavity along every such line. Jensen’s inequality for a concave function then gives the displayed expectation inequality. In general the inequality is strict when the information matrix varies across parameter values; equality holds when it is constant with prior probability one.

This result gives a useful check on a calculation, not a license to replace expected utility with utility at expected information. The upper bound may be useful as a bound, but it is not the expected log-determinant itself.

Worked Continuous-Prior Comparison

Worked Example: A Prior Average Changes Which Design Wins

Let the hypothetical scalar parameter have a uniform prior on $[1,3]$, with density $\pi(\theta)=1/2$ on that interval. Compare two candidate designs under the utility $U(\xi,\theta)=\log I(\xi;\theta)$. Suppose design A has $I(\mathrm{A};\theta)=\theta$, while design B has constant information $I(\mathrm{B};\theta)=1.95$. Both information values are positive throughout the prior support.

For A, integrate the utility over the prior:

\overline{U}(\mathrm{A}) =\frac{1}{2}\int_1^3\log(\theta)\,d\theta =\frac{1}{2}\left[\theta\log(\theta)-\theta\right]_1^3 =\frac{3\log(3)-2}{2} =0.6479.

For B, the utility is constant, so its expected value is

\overline{U}(\mathrm{B}) =\log(1.95) =0.6678.

Expected utility prefers B because $0.6678>0.6479$. To see why using expected information would give a different conclusion, first calculate the expected information for A. Under a uniform distribution on $[1,3]$, $\mathbb{E}[\theta]=(1+3)/2=2$. Thus, applying the criterion after averaging gives $\log(2)=0.6931$ for A, compared with $\log(1.95)=0.6678$ for B. That procedure prefers A. The example shows that the difference can change the selected design, even for a scalar information value.

Implementation, Diagnostics, and Limitations

In practice, expected utility may be calculated by numerical integration or by drawing parameter values from the prior. With prior draws $\theta^{(1)},\ldots,\theta^{(M)}$, a Monte Carlo estimate is

\widehat{\overline{U}}(\xi) =\frac{1}{M}\sum_{m=1}^{M}U(\xi,\theta^{(m)}).

Use the same prior draws for each candidate design when comparing them. This makes the comparison less affected by Monte Carlo noise: each design is evaluated at the same sampled parameter values. Increase the number of draws, or use more accurate quadrature where appropriate, if the estimated utility differences are small relative to numerical uncertainty.

Several checks help reveal problems before trusting a ranking:

Check the prior support. Verify that the model and utility are defined at all parameter values that receive prior weight. Pay particular attention to log-determinants, inverse matrices, and parameter boundaries.
Check integration and simulation error. Repeat the calculation with greater numerical accuracy. A design ranking that changes under modest increases in integration accuracy is not yet reliable.
Inspect performance across parameter values. A favorable average can conceal poor utility over part of the prior support. Such variation may be scientifically important even when it does not change the expected-utility calculation.
Check units and the criterion’s meaning. Confirm that candidate designs are evaluated using the same model, information definition, parameterization, and utility. Comparisons of criteria are meaningful only when they represent the same design objective.

Expected utility is not the only possible way to handle parameter uncertainty. A local design uses a specified parameter value rather than averaging over a prior; it can be appropriate when a well-supported nominal value is the intended basis for planning. Evaluating a criterion at expected information is another possible summary, but, as shown above, it is not generally the same objective as maximizing expected utility. The choice should follow the design question, not a convenience in computation.

The resulting design is conditional on the prior, utility, model, and allowed design set. A different prior can change the weighting of parameter values and thus change the preferred design. Expected utility also does not guarantee uniformly high performance across the prior support: it optimizes an average, so low utility in some regions may be offset by high utility elsewhere. These are features of the objective, not computational errors. State the prior and utility clearly so that the design’s meaning can be assessed.

Key takeaway: Bayesian expected-utility design computes the prior expectation of the design criterion itself. For nonlinear criteria, do not replace that calculation with the criterion evaluated at the expected information matrix unless equality has been established for the case at hand.

Check Your Understanding

Answer each question, then check the solutions below.

A discrete prior assigns probabilities $0.25$ and $0.75$ to two parameter values. A design has utilities 2 and 6 at those values. What is its expected utility?
For a utility defined by $U(\xi,\theta)=\log I(\xi;\theta)$, what condition is needed for the scalar log-information to be finite at a parameter value?
Two positive-definite information matrices differ across prior-supported parameter values. What inequality relates expected log-determinant to log-determinant of expected information?
Why can evaluating a criterion at expected information produce a different ranking from expected-utility design?
If two designs’ estimated expected utilities are very close, name one practical check before choosing between them.

Solutions. (1) The expected utility is $0.25(2)+0.75(6)=0.5+4.5=5$. (2) The information must be positive, $I(\xi;\theta)>0$, because the logarithm is not finite at zero or negative values. (3) By concavity of log-determinant, $\mathbb{E}_{\theta\sim\pi}[\log\det I(\xi;\theta)]\leq\log\det(\mathbb{E}_{\theta\sim\pi}[I(\xi;\theta)])$, when the expectations exist and the matrices are positive definite. (4) The criterion may be nonlinear, so applying it after averaging information does not generally equal averaging the criterion values; the two objectives can even rank designs differently. (5) Increase numerical integration accuracy or the number of prior simulation draws, and check whether the ranking is stable relative to the numerical error.

Design	\(I(\xi;\theta_1)\)	\(I(\xi;\theta_2)\)
A	1	16
B	3	3