Why Evaluate an Observation Before Collecting It?
An additional pharmacokinetic sample can improve parameter estimates, but improvement is not automatic in the sense that matters for a study. Its value depends on what the design is trying to improve: uncertainty about a parameter, expected loss from a dosing decision, or another explicitly defined objective. The sample also uses resources, such as participant time, assay capacity, and money. A principled evaluation therefore asks whether the expected improvement in the chosen objective is worth the sampling cost.
The word “expected” is essential. Before collection, the observation is unknown. Its possible values can lead to different posterior beliefs and different decisions. The value calculation averages the resulting improvements over the predictive distribution of those possible observations. It does not substitute one convenient or most likely outcome and treat that as certain.
The objective and cost must be expressed on a compatible scale before subtraction. Information gain can be measured in nats, for example, but a monetary sampling cost cannot be subtracted from nats without a stated conversion or utility scale. Alternatively, a decision objective may already be measured in monetary or clinical-loss units, allowing a cost in the same units to be subtracted directly. A positive net value favors collecting the observation under the specified assumptions; it does not establish that the observation is worthwhile under every objective.
Expected Information Gain
Let \(\theta\) denote an uncertain model parameter and \(Y\) a candidate observation. The current information is summarized by the distribution of \(\theta\), and the observation model supplies the predictive distribution of \(Y\). For each possible outcome \(y\), Bayes’ rule gives an updated distribution for \(\theta\). If uncertainty is measured by entropy, the expected information gain is the current entropy minus the predictive average of posterior entropies.
For a continuous observation, the sum is replaced by an integral over its predictive density. Entropy must be defined consistently before and after observing data. For a discrete distribution with probabilities \(p_j\), Shannon entropy is \(H=-\sum_j p_j\ln(p_j)\), measured in nats when the natural logarithm is used. This is an uncertainty summary, not a direct measure of clinical benefit.
Under a coherent probability model, expected information gain, defined as the mutual information between \(\theta\) and \(Y\), is nonnegative. When expressed as prior entropy minus expected posterior entropy, this identity requires the relevant entropies to be finite and the difference to be well-defined. A particular outcome can nevertheless leave more uncertainty than before, and sampling cost can make the net value negative. Nonnegative information gain is therefore not the same as a recommendation to collect the sample.
Worked Example: Expected Information Gain for a Binary Observation
Suppose the current probability that a parameter is in a clinically relevant high state is \(0.40\), and the probability of the low state is \(0.60\). A candidate assay result is positive or negative. Assume the probability of a positive result is \(0.60\) in the high state and \(0.40\) in the low state. A hypothetical information-equivalent sampling cost is \(0.0100\) nats.
First calculate the predictive probabilities of the assay results. For a positive result, \(P(+)=0.40(0.60)+0.60(0.40)=0.48\). Thus \(P(-)=1-0.48=0.52\). Bayes’ rule gives the posterior probability of the high state after a positive result as \(0.40(0.60)/0.48=0.50\); after a negative result it is \(0.40(0.40)/0.52=0.3077\).
The current binary entropy is \(H(\theta)=-0.40\ln(0.40)-0.60\ln(0.60)=0.6730\) nats. The posterior entropy is \(0.6931\) nats after a positive result and \(0.6172\) nats after a negative result. Therefore, using those displayed rounded entropies, expected posterior entropy is \(0.48(0.6931)+0.52(0.6172)=0.332688+0.320944=0.653632\), or \(0.6536\) nats.
The expected information gain is \(0.6730-0.6536=0.0194\) nats. Subtracting the stated cost gives a net information value of \(0.0194-0.0100=0.0094\) nats. Under this particular objective and cost conversion, the candidate has positive expected net value. The calculation averages both outcomes; it does not assume in advance that either result will occur.
Expected Value for a Dosing Decision
Information is useful in a dosing context when it can improve a decision, but an information objective and a decision objective are not interchangeable. To calculate decision value, define the available actions and a loss for each action under each possible state. The current Bayes risk is the minimum expected loss using current information. After observing \(Y=y\), calculate the minimum posterior expected loss. Average those minimized losses over possible outcomes; subtract that expected future risk from the current risk.
Here \(a\) denotes a dosing action and \(L(a,\theta)\) is its loss if the true state is \(\theta\). This quantity is the expected value of sample information before subtracting sampling cost. With the loss expressed in dollars, the value is in dollars. With a clinical-loss scale, it is in that scale. The calculation assumes the stated model, action set, and loss function adequately represent the decision being evaluated.
Worked Example: A Test That Could Change the Dose Choice
Consider two hypothetical states, high and low, with current probabilities \(0.40\) and \(0.60\). There are two dosing actions, low and high. A wrong action incurs a loss of $100, while a correct action incurs no loss. A candidate test is positive with probability \(0.80\) in the high state and \(0.20\) in the low state. Its sampling cost is $12.
Without the test, choosing the low action has expected loss \(0.40($100)+0.60($0)=$40\). Choosing the high action has expected loss \(0.60($100)+0.40($0)=$60\). The current best decision is therefore low, with Bayes risk of $40.
The predictive probability of a positive test is \(0.40(0.80)+0.60(0.20)=0.44\). Given a positive result, the probability of the high state is \(0.40(0.80)/0.44=0.7273\). The high action is then preferred, with posterior expected loss \((1-0.7273)($100)=$27.27\). The probability of a negative result is \(0.56\), and the probability of the high state after a negative result is \(0.40(0.20)/0.56=0.1429\). The low action is then preferred, with posterior expected loss \(0.1429($100)=$14.29\).
Expected risk after testing is \(0.44($27.27)+0.56($14.29)=$12.00+$8.00=$20.00\). Thus the test’s expected value before cost is \($40-$20=$20\), and its net value is \($20-$12=$8\). Under the hypothetical loss model, the test is worthwhile. This is an expected reduction in decision loss, not a guarantee that the test will improve the action for every patient.
Calculation and Interpretation
A reliable calculation begins with the objective, not with a convenient statistic. The following sequence makes the assumptions and comparison explicit.
Choose an information measure or decision loss that represents the design purpose, and state its units.
Use the current parameter uncertainty and observation model to obtain the predictive distribution of the candidate observation.
Calculate the posterior objective, or the best posterior decision and its loss, for each possible outcome.
Weight outcome-specific results by their predictive probabilities and subtract a cost expressed on the same scale.
The key justification for averaging before making a recommendation is that the outcome is not known at the design stage. For a finite set of outcomes, the future expected risk is \(\sum_y P(Y=y)R(y)\), where \(R(y)\) is the best posterior risk after outcome \(y\). The current risk is \(R_0=\min_a E[L(a,\theta)]\). Their difference \(R_0-\sum_yP(Y=y)R(y)\) measures the expected improvement. Replacing this average with \(R(y^*)\) for a single assumed result \(y^*\) answers a different, conditional question.
Worked Example: Information Value Under a Normal Model
Suppose the uncertain parameter \(\theta\) has current distribution \(\theta\sim N(0,4)\), where 4 is its variance. A candidate observation follows \(Y\mid\theta\sim N(\theta,9)\), with known observation variance 9. For this example, the objective is expected information gain, and sampling cost has been converted to \(0.0500\) nats.
For the normal prior and normal observation model, posterior precision is the sum of prior precision and observation precision. Thus the posterior variance is \((1/4+1/9)^{-1}=(13/36)^{-1}=36/13=2.769\) squared parameter units. The expected information gain for this single observation is \(\frac{1}{2}\ln(1+4/9)=\frac{1}{2}\ln(13/9)=0.1839\) nats.
The net information value is \(0.1839-0.0500=0.1339\) nats. In this known-variance normal model, the posterior variance does not depend on the observed value, so the posterior precision—and hence this information calculation—is the same for every possible observation. That is a special feature of the model, not a general reason to omit predictive averaging. The cost conversion is also essential: the number \(0.0500\) must represent an information-equivalent cost for this subtraction to be meaningful.
Assumptions, Diagnostics, and Alternatives
These calculations are only as credible as their inputs. The predictive distribution must reflect the parameter uncertainty and observation process relevant at the time of evaluation. The update must use a coherent likelihood and prior (or other appropriate uncertainty representation). For a decision-value calculation, the action set and loss function must be stated, and the loss must capture the consequences that matter. The cost should include relevant burdens and be measured on a compatible scale. If these assumptions are weak, present the value as conditional on them and examine how the conclusion changes across plausible alternatives.
- Check outcome probabilities. Predictive probabilities should sum to one, and posterior probabilities should be valid. Unexpectedly extreme outcomes can signal a coding or model specification error.
- Check the objective’s units. Nats, squared parameter units, dollars, and clinical loss are different quantities. Do not subtract a cost until a defensible conversion has been specified.
- Check computation. If predictive outcomes or posterior quantities are estimated by simulation, assess Monte Carlo error. A net value close to zero may not support a stable recommendation if its numerical uncertainty is large.
- Check sensitivity. If modest changes to the prior, observation model, or loss assumptions reverse the sign of net value, report that dependence rather than presenting the recommendation as robust.
The main alternative to expected value is a plug-in calculation that evaluates the observation at one parameter estimate or assumes one particular future result. It can be simpler, but it ignores uncertainty in the parameter or in the outcome and can misstate expected benefit. A robust criterion instead evaluates performance over a range or distribution of plausible conditions, while a decision-value calculation directly targets losses under specified actions. These approaches answer different questions; the appropriate one depends on the design objective.
A common mistake is to say, “The most likely result produces a large improvement, so the sample has large expected value.” The conclusion does not follow: less likely outcomes may produce little improvement or even a worse conditional objective, and expected value weights all outcomes by their predictive probabilities. Another mistake is to interpret positive information gain as proof that a sample improves dosing. Information can reduce uncertainty about a parameter that does not change the preferred dose. For dosing, calculate value on a decision-relevant loss scale when the aim is improved decisions.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- In the binary assay example, what is the expected posterior entropy using the displayed rounded values, and what is the net information value after a cost of \(0.0100\) nats?
- If the binary assay in question 1 instead had an information-equivalent cost of \(0.0250\) nats, what would its net information value be? What would its sign imply?
- In the dosing test example, what is the largest sampling cost for which the test has positive net value under the stated loss model?
- For the normal model with prior variance 4 and observation variance 9, calculate the posterior variance and expected information gain. If the cost is \(0.2000\) nats, is net information value positive or negative?
Solutions. (1) The expected posterior entropy is \(0.48(0.6931)+0.52(0.6172)=0.653632\), or \(0.6536\) nats. The information gain is \(0.6730-0.6536=0.0194\) nats, so net value is \(0.0194-0.0100=0.0094\) nats. (2) Net value is \(0.0194-0.0250=-0.0056\) nats, so the candidate does not have positive expected net value under this objective and cost conversion. (3) The expected value before cost is $20, so any cost below $20 gives positive net value; at exactly $20 net value is zero. (4) Posterior variance is \((1/4+1/9)^{-1}=36/13=2.769\) squared parameter units. Expected information gain is \(\frac12\ln(13/9)=0.1839\) nats, so net value is \(0.1839-0.2000=-0.0161\) nats, which is negative.