Expected Information Gain for Experiment Selection

Why Expected Information Gain Matters

When biological mechanisms remain uncertain, a candidate experiment is valuable if its possible results are expected to distinguish the mechanisms. A large response under one mechanism and a small response under another may sound promising, but the comparison is incomplete until observation variability and parameter uncertainty are included. Expected information gain (EIG) summarizes how much a proposed experiment is expected to reduce uncertainty about the model, averaging over its possible results.

EIG is a planning quantity, not a prediction of which model will be selected after the experiment. It depends on the current probabilities assigned to the models and on each model's predictive distribution for the experiment. Therefore, it can prioritize experiments only relative to those assumptions and the candidate experiments being compared.

Definition: Let $M$ denote the mechanistic model, with candidate models $M_1,\ldots,M_K$, and let $d$ denote a proposed experiment. The prior model probability is $P(M=m)$. The predictive distribution $p(y\mid M=m,d)$ describes the possible observation $y$ under model $m$ and experiment $d$, including uncertainty in model parameters and observation error. The prior predictive distribution is $$ p(y\mid d)=\sum_{m=1}^{K}P(M=m)\,p(y\mid M=m,d). $$ The expected information gain is the expected Kullback–Leibler divergence from the posterior model probabilities to the prior model probabilities: $$ \operatorname{EIG}(d) =\sum_y p(y\mid d)\sum_{m=1}^{K}P(M=m\mid y,d) \ln\!\left(\frac{P(M=m\mid y,d)}{P(M=m)}\right). $$ For continuous observations, the sum over $y$ is replaced by an integral.

With natural logarithms, EIG is measured in nats. A value of zero means that the experiment's result does not change the model probabilities, apart from events of zero predictive probability. Larger values indicate a greater expected reduction in uncertainty. The calculation assumes that the candidate models, prior model probabilities, predictive distributions, and experiment implementation are specified well enough to calculate the prior predictive distribution.

Entropy Form and Justification

For model probabilities $q_1,\ldots,q_K$, define their entropy as $H(q)=-\sum_m q_m\ln(q_m)$, using $0\ln(0)=0$. Entropy is a summary of uncertainty in a probability distribution: it is largest when the probabilities are equal and is zero when one model has probability one. Bayes' rule gives the posterior model probabilities for each possible result:

P(M=m\mid y,d) =\frac{P(M=m)\,p(y\mid M=m,d)}{p(y\mid d)}.

The entropy form follows by expanding the definition. For a fixed result $y$, the divergence from posterior to prior is $\sum_m P(M=m\mid y,d)\ln P(M=m\mid y,d)-\sum_m P(M=m\mid y,d)\ln P(M=m)$. Averaging over results, the first sum becomes the negative expected posterior entropy. In the second sum, averaging $P(M=m\mid y,d)$ over $p(y\mid d)$ returns $P(M=m)$, by the law of total probability. Thus:

\operatorname{EIG}(d) =H(M)-\mathbb{E}_{y\mid d}\!\left[H(M\mid y,d)\right].

This identity gives two useful calculation routes: average posterior-to-prior divergences over possible results, or subtract expected posterior entropy from prior entropy. It also shows why an experiment with striking predictions is not necessarily informative: if its results are noisy enough that posterior model probabilities usually remain close to their priors, the expected entropy reduction is small.

Worked Example: A Two-Model Experiment with a Negative Result

Suppose two hypothetical mechanisms have prior probabilities $P(M_1)=0.70$ and $P(M_2)=0.30$. A candidate experiment has a binary result, positive or negative. The predictive probabilities are $P(+\mid M_1,d)=0.75$ and $P(+\mid M_2,d)=0$; therefore the negative-result probabilities are 0.25 and 1.00, respectively. These are hypothetical probabilities chosen to illustrate the calculation.

The probability of a negative result is $$ P(-\mid d)=0.70(0.25)+0.30(1.00)=0.475. $$ After a negative result, Bayes' rule gives $$ P(M_1\mid -,d)=\frac{0.70(0.25)}{0.475} =\frac{0.175}{0.475}=\frac{7}{19}=0.368421, $$ and $P(M_2\mid -,d)=12/19=0.631579$.

The divergence for a negative result is $$ D_{\mathrm{KL}}(P(M\mid -,d)\Vert P(M)) =0.368421\ln(0.368421/0.70) +0.631579\ln(0.631579/0.30) \approx 0.2337\ \text{nats}. $$ The first term is approximately $-0.236472$, and the second is approximately $0.470174$; their sum is approximately $0.233702$ nats. The negative value of the first term is not an error: the divergence is the sum of all model-specific terms and is nonnegative overall.

The positive result has probability $0.70(0.75)+0.30(0)=0.525$, and its posterior probability for $M_1$ is 1. Its divergence from the prior is $\ln(1/0.70)=0.356675$ nats. Thus the expected information gain is $$ \operatorname{EIG}(d)=0.475(0.233702)+0.525(0.356675) \approx 0.2983\ \text{nats}. $$ The experiment is informative on average under the specified priors and predictions. The positive result identifies $M_1$ within this two-model set, while a negative result shifts probability toward $M_2$ without eliminating $M_1$.

Working with More Than Two Mechanisms

For several models, calculate the predictive probability of each possible result by weighting each model's predictive probability by its prior model probability. Then calculate the posterior model probabilities separately for each result. Keeping these steps explicit helps prevent a common error: comparing predictions under the models without accounting for how probable each model was before the experiment.

Worked Example: Three Mechanisms and a Binary Biomarker Result

Consider three hypothetical mechanisms with prior probabilities 0.20, 0.30, and 0.50. A proposed experiment records whether a biomarker result is positive. The probabilities of a positive result under $M_1,M_2,M_3$ are 0.80, 0.40, and 0.20, respectively; the probabilities of a negative result are therefore 0.20, 0.60, and 0.80.

The prior predictive probability of a positive result is $$ P(+\mid d)=0.20(0.80)+0.30(0.40)+0.50(0.20) =0.16+0.12+0.10=0.38. $$ The positive-result posterior probabilities are $(0.16/0.38,\ 0.12/0.38,\ 0.10/0.38)=(8/19,\ 6/19,\ 5/19)$, or approximately $(0.4211,\ 0.3158,\ 0.2632)$.

For a negative result, $$ P(-\mid d)=0.20(0.20)+0.30(0.60)+0.50(0.80) =0.04+0.18+0.40=0.62. $$ The negative-result posterior is $(0.04/0.62,\ 0.18/0.62,\ 0.40/0.62)=(2/31,\ 9/31,\ 20/31)$, or approximately $(0.06452,\ 0.2903,\ 0.6452)$.

The prior entropy is $$ H(M)=-[0.20\ln(0.20)+0.30\ln(0.30)+0.50\ln(0.50)] =1.029653\ \text{nats}. $$ Substituting the posterior probabilities into the entropy formula gives approximately 1.0795 nats after a positive result and 0.8186 nats after a negative result. The expected posterior entropy is $$ 0.38(1.0795)+0.62(0.8186)=0.9177\ \text{nats}, $$ so $\operatorname{EIG}(d)=1.029653-0.9177\approx 0.1119$ nats. This experiment reduces uncertainty on average, but not as much as the EIG of 0.2983 nats in the preceding hypothetical example. That comparison is meaningful only relative to the stated priors, predictions, and candidate designs.

Comparing Candidate Experiments

A useful comparison puts all candidate experiments on the same footing: use the same model set and prior probabilities, calculate each design's predictive distribution, and evaluate EIG using the same observation assumptions. The following example shows that a design with more separated predictions can be substantially more informative, while still allowing for the possibility of surprising outcomes.

Worked Example: Prioritizing Between Two Candidate Designs

Suppose two hypothetical mechanisms each have prior probability 0.50. Both candidate experiments have a binary positive or negative result. Under design A, the positive-result probabilities are 0.90 for $M_1$ and 0.20 for $M_2$. The prior predictive probability of a positive result is $0.50(0.90)+0.50(0.20)=0.55$, and the posterior probability of $M_1$ after a positive result is $0.45/0.55=0.8182$. After a negative result, the predictive probability is 0.45 and the posterior probability of $M_1$ is $0.05/0.45=0.1111$.

The binary entropy at posterior probability 0.8182 is approximately 0.4741 nats; at 0.1111 it is approximately 0.3488 nats. Since the prior entropy is $\ln(2)=0.6931$ nats, design A has $$ \operatorname{EIG}(A)=0.6931-[0.55(0.4741)+0.45(0.3488)] \approx 0.2754\ \text{nats}. $$

Under design B, the positive-result probabilities are 0.70 and 0.30. The positive and negative results each have prior predictive probability 0.50. Their posteriors for $M_1$ are 0.70 and 0.30, respectively; both have entropy 0.6109 nats. Thus $$ \operatorname{EIG}(B)=0.6931-0.6109=0.08228\ \text{nats}. $$ Design A is the higher-information priority under these assumptions because its possible results produce more distinct posterior model probabilities. This ranking does not imply that design A is more feasible, less costly, or more clinically valuable; those considerations require additional decision criteria.

Predictive Distributions, Diagnostics, and Limitations

In QSP applications, $p(y\mid M=m,d)$ should generally reflect both observation error and uncertainty in the parameters of model $m$. If parameters are uncertain, the model-specific predictive distribution averages the observation distribution over the parameter uncertainty. Using only a single best-fit parameter value can make predictions appear more precise and separated than they are, inflating EIG. Predictive distributions should also preserve relevant correlations and heterogeneity in the planned measurements rather than treating dependent observations as independent without justification.

Useful checks include verifying that the model probabilities sum to one for every possible result, that the prior predictive probabilities sum or integrate to one, and that posterior probabilities normalize after each result. EIG must be nonnegative, up to numerical error. If an implementation reports a materially negative value, check the probability normalization, the direction of the divergence, and the calculation of expected posterior entropy. For continuous outcomes, numerical integration or simulation may be needed; assess whether the estimate is stable to increased integration accuracy or simulation size.

A high EIG can be fragile if it depends on a prior assigning appreciable probability to a model that is otherwise implausible, or on a narrow predictive distribution that is not supported by the data. Recalculate rankings under scientifically defensible alternative priors and observation assumptions. If plausible changes reverse the ranking, report the sensitivity rather than presenting one design as unconditionally optimal. Also check whether the candidate model set omits plausible mechanisms: EIG is conditional on the models included and cannot measure information about a mechanism that has no representation in that set.

EIG differs from several other comparison criteria. Information criteria and predictive cross-validation assess model fit or predictive performance using observed data; they do not directly calculate the expected reduction in model uncertainty from a proposed experiment. A Bayes factor compares the evidence for models after a particular result is observed, whereas EIG averages over possible results before the experiment. A power calculation targets the chance of meeting a specified decision rule under specified conditions; it need not quantify average uncertainty reduction. These approaches answer different questions and are not interchangeable.

Common pitfall: “The mechanisms predict different mean responses, so this experiment must have high EIG.” Different means alone do not establish high information gain. Predictive distributions can overlap substantially because of observation noise or parameter uncertainty, and prior probabilities affect the posterior changes. Calculate the full predictive distributions and average the resulting posterior-to-prior information over possible outcomes.

Expected information gain is one component of experiment prioritization, not a complete decision rule. Feasibility, cost, ethical constraints, measurement burden, and the relevance of resolving a particular mechanism may alter the preferred experiment. In addition, EIG concerns uncertainty about the model identity; an experiment can have low model-discrimination value yet still be useful for estimating a parameter or predicting a clinical outcome. State which uncertainty the design is intended to reduce.

Specify the candidates and priors.
List the mechanistic models, assign prior model probabilities, and state the observation and parameter uncertainty represented in each predictive distribution.

Predict possible results under each design.
For each candidate experiment, calculate the model-specific predictive distribution and combine them using prior model probabilities.

Update and average.
For each possible result, calculate posterior model probabilities; then average the posterior-to-prior divergence, or subtract expected posterior entropy from prior entropy.

Check and interpret the ranking.
Verify normalization and nonnegative EIG, assess sensitivity to plausible assumptions, and describe the ranking as conditional on the model set and priors.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions below.

What two distributions are compared in the divergence definition of EIG?
If every candidate model gives the same predictive distribution for an experiment, what is its EIG?
Why should parameter uncertainty be reflected in model-specific predictive distributions?
Two experiments have EIG values of 0.12 and 0.08 nats under the same assumptions. What can be concluded, and what cannot be concluded, from this ranking?

Solutions. (1) For each possible result, EIG compares the posterior model probabilities after that result with the prior model probabilities, then averages the divergence over the prior predictive distribution. (2) The EIG is zero: if the result has the same distribution under every model, it does not update model probabilities. (3) Ignoring parameter uncertainty can understate predictive overlap, exaggerate separation between mechanisms, and overstate expected information. (4) The first experiment is expected to reduce uncertainty about model identity more under the specified model set, priors, and predictive assumptions. The ranking alone does not establish that it is cheaper, more feasible, more clinically useful, or robust to alternative assumptions.