Decision Analysis for Development Programs · Tutorial 386 of 596

Value of Information Under Model Uncertainty

Learn to calculate EVSI across plausible structural models and assess when a study’s ability to distinguish those models can change a development decision.

Advanced 10 min read

What You'll Learn

Represent uncertainty about structural models with posterior model probabilities.
Calculate current expected utility by averaging over models and model-specific parameter uncertainty.
Update model probabilities using the predictive distribution of a proposed study.
Compute EVSI when study results can change the preferred development action.
Explain why averaging model-specific EVSIs can misstate the value of information.
Diagnose sensitivity to model choice, study assumptions, and decision thresholds.

Why Structural-Model Uncertainty Changes Study Value

A development decision can depend not only on uncertain parameter values, but also on which structural description of the system is adequate. For example, two plausible exposure-response models may predict different benefits at the same dose. A proposed study can be valuable because it estimates parameters more precisely, because its results help distinguish the models, or both. Its value therefore depends on how study results alter decisions after uncertainty about the models is included.

The key is to evaluate decisions using a probability distribution over the plausible models, rather than choosing one model first and treating it as certain. Model probabilities and parameter uncertainty are combined to calculate current expected utility and the expected utility after observing the study data. The difference is the expected value of sample information (EVSI), before subtracting study cost.

Definition: Let $M$ denote a structural model, with plausible values $m=1,\ldots,K$, and let $\theta$ denote its parameters. Given current evidence $D$, write $\pi_m=P(M=m\mid D)$ for the probability assigned to model $m$, and $p(\theta\mid M=m,D)$ for the parameter distribution conditional on that model. For action $a$, let $U(a,\theta,m)$ be its utility on the chosen decision horizon. The current model-averaged expected utility is $$ V_0=\max_{a\in\mathcal A}\sum_{m=1}^{K}\pi_m\,E[U(a,\theta,m)\mid M=m,D]. $$

The action set $\mathcal A$ might include stopping, continuing development, or choosing a dose. Utility should represent the relevant consequences of each action on a consistent scale, such as net monetary value over a specified horizon. Within each model, the expectation averages over that model’s parameter uncertainty; the outer sum averages across structural models. Model probabilities are conditional on the evidence and assumptions used to construct them. They are not automatically equal simply because several models are under consideration.

How a Study Updates the Model-Averaged Decision

Let $Y$ be the future study result. Each candidate model and its parameter distribution imply a predictive distribution for that result. The model-averaged predictive distribution is $$ p(y\mid D)=\sum_{m=1}^{K}\pi_m\int p(y\mid\theta,m,D)\,p(\theta\mid m,D)\,d\theta. $$

After observing $Y=y$, the study result updates both parameter uncertainty and, when the models imply different predictions, the model probabilities. The post-study expected utility for action $a$ is $E[U(a,\theta,M)\mid D,Y=y]$. The gross EVSI is

\mathrm{EVSI} = E_{Y\mid D}\!\left[\max_{a\in\mathcal A}E[U(a,\theta,M)\mid D,Y]\right] - \max_{a\in\mathcal A}E[U(a,\theta,M)\mid D].

The first term averages the best achievable post-study expected utility over all possible study results. It does not assume that a particular result will occur. If the study cost is $C$, a simple net value is $\mathrm{EVSI}-C$, provided that cost is expressed on the same utility scale and has not already been included elsewhere in the decision calculation.

This calculation assumes that the model set and its probabilities adequately represent current uncertainty; the study’s predictive distributions are specified under each model; utilities and the available actions are defined consistently before and after the study; and the decision-maker can use the result to select an action. It also assumes the study does not alter utility except through its information and the explicitly represented costs or consequences. If the study itself changes the development program, its direct consequences need to be represented in the decision model.

Worked Example: Current Value and Perfect Information About the Model

Suppose two structural models remain plausible. Model $M_1$ has probability 0.60 and model $M_2$ has probability 0.40. The available actions are to develop or stop. Hypothetical net utilities, in millions of dollars, are as follows:

Action	Utility under $M_1$	Utility under $M_2$
Develop	100	-80
Stop	0	0

The model-averaged expected utility of developing is $0.60(100)+0.40(-80)=60-32=28$ million dollars. Stopping has expected utility 0, so the current decision is to develop, with $V_0=28$ million dollars.

If the model were known before choosing, the best expected utility would be 100 million dollars under $M_1$, and 0 under $M_2$ (stop rather than receive $-80$). Averaging these model-specific best values gives $0.60(100)+0.40(0)=60$ million dollars. Thus perfect information about which of these two models holds has value $60-28=32$ million dollars. This is an upper bound on the gross value of a study that only provides information about these models; a real study will generally be less informative than knowing the model with certainty.

Model Discrimination Can Create Value That Is Missed Within Each Model

The EVSI under model uncertainty is not generally the weighted average of EVSIs calculated after assuming each model is already known. If the models imply different preferred actions, a study can be valuable precisely because it changes the decision-maker’s relative belief in them. Conditional on knowing one model, that same study may have no decision value at all.

Worked Example: A Study That Identifies Which Action to Take

Suppose the prior probabilities of $M_1$ and $M_2$ are each 0.50. The choice is between “go” and “stop.” Go has utility 100 under $M_1$ and $-100$ under $M_2$; stop has utility 0 under either model. A proposed study has a binary result, positive or negative, with $P(+\mid M_1)=0.80$ and $P(+\mid M_2)=0.20$. Assume these probabilities already average over any parameter uncertainty within each model.

Before the study, go has expected utility $0.50(100)+0.50(-100)=0$, equal to the utility of stopping. The current best expected utility is therefore 0.

The probability of a positive result is $P(+)=0.50(0.80)+0.50(0.20)=0.50$. Bayes’ rule gives $P(M_1\mid +)=0.50(0.80)/0.50=0.80$. After a positive result, go has expected utility $0.80(100)+0.20(-100)=60$, so go is preferred.

Likewise, $P(-)=0.50$, and $P(M_1\mid -)=0.50(0.20)/0.50=0.20$. After a negative result, go has expected utility $0.20(100)+0.80(-100)=-60$, so stopping is preferred and yields 0. The expected post-study best utility is therefore $0.50(60)+0.50(0)=30$. The gross EVSI is $30-0=30$ utility units.

If the model were already known, the study would not change the action: go under $M_1$, stop under $M_2$. Its conditional EVSI would be zero in either case. The model-averaged EVSI is nevertheless 30 because the result can resolve uncertainty that matters to the choice. This is why simply averaging model-specific EVSIs can miss decision value.

Worked Structural-Model Comparison for Dose Selection

In practice, model uncertainty may arise from alternative exposure-response forms rather than from named hypotheses with simple utilities. The decision calculation is the same: each candidate form contributes to the predictive distribution of the study result and to expected utility at each available action.

Worked Example: A Study That Can Change the Preferred Dose

Consider two plausible exposure-response forms, $M_1$ and $M_2$, each assigned probability 0.50. The action is to select a low or high dose. The following hypothetical utilities are in millions of dollars:

Action	Utility under $M_1$	Utility under $M_2$
Low dose	40	70
High dose	80	20

Before the study, expected utility is $0.50(40)+0.50(70)=55$ million dollars for the low dose, and $0.50(80)+0.50(20)=50$ million dollars for the high dose. The current choice is the low dose, with value 55 million dollars.

Suppose a proposed study result is positive or negative, with $P(+\mid M_1)=0.80$ and $P(+\mid M_2)=0.20$. With equal prior model probabilities, $P(+)=P(-)=0.50$, and the posterior model probabilities are 0.80 for $M_1$ after a positive result and 0.20 for $M_1$ after a negative result.

After a positive result, the expected utility of the low dose is $0.80(40)+0.20(70)=46$ million dollars; for the high dose it is $0.80(80)+0.20(20)=68$ million dollars. The high dose is preferred. After a negative result, the low dose has expected utility $0.20(40)+0.80(70)=64$ million dollars, while the high dose has $0.20(80)+0.80(20)=32$ million dollars. The low dose is preferred.

The expected post-study best utility is $0.50(68)+0.50(64)=66$ million dollars. Therefore, gross EVSI is $66-55=11$ million dollars. If the study costs 7 million dollars on the same utility scale, the simple net value is $11-7=4$ million dollars. On these assumptions, the study has positive net value because its expected decision improvement exceeds its cost.

Why Gross EVSI Cannot Be Negative, and When It Can Mislead

For a study with no cost or other adverse consequence, EVSI cannot be negative if the decision-maker can ignore its result and retain the current action. The reason is that the expected value of the best post-study action is at least as large as the expected value of any fixed action. For every fixed action $a$, $E_Y[\max_{a'}E(U\mid D,Y)]\geq E_Y[E(U\mid D,Y)] = E(U\mid D)$. Taking the maximum over fixed actions on the right gives the current best expected utility. Subtracting it proves that gross EVSI is at least zero. Costs can, of course, make net value negative.

This result depends on a coherent comparison: the same actions, utility definition, and current evidence must be used on both sides. A calculated negative gross EVSI is a warning to check the calculation, the treatment of information, or whether the study changes something beyond knowledge. It is not evidence that information itself has negative value under the stated assumptions.

Specify candidate models and current probabilities.
Record the structural assumptions, model probabilities conditional on current evidence, and parameter uncertainty within each model.

Calculate the current action value.
For each available action, average utility first over parameters within each model and then over the model probabilities. Choose the action with the largest expected utility.

Predict study results under each model.
Use each model’s study-data distribution, including parameter uncertainty, to form the model-averaged predictive distribution.

Update and choose after each possible result.
Update the model and parameter distributions using the result, calculate expected utility for each action, and retain the best action for that result.

Average, subtract, and test sensitivity.
Average the post-study best values over possible results, subtract current best value to obtain gross EVSI, and compare with study cost. Repeat under defensible model and prior alternatives.

Diagnostics, Alternatives, and Limitations

A model-averaged EVSI is only as credible as the model set, probabilities, predictive distributions, and utilities that produce it. Check whether the study can actually distinguish the candidate forms: if their predictive distributions for $Y$ are nearly identical, the study will usually contribute little information about which form is more plausible. Check also that the study’s outcomes are defined on the same scale and time frame as the utilities, and that the predictive distribution includes relevant parameter uncertainty rather than treating uncertain parameter estimates as fixed.

A useful diagnostic is to report the current preferred action and expected utility, then show how both change across representative study results and model-probability scenarios. If small, defensible changes in prior model weights reverse the preferred action or substantially alter EVSI, the valuation is sensitive to structural assumptions. That sensitivity should be reported, not hidden by presenting a single precise estimate. Model weights derived from evidence also depend on the model set and the assumptions used to compare predictive performance; they do not establish that one candidate model is literally true.

Approach	What it does	Main caution
Model-averaged EVSI	Averages utilities and study predictions over model probabilities before choosing actions.	Results depend on the candidate model set and model probabilities.
EVSI conditional on one selected model	Calculates study value as if one structural form were known.	Can omit information value from distinguishing models and understate uncertainty.
Separate model-specific EVSIs	Shows study value within each model considered separately.	The weighted average is not generally the model-averaged EVSI.
Decision sensitivity across models	Shows how choices and values change under alternative structural assumptions.	Describes sensitivity but does not itself assign probabilities or produce a single expected value.

Model averaging is not the only way to communicate structural uncertainty. Conditional analyses and sensitivity analyses can be informative when model probabilities are not defensible, but they answer different questions from a model-averaged EVSI. Conversely, model averaging can give a misleading appearance of certainty if important plausible structures are omitted or assigned implausible probabilities. The method also assumes the decision-maker is willing to use the stated probability distribution; it does not remove the need to justify that distribution or the utility scale.

Key takeaway: Under structural-model uncertainty, calculate study value by averaging model-specific predictive and utility calculations, updating model beliefs with each possible result, and then choosing the best action. A study can be valuable because it changes which model is plausible and therefore which action is preferred.

Check Your Understanding

Try each question, then compare your reasoning with the solutions.

Two models have probabilities 0.70 and 0.30. An action has utility 50 under the first and $-20$ under the second; stopping has utility 0 under both. What is the current preferred action and expected utility?
A study has no cost, and its result can be ignored. Can its gross EVSI be negative under the stated decision framework? Give the reason.
Why can the weighted average of model-specific EVSIs differ from model-averaged EVSI?
In the dose-selection example, what would the model probability $P(M_1\mid +)$ be if the prior probability of $M_1$ were 0.70 rather than 0.50, while $P(+\mid M_1)=0.80$ and $P(+\mid M_2)=0.20$?

Solutions. (1) The action’s expected utility is $0.70(50)+0.30(-20)=35-6=29$. Since $29>0$, choose the action, with expected utility 29. (2) No. The decision-maker can ignore the result and take the current best action, so the expected value after observing information cannot be lower than the current best value. (3) Knowing a model is not the same as learning about the model. A study may have zero value conditional on either model being known but still change the preferred action when model probabilities are uncertain. (4) By Bayes’ rule, $P(M_1\mid +)=0.70(0.80)/[0.70(0.80)+0.30(0.20)]=0.56/0.62\approx0.9032$.

Back to Tutorials