Expected Value of Sample Information

Why a Study’s Information Has Decision Value

A development study can be informative without being valuable for a decision. Its results matter when they could change what the program team should do, given the payoffs of advancing or stopping. Expected value of sample information (EVSI) measures the expected improvement in decision value from observing the result of a specified study before choosing an action. It averages over all possible results, not just the result that seems most likely or the result ultimately observed.

The calculation combines three ingredients: beliefs about uncertain program outcomes before the study, the probabilities of study results under those outcomes, and the payoff for each available decision. Bayesian updating connects the first two ingredients to the decision made after each possible result. The earlier material on Expected Value of Partial Perfect Information considers learning an uncertain component perfectly; a sample study instead provides evidence that may leave uncertainty unresolved.

Definition: Let $\theta$ denote the uncertain state of the program, $a$ an available action, and $U(a,\theta)$ the payoff for taking action $a$ when the state is $\theta$. Let $Y$ be the study result. The current expected value is $V_0=\max_a E[U(a,\theta)]$. The expected value after observing the study result is $V_Y=\sum_y P(Y=y)\max_a E[U(a,\theta)\mid Y=y]$. The gross EVSI is $V_Y-V_0$, before subtracting study cost or other consequences of conducting the study.

The expectation before the study uses the prior distribution over program states. The result probability $P(Y=y)$ is predictive: it averages the likelihood of result $y$ across those states. The conditional expectation after a result uses the updated distribution over states. In a finite-state problem,

P(Y=y)=\sum_{\theta}P(Y=y\mid\theta)P(\theta)

P(\theta\mid Y=y)=\frac{P(Y=y\mid\theta)P(\theta)}{P(Y=y)}

These expressions require a coherent prior and likelihood: the state probabilities sum to one; for each state, the probabilities of all possible study results sum to one; and the result probabilities are calculated from the same prior and likelihood used for updating. Payoffs must be expressed on a consistent scale, such as incremental net present value in millions of dollars. If payoffs are already risk-adjusted, the calculation uses those stated payoffs rather than applying an additional adjustment implicitly.

From Possible Results to EVSI

For each possible study result, calculate its predictive probability, update the state probabilities, and evaluate each available action under that updated distribution. Select the action with the highest conditional expected payoff for that result. Then average those best conditional payoffs using the predictive probabilities of the results. Finally, subtract the best current expected payoff. The resulting gross EVSI is not yet a recommendation to run the study: study cost and any other consequences must be considered separately.

Value the best current decision.
Compute each action’s expected payoff under current beliefs and take the maximum.

Enumerate study results.
Use the state-specific likelihoods and prior probabilities to calculate each result’s predictive probability.

Update and decide.
For each result, calculate the posterior beliefs, evaluate the available actions, and select the action with the greatest conditional expected payoff.

Average and compare.
Weight each result-specific best payoff by its predictive probability; subtract the best current value to obtain gross EVSI.

Worked Example: A Binary Study Before a Go-or-Stop Decision

Suppose a hypothetical development program has two possible states: technical success or failure. Current probability of success is $0.30$. Advancing yields a net payoff of $+\$100$ million if successful and $-\$40$ million if unsuccessful; stopping yields $\$0$ in either state. A proposed study has sensitivity $P(+\mid\text{success})=0.80$ and false-positive probability $P(+\mid\text{failure})=0.25$. Find its gross EVSI.

Current decision. The expected payoff from advancing is $0.30(100)+0.70(-40)=30-28=\$2$ million. Stopping is worth $\$0$, so the best current decision is to advance and $V_0=\$2$ million.

Positive result. Its predictive probability is $P(+)=0.30(0.80)+0.70(0.25)=0.24+0.175=0.415$. The posterior success probability is $P(\text{success}\mid +)=0.24/0.415=0.5783$. Advancing after a positive result has conditional expected payoff $0.5783(100)+0.4217(-40)=\$40.96$ million; stopping is worth zero, so advance. Equivalently, the positive result contributes $0.24(100)+0.175(-40)=\$17.00$ million to the predictive average.

Negative result. Its predictive probability is $1-0.415=0.585$. The joint probability of success and a negative result is $0.30(0.20)=0.060$, so the posterior success probability is $0.060/0.585=0.1026$. Advancing would have conditional expected payoff $0.1026(100)+0.8974(-40)=-\$25.64$ million. Stop, worth zero, is the best action. This result therefore contributes zero to the expected post-study value.

Thus $V_Y=17.00+0=\$17.00$ million, and gross EVSI is $17.00-2.00=\$15.00$ million. Under these assumptions, the study’s information is worth up to $\$15.00$ million before study cost and other consequences. This is an expected value across possible results, not a guarantee that any particular result will be worth that amount.

A Justification and a Useful Bound

Why is gross EVSI nonnegative when the study itself has no effect other than providing information? After seeing a result, the decision-maker can always ignore it and choose the action that was best before the study. For any result $y$, the maximum conditional payoff is therefore at least the conditional expected payoff of the current best action $a_0$. Averaging gives the justification:

\sum_y P(y)\max_a E[U(a,\theta)\mid y] \geq \sum_y P(y)E[U(a_0,\theta)\mid y] =E[U(a_0,\theta)] =V_0

The equality follows because averaging the conditional expectation over all study results recovers the unconditional expectation. Subtracting $V_0$ shows $EVSI\geq 0$. This argument assumes the same actions and payoff structure are available with or without the study; it does not say that conducting a costly or disruptive study has nonnegative net value.

With the same states, actions, and payoffs, gross EVSI also cannot exceed EVPI. Perfect information reveals the state before the decision, while a sample result may leave uncertainty about the state. Conditional on any result, the expected payoff from any action cannot exceed the expected payoff from choosing the best action separately in each fully known state. Averaging over results therefore cannot exceed the perfect-information value. In Example 1, perfect information is worth $0.30(100)+0.70(0)=\$30$ million, so EVPI is $30-2=\$28$ million. The calculated EVSI of $\$15$ million is within the bound.

Worked Example: A Study With Three Possible Results

Consider a hypothetical program with prior success probability $0.40$. Advancing yields $+\$80$ million on success and $-\$50$ million on failure; stopping yields zero. A study can return favorable, ambiguous, or unfavorable. Its likelihoods are $P(F\mid S)=0.60$, $P(A\mid S)=0.30$, and $P(U\mid S)=0.10$ if the program would succeed; under failure they are $0.15$, $0.35$, and $0.50$. These likelihoods sum to one within each state.

Before the study, advancing is worth $0.40(80)+0.60(-50)=32-30=\$2$ million, so advance is preferred to stopping. The predictive result probabilities are $P(F)=0.40(0.60)+0.60(0.15)=0.33$, $P(A)=0.40(0.30)+0.60(0.35)=0.33$, and $P(U)=0.40(0.10)+0.60(0.50)=0.34$. They sum to one.

For a favorable result, the posterior success probability is $0.24/0.33=0.7273$. Advancing is worth $0.7273(80)+0.2727(-50)=\$44.55$ million, so advance. Its contribution to the predictive average is $0.24(80)+0.09(-50)=\$14.70$ million.

For an ambiguous result, the posterior success probability is $0.12/0.33=0.3636$. Advancing is worth $0.3636(80)+0.6364(-50)=-\$2.727$ million, so stop. For an unfavorable result, the posterior success probability is $0.04/0.34=0.1176$; advancing is worth $0.1176(80)+0.8824(-50)=-\$34.71$ million, so stop. Both of these results contribute zero after the best action is chosen.

The expected post-study value is therefore $\$14.70$ million, and gross EVSI is $14.70-2.00=\$12.70$ million. Notice that the ambiguous result has a positive probability, but that alone does not make it valuable: its best decision is to stop, just as the unfavorable result’s is.

Decision-Relevant Information and Diagnostics

Information has decision value when it can improve the action selected, not merely because it changes a probability estimate. For two actions, a result is decision-relevant when the updated beliefs move the preferred action across the decision threshold implied by the payoffs. In the binary-payoff examples, advancing is preferred when its expected payoff exceeds zero; this threshold depends on both success and failure payoffs, not on a conventional probability cut-off chosen independently of them.

Worked Example: Informative Results That Do Not Change the Decision

Suppose a hypothetical program has prior success probability $0.20$. Advancing yields $+\$60$ million on success and $-\$20$ million on failure; stopping yields zero. A binary study has sensitivity $0.50$ and false-positive probability $0.40$. Calculate its EVSI.

Advancing is currently worth $0.20(60)+0.80(-20)=12-16=-\$4$ million, so stop and $V_0=0$. For this payoff structure, advancing is preferable only if $60p-20(1-p)>0$, or $p>20/80=0.25$.

The positive-result probability is $0.20(0.50)+0.80(0.40)=0.42$; the posterior success probability is $0.10/0.42=0.2381$. This is below the $0.25$ threshold, so stop. The negative-result probability is $0.58$, and the posterior success probability is $0.20(0.50)/0.58=0.1724$, also below the threshold, so stop. The expected post-study value remains zero. Therefore $EVSI=0-0=0$, even though the study changes the success probability after either result.

Several checks help identify calculation errors or a model that does not represent the decision being valued:

Check probability coherence. Predictive probabilities across results must sum to one, and posterior probabilities for the states must sum to one after each result.
Check the chosen actions. Recalculate conditional expected payoffs for every available action after each result. Selecting an action using the prior instead of the posterior omits the value mechanism EVSI is intended to capture.
Check the bounds. For a study that only supplies information, gross EVSI should be nonnegative and no greater than EVPI under the same decision model. A violation commonly indicates inconsistent probabilities, an incorrect baseline, or a change in actions or payoffs that has not been handled consistently.
Check the payoff scale and scope. Include costs or consequences in the appropriate payoff if they differ by state or action. Do not subtract a study cost from gross EVSI and then describe the result as gross EVSI.
Check whether the study can affect the decision. If the same action is best after every possible result, EVSI is zero under this model, even if the study is statistically informative.

Interpreting the Result and Its Limits

EVSI differs from both the value of perfect information and the value of learning a selected uncertain component perfectly. EVPI is an upper benchmark for the gross value of information under the same actions and payoffs. EVSI refers to a specified study with specified possible results and likelihoods. In contrast to either benchmark, net value for a real decision must account for study cost and any other consequences of acquiring the information. A study with positive gross EVSI may still have negative net value if those costs exceed the expected decision improvement.

The calculation also depends on the model’s scope. It assumes that the study-result likelihoods are adequately specified, that the prior beliefs and payoff estimates are appropriate for the decision, and that the stated actions remain available after results are observed. If conducting the study delays a program, changes its outcomes, or creates other consequences, those effects must be represented in the decision model rather than treated as information alone. EVSI is not a measure of scientific importance, nor does a low value imply that a study is scientifically uninformative; it means that, for the specified decision and payoffs, expected improvement in the action’s value is small.

A common mistake is to multiply the probability of a favorable result by the payoff from advancing and call that EVSI. This ignores the other possible results, the current best decision, and the possibility that advancing is not optimal after a favorable result. The correction is to calculate the best action separately for every possible result, average those result-specific values, and subtract the best value before the study.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions.

A study has three possible results with predictive probabilities $0.20$, $0.30$, and $0.50$. The best conditional decision values are $\$8$, $\$0$, and $\$4$ million, respectively. The best current value is $\$3$ million. What is gross EVSI?
Why must result-specific values be weighted by predictive result probabilities rather than by the posterior success probabilities?
A study updates beliefs after every result, but the same action remains optimal after each one. What is gross EVSI if the study affects nothing except information?
In a consistent model with the same states, actions, and payoffs, may gross EVSI exceed EVPI? Explain.

Solutions. (1) The expected post-study value is $0.20(8)+0.30(0)+0.50(4)=1.6+0+2.0=\$3.60$ million. Gross EVSI is $3.60-3.00=\$0.60$ million. (2) The value is averaged over which result will occur, so the weights must be $P(Y=y)$. Posterior state probabilities are used within each result to calculate its conditional decision value. (3) It is zero: if the same action remains optimal, the expected value of choosing after the study equals the current expected value when the study has no other effect. (4) No. Perfect information cannot be less useful than a sample result under the same decision model, because it reveals the state rather than only providing an imperfectly informative result.