Why Compute EVSI by Simulation?
Expected value of sample information (EVSI) compares the best expected decision before a study with the expected value of decisions made after its results are observed. For simple models, the possible study outcomes and posterior decisions can sometimes be enumerated exactly. In development decisions, however, outcomes may be multidimensional, parameters may be numerous, and the likelihood or utility calculation may not have a convenient closed form. Monte Carlo computation approximates the required expectations by repeatedly simulating from the probability model.
The central discipline is to keep the simulation sequence straight: draw uncertain parameters from the current distribution, simulate a possible study result conditional on those parameters, update the parameter distribution using that result, and then evaluate the available actions under the updated distribution. The simulated parameter value generates the data; it is not itself the posterior estimate used to make the simulated decision.
This definition assumes that the probability model for \(\theta\) and the study likelihood for \(Y\mid\theta\) are specified, the action set and utility function are defined on the same decision problem, and the study result is available before the action is chosen. The expectation in the first term averages over the study’s prior-predictive distribution, \(P(Y)=\int P(Y\mid\theta)\,p(\theta)\,d\theta\). If the study cannot influence the action, its EVSI is zero. As in the earlier material on expected value of sample information, the key comparison is between decisions with and without sample information—not between a study result and a known true parameter.
From a Decision Model to a Simulation
A direct Monte Carlo calculation represents the prior-predictive expectation by a finite set of simulated study results. For each result, it calculates posterior expected utility for every action and takes the highest value. The estimated EVSI is the average of those best posterior values minus the best current expected value. A posterior update can be analytic, computed numerically, or approximated with an inner simulation; whichever method is used must represent the same conditional distribution \(p(\theta\mid Y)\).
List actions, utilities, uncertain parameters, their current probability distribution, and the study likelihood. Confirm that utilities and study costs use consistent units.
For replicate \(i\), draw \(\theta^{(i)}\) from the current parameter distribution, then draw \(Y^{(i)}\) from the study likelihood conditional on \(\theta^{(i)}\).
For the simulated result \(Y^{(i)}\), calculate \(E\{U(a,\theta)\mid Y^{(i)}\}\) for each action and retain the maximum. Do not choose the action using the generating draw \(\theta^{(i)}\).
Average the retained values over \(N\) replicates, subtract the best current expected utility, and estimate Monte Carlo error from the variability across replicates.
The simulation is valid when the parameter draws represent the specified current uncertainty, the outcomes are generated from the assumed sampling model, and the posterior calculation uses the same prior and likelihood. It also requires consistent handling of missing outcomes, trial failure, and any action-dependent payoff components that the decision model includes. Simulation cannot correct a misspecified prior, likelihood, utility, or action set; it only approximates calculations under the model supplied.
Worked Example: Exact EVSI in a Hypothetical Development Decision
Worked Example: Exact EVSI in a Hypothetical Development Decision
All values in this example are hypothetical. Let \(\theta\) be the probability that a development program achieves technical success. Suppose the current distribution is \(\theta\sim\operatorname{Beta}(2,2)\). The available actions are invest and stop. Investing has utility \(+\$100\) million if successful and \(-\$100\) million if unsuccessful; stopping has utility zero. A proposed study observes \(Y\), the number of successes in two independent Bernoulli observations with success probability \(\theta\).
For a given success probability \(\theta\), the expected utility of investing is \(100\theta-100(1-\theta)=200\theta-100\) million dollars. The invest action is preferable when \(\theta>0.5\), but the actual decision uses the expected utility under the current or updated distribution, not the unknown value of \(\theta\).
The prior mean is \(E(\theta)=2/(2+2)=0.5\). Thus the current expected utility of investing is \(200(0.5)-100=0\) million dollars, equal to stopping at zero. The best current expected utility is therefore zero.
After observing \(Y=y\), beta-binomial conjugacy gives \(\theta\mid Y=y\sim\operatorname{Beta}(2+y,4-y)\), with posterior mean \((2+y)/6\). For \(y=0,1,2\), these means are respectively \(1/3,1/2,2/3\). The corresponding posterior expected utilities of investing are \(200(1/3)-100=-\$33.33\) million, \(0\), and \(200(2/3)-100=+\$33.33\) million. Taking the better action at each result gives values \(0,0,\$33.33\) million: stop for \(y=0\), either action for \(y=1\), and invest for \(y=2\).
The prior-predictive probabilities are \(P(Y=y)=\binom{2}{y}B(2+y,4-y)/B(2,2)\). They are \(P(Y=0)=0.3000\), \(P(Y=1)=0.4000\), and \(P(Y=2)=0.3000\), which sum to one. Therefore, $$ \operatorname{EVSI} = (0.3000)(0)+(0.4000)(0)+(0.3000)(33.33)-0 \approx \$10.00\text{ million}. $$ The study’s gross expected value is \$10.00 million under these assumptions. It does not mean the study will produce a \$10 million gain in every realization; it is an expectation over possible results.
Monte Carlo Estimate and Its Error
For each simulated replicate, define \(V_i=\max_a E\{U(a,\theta)\mid Y^{(i)}\}\), the best posterior expected utility for its simulated outcome. If \(V_0=\max_a E\{U(a,\theta)\}\) is the best current expected utility, the Monte Carlo estimator is $$ \widehat{\operatorname{EVSI}}=\frac{1}{N}\sum_{i=1}^{N} V_i-V_0. $$ When replicates are independently generated and have finite variance, the Monte Carlo standard error of the estimated mean is \(s_V/\sqrt{N}\), where \(s_V\) is the sample standard deviation of the \(V_i\) values. This quantifies numerical sampling error in the computation, not uncertainty about whether the model assumptions are correct.
Worked Example: Estimating EVSI from Simulated Outcomes
Use the hypothetical decision above, where the exact value is \$10.00 million. Suppose a simulation with \(N=1{,}000\) independent replicates produces 302 outcomes with \(Y=0\), 398 with \(Y=1\), and 300 with \(Y=2\). The posterior decision values are zero for the first two outcomes and \(100/3=\$33.333\) million for \(Y=2\). The estimated EVSI is $$ \widehat{\operatorname{EVSI}} = \frac{302(0)+398(0)+300(33.333)}{1{,}000}-0 = \$10.00\text{ million}. $$
The exact sample variance of the \(V_i\) values, retaining \(100/3\) rather than rounding it during calculation, is $$ s_V^2= \frac{700(0-10)^2+300(100/3-10)^2}{999} = \frac{233{,}333.33}{999} \approx 233.57\;(\$\,\text{million})^2. $$ Thus \(s_V\approx \$15.28\) million and the Monte Carlo standard error is \(15.28/\sqrt{1{,}000}\approx \$0.4833\) million. A rough normal-approximation 95% Monte Carlo interval is \(10.00\pm1.96(0.4833)\), or approximately \([\$9.053,\$10.947]\) million. This interval describes numerical error from using 1,000 replicates; it is not a probability interval for the true EVSI, and it does not include uncertainty about the prior, likelihood, or utilities.
The simulated estimate happens to equal the exact value in this hypothetical run, but that agreement is not guaranteed. A different random run would usually give a different mean. If the interval is too wide for the decision being made, increasing \(N\) reduces the Monte Carlo standard error at approximately the rate \(1/\sqrt{N}\), provided the computation remains otherwise unchanged.
Justification, Diagnostics, and Failure Modes
The non-negativity of gross EVSI follows directly from the decision structure. For each possible result, the maximum posterior expected utility is at least the posterior expected utility of any particular action \(a\). Averaging over results gives $$ E_Y\left[\max_a E\{U(a,\theta)\mid Y\}\right] \geq E_Y\left[E\{U(a,\theta)\mid Y\}\right] = E\{U(a,\theta)\}. $$ The equality is the law of total expectation. Since this holds for every action \(a\), the first term in EVSI is at least \(\max_a E\{U(a,\theta)\}\), so EVSI is non-negative when the study creates no obligation to act differently or pay for participation within the utility definition. A negative computed gross EVSI is a warning to check the implementation, including whether the same actions and payoff definitions were used in both terms.
Useful diagnostics include checking that simulated outcome frequencies agree, within Monte Carlo error, with known predictive probabilities; checking posterior calculations against exact updates for selected outcomes; and confirming that all replicate values are finite and use the intended units. Independent batches can reveal unstable estimates or rare high-value outcomes that dominate the mean. The estimated standard error should generally decrease as the number of independent replicates increases; it need not decrease monotonically after every batch is added.
A common source of error is to select the action that would be best at each simulated true parameter draw. That calculation gives the decision-maker information about the unknown parameter itself, rather than the information provided by the study. The correction is to simulate the outcome conditional on the parameter draw, then base the action on posterior expected utilities given that outcome. Another important failure mode arises when posterior expected utilities are themselves estimated with a small inner simulation. Taking a maximum of noisy estimates tends to bias the maximum upward, because the largest estimate is preferentially one with positive simulation error. Use sufficiently precise posterior calculations, increase inner simulation effort, or assess and control this nested-simulation bias; an outer Monte Carlo standard error alone does not capture it.
Monte Carlo is most useful when exact enumeration or integration is impractical, but it is not always the best first choice. Exact summation over a small discrete outcome space avoids outer simulation error and provides a valuable benchmark. Numerical integration can be effective in low-dimensional continuous models. Monte Carlo is flexible for complex models and large outcome spaces, though it can be inefficient when rare results drive decision value. Variance-reduction methods or carefully designed simulation can improve efficiency, but they must preserve the target expectation and be validated.
Worked Example: Separating Gross EVSI from Study Cost
Suppose, again hypothetically, a one-observation study has \(Y\sim\operatorname{Bernoulli}(\theta)\), with the same \(\operatorname{Beta}(2,2)\) prior and the same invest-or-stop utilities. The predictive probabilities of \(Y=0\) and \(Y=1\) are both 0.5000. The posterior means are \(2/5=0.4000\) after \(Y=0\) and \(3/5=0.6000\) after \(Y=1\). The posterior expected invest utilities are \(200(0.4)-100=-\$20\) million and \(200(0.6)-100=+\$20\) million. The best posterior values are therefore zero and \$20 million.
The gross EVSI is $$ (0.5000)(0)+(0.5000)(20)-0=\$10.00\text{ million}. $$ If the study costs \$8.00 million and that cost is paid to obtain the information, its net expected value relative to not studying is \(\$10.00-\$8.00=\$2.00\) million, before any additional delay or other effects. The Monte Carlo computation estimates the gross EVSI; subtracting costs is a separate decision calculation. A positive gross EVSI alone does not establish that conducting the study is worthwhile.
Limitations and Practical Interpretation
An EVSI estimate is conditional on the chosen estimand, endpoint, likelihood, prior uncertainty, utility, and action set. It can be numerically precise while being decision-model dependent. Prior-predictive checks and sensitivity analyses can help assess the inputs, but simulation error should not be confused with uncertainty in those inputs. If several plausible structural models are under consideration, averaging or comparing across them requires additional model-uncertainty treatment; that issue is distinct from Monte Carlo error within a specified model.
Report the number of replicates, the method used for posterior updates, the estimated EVSI with units, and a Monte Carlo error measure. Where nested simulation is used, report enough detail to assess both outer and inner numerical error. If the estimated net value is close to zero relative to the Monte Carlo error, more computation may be needed before drawing a computational conclusion. Even then, more replicates cannot resolve a disagreement about the decision model or its assumptions.
Check Your Understanding
Answer each question, then compare with the brief solutions.
- Why must simulated actions be based on the posterior distribution given the simulated study result, rather than on the parameter value used to generate that result?
- If the sample standard deviation of replicate decision values is \$12 million and \(N=900\), what is the Monte Carlo standard error of their mean?
- What does a 95% Monte Carlo interval for an EVSI estimate describe, and what does it not describe?
- A simulation gives a negative gross EVSI. Name a key implication and one implementation check.
Solutions. 1. The decision-maker observes the study result, not the simulated true parameter; using the generating value grants perfect information and targets a different quantity. 2. \(12/\sqrt{900}=\$0.4000\) million. 3. It approximates numerical sampling uncertainty from finite simulation under the specified model; it is not a probability statement about the parameter or protection against model misspecification. 4. Gross EVSI should be non-negative when the study adds information without an obligation that reduces utility; check that the same action set, utility, and baseline value were used in both terms, along with the posterior updates and simulation code.