Why Sample Size Is a Decision Choice
A larger study can make evidence more informative, but it also costs more. The decision question is not simply whether a larger sample is statistically more precise. It is whether the additional information is likely to change a development decision enough to justify the additional study cost. Expected value of sample information (EVSI) places that question on a decision-value scale.
The calculation connects a predictive model for study results to the consequences of the actions available after the results are observed. For each candidate sample size, calculate the expected value of the best action after seeing the data, subtract the value of the best action without the study, and then account for the study’s incremental cost. The preferred candidate is the one with the greatest net decision value, not necessarily the largest sample.
The expectation over \(Y_n\) uses the predictive distribution implied by the prior uncertainty about \(\theta\) and the study’s sampling model. After each possible result, the action is chosen to maximize its expected payoff conditional on that result. The study cost should be expressed on the same value scale as the payoffs, with the same time and currency basis. If the cost has already been included in the action payoffs, it must not be subtracted a second time.
Calculation and a Basic Result
For a finite set of possible study results, the post-study value can be calculated by summing over those results. For each result \(y\), first calculate its predictive probability and the posterior expected payoff of each available action. Then select the action with the greatest conditional expected payoff. The study’s expected post-study value is the probability-weighted sum of those best conditional payoffs.
EVSI is nonnegative when the study is optional, its results can be used without changing the available actions, and study costs are excluded from gross value. To see why, a decision-maker who receives information can always ignore it and take the action that was best before the study. Therefore the best expected value after observing data cannot be lower than the value of that available strategy.
where \(a_0\) is a best action before the study. The inequality holds because the maximum conditional payoff for each result is at least the conditional payoff from \(a_0\). If candidate designs are nested—for example, the larger design contains all the observations from the smaller one—and extra observations can be ignored, gross EVSI cannot decrease with sample size. Net decision value can still decrease because cost generally increases.
Worked Example: Calculating EVSI for One and Two Observations
Suppose a hypothetical development decision has two uncertain states: favorable \(F\), with prior probability \(0.40\), and unfavorable \(U\), with probability \(0.60\). The actions are develop and stop. Developing has a payoff of \$100 million in state \(F\) and \(-\$50\) million in state \(U\); stopping has payoff \$0 in either state. A study records a binary response, with response probability \(0.60\) in state \(F\) and \(0.30\) in state \(U\). Assume responses are independent conditional on the state, and the study has no effect on the state or the payoffs.
Without a study, the expected payoff from developing is \(0.40(\$100\text{ million})+0.60(-\$50\text{ million})=\$10\) million. Stopping is worth \$0, so \(V_0=\$10\) million and the current best action is to develop.
For one observation, the predictive probability of a response is \(0.40(0.60)+0.60(0.30)=0.42\). Given a response, the posterior probability of \(F\) is \(0.40(0.60)/0.42=0.5714\). The conditional expected payoff from developing is \(0.5714(\$100\text{ million})+0.4286(-\$50\text{ million})=\$35.71\) million, so develop after a response. Given no response, the predictive probability is \(0.40(0.40)+0.60(0.70)=0.58\), and the posterior probability of \(F\) is \(0.40(0.40)/0.58=0.2759\). Developing then has expected payoff \(0.2759(100)+0.7241(-50)=-\$8.621\) million, so stop. Thus \(V(1)=0.42(\$35.71\text{ million})+0.58(\$0)=\$15.00\) million, and \(EVSI(1)=15.00-10.00=\$5.00\) million.
For two observations, let \(k\) be the number of responses. The predictive probabilities for \(k=2,1,0\), respectively, are:
For \(k=2\), the joint prior probability of \(F\) and this result is \(0.40(0.60^2)=0.144\), while that of \(U\) and the result is \(0.60(0.30^2)=0.054\). The posterior probability of \(F\) is \(0.144/0.198=0.7273\), giving a conditional develop payoff of \(0.7273(100)+0.2727(-50)=\$59.09\) million. Develop is best. For \(k=1\), the corresponding joint probabilities are \(0.40(2(0.60)(0.40))=0.192\) and \(0.60(2(0.30)(0.70))=0.252\). The posterior probability of \(F\) is \(0.192/0.444=0.4324\), giving a develop payoff of \(0.4324(100)+0.5676(-50)=\$14.86\) million. Develop is again best. For \(k=0\), the posterior probability of \(F\) is \(0.40(0.40^2)/0.358=0.1788\), giving a negative develop payoff, so stop.
The expected post-study value is therefore \(V(2)=0.198(\$59.09\text{ million})+0.444(\$14.86\text{ million})+0.358(\$0)=\$18.30\) million. Hence \(EVSI(2)=18.30-10.00=\$8.30\) million. The two-observation study has greater gross EVSI than the one-observation study, but this comparison alone does not establish which size is preferred: their costs must also be considered.
Choosing Among Candidate Sample Sizes
The practical comparison is made on a common baseline. A no-study option has \(EVSI(0)=0\) and, if it has no incremental cost, \(NDV(0)=0\). For each study size, subtract its full incremental cost from its gross EVSI. Fixed start-up costs, per-participant costs, and other relevant costs can all be included in \(C(n)\), provided they are consistently defined and not double-counted.
Worked Example: Selecting the Preferred Size After Costs
Use the hypothetical decision and EVSI calculations above. Suppose the incremental cost is \$2.50 million for one observation and \$4.00 million for two observations. No study costs \$0 and has EVSI \$0.
For one observation, \(NDV(1)=\$5.00-\$2.50=\$2.50\) million. For two observations, \(NDV(2)=\$8.30-\$4.00=\$4.30\) million. The no-study option has net value \$0. Among these candidates, the two-observation study has the greatest net decision value.
The incremental comparison gives the same conclusion: moving from one to two observations increases gross EVSI by \(\$8.30-\$5.00=\$3.30\) million and increases cost by \(\$4.00-\$2.50=\$1.50\) million. The added observation is worthwhile among these options because its expected incremental decision value exceeds its incremental cost. This conclusion is conditional on the stated candidate set and assumptions; it does not establish that two is best among sample sizes that were not evaluated.
Worked Example: When a Positive EVSI Does Not Justify a Study
Consider a second hypothetical program with the same two actions, but suppose the prior probability of a favorable state is \(0.50\). Developing pays \$80 million in the favorable state and loses \$40 million in the unfavorable state; stopping pays \$0. A one-observation study has response probabilities \(0.70\) and \(0.20\) in the favorable and unfavorable states, respectively. Assume the observation is conditionally independent of other observations and the study does not alter the available actions.
Before the study, development is worth \(0.50(80)+0.50(-40)=\$20\) million, so \(V_0=\$20\) million. The predictive response probability is \(0.50(0.70)+0.50(0.20)=0.45\). After a response, the posterior favorable-state probability is \(0.35/0.45=0.7778\), and developing is worth \(0.7778(80)+0.2222(-40)=\$53.33\) million. The predictive probability of no response is \(0.55\); the posterior favorable-state probability after no response is \(0.50(0.30)/0.55=0.2727\), so stopping is best. Therefore \(V(1)=0.45(\$53.33\text{ million})+0.55(\$0)=\$24.00\) million and \(EVSI(1)=\$24.00-\$20.00=\$4.00\) million.
If the full incremental study cost is \$3.00 million, net decision value is \(\$4.00-\$3.00=\$1.00\) million, so this study is preferred to no study on the stated values. If the cost is instead \$5.00 million, net decision value is \(-\$1.00\) million, and no study is better. A positive gross EVSI says that information has expected decision value before study costs; it does not by itself say that buying the information is worthwhile.
Assumptions, Diagnostics, and Alternatives
An EVSI-based sample-size choice is only as credible as the decision model and study model behind it. Before comparing candidates, check that the state probabilities represent current uncertainty, the sampling likelihood represents the proposed design, the payoff scale captures the consequences relevant to the decision, and the post-study actions are feasible. In particular, a response definition and analysis population must correspond to the decision being valued. If the data could change which actions are available, that change must be represented rather than treating information as the only effect of the study.
Useful diagnostics include checking that predictive probabilities over all possible results sum to one, posterior probabilities are between zero and one, and each conditional action is selected by comparing its expected payoff with the alternatives. Recalculate the result under plausible changes in prior probabilities, likelihoods, payoffs, and costs. If the chosen size changes under modest plausible changes, report that the decision is sensitive rather than implying that one sample size is unambiguously optimal. For larger or more complex outcome spaces, exact enumeration may be impractical; the calculations should still target the same expected post-study value. Computational methods for that situation require careful validation.
A common mistake is to choose the largest sample because it has the greatest gross EVSI, or to choose the design with the lowest cost per participant. Neither criterion directly maximizes net decision value. In the example above, the two-observation design has greater gross EVSI, and it is preferred only because its additional value exceeds its additional cost. A study with the greatest information per participant could still be inferior if its total cost is high or its information rarely changes the decision.
Other useful comparisons clarify what EVSI does and does not measure. Value of perfect information provides an upper bound on the gross value of learning the state before acting; it is not a substitute for comparing attainable study sizes. A decision based only on expected precision or a statistical power target addresses a different objective unless that criterion is explicitly linked to the development payoff and decision. Likewise, the largest feasible sample is not automatically preferred. The relevant comparison is expected decision value after the data, net of study cost, among the feasible designs.
The calculation has limitations. It depends on the modeled states, sampling behavior, actions, and payoffs; omitted consequences or misspecified probabilities can alter the selected size. A single expected value can also conceal important uncertainty or risk preferences in the payoffs. The result is therefore a decision aid conditional on the stated assumptions, not a guarantee that the selected study will produce a favorable result or that the decision will be correct in every realized case.
Specify uncertain states, prior probabilities, available actions, and the payoff for each action-state combination.
For every sample size, specify possible data results and their likelihood in each state; combine these with prior probabilities to obtain predictive result probabilities.
For each possible result, select the action with the highest posterior expected payoff; average these best payoffs and subtract the pre-study decision value.
Calculate \(NDV(n)=EVSI(n)-C(n)\) for every candidate, including no study if relevant, and select the candidate with the largest net value.
Check Your Understanding
Answer each question using the definitions and decision logic above. Solutions are included after the questions.
- If the current best action is worth \$12 million and the expected best action after a study is worth \$17 million, what is gross EVSI?
- A candidate study has gross EVSI of \$6 million and incremental cost of \$7 million. How does its net decision value compare with no study, assuming no-study cost is zero?
- For a nested design, why can gross EVSI not decrease when sample size increases, while net decision value can decrease?
- A study has positive EVSI. Does that establish that the study should be run? State the additional comparison needed.
Solutions. (1) \(EVSI=\$17-\$12=\$5\) million. (2) Its net value is \(\$6-\$7=-\$1\) million, below the no-study value of \$0, so no study is preferred between these options. (3) The larger design can ignore its additional observations and reproduce any decision rule available to the smaller design, so its gross expected value cannot be lower. Its cost may rise by more than its added EVSI, reducing net value. (4) No. Subtract the full incremental study cost and compare the resulting net decision value with the no-study option and other feasible sample sizes.