Valuing New Experiments by Their Potential to Change a Decision

Why Value an Experiment by Its Decision Impact?

A proposed experiment can be scientifically informative without being useful for a development decision. A biomarker study may sharpen understanding of pathway activity, for example, yet leave the target decision unchanged. Conversely, a relatively small study may be valuable if its results could distinguish between advancing at a lower dose, advancing at a higher dose, or stopping. The relevant question is therefore not only how much uncertainty an experiment could reduce, but also whether that information could change an action whose consequences matter.

Expected value of sample information formalizes this question before data are collected. It combines the current uncertainty about possible biological states, the decisions available, the consequences of those decisions, and the likelihood of each possible study result under each state. It is a prospective calculation: it evaluates potential results, rather than interpreting observed results from a completed study.

For clarity, utilities below are expressed as millions of dollars of expected development value relative to stopping development, so stopping has utility zero in every state. This is a simplifying decision model, not a claim that all consequences of a QSP program can always be reduced to money. Utilities could instead be measured in another coherent scale, provided the consequences and study costs can be compared on that scale.

A Decision Model for the Worked Case

Consider a hypothetical program evaluating a target and its dose strategy. There are three possible underlying states: the mechanism is unlikely to work at either dose (\(F\)); the lower dose is sufficient (\(L\)); or the higher dose is needed (\(H\)). The current QSP-informed probabilities are \(P(F)=0.35\), \(P(L)=0.40\), and \(P(H)=0.25\). The available decisions are to stop, advance at the lower dose, or advance at the higher dose.

The entries are hypothetical utilities in millions of dollars. They summarize the relative consequences of efficacy, safety, and development investment for this decision exercise. The calculation assumes that the state probabilities and utility table are adequate for comparing the proposed studies; those assumptions must be scrutinized in a real program.

With current information, the expected utility of each advancement decision is:

The current best decision is therefore to advance at the higher dose, with expected utility \(0.85\) million dollars relative to stopping. An experiment has decision value only to the extent that its possible results support decisions that improve on this current choice.

Why the Calculation Takes This Form

The joint probability of state \(s\) and result \(y\) is \(P(s)P(y\mid s)\). For a particular result, multiplying that joint probability by \(U(a,s)\) and summing over states gives the expected utility contribution of choosing \(a\) after that result. Choosing the action with the largest contribution, then summing across results, gives the expected value when the study result is available before the decision.

This is equivalent to first calculating \(P(y)=\sum_s P(s)P(y\mid s)\), then choosing the action with the greatest posterior expected utility given \(y\), and averaging that utility over results. The joint-probability form is convenient because it avoids rounding intermediate posterior probabilities.

A useful check is that EVSI cannot be negative under the stated assumptions when the study result is free to ignore. After every result, the decision-maker could choose the action that was best before the study. The best action after seeing a result can do at least as well as that fallback action. Averaging this inequality over results shows that expected value with information is at least the current best expected value. Study cost is separate, so net value can be negative.

Comparing Three Proposed Experiments

The same calculation can compare studies with different scientific purposes. The result probabilities below are hypothetical predictions, not observed study findings. Each study has two possible results. The positive result is defined separately for each design: it need not mean the same biological evidence across designs.

The comparison is summarized below. The study with the highest EVSI is not necessarily the study with the highest net value; the latter also depends on cost.

In this case, the biomarker study has the greatest estimated net value, narrowly exceeding the ex vivo study. The dose-ranging study has a smaller EVSI because its results are less effective at improving expected decisions in this utility model, despite being directly informative about dose. These rankings are conditional on the specified state probabilities, result likelihoods, utilities, and costs; they are not general rankings of these study types.

Assumptions, Diagnostics, and Limits

A credible comparison requires more than arithmetic. The states must represent uncertainties that matter to the decision, and the study-result likelihoods must reflect the proposed design, assay performance, population, and analysis. Utilities must include relevant benefits and harms on a comparable scale. The calculation also assumes that the decision can be made after the result is available, that results do not arrive too late to act on, and that conducting the study does not itself change the underlying state or introduce unmodeled consequences.

Several checks can reveal a fragile analysis:

• Check the decision map. For every possible result, report which action is optimal and show its expected utility. If the action is unchanged for every result, the study has zero EVSI under this model, even if it is scientifically interesting.
• Check likelihood calibration. A study with weak discrimination between states may appear highly valuable if its assumed \(P(y\mid s)\) values are too optimistic. Use evidence appropriate to the actual assay and design, and examine plausible alternatives.
• Check the utility scale. A table that omits toxicity, operational constraints, or the cost of delaying a decision can favor an action for the wrong reason. If a consequence matters to the decision, it should be represented.
• Check sensitivity. Recalculate the ranking under plausible changes in state probabilities, study costs, likelihoods, and utilities. A small difference between the biomarker and ex vivo net values here should not be treated as a robust ordering without such checks.
• Check whether the study is actionable. Information has little decision value if the result cannot be obtained in time, if the decision cannot be changed, or if another constraint prevents acting on it.

A common mistake is to rank studies by how much biological detail they generate, or by how large a change they are expected to make to a model prediction. Those quantities do not by themselves measure decision value. The correction is to calculate how each possible result changes the best available action and its expected consequences, then subtract the costs of obtaining that information. Another mistake is to interpret positive net value as proof that a study must be run. The calculation is only as reliable as its states, probabilities, utilities, and representation of constraints; it structures judgment but does not replace it.

EVSI is also not the value of perfect information. A real experiment may be noisy, may measure only part of the relevant biology, and may fail to separate states that imply different actions. Moreover, this simplified calculation does not account for every possible sequence of future experiments or decisions. Its strength is narrower: it makes explicit how a proposed experiment could affect a specified decision, and provides a consistent basis for comparing alternatives.