Avoiding Double Counting in VOI Analyses

Why Double Counting Distorts Information Value

A value-of-information (VOI) calculation compares decisions made with current information against decisions that could be made with additional information. Its apparent simplicity hides an important requirement: each piece of evidence, each cost, and each future return must enter the comparison exactly once. If an observed data set has already shaped current beliefs, reusing it as if it were new evidence can make an option appear more attractive than it is. Counting a study cost twice can make a worthwhile study look unattractive; counting a future return twice can make an unwise development decision look profitable.

The safeguard is to define the current information set, the proposed new information, and the payoff for each action before calculating. This tutorial focuses on auditing those inputs. It assumes that the decision options and payoffs are well defined, and that probabilities represent uncertainty about the relevant outcomes given the information available. Where a new study is assumed to be independent of existing data conditional on the state of interest, that assumption must be defensible for the actual data sources and study design.

The word incremental is central. The current value uses the evidence that is already available. The post-study value uses that same evidence plus the genuinely new result. A study does not create information merely because its records are placed in a new file or given a new label. Conversely, a new trial can be informative even if it uses the same endpoint as an earlier study, provided its observations are genuinely new and its relationship to existing data is modeled correctly.

Three Audits Before Calculating VOI

A practical audit traces each input through the decision calculation. First, identify which observations produced the current posterior or probability-of-success estimate. Second, specify which future observations the planned study will add and how their likelihood relates to the existing observations. Third, write down the payoff consequences by decision and state, including the timing and conditions under which costs and returns occur.

• Evidence audit: Is each data set used once in forming beliefs? Is a proposed result new, or is it a duplicate or re-expression of evidence already used?
• Cost audit: Is the cost incremental and incurred under the option being evaluated? Is it already included in a net payoff or in a study budget deducted elsewhere?
• Return audit: Is a return conditional on success, already risk-adjusted, or already net of a cost? Do not apply a success probability or subtract a cost again if the payoff already incorporates it.

When a new sample is conditionally independent of existing data given \(\theta\), its likelihood can be combined with the existing likelihood. Conditional independence means that, after conditioning on \(\theta\), the joint likelihood factors into the product of the two likelihoods. This is not a default license to multiply likelihoods: overlapping participants, reused measurements, shared controls, or common data-processing steps may invalidate the factorization. A joint model for all observations, or an explicit model of their dependence, is then needed.

Justification: Free Information Cannot Reduce Decision Value

A useful check on a VOI calculation is that free information cannot reduce expected value when the decision maker can ignore it, the available actions and payoffs do not change, and the information has no other consequence. For each possible result \(Y\), the decision maker can choose the action that performs best after seeing it. The expected value of making that choice cannot be below the value of committing to the best action before seeing \(Y\).

The inequality follows because, for any fixed action, the maximum after observing \(Y\) is at least that action's conditional expected payoff; taking expectations preserves the inequality, and it holds for the best action chosen before \(Y\) as well. The equality is the law of iterated expectation. Subtracting \(V(D)\) proves that EVSI is nonnegative under these conditions. A negative computed gross EVSI is a warning to check the calculation or the assumptions. Net value can be negative because the study has a cost or other adverse consequences.

Count Costs and Future Returns Once

Double counting can occur even when the evidence update is sound. A payoff table should make clear whether each value is gross or net, conditional or already probability-weighted, and before or after a cost. For example, if a development investment is incurred whenever a program continues, it can be included in each state-contingent continuation payoff. Alternatively, it can be subtracted once from the expected gross return. Doing both subtracts the same investment twice.

Similarly, a future return of 20 million dollars conditional on technical success contributes \(p(20)\) million dollars when the success probability is \(p\). If a reported value is already that risk-adjusted expected return, applying \(p\) again understates it. A value that is already net of development costs must not have those costs subtracted again. Costs paid before the decision and unrecoverable regardless of the chosen action are sunk for that decision; relevant future costs depend on the option being evaluated.

Diagnostics, Alternatives, and Limitations

Several checks help reveal double counting before a result is used in a development decision:

• Trace data provenance. List the observations behind the current probability or posterior and those expected from the proposed study. Check participant overlap, shared controls, repeated measurements, and whether a summary already incorporates the proposed data.
• Check the likelihood structure. Multiplying likelihoods is justified only when the conditional-independence model is appropriate. If observations overlap or share sources of variation, model their joint dependence instead of treating them as independent replicates.
• Reconcile the payoff table. Label each amount as conditional or expected, gross or net, and specify when it is paid. Confirm that a success probability, study cost, or development cost is not applied twice.
• Check gross EVSI. Under the conditions in the derivation, gross EVSI should not be negative. A negative result can indicate inconsistent conditioning, a calculation error, or a changed action set or payoff that violates those conditions.
• Compare with an alternative formulation. A joint-likelihood calculation updates from all data at once; a sequential calculation first updates with \(D\), then with \(Y\). When both use the same valid probability model, they should agree. A disagreement often signals duplicated evidence, inconsistent priors, or a mismatched likelihood.

Adding separate VOI estimates is not generally a substitute for evaluating the combined information. Two studies can overlap, their value can depend on which decisions are available after both, and the first study can change the usefulness of the second. The reliable comparison specifies the information available in each scenario and calculates the corresponding optimal expected value. This tutorial's calculations also assume that the decision options and payoff structure are adequately represented; misspecified payoffs, dependence, timing, or study outcomes can still produce misleading VOI even when there is no literal duplication.

Common mistake: “The trial database is separate from the existing database, so it must be independent evidence.” A separate file or analysis does not establish independence. The correction is to check what observations and sources generated each data set and to model overlap or dependence explicitly. Likewise, do not call a previously risk-adjusted return “conditional on success” and apply the success probability again: first determine what the reported amount already includes.