Setting Evidence Standards by Decision Consequence

Why Evidence Standards Depend on the Decision

A model-informed result is not an end in itself. It matters because it may support an action: advance a development program, choose a dose, defer a study, or limit an initial use. The evidence needed to support an action should therefore reflect what could happen if the action is wrong. A mistaken choice that can be quickly reversed may have modest consequences; a mistaken choice that commits substantial resources or exposes patients to risk may have much greater consequences.

This does not mean that a consequential decision always requires a particular sample size, a particular model, or a universal numerical cutoff. The required evidence depends on the decision, the alternatives, the possible consequences, and how uncertain the evidence is. Earlier material on decision rules provides the basic link between model outputs and actions; here the emphasis is on why the strength and precision of supporting evidence should be proportionate to the consequences of those actions.

Definition: An evidence standard is the pre-specified degree and kind of support required before using evidence to make a particular decision. It should be stated in relation to a defined context of use and decision, including the outcomes that matter if the decision is wrong. It is not a universal property of a model.

Evidence strength and evidence precision are related but distinct. Strength concerns whether the evidence is relevant and credible for the decision at hand. Precision concerns how narrowly the evidence constrains the decision-relevant quantity. A precise estimate from data that do not represent the intended use may still be weak evidence for that decision. Conversely, uncertainty in an estimate may be tolerable for a reversible, low-impact choice but unacceptable for an action with large potential harm.

Expected Loss as a Decision Framework

A simple way to make the relationship explicit is to describe possible states of the world, available actions, and the loss associated with each action-state combination. Loss may represent clinical harm, development time, financial cost, or another consequence relevant to the decision. The quantities should use a consistent scale and should include important consequences of the action, not just the cost of obtaining more data.

Definition: The expected loss of an action is the probability-weighted average of its losses across the possible states of the world. If state \(j\) has probability \(p_j\) and the action has loss \(L_j\) in that state, its expected loss is \(\sum_j p_j L_j\). Among the actions being compared, the action with the lower expected loss is preferred under the stated probabilities, losses, and assumptions.

For a two-state illustration, let \(q\) be the probability that a proposed action is suitable, and \(1-q\) the probability that it is unsuitable. Suppose “proceed” has loss \(C_{\mathrm{FP}}\) if the action is unsuitable and zero loss if it is suitable. Suppose “defer” has loss \(C_{\mathrm{FN}}\) if the action is suitable and zero loss if it is unsuitable. These labels distinguish two kinds of wrong decision: proceeding when unsuitable, and deferring when suitable. They do not imply that the losses must be equal.

\begin{aligned} E[L(\mathrm{proceed})] &= (1-q)C_{\mathrm{FP}},\\ E[L(\mathrm{defer})] &= qC_{\mathrm{FN}}. \end{aligned}

Proceeding is preferred when its expected loss is lower:

\begin{aligned} (1-q)C_{\mathrm{FP}} &< qC_{\mathrm{FN}},\\ C_{\mathrm{FP}} &< q(C_{\mathrm{FP}}+C_{\mathrm{FN}}),\\ q &> \frac{C_{\mathrm{FP}}}{C_{\mathrm{FP}}+C_{\mathrm{FN}}}. \end{aligned}

The final expression is the decision threshold for this simplified loss structure. If the loss of a mistaken proceed action is large relative to the loss of deferring, the threshold rises. If the loss of deferring a suitable action is large, the threshold falls. This derivation assumes that the two states are exhaustive, the probabilities are appropriate for the decision, the losses represent the relevant consequences, and the choices are limited to the two actions described. If those assumptions do not hold, the calculation must be expanded rather than applied mechanically.

Worked Example: A Reversible, Low-Impact Choice

Suppose a development team is considering whether to use a model-supported workflow for an internal planning task. The workflow can be stopped quickly if it performs poorly. For this hypothetical decision, the team assigns a loss of 6 staff-days to adopting an unsuitable workflow and a loss of 4 staff-days to deferring a suitable workflow while using the existing process. Assume the model-based evidence implies \(q=0.68\) that the workflow is suitable for this task. The probabilities and losses are illustrative, not results from a real program.

Check the setup. The two states are “suitable” and “unsuitable,” and the actions are “adopt” and “defer.” The stated losses are on the same scale of staff-days, and the suitability probability is assumed to apply to this task. The calculation does not include other consequences, such as patient impact; if those mattered, they would need to be included.

The decision threshold is:

q_{\mathrm{threshold}}=\frac{6}{6+4}=0.600.

At \(q=0.68\), the expected loss of adopting is \((1-0.68)(6)=0.32(6)=1.92\) staff-days. The expected loss of deferring is \((0.68)(4)=2.72\) staff-days. Under these assumptions, adoption has the lower expected loss by \(2.72-1.92=0.80\) staff-days.

This calculation supports adoption for the stated task; it does not establish that the workflow is suitable for other tasks. Because the choice is reversible and the assumed losses are modest, a team might reasonably accept less precise evidence than it would require for a difficult-to-reverse decision with substantial consequences. Reversibility does not eliminate the need for relevant evidence; it changes the consequences that must be weighed.

Consequential Decisions and Robust Support

For a high-impact decision, the loss from a mistaken action may be much larger. That can change the decision threshold when the losses are asymmetric. It can also make the consequences of acting on a fragile estimate more concerning, even when the threshold itself does not change. These are related but not identical points: the loss structure determines the threshold in the calculation, while the credibility and precision of the evidence determine how confidently the relevant probability can be assessed.

Worked Example: A Consequential Development Choice

Consider a hypothetical decision to advance a development option. If the option is unsuitable, advancing it incurs a loss of 120 resource units. If the option is suitable, deferring it incurs a loss of 15 resource units through delay and lost opportunity. Let \(q\) denote the probability that it is suitable for this specific decision. Assume these are the only losses being compared.

The threshold for advancing is:

q_{\mathrm{threshold}}=\frac{120}{120+15} =\frac{120}{135}=0.8889.

If the model-based assessment gives \(q=0.91\), the expected loss of advancing is \((1-0.91)(120)=0.09(120)=10.8\) resource units. The expected loss of deferring is \((0.91)(15)=13.65\) resource units. On the point estimate and stated loss assumptions, advancing has the lower expected loss by \(13.65-10.8=2.85\) resource units.

Now suppose reasonable changes to the assumptions and inputs produce suitability assessments from 0.82 to 0.94. This is a scenario range, not a confidence or credible interval. At 0.82, advancing has expected loss \((0.18)(120)=21.6\), while deferring has expected loss \((0.82)(15)=12.3\); deferring is preferred. At 0.94, advancing has expected loss \((0.06)(120)=7.2\), while deferring has expected loss \((0.94)(15)=14.1\); advancing is preferred. Because plausible assessments fall on both sides of the threshold, the decision is sensitive to the evidence and assumptions. A point estimate just above the threshold is not, by itself, a strong reason to treat the decision as settled.

For a consequential choice, the team may need evidence that more convincingly supports the decision, or a way to reduce the consequences of being wrong. Whether it is worth delaying for more evidence also depends on the cost of delay. The expected-loss comparison makes those trade-offs explicit; it does not automatically dictate that more evidence must always be collected.

Reversibility and Staged Actions

A decision need not be limited to “proceed fully” or “defer.” A capped, monitored, or otherwise reversible first step can limit exposure while preserving an opportunity to learn. Such an option is not simply a lower evidence standard for the same action: it is a different action with a different set of consequences. Its losses and decision rule should be considered on their own terms.

Worked Example: Comparing Full Advance With a Capped Pilot

Continue the hypothetical development setting, but add a capped pilot. Assume that a mistaken full advance has loss 120 units and that deferring a suitable option has loss 15 units, as above. For the pilot, assume the maximum loss from starting it when the option is unsuitable is 24 units; declining the pilot when the option is suitable still has opportunity loss 15 units. These figures summarize consequences over the decision horizon being considered.

For full advance, the threshold remains \(120/(120+15)=0.8889\). For the capped pilot, the threshold is:

q_{\mathrm{threshold,pilot}}= \frac{24}{24+15} =\frac{24}{39}=0.6154.

If \(q=0.75\), the expected loss of full advance is \((1-0.75)(120)=30\) units, while deferring has expected loss \((0.75)(15)=11.25\) units. Full advance is not preferred under this comparison. For the pilot, expected loss is \((1-0.75)(24)=6\) units, compared with \(11.25\) units for declining it. The pilot is preferred under its stated loss assumptions.

The lower threshold for the pilot follows from limiting the loss of a mistaken start, not from treating the model evidence as more certain. The result depends on the assumed pilot losses and on the pilot being genuinely capped as described. If it exposes patients to substantial risk or creates downstream commitments, those consequences must be represented rather than assumed away.

Setting and Reviewing an Evidence Standard

A practical evidence standard should be stated before the result is interpreted as support for an action. It should identify the decision, the consequences that matter, and what degree of support is considered adequate. This is not a formula that converts a consequence directly into a required number of subjects or a single model-performance statistic. Rather, it gives a disciplined basis for judging whether the available evidence is proportionate to the decision.

Specify the decision and alternatives.
State what action is being considered, what it would be compared with, and the intended context in which the evidence will be used.

Describe the important wrong-decision outcomes.
Identify who or what could be affected, how severe the consequences could be, and whether the action can be reversed or limited.

Connect evidence to the decision.
State which model-based quantities inform the choice and what evidence would make the decision change. The model output must be relevant to the defined use.

Judge whether support is adequate.
Consider whether the evidence is sufficiently credible and precise for the consequences at stake, and whether plausible alternative assessments would lead to a different action.

This approach is different from applying one fixed statistical cutoff to every MIDD decision. A significance level \(\alpha\), for example, controls a specified long-run error rate under the assumptions of a particular testing procedure; it does not by itself say whether a development action has acceptable expected consequences. Nor does a high model fit statistic establish that the model is adequate for a decision. Those measures can be relevant evidence, but their meaning must be connected to the intended action.

The assumptions behind a consequence-based standard also require scrutiny. Probabilities may be poorly estimated or may not apply to the population or setting relevant to the decision. Losses may omit important patient, operational, or opportunity costs. Actions may have consequences that unfold over different time horizons. If these problems are material, a precise calculation can create false confidence. State what is included, what is omitted, and which assumptions drive the choice.

Common Pitfalls and Limitations

Using a universal evidence cutoff. “Every decision needs the same probability or p-value” ignores differences in reversibility, alternatives, and consequences. Define the decision and its loss structure first.
Assuming that higher consequences always mean a higher probability threshold. The threshold depends on the relative losses of the two types of error. If both losses are multiplied by the same factor, their ratio and threshold are unchanged, although the absolute consequences are larger.
Calling a point estimate sufficient because it crosses the threshold. If credible alternative assessments fall on both sides of the threshold, the preferred action may not be robust to uncertainty. Report that decision sensitivity rather than presenting the point estimate as decisive.
Counting only the cost of a wrong proceed decision. Deferring or stopping can also cause harm or lost opportunity. Leaving those costs out can distort the comparison.
Treating a staged action as the same action with weaker evidence. A capped pilot, a full advance, and a deferral have different consequences and should be compared as distinct choices.
Confusing statistical significance with decision sufficiency. A p-value does not state the expected loss of an action and does not establish that evidence is adequate for every context of use.

Expected-loss calculations are aids to structured reasoning, not objective answers independent of their inputs. They are limited by the quality of the probabilities, the completeness of the loss descriptions, and the actions included in the comparison. Their main value is to make explicit why a particular evidence standard is proportionate to a particular decision—and which assumptions would need to change for the decision to change.

Check Your Understanding

Try each problem, then compare your reasoning with the solutions.

A reversible choice has loss 8 units if an unsuitable option is adopted and loss 2 units if a suitable option is deferred. What is the adoption threshold?
For a consequential decision, the loss of proceeding when unsuitable is 50 units and the loss of deferring when suitable is 10 units. If \(q=0.80\), which action has lower expected loss, and by how much?
Why can multiplying both wrong-decision losses by 10 leave the probability threshold unchanged, even though the decision may deserve more careful attention?
A model estimate lies above the decision threshold, but plausible alternative assessments lie below it. What should be reported, and why is the estimate alone not enough to call the evidence decisive?

Solutions. (1) The threshold is \(8/(8+2)=0.800\); adoption is preferred only when \(q>0.800\) under this loss structure. (2) The threshold is \(50/(50+10)=0.8333\), so \(q=0.80\) is below it. Proceeding has expected loss \((1-0.80)(50)=10\) units; deferring has expected loss \((0.80)(10)=8\) units. Defer, with 2 units lower expected loss. (3) Multiplying both losses by 10 multiplies the numerator and denominator of the threshold by 10, leaving their ratio unchanged. The absolute expected losses are ten times larger, so the stakes remain greater even though the threshold is unchanged. (4) Report that plausible assessments cross the threshold and therefore change the preferred action. The point estimate does not show that the decision is robust to the uncertainty in the evidence and assumptions.