Why Test a Decision Against Its Assumptions?
A development recommendation can look decisive while depending heavily on one uncertain input. A treatment may be favored if baseline risk is high, but not if it is low; a modest change in the assumed treatment effect or in the value assigned to preventing an event may also reverse the comparison. The relevant question is not only whether a recommendation follows from a chosen set of inputs, but whether it remains the same across reasonable alternatives.
This tutorial uses a deliberately simple benefit-risk model to make that question concrete. The calculation is a structured sensitivity analysis: hold all but one input fixed, change that input, and recalculate the decision. It does not establish which input value is correct, nor does it measure the probability that a recommendation will succeed.
A Simple Benefit-Risk Model
Suppose a hypothetical treatment is considered for a defined patient population and time horizon. The beneficial outcome is an event the treatment may prevent, and the harm is a treatment-related adverse event. Let \(p_0\) be the probability of the beneficial outcome in the control group, and let \(r\) be the relative risk reduction with treatment. Thus, the risk under treatment is \(p_0(1-r)\), and the absolute reduction in risk is \(p_0r\). Let \(q\) be the absolute increase in adverse-event risk attributable to treatment.
Assign \(u_B\) utility points to preventing one beneficial-outcome event and \(u_H\) utility points as the disutility of one additional adverse event. These weights must refer to the same decision context and time horizon. Under an additive expected-utility calculation, incremental net benefit per patient is the expected utility with treatment minus the expected utility with control:
The first term is the expected utility from beneficial events prevented; the second is the expected utility lost to excess harms. If \(\Delta U>0\), treatment has greater expected utility under this model. If \(\Delta U<0\), control has greater expected utility. If it equals zero, the model assigns equal expected utility. A decision-maker may require a positive margin rather than merely a value above zero; denote that required margin by \(C\), in utility points per patient. The decision rule is then to favor treatment when \(\Delta U\ge C\).
For communication, results are often expressed per 1,000 treated patients. The net benefit on that scale is \(1000\Delta U\), and a threshold \(C\) per patient corresponds to \(1000C\) points per 1,000 patients. The event counts used in this presentation are expected counts, not predictions that a particular group will experience fractional events.
Worked Example: Calculating the Base-Case Recommendation
Worked Example: Calculating the Base-Case Recommendation
Consider a hypothetical treatment for which the control risk of the outcome to be prevented is \(p_0=0.30\), the relative risk reduction is \(r=0.30\), and the absolute increase in adverse-event risk is \(q=0.03\). Suppose preventing one outcome is assigned \(u_B=10\) utility points and one additional adverse event has disutility \(u_H=20\) points. Use a decision threshold of \(C=0\).
The absolute reduction in the beneficial outcome is \(p_0r=0.30(0.30)=0.09\), or 90 events prevented per 1,000 treated. The treatment-related increase in adverse events is \(q=0.03\), or 30 excess events per 1,000.
The incremental expected utility is
Thus, the expected net benefit is \(0.30\) utility points per patient, or \(1000(0.30)=300\) utility points per 1,000 treated. Because \(0.30>0\), the model favors treatment under the stated assumptions and decision rule. This is a model-based recommendation, not a claim that every treated patient benefits or that the inputs are known without uncertainty.
Deriving and Using Break-Even Values
A break-even value makes the decision rule transparent. With \(C=0\), treatment and control have equal expected utility when the benefit term equals the harm term. Solving the model equation for the relative risk reduction gives:
If \(p_0\), \(u_B\), and \(q u_H\) are positive, treatment has greater expected utility when \(r>r_{\mathrm{crit}}\). The same equality can be solved for baseline risk:
When the treatment effect and utility weights are fixed, the model favors treatment for \(p_0>p_{\mathrm{crit}}\). These relationships also show why a baseline risk is decision-relevant: the same relative risk reduction prevents more events when the untreated risk is higher.
If the decision-maker requires a positive margin \(C\), the break-even condition changes from \(\Delta U=0\) to \(\Delta U=C\). Solving for baseline risk gives:
A higher required margin raises the baseline risk needed to favor treatment. The threshold is meaningful only under the chosen model, units, and policy rule; it is not a universal clinical cutoff.
Worked Example: Finding the Baseline-Risk Threshold
Use the base-case assumptions \(r=0.30\), \(q=0.03\), \(u_B=10\), and \(u_H=20\), with \(C=0\). The break-even baseline risk is
The model is at break-even when the control risk is 20%. For example, at \(p_0=0.15\), \(\Delta U=(0.15)(0.30)(10)-(0.03)(20)=0.45-0.60=-0.15\), so control is favored. At \(p_0=0.30\), \(\Delta U=0.90-0.60=0.30\), so treatment is favored. These conclusions follow from changing baseline risk while keeping the other inputs fixed.
One-Way Sensitivity Analysis in Practice
A useful analysis selects a plausible range for the assumption being varied, calculates the decision across that range, and reports where the recommendation changes. The range should be justified by available evidence, relevant clinical settings, or explicitly stated decision scenarios. A grid of arbitrary values can show mathematical behavior, but it does not establish that those values are plausible.
Worked Example: Varying Baseline Risk
Keep \(r=0.30\), \(q=0.03\), \(u_B=10\), and \(u_H=20\) fixed. The net benefit per 1,000 patients is
| Baseline risk \(p_0\) | Expected events prevented per 1,000 | Expected excess adverse events per 1,000 | Net benefit (points per 1,000) | Recommendation at \(C=0\) |
|---|---|---|---|---|
| 0.15 | 45 | 30 | -150 | Control |
| 0.20 | 60 | 30 | 0 | Break-even |
| 0.30 | 90 | 30 | 300 | Treatment |
| 0.40 | 120 | 30 | 600 | Treatment |
For example, at \(p_0=0.15\), \(1000\Delta U=1000(3(0.15)-0.60)=1000(-0.15)=-150\) points per 1,000. At \(p_0=0.30\), the corresponding result is \(1000(0.90-0.60)=300\). The recommendation changes at the previously derived 20% baseline-risk threshold. This is a decision boundary for the specified model, not a claim that risks just above or below 20% are known precisely.
Worked Example: Varying the Treatment Effect
Now hold baseline risk at \(p_0=0.30\), with \(q=0.03\), \(u_B=10\), and \(u_H=20\). Vary only the relative risk reduction \(r\). The net benefit per 1,000 is \(1000(3r-0.60)\).
At \(r=0.15\), the calculation is \(1000(3(0.15)-0.60)=1000(-0.15)=-150\) points per 1,000, favoring control. At \(r=0.20\), it is \(1000(0.60-0.60)=0\), the break-even value. At \(r=0.30\), it is \(1000(0.90-0.60)=300\), favoring treatment. The critical treatment effect is therefore \(r_{\mathrm{crit}}=(0.03)(20)/((0.30)(10))=0.20\), or a 20% relative risk reduction.
This calculation tests how much the decision relies on the assumed effect. It does not tell us whether a 20% effect is plausible; that requires evidence about the treatment effect and its applicability to the target population.
Worked Example: Changing Utility Weights and the Required Margin
Return to the base-case event risks and effects: \(p_0=0.30\), \(r=0.30\), and \(q=0.03\). The number of prevented events per excess adverse event is \(p_0r/q=0.09/0.03=3\). Treatment has positive net utility when
With \(u_B=10\), the break-even harm weight is \(u_H=30\). At the base-case \(u_H=20\), net benefit is \(0.30\) points per patient. If \(u_H=35\), then \(\Delta U=(0.09)(10)-(0.03)(35)=0.90-1.05=-0.15\), so the model instead favors control.
A separate change is to the decision threshold. If the required margin is \(C=0.20\) points per patient, the break-even baseline risk becomes \(((0.03)(20)+0.20)/((0.30)(10))=0.80/3.00=0.2667\), or about 26.7%. At \(p_0=0.30\), net benefit is \(0.30\), which exceeds \(C=0.20\); at \(p_0=0.25\), net benefit is \(1000\)-scaled as \(1000(3(0.25)-0.60)=150\) points per 1,000, equivalent to \(0.15\) per patient, below the required margin. The decision rule therefore favors treatment at the former value but not the latter.
Interpreting Robustness, Diagnostics, and Alternatives
A recommendation is robust to a particular assumption over a stated range if it does not change across that range. If the sign of net benefit changes within the range, the decision is sensitive to that assumption. It can then be useful to report the break-even value alongside the evidence-supported range. If the entire range lies on one side of the threshold, that specific assumption is less likely to reverse the recommendation within the model.
Check the calculation before interpreting it. Confirm that \(p_0\) and \(q\) are probabilities on the same time horizon, that \(r\) is a relative reduction rather than an absolute reduction, and that \(p_0r\) does not exceed the control risk. Check that the utility weights refer to compatible outcomes and units, and that event counts and net-benefit totals scale consistently. A threshold outside the permissible range for a probability or treatment effect may indicate that the decision will not reverse within that input’s feasible range, or that the model has been specified or interpreted incorrectly.
One-way analysis is easy to communicate and helps isolate which assumption drives a decision, but it cannot show what happens when assumptions change together. A table or threshold calculation is often clearer than reporting only a single base case. A tornado plot can summarize one-way changes across multiple inputs, while a scenario analysis can change several inputs together to represent a coherent clinical setting. Both remain dependent on the selected values. Analyses that represent uncertainty with probability distributions answer a different question and require additional assumptions; a one-way sensitivity result should not be interpreted as the probability that a treatment is beneficial.
The model also has substantive limitations. It treats each event consequence as if it could be summarized by a fixed utility weight, does not represent the timing or severity distribution of events, and may omit outcomes that matter to patients or decision-makers. If beneficial and adverse events can occur in the same patient, simple multiplication of marginal risks and weights may not capture their joint consequences. Heterogeneous baseline risks or treatment effects may also make a population-average calculation misleading. The earlier material on benefit-risk weighting and subgroup differences provides context for defining relevant weights and populations; the present calculation tests how a recommendation responds to specified assumptions.
Common Pitfall
A frequent error is to say, “The treatment remains preferred in my base case, so the decision is robust.” A base case checks only one combination of inputs. Robustness requires evaluating the decision over a stated range or set of alternative assumptions. The reverse error is also common: seeing a recommendation change under an extreme value and concluding that the decision is necessarily unreliable. First establish whether that value is plausible for the target population and time horizon. Sensitivity analysis identifies dependence on assumptions; it does not decide whether those assumptions are credible.
Check Your Understanding
Answer each question, then compare with the brief solutions.
- In the model above, \(p_0=0.25\), \(r=0.20\), \(q=0.02\), \(u_B=10\), and \(u_H=15\). Calculate \(\Delta U\) and state the recommendation when \(C=0\).
- For \(p_0=0.25\), \(q=0.02\), \(u_B=10\), and \(u_H=15\), find the break-even relative risk reduction.
- At the values in question 1, what is the expected net benefit per 1,000 patients? Explain why it is not a guaranteed event count or outcome.
- What does it mean if the decision changes as baseline risk is varied across a plausible range? Does that result establish which baseline risk is correct?
Solutions. (1) \(\Delta U=(0.25)(0.20)(10)-(0.02)(15)=0.50-0.30=0.20\) utility points per patient, so treatment is favored at \(C=0\). (2) \(r_{\mathrm{crit}}=q u_H/(p_0u_B)=(0.02)(15)/((0.25)(10))=0.30/2.50=0.12\), or 12%; the assumed 20% effect exceeds this threshold. (3) \(1000(0.20)=200\) utility points per 1,000. It is an expected net utility calculated from model probabilities and weights, not a prediction that a particular 1,000 patients will realize exactly that total. (4) The recommendation is sensitive to baseline risk over the range assessed. This identifies a decision-relevant assumption but does not establish the true risk; evidence appropriate to the target population and time horizon is needed to assess it.