Why Use Scenarios for Benefit-Risk Uncertainty?
A benefit-risk model can give a clear treatment ranking for a specified set of efficacy, safety, and utility inputs. The ranking may be less clear when those inputs are uncertain. A useful response is to ask how the comparison changes under a set of clinically interpretable alternative states: for example, a central estimate, a state with a stronger efficacy effect, or a state in which a safety concern is more pronounced. A scenario distribution makes these alternatives explicit and, when justified, assigns probability weights to them.
The purpose is not to choose whichever scenario produces the preferred answer. It is to represent uncertainty transparently, including plausible combinations that matter to the decision. This complements earlier material on joint uncertainty in benefits and harms and on separating parameter and preference uncertainty. Here the emphasis is on constructing coherent alternatives and comparing preferences across them.
For clarity, consider a treatment compared with a common comparator. Let \(\Delta E_j(s)\) be the difference in efficacy for treatment \(j\) versus the comparator in scenario \(s\), expressed as additional responders per 100 treated. Let \(\Delta H_j(s)\) be the corresponding difference in a specified harm, expressed as additional events per 100 treated. A simple additive score is
where \(w_E(s)\) and \(w_H(s)\) are utility points per additional event. Thus \(U_j(s)\) is a utility-score difference in points per 100 treated, relative to the comparator. A positive score favors the treatment over the comparator. If a harm difference is negative, the treatment has fewer harms and the harm term increases its score. Unless the weights are calibrated to a validated utility scale, this score is an index for comparing options, not a direct measure of quality-adjusted survival or individual patient utility.
A scenario can specify fixed values, ranges, or a probability distribution for inputs. The essential requirement is that the specification be clinically meaningful. If efficacy and toxicity may be related through dose, exposure, or patient susceptibility, a scenario should not combine values as though those inputs were unrelated without a rationale. Preference weights may also vary across scenarios, but this represents preference uncertainty only if the scenario construction explicitly says so.
Constructing Scenarios That Make Clinical Sense
Start with the decision and identify which uncertain inputs could change the treatment ranking. Then define a small set of interpretable alternatives. For example, scenarios might distinguish a central clinical estimate, a favorable efficacy state, and a safety-concern state. State clearly whether the scenarios represent alternative evidence interpretations, patient or clinical settings, preference profiles, or a combination. Avoid labels such as “best case” unless the values and the sense in which they are best are specified.
Clinical coherence is more important than filling every cell in a grid of possible inputs. Independently crossing low, central, and high values for every input can create combinations that are inconsistent with the biology or evidence. A high-efficacy state might plausibly accompany a higher toxicity risk if both are related to exposure. Conversely, if the evidence supports no such dependence, the scenarios should not impose one merely because it seems intuitive.
When scenario weights are used as probabilities, justify them from evidence, elicitation, or a clearly stated decision convention. Equal weights are not automatically objective probabilities. If weights are only used to summarize a set of alternatives, call them weights rather than probabilities and avoid probability interpretations. The distinction matters: a weighted average can be calculated either way, but its meaning depends on what the weights represent.
Worked Example: Comparing Two Treatments Across Clinical Scenarios
Suppose two hypothetical treatments, A and B, are compared with the same standard treatment. Use \(w_E=1\) utility point per additional responder and \(w_H=2\) utility points per additional serious harm. The scenario values below are differences per 100 treated. The scenario weights are assumed, for this illustration, to represent probabilities.
| Scenario | Weight | A: ΔE | A: ΔH | A: score | B: ΔE | B: ΔH | B: score |
|---|---|---|---|---|---|---|---|
| Central clinical state | 0.50 | 8 | 2 | 8 − 2(2) = 4 | 6 | 1 | 6 − 2(1) = 4 |
| Safety concern for A | 0.30 | 8 | 4 | 8 − 2(4) = 0 | 6 | 1.5 | 6 − 2(1.5) = 3 |
| Stronger efficacy for A | 0.20 | 10 | 2 | 10 − 2(2) = 6 | 6 | 1.5 | 6 − 2(1.5) = 3 |
The weights sum to \(0.50+0.30+0.20=1.00\), and each row specifies a joint set of efficacy and harm inputs for both treatments. Treatment A ties B in the central state, loses in the safety-concern state, and wins in the stronger-efficacy state.
The weighted mean score for A is \(0.50(4)+0.30(0)+0.20(6)=2.0+0+1.2=3.2\) points per 100 treated. For B it is \(0.50(4)+0.30(3)+0.20(3)=2.0+0.9+0.6=3.5\) points per 100 treated. On this weighted-mean criterion, B has the higher score by \(3.5-3.2=0.30\) points per 100 treated. Under the stated scenario probabilities, A has the higher score in 0.20 of the probability mass, B in 0.30, and they tie in 0.50. The average-score comparison and the probability of a strict win answer different questions; neither should be substituted for the other.
The result depends on the specified scenarios, weights, and utility weights. In particular, the three scenario weights are illustrative assumptions, not evidence-based probabilities. A decision maker should not interpret the numerical preference as robust unless the scenarios and weights are adequately supported.
Expected Scores and Scenario-Specific Preferences
For scenarios \(s=1,\ldots,K\) with probabilities \(q_s\), where \(q_s\geq 0\) and \(\sum_{s=1}^{K}q_s=1\), the expected score for treatment \(j\) is the probability-weighted mean of its scenario scores:
This follows by partitioning the uncertainty into the mutually exclusive scenarios. In scenario \(s\), the score equals \(U_j(s)\); that score contributes to the overall mean in proportion to the probability \(q_s\) of that scenario. Summing those contributions gives the expression above. For two treatments, compare their expected scores using the same scenarios and weights. This is appropriate when the decision criterion is expected score; it does not imply that every scenario favors the treatment with the larger mean.
A separate summary is the probability that one treatment has the higher score. For treatments A and B, with no ties, this is \(\sum_s q_s I\{U_A(s)>U_B(s)\}\), where \(I\{\cdot\}\) is 1 when the condition is true and 0 otherwise. If ties occur, report their probability mass separately. Reporting both summaries can clarify whether a mean difference reflects a small advantage across many scenarios or a large advantage in only a few.
If a scenario contains a distribution rather than fixed input values, the score may also vary within that scenario. The overall expected score then averages the scenario-specific expected scores, weighted by \(q_s\). Be explicit about whether a table shows fixed scenario values, scenario means, or the full distribution; those are not interchangeable.
Worked Example: A Four-State Distribution for Efficacy and Harm
Consider one hypothetical treatment versus a comparator, with \(w_E=1\) and \(w_H=2\). Construct four joint states, each specifying additional responders and additional serious harms per 100 treated. Assign the stated probabilities to the paired inputs rather than combining their marginal distributions independently.
| Joint state | Probability | ΔE | ΔH | Score \(U=\Delta E-2\Delta H\) |
|---|---|---|---|---|
| 1 | 0.30 | 10 | 2 | 10 − 2(2) = 6 |
| 2 | 0.20 | 6 | 3 | 6 − 2(3) = 0 |
| 3 | 0.30 | 8 | 3 | 8 − 2(3) = 2 |
| 4 | 0.20 | 4 | 6 | 4 − 2(6) = −8 |
The probabilities sum to \(0.30+0.20+0.30+0.20=1.00\). The expected score is
Thus the mean preference score is \(0.80\) points per 100 treated in favor of the treatment relative to the comparator. The score is strictly positive in states 1 and 3, so the probability of a strictly positive score is \(0.30+0.30=0.60\). State 2 is a tie, with probability 0.20; state 4 favors the comparator, with probability 0.20. The positive mean does not mean the treatment wins in every state. It also is not a 60% probability that the treatment is clinically beneficial in some broader sense; it is the probability of a positive score under this specified model and scenario distribution.
The paired values should be reviewed for clinical plausibility. For example, the combination of 4 additional responders and 6 additional serious harms in state 4 should be retained only if such a joint state is credible given the evidence and the population. If it is not, replace it with a clinically defensible state and reassess the results.
Preference Uncertainty Can Change the Ranking
Clinical inputs and utility weights represent different kinds of uncertainty. A clinical scenario can vary efficacy and harm effects while holding weights fixed. A preference scenario can hold clinical effects fixed while varying how strongly decision makers value efficacy relative to harm. A combined scenario may vary both, but should identify which source is changing. This distinction prevents a change in treatment ranking from being incorrectly attributed to new clinical evidence when it actually reflects a different trade-off.
Worked Example: Utility Profiles Reverse a Treatment Preference
Suppose hypothetical treatment A, relative to treatment B, provides 3 additional responders per 100 patients and 1 additional serious harm per 100 in a lower-harm-aversion clinical state. In a second state it provides 2 additional responders and 1 additional serious harm per 100. Consider two elicited preference profiles: a lower-harm-aversion profile with \(w_E=1\), \(w_H=1\), and a higher-harm-aversion profile with \(w_E=1\), \(w_H=3\). The scenario distribution pairs the lower-harm-aversion profile with the first clinical state at probability 0.60 and the higher-harm-aversion profile with the second at probability 0.40.
In the first scenario, the score difference favoring A over B is \(1(3)-1(1)=2\) utility points per 100 treated. In the second, it is \(1(2)-3(1)=-1\) point per 100 treated, so B is preferred there. The expected score difference is \(0.60(2)+0.40(-1)=1.20-0.40=0.80\) points per 100 treated. A has the higher score in 0.60 of the scenario probability mass; B has the higher score in 0.40. This comparison shows both the influence of preference weights and the influence of the paired clinical assumptions. It does not establish that these two profiles represent a particular patient population unless that distribution has been justified.
Diagnostics, Alternatives, and Limitations
A scenario analysis is only as informative as its construction. Check the following before interpreting the rankings:
- Input coherence: Are efficacy and harms expressed for the same population, time horizon, and treatment comparison? Are paired values compatible with the evidence and clinical mechanism?
- Coverage: Do the scenarios include plausible states that could reverse the decision, including unfavorable states and relevant safety concerns?
- Weight interpretation: Are weights probabilities, elicited degrees of plausibility, or merely analytic weights? Do they sum to 1 when used in a probability-weighted mean?
- Preference clarity: Are the utility weights fixed or varied? If varied, is that change distinguishable from clinical uncertainty?
- Decision stability: Does the preferred treatment remain the same across most plausible scenarios, or does the ranking depend on one scenario or one weight?
A common failure is to vary each input separately and then present the resulting combinations as a joint distribution. One-way sensitivity checks can show how a score changes when one input moves while others are fixed, but they do not establish that the resulting combinations occur jointly or quantify their probabilities. A scenario distribution instead specifies coherent combinations directly. A fully probabilistic model can represent continuous uncertainty in more detail, but it also requires defensible joint distributions and dependence assumptions. Scenario analysis is often easier to communicate and audit, though it may omit important states or conceal variation within each state.
Another pitfall is to treat scenario weights as objective simply because they are written as decimals. If probability weights cannot be justified, present results scenario by scenario, or show how the comparison changes over alternative weight sets. Do not report a weighted expected score as though it were a uniquely determined clinical fact. The score also inherits the limitations of the utility model: omitted outcomes, uncertain weights, or poorly measured effects can change the ranking. A scenario analysis cannot repair an inappropriate outcome definition or a clinically implausible model.
Check Your Understanding
Answer each question, then compare your reasoning with the brief solutions.
- Why is independently combining every low and high efficacy and harm value not automatically a valid scenario distribution?
- For two scenarios with probabilities 0.70 and 0.30, treatment A has scores 2 and −4. Calculate its expected score and the probability that its score is strictly positive.
- What information is needed before interpreting scenario weights as probabilities?
- A treatment has a higher expected score than its comparator but loses in one scenario. Is this necessarily a contradiction? Explain.
Solutions. (1) The crossed combinations may be clinically incompatible or may ignore dependence; scenarios should represent credible joint states. (2) The expected score is \(0.70(2)+0.30(-4)=1.4-1.2=0.20\) points in the stated score units; the probability of a strictly positive score is 0.70. (3) The scenarios must be mutually exclusive and collectively exhaustive for the uncertainty represented, and their probabilities need a defensible basis. (4) No. An expected score averages over all scenarios, so a treatment can have the higher mean while losing in one or more states.