Why Test Context-of-Use Assumptions?
A model-informed decision is conditional: it depends on what population the model represents, how an outcome is defined, when it is assessed, and what rule turns model output into an action. A result that supports an action under one set of assumptions may not support it under another. Sensitivity analysis makes those dependencies visible. It asks, in a structured way, which plausible changes in context would change the action, and which would leave it unchanged?
The goal is not to make every possible scenario look favorable or to replace the evidence supporting the model. It is to show how the decision responds to specified changes in assumptions. A scenario is a conditional calculation: if this population, endpoint, horizon, or threshold is the relevant one, what action follows from the corresponding model output? Unless probabilities are explicitly assigned to scenarios, the analysis does not say how likely any scenario is.
Consider a hypothetical development decision: whether to advance a regimen for adults with a chronic condition. The model predicts the proportion who meet a response criterion, and the decision rule is to advance if that predicted proportion is at least a specified threshold. This simple setup will illustrate sensitivity analysis. The same reasoning applies to other model outputs and actions, provided the decision rule is stated clearly.
Set Up the Decision Before Varying Assumptions
Let \(z\) denote a specified context, including the population, outcome definition, and time horizon. Let \(p(z)\) be the model-predicted response proportion in that context, and let \(c\) be the decision threshold. In this example, the action is “advance” when the predicted proportion meets or exceeds the threshold; otherwise, the action is “do not advance.”
The threshold is part of the decision rule, not a property of the model prediction. It may represent a prespecified minimum response level needed to justify further development. Changing the threshold therefore asks whether the action is robust to a different decision standard. Changing the context asks whether the prediction, and thus the action, is robust to a different use of the model.
A useful summary is the decision margin, \(m=p(z)-c\). A positive margin means the prediction is above the threshold; a negative margin means it is below. The size of the margin indicates how far the prediction is from the decision boundary, but it is not itself a measure of statistical uncertainty.
A Simple Justification for the Decision Margin
Suppose the base-case prediction and threshold are \(p_0\) and \(c_0\). Under an alternative scenario, the prediction changes by \(\delta_p\) and the threshold by \(\delta_c\). The scenario margin is therefore
The action is unchanged if \(m_0\) and \(m_1\) are on the same side of zero (with a margin of exactly zero assigned to “advance” under the rule above). In particular, if \(m_0>0\) and \(\delta_p-\delta_c>-m_0\), then \(m_1>0\), so the action remains “advance.” If \(m_0<0\) and \(\delta_p-\delta_c<-m_0\), then \(m_1<0\), so it remains “do not advance.” This shows why a small base-case margin can signal a decision that is easy to change: a smaller shift in the prediction or threshold may cross zero.
This is a calculation about the stated rule and the supplied scenario values. It does not establish that the alternative prediction is correct, nor does it quantify the chance that the action is wrong. Those require evidence and uncertainty assessments beyond a scenario comparison.
Worked Example: Sensitivity to the Decision Threshold
Hypothetical data: For a defined adult population, a model predicts that 0.64 of patients will meet the prespecified response outcome at week 12. The base-case decision threshold is 0.60. Assume for this example that the prediction is appropriate for this population, outcome, and horizon, and that the stated rule is to advance when the prediction is at least the threshold.
At the base case, the decision margin is
Because \(0.64\geq0.60\), the base-case action is advance. Now consider a stricter threshold of 0.65 while holding the model prediction and all other assumptions fixed:
Since \(0.64<0.65\), the action under this threshold is do not advance. In this hypothetical example, the conclusion is sensitive to whether the required response proportion is 0.60 or 0.65. This comparison does not tell us which threshold is appropriate; that standard must be justified for the decision at hand.
Vary Population, Outcome, and Time Horizon
The population assumption identifies the people to whom the model output is meant to apply. A population scenario could alter age distribution, disease severity, concomitant treatment, or another defining feature. The model output should be obtained or justified for that population; simply relabeling the original prediction does not make it valid there.
The outcome definition specifies what counts as success. For example, “at least a 30% reduction in symptom score” is a different outcome from “at least a 50% reduction.” The associated predicted proportions need not be the same. A time horizon is similarly substantive: a predicted response at week 12 is not automatically a prediction at week 24. For each changed assumption, state which model output is being used and why it corresponds to the changed context.
Worked Example: Sensitivity to Population and Outcome Definition
Hypothetical model outputs: The base context is adults with moderate-to-severe disease, using at least a 30% symptom reduction at week 12 as the response definition. The model predicts a response proportion of 0.64. The decision threshold remains 0.60. Alternative model outputs are shown below. Assume these outputs are available for the listed scenarios and that the same threshold rule applies to each.
| Scenario | Predicted response proportion | Margin versus 0.60 | Action |
|---|---|---|---|
| Base population; at least 30% reduction | 0.64 | 0.64 − 0.60 = 0.04 | Advance |
| Broader population including older adults with comorbidity; at least 30% reduction | 0.58 | 0.58 − 0.60 = −0.02 | Do not advance |
| Base population; at least 50% reduction | 0.49 | 0.49 − 0.60 = −0.11 | Do not advance |
The base case supports advancement, while both alternatives do not meet the threshold. The first comparison changes the target population and holds the outcome definition and horizon fixed. The second changes the outcome definition and holds the base population and horizon fixed. These are separate comparisons, so they help identify which assumption is associated with a changed action in the specified scenarios.
The table does not prove that age or comorbidity caused the lower prediction, and it does not show that the stricter outcome is preferable. It reports conditional model outputs. Before using either alternative to inform a real decision, one would need to establish that the model can support predictions in that population or for that outcome.
Worked Example: Sensitivity to Time Horizon and Combined Changes
Hypothetical model outputs: Retain the base population and the at-least-30% response definition. The model predicts a response proportion of 0.64 at week 12 and 0.59 at week 24. First, keep the threshold at 0.60:
The week-12 action is “advance,” while the week-24 action is “do not advance.” The conclusion therefore depends on the assessment horizon in these hypothetical outputs.
Now examine a combined scenario: the broader population from the previous example, the stricter at-least-50% outcome, week 24, and a threshold of 0.55. Suppose the model provides a hypothetical predicted proportion of 0.38 for this complete scenario. Then
Because \(0.38<0.55\), the action is “do not advance.” This combined scenario illustrates that several assumptions can change together, but it cannot identify which one drives the result. The one-at-a-time comparisons and the combined comparison answer different questions: the former isolates specified changes, while the latter tests the decision under a joint set of changes.
Design, Diagnose, and Interpret the Analysis
A transparent sensitivity analysis makes the scenario choices traceable. Begin with a precise context and decision rule, then identify assumptions that could reasonably differ and matter to the action. Record the prediction, threshold, margin, and action for every scenario. When several assumptions may interact, include both focused comparisons and plausible joint scenarios.
Name the target population, outcome definition, time horizon, model output, threshold, and action rule. Calculate the base-case action and margin.
Vary assumptions that could plausibly differ for the intended decision. State each alternative precisely rather than describing it as merely “broader” or “stricter.”
Use predictions appropriate to each changed context. Explain the evidence or model basis for the outputs; do not assume the base prediction transports unchanged.
Calculate each margin and action. If the rule itself changes, record that as a threshold scenario rather than presenting it as a changed model prediction.
State which examined scenarios change the action, which do not, and why the analysis does not cover every possible context or establish scenario probabilities.
Several diagnostic signals deserve attention. A decision margin close to zero means modest changes may flip the action, so report adequate numerical precision and do not rely on rounded values alone. If a prediction changes abruptly across nearby scenarios, check whether the model output or decision rule has a discontinuity and whether the scenario definitions were applied consistently. If a prediction is identical across population or horizon changes, verify that this is a justified model result rather than an unexamined assumption.
A one-at-a-time analysis is useful for showing the effect of each specified change while holding other assumptions fixed. Its limitation is that it can miss interactions: for example, an outcome definition may matter more in one population than another. A joint scenario can reveal such combined sensitivity, but if several assumptions change at once, it cannot attribute the change to any single assumption. Using both approaches can provide a clearer picture.
Sensitivity analysis also differs from probabilistic uncertainty analysis. A scenario table lists outcomes conditional on chosen assumptions; it does not assign probabilities to those assumptions or to the resulting actions. Nor does a stable action across a handful of scenarios prove the decision is safe, correct, or robust to every relevant uncertainty. The result is limited by the scenarios considered and by whether the model can supply credible predictions in those contexts.
Common Pitfalls and Practical Value
- Calling a scenario probability. A prediction under an alternative population is not the probability that the population assumption is true. Report it as a conditional model result unless probabilities have separately been specified.
- Changing the context without changing the output basis. A week-12 estimate cannot simply be described as a week-24 estimate. Identify the model output that supports each context.
- Confounding prediction and threshold changes. A stricter decision threshold can change the action even when the model prediction is fixed. Show the prediction and threshold separately.
- Testing only convenient alternatives. A scenario analysis can look stable because it omitted assumptions most likely to challenge the action. Explain how scenarios were selected and include plausible decision-relevant alternatives.
- Overinterpreting a stable result. Stability across selected scenarios means only that those specified changes did not flip the action under the stated rule. It is not a guarantee about unexamined contexts.
This analysis matters because it distinguishes a decision that rests on a well-separated margin from one that depends on a narrow choice of context or threshold. That distinction can guide discussion of what assumptions need stronger justification, which populations or outcomes need additional evidence, and whether the proposed action is conditional on a particular use of the model.
Check Your Understanding
Answer each question, then compare with the solutions below.
- A model predicts a response proportion of 0.57, and the rule is to advance if the prediction is at least 0.60. What is the margin and action?
- The base-case margin is 0.03. Under an alternative context, the prediction rises by 0.01 and the threshold rises by 0.02. Does the action remain above the threshold?
- Why does a combined scenario that changes the population and outcome definition fail to show which change caused an action switch?
- A decision remains “advance” for every scenario examined. What can, and cannot, be concluded?
Solutions. (1) The margin is \(0.57-0.60=-0.03\), so the action is “do not advance.” (2) The new margin is \(0.03+0.01-0.02=0.02\), which remains positive; the action remains “advance.” (3) Both assumptions change together, so the comparison does not isolate either one. Separate comparisons that vary one assumption at a time can help identify their individual effects, while a joint scenario assesses their combined effect. (4) The action is stable across the specific scenarios tested under the stated rule. It cannot be concluded that the action is stable for untested assumptions, that the scenarios are equally likely, or that the decision is correct in every context.