Why Decision Robustness Requires More Than a Base Case
A model-informed recommendation depends not only on data and estimated parameters, but also on assumptions used to turn evidence into a forecast or decision. A base-case result can look compelling while relying on one uncertain assumption: for example, how missed doses affect exposure, how a dose-response curve behaves outside the observed range, or how an operational cost is counted. Sensitivity analysis asks whether changing such assumptions within defensible bounds changes the result that matters.
The goal is not to find a collection of alternative numbers that all support a preferred conclusion. It is to identify assumptions that could plausibly differ, explain why they matter to the decision, and report how the model estimate and recommendation respond. The earlier material on model specification and assumption disclosure establishes why assumptions must be made explicit; here, the focus is their decision relevance and the robustness of the resulting action.
A useful sensitivity analysis therefore makes four items visible: the reference case, the assumption being varied, the range or alternative scenarios considered, and the decision rule. It also reports both the changed estimate and whether the decision changes. If the action rule is “proceed when predicted benefit is at least 2.0 days,” the rule is part of the analysis, not an afterthought.
Selecting Assumptions That Matter
Sensitivity analyses are most informative when they prioritize assumptions that are both uncertain and capable of changing the decision. A parameter may be uncertain but immaterial if even its plausible extremes leave the recommendation unchanged. Conversely, a modest uncertainty can matter greatly when the base-case estimate lies near a decision threshold.
Set the range using relevant evidence: observed data, external evidence, design constraints, expert elicitation, or a clearly stated conservative bound. Do not choose a range just because it produces a desired result. State what is held fixed in each analysis. In a one-way analysis, one assumption changes while the others remain at their reference values. This isolates its effect but does not show what happens when several assumptions shift together. A scenario analysis changes a coherent combination of assumptions—for example, a lower treatment effect together with a higher implementation cost—when that combination has a defensible rationale.
Sensitivity analysis should distinguish two questions. First, how much does the numerical estimate change? Second, does that change alter the selected action? A forecast can move noticeably without crossing the decision threshold; a small shift can also reverse a recommendation when the base case is close to the boundary. Neither the size of an estimate change nor the label “robust” is meaningful without the decision context.
Worked Example: Missed Doses and a Benefit Threshold
Suppose a hypothetical dosing analysis estimates benefit, in additional symptom-free days over a fixed period, as \(B(r)=3.6(1-r)\), where \(r\) is the fraction of doses missed. The stated decision rule is to recommend the regimen when \(B(r)\geq 2.0\) days. The reference assumption is \(r=0.20\). This simple relationship is treated as the model for this illustration; in an actual analysis, its form and the plausible range for \(r\) would need support.
At the reference value, the predicted benefit is \(B(0.20)=3.6(1-0.20)=3.6(0.80)=2.88\) days. Since \(2.88\geq2.0\), the recommendation is to proceed under the stated rule.
To find the tipping point, solve the decision inequality:
Thus, \(r=4/9\), approximately \(0.444\) or 44.4%, is the boundary. At exactly \(r=4/9\), the predicted benefit is 2.0 days, so the recommendation still holds because the rule includes equality. It changes only for \(r>4/9\). For example, at \(r=0.45\), \(B(0.45)=3.6(0.55)=1.98\) days, which is below the threshold. The analysis identifies not only a change in the estimate but the assumption value at which the action changes.
One-Way and Joint Scenario Analyses
A one-way analysis is useful for locating influential assumptions and communicating their individual effects. A joint scenario analysis is needed when assumptions may plausibly move together or when a decision depends on their combined effect. Scenarios are not automatically probabilities: unless probabilities are assigned and justified, a “worst case” or “conservative case” is a conditional calculation, not a statement about how likely that case is.
A tipping point is the value of an assumption at which the decision rule reaches its boundary. Its interpretation depends on the rule’s equality convention. For a rule requiring an estimate to be at least a threshold, equality satisfies the rule; the recommendation changes only beyond the boundary in the direction that fails the inequality. Report sufficient precision to make this clear, and avoid describing a rounded approximation as if it were the exact switching value.
Worked Example: Combining Success Probability and Cost
Consider a hypothetical development decision with utility \(U=100p-c\), where \(p\) is the model-predicted probability of success, 100 is the utility assigned to success, and \(c\) is cost in the same utility units. The rule is to proceed when \(U\geq0\). In the reference case, \(p=0.62\) and \(c=58\), giving \(U=100(0.62)-58=62-58=4\) utility units. The reference recommendation is to proceed.
Suppose the review identifies \(p=0.58\) to \(0.62\) and \(c=58\) to \(62\) as plausible ranges. A one-way change in probability, with cost fixed at 58, gives \(U=100(0.58)-58=0\) at the lower probability bound; the recommendation still holds at equality. At \(p=0.56\), if that value were included in a broader scenario, utility would be \(56-58=-2\), and the recommendation would not hold. A one-way change in cost, with \(p=0.62\), gives \(U=62-62=0\) at cost 62, also still meeting the rule.
Now consider their joint lower-probability, higher-cost scenario: \(p=0.58,\ c=62\). Then \(U=100(0.58)-62=58-62=-4\), so the recommendation changes. In general, the decision rule is \(100p-c\geq0\), or \(p\geq c/100\). The reference case proceeds, but the recommendation is not robust across the full stated rectangle of plausible values, because at least one combination fails the rule. These scenarios do not imply that the joint case has any particular probability; that would require a separate, justified probability model.
Assumption Form and Structural Alternatives
Changing a numerical value within one model is not the only useful sensitivity analysis. A recommendation may depend on the model form itself—for example, on whether an exposure-response curve is assumed to saturate in a particular way. Where credible alternative forms exist, compare their decision-relevant predictions under comparable conditions. Do not treat a different model form as merely another parameter value unless that equivalence is justified.
Worked Example: A Dose-Response Assumption Near a Threshold
Suppose a hypothetical exposure-response model is \(E(C)=E_{\max}C/(EC_{50}+C)\), where \(E(C)\) is predicted response in response units at exposure \(C\). Assume \(E_{\max}=100\), \(C=5\) exposure units, and a decision rule to recommend the dose if predicted response is at least 45 units. These values are illustrative; the analysis assumes the model form is appropriate over the exposure range considered.
With the reference assumption \(EC_{50}=5\) exposure units, \(E(5)=100(5)/(5+5)=500/10=50\) response units. The rule is met. If \(EC_{50}=6\), then \(E(5)=500/11=45.45\) units, still meeting the rule. If \(EC_{50}=6.2\), then \(E(5)=500/11.2=44.64\) units, so the rule is not met.
The boundary follows by solving \(500/(EC_{50}+5)\geq45\). Since the denominator is positive, \(500\geq45(EC_{50}+5)\), so \(275\geq45EC_{50}\), and hence \(EC_{50}\leq275/45\approx6.111\) exposure units. At exactly \(275/45\), predicted response is 45 units and the recommendation still holds; it fails for larger \(EC_{50}\). If a defensible range for \(EC_{50}\) includes values on both sides of this boundary, the recommendation is assumption-dependent over that range.
How to Review and Report the Analysis
A decision-focused report should allow a reviewer to reconstruct why the conclusion is or is not robust. It should name the assumptions varied, the reference values, the rationale and bounds for alternatives, the quantities held fixed, the resulting estimates, and the decision rule. If the analysis uses a set of scenarios rather than a probability distribution, say so. Report any tipping point with units and clarify whether the decision holds at equality.
A practical review can proceed in the following order:
Name the action under consideration, the decision-relevant estimate, and the threshold or rule used to select the action.
Identify which assumptions could materially affect the action and explain why the chosen alternatives are plausible.
Show the reference and alternative results, including units, and state explicitly where the recommendation changes.
Describe the assumptions over which the action is stable and identify any important uncertainty not covered by the analysis.
Diagnostics for a weak analysis include a narrow or unexplained range, varied inputs that are irrelevant to the decision, contradictory assumptions combined without rationale, and results reported without the corresponding action. Check calculations at the boundary and just beyond it; a rounding error can incorrectly label the tipping point. Also check that the “plausible” range is consistent with the evidence and that the model is being used within a domain where its predictions are interpretable.
One-way analysis is easier to interpret than a large set of combined scenarios, but can miss interactions and joint changes. Coherent scenario analysis addresses combinations but may obscure which assumption drives a result. A local sensitivity measure can summarize how rapidly an estimate changes near the reference case, but a local slope alone cannot establish behavior over a broad range or whether a threshold is crossed. Probabilistic sensitivity analysis can incorporate distributions over uncertain inputs, but its conclusions depend on those distributions and their dependence structure; it is not interchangeable with an unweighted scenario set.
No sensitivity analysis proves robustness outside its tested assumptions, resolves poor data, or repairs an unsuitable model. It also cannot establish that a decision rule itself is appropriate. Its contribution is narrower and valuable: it shows how much the recommendation depends on specified choices and gives decision makers a clearer account of what evidence or assumption would change the action.
Common Pitfall
A common error is to say that a recommendation “changes at 44.4%” after calculating a boundary of \(4/9\), without checking the inequality or equality case. That wording can be wrong when the decision rule includes equality. For the missed-dose example, \(r=4/9\) gives exactly 2.0 days and still meets the rule \(B(r)\geq2.0\); the recommendation changes only for \(r>4/9\). Another error is to call a decision robust merely because several tested scenarios agree, when the scenarios were chosen arbitrarily or exclude a plausible decision-relevant assumption. Robustness must always be stated relative to the assumptions and range actually examined.
Check Your Understanding
Answer each question using the stated decision rule and assumptions. Solutions follow.
- A model predicts benefit \(B(x)=5-2x\) units. The rule is to proceed when \(B(x)\geq3\). Find the boundary value of \(x\), and state whether the recommendation holds at the boundary.
- In a one-way analysis, the estimate changes substantially but remains above the action threshold for every tested value. What can be concluded, and what cannot be concluded?
- Why is a joint low-efficacy, high-cost scenario not automatically a probability statement?
- Name two features that make a sensitivity range more defensible than a range selected solely to produce favorable results.
Solutions. (1) Solve \(5-2x\geq3\): \(-2x\geq-2\), so \(x\leq1\). The recommendation holds at \(x=1\), where benefit equals 3 units, and fails for \(x>1\). (2) The action is robust over the tested values, although the estimate is sensitive; this does not establish robustness outside those values or under other untested assumptions. (3) A scenario specifies a combination of values, not how likely that combination is. A probability claim requires justified probability distributions and, where relevant, a dependence structure. (4) The bounds can be supported by observed or external evidence, design constraints, or documented expert elicitation, and the report can explain why the assumptions are relevant to the decision.