Why Stress-Test a Decision?
A QSP model can make a dose or development recommendation look precise even when its predictions depend on uncertain parameter values or on a choice among plausible model structures. A useful decision analysis therefore asks more than whether a reference simulation meets a criterion: does the recommendation remain the same across the plausible alternatives that matter? A recommendation that persists across a deliberately chosen set of stress tests is less sensitive to those assumptions than one that changes under a plausible alternative.
This is a conditional assessment, not a claim that every possible model has been considered. Its result depends on the decision rule, the range of plausible scenarios, and the consistency of the simulations used to evaluate them. The earlier material on mapping QSP outputs to development decision rules and setting quantitative go-or-no-go criteria provides the decision context; here the focus is how to test that decision against model uncertainty.
State the Rule Before Testing Scenarios
For a concrete rule, suppose a candidate dose must achieve at least \(B_{\min}\) percent reduction in a disease-relevant measure and must keep the predicted probability of a specified adverse event at or below \(S_{\max}\) percent. Let \(B_j\) and \(S_j\) be the predicted benefit and safety measure under scenario \(j\). The decision is:
Here \(A_j=1\) means advance under scenario \(j\), and \(A_j=0\) means do not advance. The rule is conjunctive: both criteria must pass for advancement. A scenario that misses either criterion recommends do not advance. The same logic applies to other prespecified QSP outputs and thresholds, but the output definitions, units, and boundary conventions must be fixed before comparing scenarios.
For a finite set of \(J\) scenarios, the recommendation is invariant exactly when all \(A_j\) values are equal. In particular:
The second condition is important. Failing to meet a threshold across the entire scenario set can still produce a robust decision: if every scenario recommends do not advance, the recommendation is invariant. Conversely, it is not enough that some scenarios fail a criterion to call the decision non-invariant; at least one scenario must pass both criteria and recommend advance.
How to Construct a Useful Stress Test
A stress test begins with a defined decision question, not an unstructured collection of simulations. Specify the dose or advancement choice, the decision rule and its thresholds, the QSP outputs and units, and which uncertainties are being varied. Parameter scenarios may represent plausible alternative values for uncertain parameters; structural scenarios may represent scientifically defensible alternative equations, feedbacks, or pathway representations. A scenario should be interpretable: when a recommendation changes, the analyst should be able to identify what assumption changed.
Parameter combinations should be coherent. If two parameters are linked by physiology, data, or a shared mechanism, pairing extreme values independently may create a combination that is not plausible. Likewise, structural alternatives should be credible enough to test, rather than arbitrary changes chosen only to produce a reversal. The scenario set should include a reference case and alternatives motivated by the evidence, model qualification, or mechanistic uncertainty.
Record the thresholds, output definitions, units, dose, and whether equality counts as passing. Do not adjust a threshold after seeing which scenarios pass.
List the parameter values and structural assumptions to vary, and explain why each scenario is scientifically defensible.
For each scenario, calculate the same decision outputs using consistent implementation and solver settings, then apply the rule separately.
State whether recommendations agree across scenarios and identify the particular assumption or combination associated with any change.
Worked Decision Checks
Worked Example: Advancement Is Robust Across Parameter Scenarios
A hypothetical development team evaluates one dose using benefit \(B\), the predicted percent reduction in a disease-relevant measure, and safety measure \(S\), the predicted adverse-event probability in percent. The prespecified rule is \(B_{\min}=30\%\) and \(S_{\max}=8\%\). Three parameter scenarios give the following outputs.
| Scenario | Benefit \(B_j\) (%) | Safety measure \(S_j\) (%) | Decision |
|---|---|---|---|
| Reference parameters | 36 | 6.0 | Advance |
| Lower target potency | 32 | 7.5 | Advance |
| Higher clearance | 34 | 5.0 | Advance |
For the lower-potency scenario, for example, \(32\% \geq 30\%\) and \(7.5\% \leq 8\%\), so both criteria pass. In the other two scenarios, the benefit values are \(36\%\) and \(34\%\), each at least \(30\%\), while the safety values \(6.0\%\) and \(5.0\%\) are each at most \(8\%\). Thus \(A_1=A_2=A_3=1\): within this specified parameter set, the advancement recommendation is robust. It does not establish robustness to untested parameter values or structural alternatives.
Worked Example: A Robust Do-Not-Advance Recommendation
Keep the same hypothetical thresholds, \(B_{\min}=30\%\) and \(S_{\max}=8\%\), but consider three different scenarios. The team wants to know whether a uniform recommendation can result even when different scenarios miss different criteria.
| Scenario | Benefit \(B_j\) (%) | Safety measure \(S_j\) (%) | Reason for do not advance |
|---|---|---|---|
| Reference parameters | 25 | 5.0 | Benefit below threshold |
| Higher pathway drive | 35 | 10 | Safety measure above limit |
| Alternative feedback strength | 28 | 9.0 | Both criteria fail |
In scenario 1, \(25\% < 30\%\), so the rule recommends do not advance even though \(5.0\% \leq 8\%\). In scenario 2, benefit passes because \(35\% \geq 30\%\), but safety fails because \(10\% > 8\%\). In scenario 3, \(28\% < 30\%\) and \(9.0\% > 8\%\). No scenario passes both criteria, so all three have \(A_j=0\). The recommendation is robustly do not advance across this set. A criterion failing in some or even all scenarios does not by itself imply that recommendations differ.
Worked Example: Structural Assumptions Reverse the Recommendation
Suppose the same thresholds are applied to a reference model and two plausible alternatives. The alternative structural model includes a feedback pathway that is absent from the reference model; a separate parameter scenario uses lower potency. The following numbers are hypothetical model predictions.
| Scenario | Assumption changed | Benefit \(B_j\) (%) | Safety measure \(S_j\) (%) | Decision |
|---|---|---|---|---|
| Reference | Reference parameters and structure | 34 | 7.0 | Advance |
| Parameter alternative | Lower potency | 29 | 6.0 | Do not advance |
| Structural alternative | Feedback pathway included | 33 | 9.0 | Do not advance |
The reference scenario passes because \(34\% \geq 30\%\) and \(7.0\% \leq 8\%\). The lower-potency scenario fails the benefit criterion, \(29\% < 30\%\), even though its safety result passes. The feedback scenario meets the benefit criterion, \(33\% \geq 30\%\), but fails the safety criterion, \(9.0\% > 8\%\). Therefore the set contains at least one advance and at least one do-not-advance recommendation; the decision is non-invariant. The table also identifies two distinct potential reversal drivers: lower potency changes efficacy classification, whereas the structural feedback alternative changes safety classification. Follow-up interpretation should preserve that distinction rather than attributing the reversal vaguely to “model uncertainty.”
Justifying the Classification
The classification follows directly from the binary decision rule. Define \(A_j\) to be 1 only if both criteria pass, and 0 otherwise. If every scenario passes both, then \(A_j=1\) for all \(j\), which is robust advancement. If every scenario fails at least one, then \(A_j=0\) for all \(j\), which is robust non-advancement. If the recommendations are not all equal, the binary values must include both 0 and 1, which is exactly the non-invariant case.
For robust advancement, checking the worst-case benefit and safety values is sufficient for this two-threshold rule: \(\min_j B_j \geq B_{\min}\) and \(\max_j S_j \leq S_{\max}\). For robust non-advancement, the relevant check is instead whether each scenario fails at least one criterion. In general, it is not sufficient to inspect marginal ranges alone: a low benefit value and a high safety value might occur in different scenarios, and the joint scenario-level rule determines the recommendation.
Diagnostics, Alternatives, and Limitations
A decision stress test is only as reliable as the simulations and scenario definitions it uses. Check that every scenario was run with the same implementation conventions, appropriate solver settings, and consistent event handling; reproducibility review and regression tests help detect computational differences that could masquerade as scientific uncertainty. Inspect the predicted outputs near each threshold, where small numerical or parameter changes can flip a binary classification. If a result is close to a threshold, report the distance from it and verify that numerical tolerances are small enough for the decision.
A useful report presents scenario-level outputs, not merely the number of scenarios that recommend advance. It should state which uncertainties were varied, which were held fixed, why the alternatives are plausible, and whether conclusions depend on a particular combination. A reversal may reflect a parameter, a structural assumption, or their interaction. If several assumptions change at once, the cause is not isolated; targeted comparisons that change one defensible feature at a time can clarify the driver, while retaining combined scenarios when interactions are plausible.
| Approach | What it answers | Key caution |
|---|---|---|
| Reference-case decision | Does the chosen reference model meet the rule? | Does not assess whether alternatives change the recommendation. |
| Scenario stress test | Does the recommendation persist across specified plausible alternatives? | Conclusion is conditional on scenario coverage and plausibility. |
| Probabilistic uncertainty analysis | How often or with what probability does a decision outcome occur under a specified uncertainty distribution? | Requires defensible probability distributions and dependence assumptions; a scenario count is not a probability. |
| Average predicted outputs | What is the mean output across a chosen set or distribution? | An average can meet a threshold even when the decision reverses across individual scenarios. |
Scenario analysis is not a substitute for assigning probabilities when those probabilities are needed, and a collection of equally listed scenarios does not imply equal likelihood. Nor can a finite set prove a recommendation holds over every scientifically possible model. Its value is disciplined transparency: it reveals whether a recommendation survives the alternatives judged relevant, and shows where a decision depends on assumptions. If the result is non-invariant, report the reversal conditions plainly; do not conceal them by averaging benefit and safety outputs or by selecting only the reference model.
Common Pitfall
- Wrong reasoning: “At least one scenario misses the benefit threshold, so the recommendation is not robust.” This confuses criterion failure in a scenario with disagreement among scenario-level recommendations.
- Correction: Apply the full rule separately in every scenario. A robust do-not-advance result occurs when every scenario fails at least one criterion, even if the reason differs. The decision is non-invariant only if some scenarios pass both criteria and others fail at least one.
- Another warning: Do not call the fraction of scenarios that advance a probability of advancement unless the scenarios have a justified probability distribution and are analyzed accordingly.
Check Your Understanding
Answer each question using the scenario-level decision rule; solutions follow each question.
- With \(B_{\min}=30\%\) and \(S_{\max}=8\%\), scenarios have \((B,S)=(27\%,5\%)\) and \((28\%,7\%)\). Is the recommendation invariant? Solution: Both scenarios fail benefit because \(27<30\) and \(28<30\), while both pass safety. Each recommends do not advance, so the recommendation is robustly do not advance.
- With the same thresholds, scenarios have \((32\%,7\%)\), \((29\%,6\%)\), and \((35\%,9\%)\). Is the recommendation invariant? Solution: The first passes both criteria and advances. The second fails benefit; the third fails safety. There are both advance and do-not-advance outcomes, so the recommendation is non-invariant.
- Why does a failed benefit threshold in one scenario not by itself establish a decision reversal? Solution: That scenario recommends do not advance, but all other scenarios could also recommend do not advance, for example by failing benefit or safety. A reversal requires at least one scenario to pass both criteria and recommend advance as well.
- Across a stress-test set, the smallest benefit is \(31\%\) and the largest safety measure is \(7.8\%\). With thresholds \(30\%\) and \(8\%\), respectively, what follows? Solution: Every benefit is at least \(31\%\geq30\%\), and every safety measure is at most \(7.8\%\leq8\%\). Therefore every scenario passes both criteria and advancement is robust within the tested set.
- Why is the fraction of listed scenarios recommending advance not automatically a probability? Solution: Scenarios may have been selected to explore assumptions rather than sampled from a probability distribution. Without justified scenario probabilities and dependence assumptions, their count does not quantify the probability of advancement.