Why Decision Robustness Is a Separate Question
A PBPK model can have unresolved limitations without every decision based on it being equally uncertain. A limitation may change predicted exposure substantially while leaving the action unchanged, if the predictions remain on the same side of a decision threshold. Conversely, a modest change in prediction can matter greatly when it crosses a threshold. The relevant question is therefore not only, “How much can the prediction move?” but also, “Can a plausible interpretation of the limitation change the decision?”
This assessment requires a defined context of use, a specified decision rule, and a finite set of plausible scenarios representing the unresolved issue. The scenarios are alternative ways of handling that issue in the model, not automatically a statistical sample from a probability distribution. Their predictions should be compared with the threshold that actually governs the intended decision.
For example, suppose a decision rule triggers a dose adjustment when a predicted exposure ratio is greater than or equal to a specified threshold. Each scenario produces an exposure ratio, and each ratio is classified using that same rule. The comparison is meaningful only if the prediction metric, comparator, population, dosing conditions, and threshold are held consistent across scenarios. Otherwise, a difference in classification may reflect a changed question rather than a changed interpretation of the limitation.
Set Up the Threshold Comparison
First state the decision in operational terms. Identify the model output, its units or scale, the threshold or acceptable interval, and what action follows each possible classification. A threshold might be an exposure ratio, a predicted concentration, or another context-specific quantity. It should be chosen for the intended decision before inspecting which scenario results fall on either side.
Next, specify the unresolved limitation and describe the alternative interpretations that define the scenarios. For instance, a tissue partitioning limitation might be represented by two mechanistically defensible parameterizations, or uncertainty about an absorption process might be represented by two supported model structures. The alternatives should be plausible in light of the available evidence and model purpose. They should not be selected merely because they produce reassuring or alarming results.
For a one-sided rule that triggers action when a prediction \(Y\) is greater than or equal to threshold \(T\), let \(Y_i\) be the prediction under scenario \(i\). The classification is “no trigger” when \(Y_i<T\), and “trigger” when \(Y_i\geq T\). For a finite scenario set, all scenarios give “no trigger” exactly when the largest prediction is below \(T\). All scenarios give “trigger” exactly when the smallest prediction is at least \(T\).
This follows directly from the decision rule. If the largest scenario prediction is below \(T\), then every \(Y_i\) is below \(T\). If any \(Y_i\) is at least \(T\), the maximum cannot be below \(T\). Similarly, if the smallest prediction is at least \(T\), every scenario triggers action; if any prediction is below \(T\), the minimum is below \(T\). The strict and non-strict inequalities matter: under this rule, a prediction exactly equal to \(T\) triggers action.
For an acceptable interval \([L,U]\), where the decision is acceptable only if \(L\leq Y_i\leq U\), all scenarios are acceptable exactly when the minimum is at least \(L\) and the maximum is at most \(U\). This test evaluates classification stability, not the probability that a prediction lies in the interval.
Worked Scenario Comparisons
Worked Example: Stable Decision Below an Exposure Threshold
Hypothetical situation. A PBPK analysis is being used to decide whether a modeled interaction triggers a dose-management action. The specified rule is to trigger the action if the ratio of AUC in the interaction condition to AUC in the reference condition is at least 1.25. A limitation in the representation of hepatic clearance has two plausible interpretations, in addition to the current model specification. The reference-condition AUC is 16.0 mg·h/L in each scenario.
The interaction-condition AUC predictions are 18.0, 18.4, and 19.2 mg·h/L. The corresponding exposure ratios are calculated as follows:
The largest ratio is 1.200, which is below the threshold of 1.25. Each scenario therefore leads to no trigger. The decision is robust across these specified interpretations, although the predicted interaction ratio varies by 0.075 between the smallest and largest value. The distance from the largest prediction to the threshold is \(1.25-1.20=0.05\) ratio units.
This result does not show that the true ratio is below 1.25, nor that all plausible model limitations have been covered. It says only that these justified scenarios, evaluated under the stated rule, agree on the action. The relatively small threshold margin also makes the scope of the scenario set important.
Worked Example: An Acceptability Decision That Changes
Hypothetical situation. A PBPK analysis is used to assess whether a modeled dose produces an acceptable AUC relative to a reference exposure. The pre-specified acceptable range is 0.80 to 1.25, inclusive. An unresolved absorption limitation is represented by three plausible scenarios. For a reference AUC of 25.0 mg·h/L, the scenarios predict AUC values of 19.5, 26.0, and 32.25 mg·h/L.
Divide each scenario AUC by the same reference AUC:
The first ratio is below the lower bound of 0.80, the second lies inside the interval, and the third is above the upper bound of 1.25. Thus the scenarios do not produce a common classification: the acceptable-exposure decision changes with the interpretation of the limitation. The minimum ratio is 0.780 and the maximum is 1.290, so the scenario range extends beyond both bounds.
It would be misleading to summarize these results by saying that the “average scenario” is acceptable: the mean ratio, \((0.780+1.040+1.290)/3=1.037\), is inside the range, but averaging classifications or predictions does not make the individual scenario decisions agree. The decision rule applies to each scenario prediction. The appropriate conclusion is that the decision is not robust to this specified set of absorption interpretations.
Worked Example: Stable Action Despite Different Limiting Metrics
Hypothetical situation. A dose-reduction action is specified if either the AUC ratio is greater than 1.25 or the Cmax ratio is greater than 1.50. The two metrics are both required because the intended decision is sensitive to either total exposure or peak exposure. Plausible interpretations of an absorption limitation produce the following ratios:
| Scenario | AUC ratio | Cmax ratio |
|---|---|---|
| Current specification | 23.6/20.0 = 1.180 | 14.2/10.0 = 1.420 |
| Faster absorption | 24.2/20.0 = 1.210 | 15.6/10.0 = 1.560 |
| Lower effective clearance | 25.6/20.0 = 1.280 | 14.7/10.0 = 1.470 |
The current specification crosses neither threshold, so it does not trigger dose reduction. Faster absorption gives a Cmax ratio of 1.560, which exceeds 1.50 and triggers the action, even though its AUC ratio remains below 1.25. Lower effective clearance gives an AUC ratio of 1.280, which exceeds 1.25 and triggers the action, even though its Cmax ratio remains below 1.50. Therefore, all scenarios do not agree: the action is not robust. The reason for triggering differs by scenario, so reporting only AUC or only Cmax would miss a decision-relevant result.
This example also illustrates why a rule involving several metrics must be evaluated as written. Under the “either metric” rule, exceeding one threshold is sufficient. If the intended policy instead required both thresholds to be exceeded, the classifications would differ; that would be a different decision rule and must not be substituted after seeing the results.
Interpret the Result and Diagnose Weak Comparisons
A useful report gives more than a label such as “robust” or “not robust.” State the scenarios and their rationale, the prediction for each, the applicable threshold, each scenario’s classification, and the closest distance to a threshold. For a one-sided rule, that distance is the threshold minus the largest prediction when all predictions are below the threshold, or the smallest prediction minus the threshold when all predictions are above it. The distance has the same units as the prediction; for a ratio it is expressed in ratio units.
Robustness can be falsely reassuring if the scenario set is too narrow. Omitting a plausible interpretation, changing only a parameter that is easy to vary, or treating an unsupported extreme as the full range of uncertainty can all distort the conclusion. Scenarios should be tied to the particular unresolved limitation and to evidence about plausible alternatives. Each scenario should also be run and checked consistently, including numerical diagnostics relevant to the model output.
- Check the decision rule. Confirm the metric, comparator, population, threshold, boundary convention, and action are stated consistently.
- Check scenario comparability. Hold the intended use and other assumptions fixed unless they are part of the limitation being examined.
- Check prediction quality. Investigate implausible outputs, numerical problems, and unexpected changes in other model quantities before interpreting a threshold crossing.
- Check the margin. Agreement very close to a threshold may be fragile to an omitted scenario or a small change in an input, even when the current scenario classifications agree.
- Check what the scenarios represent. A finite set of alternatives does not establish coverage of every possible interpretation or quantify how likely each one is.
A common mistake is to call a decision robust because the scenario predictions are “similar.” That reasoning is insufficient: two similar predictions can lie on opposite sides of a threshold, while substantially different predictions can remain on the same side. The correction is to apply the pre-specified decision rule to every scenario and compare classifications; prediction spread and decision stability answer different questions.
How This Differs From Other Analyses
Threshold-focused scenario comparison complements, but does not replace, sensitivity analysis or probabilistic uncertainty analysis. Sensitivity analysis asks how model outputs change as inputs or assumptions vary. It can help identify influential limitations, but an influential input does not necessarily change the decision, and a small output change can matter near a threshold. Probabilistic uncertainty analysis uses probability distributions and a defined uncertainty model to characterize a distribution of predictions. Scenario comparison instead asks whether explicitly described plausible interpretations yield the same decision; without justified scenario probabilities, it does not estimate the probability of either decision.
| Approach | Main question | What it does not establish by itself |
|---|---|---|
| Threshold-focused scenario comparison | Do specified plausible interpretations lead to the same classification? | Probability that a classification is correct or that scenarios cover all uncertainty |
| Sensitivity analysis | How does a prediction respond to changes in selected inputs or assumptions? | Whether the decision changes, unless results are explicitly assessed against its rule |
| Probabilistic uncertainty analysis | What distribution of predictions follows from specified probability assumptions? | Validity of the probability model or decision robustness to unrepresented structural limitations |
The methods can inform one another. Sensitivity results may identify which limitations deserve explicit scenarios, while probability-based analyses may characterize uncertainty within a specified model. Neither removes the need to define the intended decision and inspect the threshold directly. A stable classification across selected scenarios is a conditional robustness finding, not a substitute for evaluating the evidence supporting the model or the scenarios.
Check Your Understanding
Answer each question using the stated decision rule. Brief solutions follow the questions.
- A rule triggers action when \(Y\geq 2.00\). Three plausible scenarios predict 1.72, 1.94, and 1.88. Is the decision robust, and what is the margin to the threshold?
- An acceptable interval is \([0.90,1.10]\). Scenarios predict 0.95, 1.08, and 1.12. Do all scenarios support acceptability?
- Why does an average prediction inside a threshold range not prove that scenario decisions are robust?
- What does “robust across the examined scenarios” not establish about the probability of the decision?
Solutions. (1) All predictions are below 2.00, so the no-trigger decision is robust across these scenarios. The margin from the largest prediction is \(2.00-1.94=0.06\) prediction units. (2) No. The value 1.12 exceeds the upper bound of 1.10, so the scenario classifications do not all support acceptability. (3) The decision rule applies to each scenario prediction; averaging can conceal predictions on different sides of a threshold. The average’s classification does not establish agreement among the scenarios. (4) It does not establish a probability for the decision. Unless justified probabilities are assigned to scenarios under a defined probability model, the comparison is a conditional assessment of those specified alternatives.