Why Compare Decisions Across Uncertain Inputs?
A decision based on one set of input estimates can be clear yet fragile: a modest change in efficacy, safety, cost, or another assumption may change which dose is preferred. Probabilistic sensitivity analysis (PSA) makes that uncertainty explicit. Rather than evaluating each action only at one best-estimate input vector, it evaluates the actions across joint draws from specified input distributions and records which action performs best on each draw.
The resulting summary answers a particular question: under the specified uncertainty model, how often is each action preferred? It does not, by itself, state the probability that a treatment succeeds, the probability that an action is truly optimal in an unknowable sense, or the probability that a patient benefits. Those are different quantities and require their own definitions.
Before interpreting a PSA, the decision model must define the available actions, the inputs treated as uncertain, and a utility scale that permits meaningful comparisons among actions. Utility may combine benefits and harms, or other decision-relevant outcomes, but its direction must be consistent: higher values must always mean better outcomes. The model and uncertainty distributions are assumptions for the analysis, not facts established by the simulation.
Each draw is one coherent input vector. Evaluating all actions using the same draw makes their comparison conditional on the same assumed state of the uncertain inputs. The draws should represent the joint uncertainty distribution specified for the analysis; any dependence included in that distribution should be retained. The interpretation also depends on the model adequately representing how inputs map to utility. PSA does not repair a misspecified utility function or an implausible uncertainty distribution.
Estimand and Calculation
On a draw, the preferred action is the action with the largest utility. If there is a unique maximizer, it receives all the allocation for that draw. If \(k\) actions tie for the largest utility, this tutorial allocates \(1/k\) to each tied action. Define \(I_m(a)\) as that allocation weight for action \(a\) on draw \(m\), and define the tie-allocated preferred-action probability as its expected weight under the specified joint uncertainty distribution.
Here \(\pi_a\) is the target probability and \(\hat{\pi}_a\) is its Monte Carlo estimate from \(M\) draws. This tie allocation is a convention that produces a set of probabilities summing to one. A different question is the probability that an action is among the maximizers: its indicator is one for every tied maximizer, so those probabilities can sum to more than one when ties have positive probability. Do not mix that alternative estimand with the tie-allocated probabilities.
The first equality follows because a draw has at least one maximizer: either one action receives weight one, or the \(k\) tied maximizers each receive \(1/k\), whose total is one. Taking expectations or averaging over draws preserves that total. This is a useful arithmetic check, but it cannot establish that the inputs or model are appropriate.
Worked Example: Comparing Two Doses on Joint Draws
Worked Example: Comparing Two Doses on Joint Draws
Hypothetical illustration. Suppose utility is defined as \(U=100p-200r\), where \(p\) is a benefit probability and \(r\) is an adverse-event probability. The coefficients convert both inputs to the same utility scale; for this example, all probabilities are between zero and one and higher utility is preferred. Five joint draws give the following values.
| Draw | Low-dose \(p\) | Low-dose \(r\) | Low-dose utility | High-dose \(p\) | High-dose \(r\) | High-dose utility | Outcome |
|---|---|---|---|---|---|---|---|
| 1 | 0.50 | 0.04 | 42 | 0.62 | 0.07 | 48 | High |
| 2 | 0.65 | 0.05 | 55 | 0.70 | 0.10 | 50 | Low |
| 3 | 0.60 | 0.05 | 50 | 0.70 | 0.10 | 50 | Tie |
| 4 | 0.55 | 0.08 | 39 | 0.60 | 0.125 | 35 | Low |
| 5 | 0.51 | 0.05 | 41 | 0.66 | 0.13 | 40 | Low |
For example, on draw 5, low-dose utility is \(100(0.51)-200(0.05)=51-10=41\), while high-dose utility is \(100(0.66)-200(0.13)=66-26=40\). Low dose, not high dose, is preferred on that draw. On draw 3, both utilities equal \(100(0.60)-200(0.05)=50\) and \(100(0.70)-200(0.10)=50\), so each dose receives allocation \(1/2\).
Low dose receives three full allocations, from draws 2, 4, and 5, plus one-half from the tie. High dose receives one full allocation, from draw 1, plus one-half from the tie. Thus:
The estimates sum to one. In this five-draw illustration, low dose is preferred more often under the tie-allocation rule. The estimate is based on only these five hypothetical draws and should not be treated as a precise estimate of a real decision probability.
Preferred-Action Probability Is Not Expected Utility
The probability an action is preferred and its expected utility summarize different features of the uncertainty distribution. The former counts how often an action has the highest utility, with the stated tie rule. The latter averages utility values, including how large gains or losses are. If the decision rule is to maximize expected utility, the selected action is the one with the largest expected utility, not necessarily the action with the largest preferred-action probability.
Worked Example: Frequent Wins Do Not Guarantee the Best Expected Utility
Hypothetical illustration. Compare action A with action B across ten joint draws. B has utility \(0\) on every draw. A has utility \(10\) on nine draws and utility \(-100\) on one draw. There are no ties: A is preferred on nine draws and B on one, so their estimated preferred-action probabilities are \(9/10=0.90\) and \(1/10=0.10\), respectively.
The mean utilities are different:
If the decision criterion is to choose the action with the greater expected utility, B is selected because \(0>-1\), even though A is preferred on 90% of these draws. The size of A's loss on the remaining draw changes its mean utility but counts as just one draw in the win frequency. Neither summary is inherently wrong; each answers a different question. A decision report should state which criterion informs the choice.
The same distinction applies when a single best-estimate input vector is used. Evaluating utilities at that vector produces a deterministic comparison conditional on those inputs. It does not reveal how often the ranking changes across uncertainty draws. Conversely, a preferred-action probability does not specify the utility difference when an action wins or loses. Reporting both can be informative, but they should not be substituted for one another.
Worked Example: Three Actions and a Tie
Worked Example: Three Actions and a Tie
Hypothetical illustration. Consider three actions, A, B, and C, evaluated on five joint draws. Their utilities, in the same decision-relevant units, are:
| Draw | Utility A | Utility B | Utility C | Maximizer(s) |
|---|---|---|---|---|
| 1 | 8 | 6 | 4 | A |
| 2 | 3 | 9 | 5 | B |
| 3 | 6 | 6 | 2 | A and B tie |
| 4 | 2 | 4 | 7 | C |
| 5 | 10 | 8 | 1 | A |
A wins draws 1 and 5 and receives one-half on draw 3, for total allocation \(2.5\). B wins draw 2 and receives one-half on draw 3, for total allocation \(1.5\). C wins draw 4, for total allocation \(1\). Dividing each by \(M=5\) gives:
The estimates sum to \(0.50+0.30+0.20=1.00\). These figures describe tie-allocated preferred-action frequencies over the displayed draws; they do not imply that A has the highest mean utility. That would require separately calculating and comparing the mean utilities.
Assumptions, Diagnostics, and Limitations
Interpretation requires the uncertainty distribution to represent the uncertainty relevant to the decision, the decision model to evaluate each action consistently, and the utility scale to reflect the stated preference ordering. The simulated draws must be generated from the specified joint distribution. These conditions are substantive: a large number of draws cannot make an incorrect distribution, an omitted action, or an unsuitable utility definition correct.
Several practical checks help detect implementation errors:
- Check the utility direction and units. Confirm that larger values mean better outcomes for every action and that terms combined in a utility calculation are on a compatible scale.
- Check action-by-action calculations. Recalculate selected draws by hand, especially near ties. A single arithmetic or coding error can change the recorded maximizer.
- Check the tie rule. Apply the same rule on every draw. With equal splitting, allocations across all actions must sum to one on each draw and the estimated preferred-action probabilities must sum to one.
- Check the draw construction. Compare actions using the same joint draw of uncertain inputs. Using unrelated input draws for different actions would not represent their comparison under a common uncertainty state.
- Inspect the decision model and uncertainty inputs. Implausible input ranges, inconsistent definitions, or missing relevant uncertainty can produce misleading summaries even when the arithmetic is correct.
A common mistake is to say, “Action A has a 70% probability of being the best, so it should be selected.” That conclusion does not follow without a decision rule that selects the most frequently preferred action. Under an expected-utility rule, the action with the greatest expected utility is selected; the frequency of being preferred is a complementary description of ranking uncertainty. Another mistake is to interpret the estimate as a probability that the model is correct. It is conditional on the specified model, inputs, distributions, action set, and tie convention.
PSA complements rather than replaces other forms of sensitivity analysis. A one-way or two-way analysis varies selected inputs to show how a decision changes over chosen values; scenario analysis compares specified alternatives. PSA summarizes action rankings across a joint distribution of inputs, but its result depends on that distribution and can hide the magnitude of utility differences. It also does not, by itself, establish that a decision is robust to assumptions outside the distributions used. The preferred-action probability is therefore best reported alongside the decision criterion and a concise description of what uncertainty was represented.
Check Your Understanding
Answer each question using the definitions and tie rule in this tutorial. Short solutions follow the questions.
- On one draw, three actions have utilities 7, 7, and 4. What allocation does each action receive?
- Across 20 draws, action A has 12 unique wins, action B has 6 unique wins, and two draws are ties between A and B. What are their tie-allocated preferred-action probabilities?
- In one analysis, A is preferred on 80% of draws, but B has the higher expected utility. Which action is selected under an expected-utility decision rule?
- Why should all actions be evaluated using the same joint input draw?
Solutions. (1) Each of the two tied maximizers receives \(1/2\), and the third action receives zero. (2) A receives \(12+2(1/2)=13\) allocations and B receives \(6+2(1/2)=7\), so \(\hat{\pi}_A=13/20=0.65\) and \(\hat{\pi}_B=7/20=0.35\); they sum to one. (3) B is selected, because an expected-utility rule selects the action with the higher expected utility, not the higher preferred-action frequency. (4) A shared joint draw compares actions under the same assumed input state; unrelated draws would not provide that like-for-like comparison.