Why Examine Decision Sensitivity?
A Bayesian arm-selection decision depends on both the observed data and the prior model. In a multi-arm setting, reasonable changes to prior means, prior scales, or the amount of borrowing between arms can change posterior probabilities. If a decision rule selects an arm only when its posterior probability of clinically meaningful benefit exceeds a threshold, even a modest change in a probability near that threshold may change the action.
Sensitivity analysis evaluates that dependence systematically. It does not identify a uniquely correct prior by trying several and choosing the one that gives a preferred decision. Instead, it asks whether the conclusion is stable across a prespecified set of scientifically plausible assumptions, and explains which assumptions matter if it is not. The introductory material on robust and skeptical priors for arm decisions motivates alternative prior specifications; here the emphasis is on comparing their consequences for selection.
A stable decision does not require identical posterior probabilities under every prior. Conversely, similar probabilities do not guarantee a stable decision if they lie close to the decision threshold. Report both the probability changes and the resulting actions.
A Model for Separating Prior Mean and Borrowing
Consider two treatment effects, \(\theta_1\) and \(\theta_2\), measured as risk differences: positive values favor treatment, and a value of \(0.10\) represents a 10-percentage-point increase in response. Write their common level and half-difference as \(c=(\theta_1+\theta_2)/2\) and \(d=(\theta_1-\theta_2)/2\), so \(\theta_1=c+d\) and \(\theta_2=c-d\). A prior on \(c\) controls the overall level of the effects, while a prior on \(d\) controls how similar the arms are expected to be. A smaller prior standard deviation for \(d\) means stronger borrowing: the arms are more strongly shrunk toward one another.
For a transparent worked model, suppose the estimated effects \(Y_1\) and \(Y_2\) have independent Normal sampling distributions with known variance \(0.016\) each. Thus \(Y_i\mid\theta_i\sim N(\theta_i,0.016)\). Put independent priors \(c\sim N(m,s_c^2)\) and \(d\sim N(0,\tau_d^2)\). Here \(m\) and \(s_c\) describe the prior for the common level, and \(\tau_d\) is the prior scale for between-arm differences. The likelihood and priors are Normal, the sampling variances are treated as known, and the two estimates are assumed independent for this illustration. These are modeling assumptions, not facts guaranteed by the data.
The transformed estimates are \(\bar Y=(Y_1+Y_2)/2\) and \(\hat d=(Y_1-Y_2)/2\). Under the stated independent sampling model, each has sampling variance \(0.016/2=0.008\); they are independent. Conjugacy gives posterior variances \(C\) for \(c\) and \(D\) for \(d\), and posterior means \(\mu_c\) and \(\mu_d\):
Because the transformed likelihood and priors are independent across \(c\) and \(d\), the posterior distributions of these quantities are independent. Consequently, each arm has posterior variance \(C+D\), while the arm-specific posterior means are \(\mu_c+\mu_d\) and \(\mu_c-\mu_d\). The two arm effects are not independent in the posterior: they share \(c\) and \(d\), and their covariance is \(C-D\). For this tutorial’s individual-arm probabilities, the marginal Normal distributions suffice.
Worked Example: Baseline Posterior Decision
Worked Example: Baseline Posterior Decision
Suppose a hypothetical two-arm trial produces estimated risk differences \(Y_1=0.26\) and \(Y_2=0.22\), each with known sampling variance \(0.016\). Consider the priors \(m=0.20\), \(s_c=0.20\), and \(\tau_d=0.20\). The rule is to select an arm if \(q_i\geq0.90\); if both qualify, select the one with the larger \(q_i\). Benefit means a risk difference exceeding \(\Delta=0.10\).
First, \(\bar Y=(0.26+0.22)/2=0.24\) and \(\hat d=(0.26-0.22)/2=0.02\). Since \(s_c^2=\tau_d^2=0.04\), the posterior variances are \(C=D=(125+25)^{-1}=0.006667\), where \(125=1/0.008\) and \(25=1/0.04\). The posterior means are \(\mu_c=0.006667(125(0.24)+25(0.20))=0.2333\) and \(\mu_d=0.006667(125(0.02))=0.01667\).
The arm posterior means are \(0.2500\) and \(0.2167\), and their common marginal posterior standard deviation is \(\sqrt{0.006667+0.006667}=0.1155\). Therefore, \(q_1=\Phi((0.2500-0.10)/0.1155)=\Phi(1.299)\approx0.903\), while \(q_2=\Phi((0.2167-0.10)/0.1155)=\Phi(1.010)\approx0.844\). Only arm 1 meets the 0.90 threshold, so the rule selects arm 1. These probabilities describe uncertainty about the effects conditional on this model, prior, and observed data; they are not probabilities that the prior assumptions are correct.
What the Posterior Calculation Is Doing
The update for each transformed parameter is a precision-weighted average. For example, the posterior mean of \(c\) weights the data estimate \(\bar Y\) by its likelihood precision \(1/0.008\) and the prior mean \(m\) by its prior precision \(1/s_c^2\). To see this, the terms in the Normal likelihood and prior involving \(c\) have exponent proportional to
Collecting the coefficients of \(c^2\) gives posterior precision \(1/0.008+1/s_c^2\); collecting the linear terms gives posterior mean \(C\{\bar Y/0.008+m/s_c^2\}\). The same completion-of-the-square argument for \(d\), whose prior mean is zero, gives precision \(1/0.008+1/\tau_d^2\) and mean \(D\hat d/0.008\). This derivation also shows why a prior scale affects both the amount of shrinkage and posterior uncertainty.
For this example, the data contribute more precision than either prior: \(125\) compared with \(25\). That fact does not make prior sensitivity irrelevant. A decision can be sensitive when the posterior probability is near its threshold, and changing \(\tau_d\) affects both the estimated difference and the posterior variance.
Worked Example: Changing the Prior Mean
Worked Example: Changing the Prior Mean
Keep the data, \(s_c=0.20\), \(\tau_d=0.20\), threshold \(\Delta=0.10\), and decision rule from the baseline example. Vary only the common-level prior mean over \(m=0.10, 0.20,\) and \(0.30\). The posterior variances remain \(C=D=0.006667\), and \(\mu_d=0.01667\). Since \(\mu_c=0.006667(125(0.24)+25m)=0.2000+m/6\), the calculations are:
| Prior mean \(m\) | Posterior arm means \((\theta_1,\theta_2)\) | \(q_1\) | \(q_2\) | Decision |
|---|---|---|---|---|
| 0.10 | (0.2333, 0.2000) | 0.876 | 0.807 | No arm qualifies |
| 0.20 | (0.2500, 0.2167) | 0.903 | 0.844 | Select arm 1 |
| 0.30 | (0.2667, 0.2333) | 0.926 | 0.876 | Select arm 1 |
For example, when \(m=0.10\), arm 1 has \(q_1=\Phi((0.2333-0.10)/0.1155)=\Phi(1.155)\approx0.876\); arm 2 has \(q_2=\Phi((0.2000-0.10)/0.1155)=\Phi(0.866)\approx0.807\). Neither reaches 0.90. This grid therefore shows a change in the operational conclusion: the baseline prior leads to arm 1, while the lower prior mean leads to no selected arm. It does not show that one of these prior means is correct. Whether the alternatives are plausible must be justified scientifically.
Worked Example: Prior Scale and Borrowing Strength
Worked Example: Prior Scale and Borrowing Strength
Return to \(m=0.20\) and \(s_c=0.20\), and compare \(\tau_d=0.10, 0.20,\) and \(0.40\). The data difference remains \(\hat d=0.02\). For \(\tau_d=0.10\), the prior variance is \(0.01\), so \(D=(125+100)^{-1}=0.004444\) and \(\mu_d=0.004444(125(0.02))=0.01111\). The posterior arm means are \(0.2444\) and \(0.2222\), with standard deviation \(\sqrt{0.006667+0.004444}=0.1054\). Hence \(q_1=\Phi(1.370)\approx0.915\) and \(q_2=\Phi(1.160)\approx0.877\); arm 1 qualifies.
For \(\tau_d=0.20\), the baseline calculation gives \(q_1\approx0.903\) and \(q_2\approx0.844\), so arm 1 qualifies. For \(\tau_d=0.40\), \(D=(125+6.25)^{-1}=0.007619\) and \(\mu_d=0.007619(2.5)=0.01905\). The arm means are \(0.2524\) and \(0.2143\), and the standard deviation is \(\sqrt{0.006667+0.007619}=0.1195\). Thus \(q_1=\Phi((0.2524-0.10)/0.1195)=\Phi(1.275)\approx0.899\), while \(q_2=\Phi((0.2143-0.10)/0.1195)=\Phi(0.956)\approx0.831\). Neither arm meets the 0.90 threshold.
Here the decision changes between the smaller and larger difference scales. The larger \(\tau_d\) permits more arm-to-arm separation and shrinks the observed difference less strongly toward zero; it also produces a larger posterior variance for \(d\). Both changes affect each arm’s marginal probability. It is therefore unsafe to explain the probability change by the posterior mean alone.
How to Conduct and Report the Analysis
A useful sensitivity analysis changes assumptions in a structured way, rather than changing many inputs at once and leaving the cause of a decision change unclear. A small grid can vary prior means, common-level scales, and difference scales. The ranges should be scientifically defensible and specified without selecting them to force a desired outcome.
State each arm effect, its units, the clinically meaningful threshold \(\Delta\), the posterior probability \(q_i\), and the qualifying threshold. Keep them fixed across prior scenarios.
Vary the prior mean and scale for the common level and the difference scale governing borrowing. Explain why each value is plausible before interpreting results.
For each scenario, calculate posterior means, variances, arm probabilities, and the action implied by the same rule. Check calculations near the decision boundary with adequate numerical precision.
Report the full scenario range or grid, distinguish changes in probability from changes in decision, and identify which prior dimension drives any instability.
Diagnostics, Alternatives, and Limitations
Check that the model used for the sensitivity calculations matches the data structure. In particular, the worked examples assume independent treatment-effect estimates with known variances. In a shared-control trial, treatment-effect estimates can be correlated because they use the same control observations. Their joint likelihood must retain that covariance; treating correlated estimates as independent can give incorrect posterior variances and probabilities. The earlier material on shared-control posterior covariance gives the relevant covariance structure. If the likelihood is not Normal or variances are uncertain, the conjugate formulas shown here may not apply, though the principle of recomputing the same decision under alternative priors still applies.
A practical diagnostic is to compare posterior means, posterior standard deviations, probabilities, and actions side by side. A probability that changes from 0.91 to 0.89 may be numerically modest but operationally important under a 0.90 rule. Conversely, larger probability changes far from the threshold may leave the action unchanged. Also check that the direction of shrinkage is sensible: when the prior for \(d\) is centered at zero, stronger borrowing should move the posterior difference toward zero, not away from it.
The main alternative is to report results under one chosen prior only. That is simpler, but conceals how dependent the decision is on the chosen assumptions. Another alternative is to use a robust mixture prior, which can average a skeptical component with a more permissive component; such a model is itself a prior choice and should also be examined for sensitivity. Neither approach replaces transparent scenario comparisons. Sensitivity analysis is not a frequentist error-rate correction, does not prove that a prior is valid, and cannot protect against every misspecified likelihood or implausible range.
Common Mistakes
- Changing the decision rule along with the prior. If the qualifying threshold or clinical benefit threshold changes across scenarios, the differences cannot be attributed to the prior alone. Hold the rule fixed when assessing prior sensitivity.
- Calling similar probabilities a robust decision without checking the cutoff. A small numerical change can switch the action when a probability is near the threshold. State the action for every scenario.
- Treating the borrowing scale as a vague “amount of shrinkage.” In this model, \(\tau_d\) is a prior standard deviation for the half-difference \(d\). Smaller values impose stronger similarity; they do not directly specify a fixed percentage of borrowing.
- Ignoring dependence induced by a shared control. Independent marginal calculations are not a substitute for a joint model when treatment-effect estimates are correlated. Use the correct sampling covariance.
- Selecting the prior that gives the desired result. A sensitivity grid assesses dependence on plausible assumptions; it is not a method for tuning a prior after seeing which arm qualifies.
Check Your Understanding
Answer each question using the model and decision rule in the worked examples. Brief solutions are included after the questions.
- What does decreasing \(\tau_d\) mean in this model, and which parameter does it shrink?
- In the baseline scenario, why is the posterior variance of each arm effect \(C+D\)?
- At \(m=0.10\) in the prior-mean example, what action follows from the 0.90 threshold?
- Why can a modest change in \(q_i\) be operationally important?
- What must change in the worked likelihood if the two treatment-effect estimates share a control group and are correlated?
Solutions. (1) A smaller \(\tau_d\) gives a more concentrated prior for \(d\) around zero, shrinking the arm difference more strongly. (2) Since \(\theta_1=c+d\) and \(\theta_2=c-d\), and the posterior \(c\) and \(d\) are independent, each arm variance is \(\operatorname{Var}(c)+\operatorname{Var}(d)=C+D\). (3) Neither arm qualifies: \(q_1\approx0.876\) and \(q_2\approx0.807\), both below 0.90. (4) A small change can move a probability across the fixed decision threshold and change selection. (5) The joint likelihood must include the sampling covariance between the estimates; otherwise the posterior uncertainty and decision probabilities can be wrong.