Why a Sensitivity Set Matters
A Bayesian submission presents conclusions conditional on a prior distribution, a data source, and a statistical model. A reviewer therefore needs more than a single posterior result: the evidence package should show how conclusions change under other scientifically defensible choices. A sensitivity analysis is not a search for a favorable specification. It is a planned comparison that identifies which assumptions materially influence the answer to the regulatory question.
The reference analysis is the prespecified primary analysis used for the principal conclusion. Sensitivity analyses alter selected influential assumptions while keeping other choices fixed where possible. This makes changes interpretable: if a posterior probability moves after only the prior changes, the difference can be attributed to the prior choice rather than to several simultaneous modifications. The assumptions varied should be plausible alternatives, not arbitrary extremes chosen to produce a desired outcome.
Start with the regulatory question and the quantity that answers it: for example, a response probability, a treatment effect in specified units, or a posterior probability that an effect exceeds a clinically meaningful value. State the decision-relevant interpretation before comparing analyses. The set should then target assumptions capable of changing that quantity or its interpretation. Typical dimensions include the prior distribution, the inclusion or weighting of external data, and the likelihood or variance structure.
Designing a Focused Set of Analyses
A useful set has a clear reference analysis, a rationale for each alternative, and a comparison of the resulting decision-relevant quantities. It is usually more informative to vary one important assumption at a time before adding a small number of joint alternatives. If every analysis changes the prior, data source, outcome definition, and model simultaneously, a changed conclusion cannot be attributed to any one choice.
Name the estimand and the posterior summary or decision-relevant probability used to answer the regulatory question.
Record its prior, included data, likelihood, and key assumptions so that it can be reproduced and compared.
Use clinical knowledge, data provenance, and model diagnostics to explain why each alternative is credible and what concern it addresses.
Report the same estimand and summary for each analysis, alongside changes in data or assumptions and any relevant diagnostics.
State which conclusions persist, which change, and how that affects the strength or scope of the evidence.
This structure supports transparent review without implying that every imaginable analysis must be performed. Prior-data conflict deserves particular attention: earlier material on prior-data conflict helps identify situations in which prior information and trial data may pull in different directions. The sensitivity set should then make the consequences of plausible prior choices visible.
Varying an Influential Prior
For a binary outcome with \(y\) responses among \(n\) participants, suppose the response probability is \(p\), and the likelihood is binomial: \(Y\mid p\sim\operatorname{Binomial}(n,p)\). With a beta prior \(p\sim\operatorname{Beta}(a,b)\), the posterior is \(\operatorname{Beta}(a+y,b+n-y)\). This conjugate result assumes independent Bernoulli outcomes with a common response probability and a prior specified independently of the observed trial outcomes.
Worked Example: Comparing Two Plausible Priors
Hypothetical trial data: 52 of 100 participants respond. Use a uniform reference prior, \(\operatorname{Beta}(1,1)\), and compare it with a prespecified skeptical prior, \(\operatorname{Beta}(20,30)\), centered at 0.40. Both analyses use the same binomial likelihood and trial data. For illustration, compare the posterior probability that \(p\) exceeds 0.45.
With the uniform prior, the posterior is \(\operatorname{Beta}(1+52,1+48)=\operatorname{Beta}(53,49)\). Its posterior mean is \(53/(53+49)=53/102\approx0.5196\). Its variance is \(53(49)/[102^2(103)]\approx0.002423\), giving a posterior standard deviation of about 0.04923. A normal approximation to this beta posterior gives \(z=(0.5196-0.45)/0.04923\approx1.414\), so \(P(p>0.45\mid\text{data})\approx0.921\).
With the skeptical prior, the posterior is \(\operatorname{Beta}(20+52,30+48)=\operatorname{Beta}(72,78)\). Its mean is \(72/150=0.4800\), and its variance is \(72(78)/[150^2(151)]\approx0.001653\), giving a standard deviation of about 0.04065. The corresponding normal approximation gives \(z=(0.4800-0.45)/0.04065\approx0.738\), or \(P(p>0.45\mid\text{data})\approx0.770\).
These tail probabilities are normal approximations; the posterior distributions themselves are exact beta distributions, and their tail probabilities can instead be calculated from the beta distribution function. The posterior mean exceeds 0.45 under both priors, but the probability of exceeding 0.45 changes substantially. A regulatory interpretation should report that prior sensitivity rather than treating the two summaries as interchangeable.
A prior alternative should have a scientific rationale: for example, it may represent less optimistic expert judgment or a justified difference in the amount of prior information. Report its parameters and their interpretation, not only a label such as “skeptical.” Do not choose a prior solely because it yields a desired conclusion. If a prior is informative, explain which conclusions depend on it.
Varying the Data Source
External data can affect a posterior even when the current trial data are unchanged. The analysis should show how conclusions compare when external observations are excluded, discounted, or incorporated under the justified primary approach. These comparisons are meaningful only if the external source is sufficiently relevant to the population, endpoint, and conduct of the current trial for the proposed use. Discounting reduces the external data’s contribution; it does not remove concerns about systematic differences.
Worked Example: Excluding, Discounting, or Fully Using External Data
Hypothetical data: In a current trial, 40 of 80 participants respond. A potentially relevant external cohort has 45 responses among 75 participants. Use a uniform \(\operatorname{Beta}(1,1)\) starting prior. For a transparent illustration, represent the external observations with a fixed power weight \(a\): each external likelihood contribution is raised to power \(a\). This assumes a binomial likelihood for both sources and treats the selected discount as fixed; it does not model uncertainty about how comparable the cohorts are.
With no external borrowing, the posterior is \(\operatorname{Beta}(1+40,1+40)=\operatorname{Beta}(41,41)\), with mean \(41/82=0.5000\). With discount \(a=0.5\), the posterior is \(\operatorname{Beta}(1+40+0.5(45),1+40+0.5(30))=\operatorname{Beta}(63.5,56)\). Its mean is \(63.5/119.5\approx0.5314\). With full borrowing, \(a=1\), the posterior is \(\operatorname{Beta}(1+40+45,1+40+30)=\operatorname{Beta}(86,71)\), with mean \(86/157\approx0.5478\).
The posterior mean increases as more weight is given to the external cohort, whose observed response proportion is \(45/75=0.6000\). This comparison reveals the direction and size of the borrowing effect. It does not establish that the external cohort is exchangeable with the trial, nor that a fixed discount is adequate. The submission should explain the source’s relevance and quality, justify the primary weighting approach, and state whether the main conclusion changes when the external data are excluded or downweighted.
When data-source sensitivity is important, present the amount of information contributed by each source and identify the assumptions needed to combine them. A dramatic change after including external data is a signal to examine comparability and borrowing assumptions carefully, not an automatic reason to choose whichever analysis appears more favorable.
Varying Modeling Assumptions
A model sensitivity analysis asks whether conclusions depend on how the sampling variation or outcome structure is represented. For a continuous treatment effect, a normal approximation may be reasonable when the estimator is sufficiently well behaved. But uncertainty can be underestimated if a model ignores clustering or other dependence. An alternative variance assumption can therefore be informative even when the estimated treatment effect is unchanged.
For an observed estimate \(Y\) with \(Y\mid\theta\sim N(\theta,s^2)\) and prior \(\theta\sim N(m_0,v_0)\), where \(s^2\) is treated as known, the posterior is normal with variance \(v_1=(1/v_0+1/s^2)^{-1}\) and mean \(m_1=v_1(m_0/v_0+Y/s^2)\). These expressions follow by multiplying the likelihood and prior densities and collecting the terms involving \(\theta\):
Here \(C\) collects terms that do not depend on \(\theta\). The coefficient of the squared term determines the posterior variance, and its center determines the posterior mean. This derivation depends on the stated normal likelihood and prior; changing the assumed sampling variance changes the posterior.
Worked Example: Sensitivity to the Assumed Standard Error
Hypothetical result: A treatment effect estimate is 2.4 points, with units of points on the clinical scale. Use the prior \(\theta\sim N(0,4)\), so \(m_0=0\) and \(v_0=4\). The primary model gives standard error \(s=1.0\). Compare it with an alternative that accounts for within-site dependence and gives \(s=1.5\). Assume a normal likelihood with each stated standard error treated as known.
For \(s=1.0\), \(v_1=(1/4+1/1^2)^{-1}=0.8000\), and \(m_1=0.8000(0/4+2.4/1^2)=1.920\) points. The posterior standard deviation is \(\sqrt{0.8000}\approx0.8944\) points. Thus \(P(\theta>1\mid\text{data})\) is approximately \(\Phi[(1.920-1)/0.8944]=\Phi(1.029)\approx0.848\).
For \(s=1.5\), \(v_1=(1/4+1/2.25)^{-1}=1.440\), and \(m_1=1.440(2.4/2.25)=1.536\) points. The posterior standard deviation is \(\sqrt{1.440}=1.200\) points. The corresponding probability is \(\Phi[(1.536-1)/1.200]=\Phi(0.447)\approx0.673\). The estimate remains 2.4 points, but the increased uncertainty materially lowers the posterior probability that the effect exceeds 1 point. The sensitivity should be described in terms of this clinically interpretable quantity and the model assumption behind the larger standard error.
Reading the Results and Recognizing Failure Modes
A concise comparison table helps distinguish stable findings from assumption-dependent ones. For every row, report the assumption changed, the resulting estimand or posterior summary, and whether the interpretation changes. Include units for effects and enough detail about priors, data sources, or model specifications that the analysis can be reproduced. If the primary conclusion is based on a decision criterion, report the relevant posterior quantity in each analysis; do not substitute a different summary in an alternative analysis without explaining why.
| Sensitivity dimension | Useful comparison | Question for interpretation |
|---|---|---|
| Prior | Reference prior versus scientifically justified alternatives | Does the prior materially change the posterior summary or decision-relevant probability? |
| Data source | External data excluded, discounted, or incorporated as specified | How much does the conclusion depend on the external evidence and its assumed relevance? |
| Model | Primary likelihood or variance structure versus plausible alternatives | Does a defensible representation of sampling variability change the conclusion? |
| Joint alternatives | A limited set combining important plausible changes | Do the main assumptions together produce a conclusion not seen in one-at-a-time analyses? |
Several warning signs merit explanation. A sensitivity set is weak if every alternative is less plausible than the reference analysis, if the alternatives are selected after seeing which produces a favorable result, or if important choices change together so their effects cannot be separated. It is also uninformative to report only that results were “similar” without giving the values being compared. Conversely, a changed result is not automatically evidence that the primary model is wrong: it identifies dependence on an assumption and prompts scientific interpretation.
One common mistake is to call a conclusion robust because the posterior mean has the same direction in every analysis. That can conceal a substantial change in uncertainty or in the probability relevant to the regulatory question. The correction is to compare the same decision-relevant summary across analyses and state its numerical change. Another mistake is to treat a wide, arbitrary stress test as a menu from which to select the preferred result. Alternatives should instead be justified before results are interpreted, and unfavorable as well as favorable findings should be reported.
Sensitivity analysis has limits. Stability across selected alternatives cannot rule out bias shared by every model, an unrecognized data-quality problem, or an implausible scientific assumption common to the set. It also does not replace a clear primary analysis or the justification for using external information. Report the scope of the set and any important assumptions it does not address. When findings are assumption-dependent, describe that dependence directly and consider whether the conclusion should be narrower or more qualified.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- A trial’s reference analysis uses a prior informed by expert opinion. What should a prior sensitivity analysis vary, and what should remain comparable across analyses?
- In a hypothetical binomial trial, 18 of 30 participants respond. With prior \(\operatorname{Beta}(2,2)\), find the posterior distribution and mean.
- A hypothetical trial has 8 responses among 20 participants, and an external cohort has 18 among 30. Under a \(\operatorname{Beta}(1,1)\) prior and fixed external-data weight \(a=0.25\), find the posterior distribution and mean using a power-weighted binomial likelihood.
- Suppose all analyses except one plausible alternative model support the same regulatory conclusion. What should the report say, and what should it avoid claiming?
Solutions. (1) Compare the reference prior with scientifically justified alternatives, such as a less informative or differently centered prior, while keeping the data and likelihood fixed where possible. Report the same estimand and decision-relevant posterior summary for each. (2) The posterior is \(\operatorname{Beta}(2+18,2+12)=\operatorname{Beta}(20,14)\), with mean \(20/34\approx0.5882\). (3) The external contributions are \(0.25(18)=4.5\) responses and \(0.25(12)=3\) nonresponses. The posterior is \(\operatorname{Beta}(1+8+4.5,1+12+3)=\operatorname{Beta}(13.5,16)\), with mean \(13.5/29.5\approx0.4576\). (4) Report which plausible model changes the result, give the comparable numerical summaries, and explain the model difference and its scientific relevance. Do not claim robustness across all plausible analyses or hide the result by reporting only the analyses that agree.