Why Prior Sensitivity Matters Near a Decision Boundary
Predictive success asks how likely the trial is to meet its specified final success rule, given the evidence observed so far. That probability depends on the model for future data and on the prior distribution used to update the treatment-effect parameter. When interim evidence is far from the decision boundary, plausible priors may lead to similar decisions. Near the boundary, even modest prior-driven changes in the predictive distribution can change whether a future outcome is expected to cross the rule.
A useful sensitivity comparison changes one ingredient at a time. To isolate prior effects, hold the interim data, remaining sample size, outcome model, and final decision rule fixed. Then compare priors that are plausible for the clinical context. If the interim data change as well, the resulting difference in predictive success cannot be attributed to the prior alone.
This is not the probability that the treatment is effective, nor the probability that the null hypothesis is true. It is a probability about a future decision under a fully specified analysis. The prior matters because it affects both the posterior distribution given the interim data and the predictive distribution for future observations.
A Beta-Binomial Setup Near the Boundary
Consider a hypothetical binary-outcome trial. Let \(p\) be the treatment-arm response probability, and suppose responses are conditionally independent Bernoulli observations with common probability \(p\). A prior \(p\sim\mathrm{Beta}(a,b)\) gives, after \(x\) responses among \(n\) observed participants, the posterior distribution \(p\mid y\sim\mathrm{Beta}(a+x,b+n-x)\). The beta-binomial predictive distribution follows by averaging the binomial likelihood for future responses over this posterior.
For a transparent illustration, suppose the final rule declares success when the final posterior probability that \(p\) exceeds 0.50 is greater than 0.60. The threshold 0.50 and probability cutoff 0.60 are hypothetical choices for this example, not universal clinical standards. Assume five responses have been observed among ten participants, and exactly one participant remains. The rule is deliberately close enough to the boundary that the last outcome may change the final decision.
Worked Example: A Uniform Prior and One Remaining Participant
Use the hypothetical data above, \(x=5\) responses among \(n=10\), and a uniform prior \(p\sim\mathrm{Beta}(1,1)\). Calculate predictive success under the stated final rule.
The interim posterior is \(\mathrm{Beta}(1+5,1+10-5)=\mathrm{Beta}(6,6)\). Its posterior probability that \(p>0.50\) is 0.50 by symmetry, so the success rule is not currently met. The predictive probability of a response in the remaining participant is \[ P(Y_{\mathrm{rem}}=1\mid y)=\frac{6}{6+6}=0.5000. \]
If the remaining participant responds, the final posterior is \(\mathrm{Beta}(7,6)\). For integer beta parameters, the identity \(P\{p>0.5\}\), for \(p\sim\mathrm{Beta}(A,B)\), equals \(P\{X\leq A-1\}\) for \(X\sim\mathrm{Binomial}(A+B-1,0.5)\). Consequently, \[ P(p>0.50\mid y,Y_{\mathrm{rem}}=1) =\frac{1+12+66+220+495+792+924}{4096} =\frac{2510}{4096} =0.6128. \] This exceeds 0.60, so a response leads to final success. If there is no response, the final posterior is \(\mathrm{Beta}(6,7)\). By symmetry its tail probability is \(1-0.6128=0.3872\), so that outcome does not lead to success. Therefore, \[ \mathrm{PPOS}_{\mathrm{Beta}(1,1)}(y) =0.5000(1)+0.5000(0)=0.5000. \]
The interpretation is that, under this prior, model, interim data, remaining sample size, and final rule, the probability of meeting the final rule is 0.5000. It is not a 50% posterior probability that \(p>0.50\); that current posterior probability is exactly 0.50.
Compare Priors While Holding Everything Else Fixed
A useful comparison can vary prior concentration while keeping prior means equal. For beta priors, the mean is \(a/(a+b)\); increasing both parameters around the same mean concentrates more prior probability near that mean. The choices below are hypothetical sensitivity scenarios. Whether they are plausible for a particular trial requires subject-matter justification; mathematical convenience alone does not make a prior clinically plausible.
Worked Example: Equal Prior Means, Different Prior Strengths
Keep the same five responses among ten participants, one remaining participant, and final rule. Compare \(\mathrm{Beta}(1,1)\), \(\mathrm{Beta}(2,2)\), and \(\mathrm{Beta}(4,4)\). All have prior mean 0.50, but their concentrations differ.
Under \(\mathrm{Beta}(2,2)\), the interim posterior is \(\mathrm{Beta}(7,7)\), and the predictive probability of a response is \(7/(7+7)=0.5000\). A response gives \(\mathrm{Beta}(8,7)\) at the end. Its probability above 0.50 is \[ \frac{1+14+91+364+1001+2002+3003+3432}{16384} =\frac{9908}{16384} =0.6047. \] It therefore meets the rule, though only narrowly. A nonresponse gives the symmetric posterior \(\mathrm{Beta}(7,8)\), whose probability above 0.50 is \(1-0.6047=0.3953\), so it does not meet the rule. The predictive success probability is \(0.5000(1)+0.5000(0)=0.5000\).
Under \(\mathrm{Beta}(4,4)\), the interim posterior is \(\mathrm{Beta}(9,9)\), again with predictive response probability \(9/(9+9)=0.5000\). A response leads to \(\mathrm{Beta}(10,9)\), for which \[ P(p>0.50\mid y,Y_{\mathrm{rem}}=1) =\frac{155382}{262144} =0.5927. \] A nonresponse leads to the symmetric \(\mathrm{Beta}(9,10)\), with tail probability \(0.4073\). Neither possible final outcome meets the strict \(0.60\) rule. Thus predictive success is \(0.0000\) under this prior.
The three predictive success values are \(0.5000\), \(0.5000\), and \(0.0000\), respectively. The equality for the first two is specific to this small example: the final response crosses the rule under both, and the predictive response probability is one-half. The more concentrated prior pulls the final posterior probability closer to 0.50, so even the response does not cross the cutoff. The change from 0.5000 to 0.0000 is prior sensitivity, not a change in the observed data or the rule.
This example also illustrates a boundary effect. Predictive success is based on an indicator: a final result either meets the rule or does not. A small change in a posterior tail probability from 0.6047 to 0.5927 crosses the fixed cutoff and changes the predicted success event for that future outcome. The underlying posterior distributions differ continuously; the reported predictive success can change abruptly because the decision rule is discontinuous.
Separate Prior Effects From Data Effects
A prior comparison is interpretable only when the observed data and the rest of the analysis are held fixed. Conversely, to study how interim data affect predictive success, keep the prior and rule fixed and vary the data. The following comparison changes the interim response count while retaining the \(\mathrm{Beta}(4,4)\) prior and the same one-participant remainder.
Worked Example: More Favorable Interim Data Under the Same Prior
Suppose the hypothetical trial instead has six responses among ten participants. Under \(\mathrm{Beta}(4,4)\), the interim posterior is \(\mathrm{Beta}(10,8)\), so the predictive probability of a response is \(10/18=0.5556\). A response yields \(\mathrm{Beta}(11,8)\), with posterior probability above 0.50 equal to \[ P\{X\leq 10\},\quad X\sim\mathrm{Binomial}(18,0.5) =\frac{155382+43758}{262144} =0.7597. \] This meets the 0.60 rule. A nonresponse yields \(\mathrm{Beta}(10,9)\), whose corresponding probability is \(0.5927\), below the rule. Therefore, \[ \mathrm{PPOS}_{\mathrm{Beta}(4,4)}(y) =\frac{10}{18}(1)+\frac{8}{18}(0) =0.5556. \]
With five responses and the same prior, predictive success was 0.0000; with six responses it is 0.5556. This is a data-driven change, because the prior, final rule, and remaining sample size did not change. A comparison that changed both the prior and the response count would mix this data effect with prior sensitivity and would not identify either contribution by itself.
How to Conduct and Report a Sensitivity Analysis
A practical comparison starts by specifying the analysis that defines success: the outcome model, the prior, the final posterior rule, and how much data remain. Then choose a small set of clinically defensible priors. Explain why each is plausible—for example, by its prior mean and concentration, or by historical information—and calculate predictive success for each using identical interim data and the identical decision rule.
For each prior, inspect not only the final predictive success probability but also the posterior predictive outcomes that contribute to it. In the one-participant examples, this means showing which final outcome crosses the rule and the probability of that outcome. With more participants remaining, the calculation sums over all possible future response counts, weighting each by its posterior predictive probability. Simulation can approximate this sum when direct enumeration is inconvenient, but Monte Carlo error should be distinguished from actual prior sensitivity. The earlier material on Monte Carlo error in predictive success estimates addresses that simulation uncertainty.
A clear report can give a table or range across the chosen priors, while stating that the range is conditional on those particular prior scenarios. It is not a universal uncertainty interval and does not assign probabilities to the priors. If there is a principled probability distribution over prior models, model averaging is a distinct analysis; merely listing several priors does not create such weights.
| What changes? | What is held fixed? | What the comparison reveals |
|---|---|---|
| Prior distribution | Interim data, model, remaining sample, and decision rule | Prior sensitivity of predictive success |
| Interim data | Prior, model, remaining sample, and decision rule | Dependence on the evidence observed so far |
| Decision rule | Prior and interim data | Rule sensitivity, a different question from prior sensitivity |
| Monte Carlo random seed or replicate count | Statistical model and inputs | Numerical simulation variation, not a change in the underlying predictive probability |
Diagnostics, Limitations, and Common Mistakes
Check that each prior is stated on the parameter scale used by the model and that its implications are reasonable for the clinical setting. For a beta prior, inspect its mean and concentration; for other models, examine prior predictive distributions for observable outcomes. A prior that produces implausible response rates or trial outcomes is not made defensible merely by including it in a sensitivity table.
Near a decision boundary, report the final posterior probabilities for the future outcomes that matter, not only the rounded predictive success estimate. This reveals whether a result is robustly above or below the cutoff or depends on a very small crossing. Check that the final rule has been applied consistently, especially whether it uses \(>\) or \(\geq\). If success depends on a posterior probability of 0.6000, rounding before applying the rule can change the classification.
- Mistake: changing the prior and the interim data together, then calling the difference prior sensitivity. Correction: hold the data fixed across priors. Change the data separately if the goal is to study data sensitivity.
- Mistake: interpreting predictive success as current evidence of efficacy. Correction: predictive success is a probability over future data and a specified final decision rule. Report the current posterior probability separately if it is relevant.
- Mistake: treating a small difference in predictive success as proof that two priors are interchangeable. Correction: consider whether the difference matters for the planned action, and whether the chosen priors span plausible prior information.
- Mistake: treating a sensitivity range as a probability distribution over priors. Correction: a range summarizes the selected scenarios only; it has no probability interpretation unless a model-averaging scheme explicitly supplies prior-model weights.
Prior sensitivity is most informative when it exposes an important dependence of the planned decision on assumptions that are uncertain but scientifically plausible. It does not identify which prior is correct. The appropriate response to sensitivity may be to justify the primary prior more carefully, collect more information, or acknowledge that the decision is not robust across the plausible scenarios examined.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- For \(p\sim\mathrm{Beta}(2,2)\) and five responses among ten participants, what is the posterior distribution? What is the predictive probability of a response if one participant remains?
- In the equal-mean prior comparison, why does \(\mathrm{Beta}(4,4)\) produce predictive success 0.0000 with five responses among ten, even though a response is possible?
- What must remain fixed to attribute a difference in predictive success to the prior?
- Does a reported predictive success of 0.5000 mean the current posterior probability that \(p>0.50\) is 0.5000? Explain.
Solutions. (1) The posterior is \(\mathrm{Beta}(2+5,2+10-5)=\mathrm{Beta}(7,7)\). The predictive response probability is \(7/(7+7)=0.5000\). (2) Under \(\mathrm{Beta}(4,4)\), a response gives \(\mathrm{Beta}(10,9)\), whose probability above 0.50 is 0.5927, below the strict 0.60 cutoff. A nonresponse gives 0.4073, also below the cutoff, so neither possible outcome is a success. (3) Hold the interim data, likelihood and outcome model, remaining sample size, and final decision rule fixed; vary only the prior. (4) No. Predictive success is the probability that future data lead to meeting the final rule. In the uniform-prior example, the current posterior probability above 0.50 is 0.5000 as well, but that equality is coincidental; the two quantities describe different events.