Why Stress-Test Prior-Data Conflict?
A Bayesian analysis combines current trial data with information represented by a prior distribution. That combination is useful when the prior is relevant, but it also creates a practical question: what if the current results disagree with the prior information? The posterior can still be calculated, but the disagreement may affect its location, uncertainty, or the decision made from it. A stress test makes those consequences visible rather than assuming that the prior and trial data align.
For regulatory interpretation, the aim is not to declare a prior “right” or “wrong” from one observed trial. It is to assess what the prespecified analysis says under plausible conflict scenarios, how surprising the data were under the prior, and whether the decision changes. The prior scenarios and decision criteria should be set out clearly; selecting whichever prior produces the preferred conclusion after seeing the results does not constitute a credible stress test.
Model, Assumptions, and a Conflict Diagnostic
For a binary endpoint, let \(Y\) be the number of responses among \(n\) participants, and let \(p\) be the response probability in the population described by the trial. A basic model is \(Y\mid p\sim\operatorname{Binomial}(n,p)\). It assumes that outcomes are binary, participants’ outcomes are conditionally independent given \(p\), and the participants share the same response probability. If trial features or participant differences make these assumptions unreasonable, a simple binomial conflict calculation may not describe the actual data-generating process.
With a \(\operatorname{Beta}(a,b)\) prior, where \(a>0\) and \(b>0\), the posterior after observing \(y\) responses is \(\operatorname{Beta}(a+y,b+n-y)\). This conjugate update is useful here because it shows transparently how the prior and current data contribute. Its assumptions are the beta-binomial model just stated; it does not establish that the prior is clinically relevant or that the model fits the trial.
A prior-predictive distribution describes what trial results the prior and sampling model predict before the current outcomes are observed. For the beta-binomial model, its mean and variance are
An observed count far from the prior-predictive mean can flag a potential conflict. The distance in standard deviations is a descriptive diagnostic, not a calibrated conflict-test p-value. A more direct check is the prior-predictive probability of an event at least as surprising as the observed result, with “at least as surprising” specified in advance. For a two-sided discrepancy, the rule for defining extremeness must also be stated; there is no universal two-sided tail definition for every discrete predictive distribution.
Worked Example: Trial Results Against Low, Central, and High Priors
Worked Example: Trial Results Against Low, Central, and High Priors
Suppose a hypothetical trial observes 10 responses among 20 participants. The endpoint is response by a prespecified assessment time. Assume the binomial model is appropriate, with independent outcomes and a common response probability \(p\). Before seeing these data, the analysis plan identifies three prior scenarios with the same total concentration but different centers: \(\operatorname{Beta}(2,8)\), \(\operatorname{Beta}(5,5)\), and \(\operatorname{Beta}(8,2)\). Consider the illustrative regulatory decision rule \(P(p>0.50\mid Y)>0.85\).
The prior means are \(2/10=0.20\), \(5/10=0.50\), and \(8/10=0.80\). The prior-predictive mean counts among 20 participants are therefore \(20(0.20)=4\), \(20(0.50)=10\), and \(20(0.80)=16\). The observed count, 10, is at the central prior’s predictive mean, but differs from the low- and high-prior predictive means by 6 responses in opposite directions.
For the low prior, the prior-predictive variance is \(\{20(2)(8)(10+20)\}/\{10^2(11)\}=8.727\), so its standard deviation is \(\sqrt{8.727}=2.954\). The observation is \((10-4)/2.954=2.031\) standard deviations above that predictive mean. For the high prior, the variance is the same and the observation is \((10-16)/2.954=-2.031\) standard deviations below its predictive mean. For the central prior, the variance is \(\{20(5)(5)(10+20)\}/\{10^2(11)\}=13.64\), with standard deviation \(3.693\); the observed count equals its predictive mean. These standardized distances flag the contrast, but do not replace a prespecified predictive-tail calculation.
Updating each prior gives the following posteriors:
For integer \(a,b\), the beta-tail identity gives \(P(p>c)=P\{Z\leq a-1\}\), where \(Z\sim\operatorname{Binomial}(a+b-1,c)\). At \(c=0.50\), the posterior probabilities are approximately 0.1324 for \(\operatorname{Beta}(12,18)\), 0.5000 for \(\operatorname{Beta}(15,15)\), and 0.8676 for \(\operatorname{Beta}(18,12)\). Thus only the high-center scenario exceeds the stipulated 0.85 threshold. Although the observed response rate is exactly \(10/20=0.50\), the high prior moves the posterior enough to change the decision. This is an interpretable consequence of the prior-data conflict, not evidence that the high-prior conclusion is established by the current data alone.
A Short Justification for the Beta-Tail Calculation
The beta-tail identity makes the posterior probabilities in the example reproducible using a binomial cumulative probability. For integer \(a,b>0\), set \(m=a+b-1\) and define \(G(c)=\sum_{j=0}^{a-1}{m\choose j}c^j(1-c)^{m-j}\). Differentiating the finite sum and cancelling adjacent terms gives
The beta survival probability \(S(c)=\int_c^1 t^{a-1}(1-t)^{b-1}/B(a,b)\,dt\) has the same derivative, \(S'(c)=-c^{a-1}(1-c)^{b-1}/B(a,b)\). Also, \(G(1)=S(1)=0\). Since the two functions have equal derivatives and agree at 1, \(G(c)=S(c)\). This proves \(P(p>c)=P\{Z\leq a-1\}\) for \(Z\sim\operatorname{Binomial}(a+b-1,c)\), including the calculations used above.
Worked Example: Interpreting a Posterior Tail Near a Decision Threshold
Worked Example: Interpreting a Posterior Tail Near a Decision Threshold
In a second hypothetical analysis, the prior for a response probability is \(\operatorname{Beta}(6,12)\), and 2 responses are observed among 2 participants. Assume the same independent, common-probability binomial model. The posterior is \(\operatorname{Beta}(6+2,12+0)=\operatorname{Beta}(8,12)\). Suppose the decision criterion is \(P(p>0.30\mid Y)\geq0.80\).
Applying the identity with \(a=8\), \(b=12\), and \(c=0.30\) gives \(P(p>0.30\mid Y)=P\{Z\leq7\}\), where \(Z\sim\operatorname{Binomial}(19,0.30)\). Substitution into the binomial cumulative probability gives \(\sum_{j=0}^{7}{19\choose j}(0.30)^j(0.70)^{19-j}\approx0.8177\). Because \(0.8177\geq0.80\), the illustrative rule is met. This is a posterior probability conditional on the stated model, prior, and observed data; it is not the probability that the decision rule itself is correct. The value is only modestly above the threshold, so a report should show the numerical margin and the prior assumptions, rather than reducing the result to “pass.”
Worked Example: Same Prior Center, Different Prior Strength
Worked Example: Same Prior Center, Different Prior Strength
Return to the hypothetical result of 10 responses among 20 participants. Compare a lower-concentration prior \(\operatorname{Beta}(2,8)\) with a higher-concentration prior \(\operatorname{Beta}(20,80)\). Both have mean \(0.20\), but their total concentrations are 10 and 100. These are prespecified scenarios for illustrating the consequence of prior strength; their parameters should not automatically be interpreted as literal counts of historical patients.
Under the binomial model, the posteriors are \(\operatorname{Beta}(12,18)\) and \(\operatorname{Beta}(30,90)\). The beta mean is \(a/(a+b)\), so the posterior means are \(12/30=0.4000\) and \(30/120=0.2500\), respectively. If, solely for illustration, a decision rule were to proceed when the posterior mean response probability exceeds 0.35, the lower-concentration scenario meets it and the higher-concentration scenario does not. This is a consequence for that particular mean-based rule, not a claim that the rule is appropriate for a particular trial. If the protocol instead specifies a posterior probability threshold, that exact probability must be calculated under each scenario; posterior means cannot be substituted for it.
How to Conduct and Report the Stress Test
A useful stress test follows a transparent sequence. The scenarios should reflect scientifically plausible ways in which current evidence might disagree with the prior, rather than arbitrary alternatives chosen to change the result.
Name the parameter, endpoint, analysis population, likelihood, and assumptions needed for the calculation.
Describe which prior features vary, why the scenarios are plausible, and how they were selected before the current outcomes are known.
Report a prior-predictive diagnostic with its definition. Distinguish a descriptive distance from an explicitly calculated predictive tail probability.
For each scenario, report the relevant posterior summary and whether the prespecified decision rule changes. Include the size and direction of any change.
State what the scenarios do not examine, and explain how model assumptions or data-quality concerns could affect the interpretation.
A comparison table can keep the decision consequences clear. It should name the prior scenario, the posterior quantity relevant to the rule, the rule’s result, and any diagnostic used to characterize conflict. Avoid a single label such as “robust” unless the scope of scenarios and the meaning of robustness are made explicit.
Comparisons, Diagnostics, and Common Pitfalls
Prior-predictive checking and posterior decision comparison answer related but distinct questions. A prior-predictive check asks whether the observed data were unusual under the prior and likelihood. A posterior comparison asks how the specified priors and data together affect the quantity used for inference or action. A prior can assign relatively little predictive weight to the observed result while the posterior still supports the same decision; conversely, a modest posterior shift can move a result across a decision threshold. Both perspectives are useful, and neither should be presented as a substitute for the other.
A likelihood-only analysis can show what the current data contribute without the prior, but it is not the same as a Bayesian decision analysis and does not by itself establish which prior is appropriate. Likewise, changing prior location or concentration is only a targeted conflict stress test; it is not a complete account of every modeling, data, or decision uncertainty. The earlier material on borrowing and discounting choices addresses how borrowing strategies are justified; a conflict stress test examines what happens when the resulting prior information and current results disagree.
- Do not call a standardized predictive discrepancy a p-value. A distance in predictive standard deviations is descriptive unless a calibrated reference distribution and tail rule are specified.
- Do not treat a prior-predictive tail probability as the probability that the prior is false. It measures how much probability the specified model assigns to a defined set of data outcomes.
- Do not choose scenarios after seeing which one supports the desired decision. Post hoc selection can conceal rather than reveal prior influence.
- Do not infer that agreement proves validity. A prior and trial can agree even if the sampling model, population relevance, or data are problematic.
- Do not omit the decision rule. Posterior summaries become decision-relevant only in relation to the criterion actually specified for the trial.
Stress testing is limited by the scenarios examined and the adequacy of the likelihood. If the trial population differs from the population represented by external information, or if response probabilities vary across participants, simple beta-binomial calculations may not capture the important source of disagreement. A favorable result across a small set of priors is not proof of general robustness. The analysis should therefore report the scope of the scenarios, the quantities compared, and the assumptions under which conclusions follow.
Check Your Understanding
Answer each question, then compare with the solutions below.
- Under a beta-binomial model, what is the posterior after a \(\operatorname{Beta}(3,7)\) prior and 4 responses among 10 participants?
- For that posterior, calculate the posterior mean response probability.
- In the 10-of-20 example, why is the standardized discrepancy under the low prior not a p-value?
- What does a prior-predictive tail probability describe, and what does it not describe?
- Why should conflict scenarios be specified before the trial outcomes are reviewed?
Solutions. (1) The posterior is \(\operatorname{Beta}(3+4,7+10-4)=\operatorname{Beta}(7,13)\). (2) Its mean is \(7/(7+13)=0.3500\). (3) The standardized discrepancy is an observed distance from the predictive mean in predictive standard deviations; without a defined calibrated tail calculation, it is not a p-value. (4) The tail probability is the probability, under the specified prior and sampling model, of a prespecified set of outcomes at least as extreme as the observed data. It is not the probability that the prior is false. (5) Prespecification helps ensure the scenarios reflect plausible scientific concerns rather than being selected after seeing which prior produces a preferred decision.