What Makes Bayesian Evidence Credible?
A Bayesian analysis can produce a precise posterior probability and still provide weak support for a clinical decision. The model might not represent the trial, the effect might be too small to matter to patients, or important assumptions might be difficult to evaluate because they were not clearly reported. Regulatory assessment therefore asks more than whether a calculation is correct: it asks whether the analysis is valid for the data, whether the result matters clinically, and whether the evidence and assumptions can be understood and scrutinized.
These are related but distinct questions. Statistical validity concerns whether the design, data, model, and computation support the stated inference. Clinical relevance concerns whether the estimated treatment effect is large enough, for long enough, and on an outcome that matters to patients or clinical practice. Transparency concerns whether reviewers can identify what was planned, what was done, and how reasonable changes to assumptions affect the conclusion. A strong result in one dimension cannot automatically repair a serious weakness in another.
Start With the Question and the Analysis
Before interpreting a posterior probability, reviewers need to understand the question it answers. A statement such as “the posterior probability of benefit is 97%” is incomplete unless “benefit” is defined. It might mean any positive effect, an effect exceeding a clinically meaningful threshold, or success on a specified endpoint at a specified follow-up. These are different claims. The decision rule, target population, endpoint, direction of benefit, and analysis time point should be clear enough that the result can be connected to the proposed clinical decision.
Statistical validity also depends on how the trial was conducted and how its data are represented. Review includes whether the design supports the intended comparison, whether the endpoint is measured consistently, and whether exclusions, missing observations, protocol deviations, or changes in treatment are handled in a defensible way. For a model, reviewers need to understand its likelihood, prior, and assumptions—for example, whether outcomes are treated as independent and whether participants are assumed to share a common event probability. Computation should be reproducible, and numerical methods should be checked when the result depends on simulation or approximation.
A prespecified analysis is generally easier to assess than one selected after seeing the results, because the latter can make it difficult to separate planned evidence from analytical flexibility. Prespecification does not make an analysis correct by itself; a prespecified model can still be unsuitable. Conversely, an additional analysis can be informative if its purpose, assumptions, and relationship to the primary analysis are reported clearly.
Worked Example: Checking the Statistical Meaning of a Posterior Probability
Suppose a hypothetical single-arm trial observes 15 responses among 20 patients. For illustration, assume each response is an independent Bernoulli outcome with the same response probability \(p\), and use a uniform prior \(p\sim\operatorname{Beta}(1,1)\). The question is the posterior probability that \(p\) exceeds 0.50.
For \(x\) responses in \(n\) patients, the likelihood is proportional to \(p^x(1-p)^{n-x}\). Multiplying by a \(\operatorname{Beta}(a,b)\) prior gives the posterior distribution:
Here \(a=1\), \(b=1\), \(x=15\), and \(n=20\), so \(p\mid x,n\sim\operatorname{Beta}(16,6)\). For integer beta parameters, the beta CDF at \(0.5\) can be evaluated using a binomial tail:
Under the stated model and prior, the posterior probability that the response probability exceeds 0.50 is 98.67%. This is not, by itself, evidence that the treatment is clinically worthwhile: the 0.50 benchmark must be relevant to the clinical question, and a single-arm response rate does not establish a comparative treatment effect. The calculation is also conditional on the common-probability and independence assumptions. If outcomes vary systematically across sites or patient groups, or the endpoint is not measured consistently, the simple model may not adequately represent the trial.
Clinical Relevance Is Not the Same as Statistical Evidence
A posterior probability can be statistically well calculated yet answer a clinically unhelpful question. For example, a high probability that a treatment effect is greater than zero does not necessarily show that the effect is large enough to justify treatment burden, risks, cost, or changes in care. The clinical threshold should be interpreted in the scale of the endpoint and the decision context. For an absolute risk reduction, that scale may be percentage points over a stated follow-up period; for a continuous outcome, it may be units on a validated measure.
The posterior distribution also conveys uncertainty about the size of the effect. Reporting only a probability of crossing a threshold can conceal whether plausible effects cluster just above that threshold or extend across a wide range. Useful interpretation therefore considers both the probability relevant to the decision and the range of effect sizes supported by the posterior.
Worked Example: Separating Probability of Benefit From Clinical Importance
Consider a hypothetical randomized trial in which the treatment’s absolute reduction in event risk, \(\delta\), has posterior distribution \(N(0.10,0.025^2)\). Here \(\delta\) is measured as a proportion, and positive values favor treatment. Suppose the decision-relevant threshold is a reduction of at least 0.05, or 5 percentage points, over the stated follow-up period.
The standardized threshold is \((0.05-0.10)/0.025=-2\). Therefore:
The posterior mean reduction is 0.10, equivalent to 10 fewer events per 100 treated patients at that follow-up, if the estimated effect applies to the trial population. A central 95% posterior interval is approximately \(0.10\pm1.96(0.025)\), or 0.051 to 0.149. Under the assumed posterior model, there is a 97.72% probability that the reduction exceeds 5 percentage points. Whether this is sufficient evidence for a decision depends on whether the threshold and outcome are clinically justified and on the treatment’s other benefits and harms. The probability alone does not settle those questions.
Transparency and Robustness
A reviewer should be able to follow the path from trial question to reported conclusion. That means clearly identifying the data included in the analysis, the model and prior, the decision threshold, and the computational approach. It also means explaining how missing data and other departures from the intended design are handled. If a result depends strongly on an assumption, the report should make that dependence visible rather than presenting one analysis as if it were assumption-free.
Sensitivity analyses examine whether the main conclusion changes under reasonable alternative assumptions. Their purpose is not to search for a version that gives the preferred answer. They help show which features of the evidence drive the result. A robust conclusion need not have exactly the same numerical posterior probability under every plausible analysis, but decision-relevant changes should be understood and reported.
Worked Example: Prior Sensitivity Changes the Interpretation
Suppose a hypothetical single-arm trial observes 8 responses among 10 patients. Compare two analyses under the same independent, common-probability binomial likelihood. With a uniform prior, \(p\sim\operatorname{Beta}(1,1)\), the posterior is \(\operatorname{Beta}(9,3)\). With a more concentrated prior \(p\sim\operatorname{Beta}(10,10)\), the posterior is \(\operatorname{Beta}(18,12)\). The second prior places more weight near 0.50 before the trial data are observed.
For the first analysis, the binomial-tail identity gives:
For the second analysis:
The same 8 responses produce a notably different posterior probability under these two priors. This comparison does not establish that either prior is appropriate; that requires a defensible rationale. It does show why a decision near a probability threshold should not be reported without making the prior choice and its influence assessable. In a real review, the relevant alternatives should be clinically and scientifically plausible, not chosen merely to create a favorable comparison.
Diagnostics, Alternatives, and Limits
Warning signs include a conclusion that changes under modest, defensible analysis choices; unexplained discrepancies between the planned and reported analysis; poor agreement between the model and observed data; and claims that rely on an endpoint or threshold whose clinical meaning is unclear. For complex models, computation should be checked for convergence and adequate precision. For simpler models, reviewers should still check that the likelihood, prior update, and reported probabilities match the data. In every case, an apparent computational success cannot compensate for poor data quality or an inappropriate model.
Bayesian and frequentist analyses can provide different summaries of the same trial. A Bayesian analysis can directly report a posterior probability about a parameter, conditional on the model and prior. Frequentist methods commonly report quantities such as confidence intervals, p-values, or error rates under repeated sampling. These summaries answer different inferential questions; a confidence interval is not a posterior probability statement, and a posterior probability is not automatically a measure of long-run error control. Reporting both approaches may help characterize evidence when each has a clear purpose, but agreement between them does not guarantee that the trial is clinically persuasive.
No statistical framework can remove the limitations of the evidence source. A small study may leave substantial uncertainty; a single-arm design may not separate treatment effects from changes over time or patient selection; and an endpoint may not fully capture outcomes important to patients. Bayesian modeling can express uncertainty under stated assumptions, but it cannot turn weak data into strong evidence merely by producing a precise-looking number.
| Assessment dimension | Core question | Warning sign |
|---|---|---|
| Statistical validity | Do the design, data, model, and computation support the inference? | Unjustified assumptions, unexplained analysis changes, or inadequate data handling |
| Clinical relevance | Does the result address a meaningful outcome and decision threshold? | Emphasis on any benefit, without considering its size or context |
| Transparency | Can a reviewer understand and evaluate the analysis choices? | Unclear priors, thresholds, exclusions, or sensitivity results |
A Practical Review Sequence
Identify the population, outcome, effect measure, direction of benefit, and clinical threshold behind the reported result.
Assess whether the trial design and data support the likelihood, assumptions, and computation used to obtain the posterior.
Consider both the probability of crossing the threshold and the plausible range of effect sizes, in clinical context.
Check whether key choices are explained and whether reasonable alternatives materially alter the conclusion.
The sequence helps prevent a common mistake: treating a large posterior probability as a complete regulatory argument. The wrong reasoning is, “The probability of benefit is high, so the evidence is sufficient.” The correction is to ask what benefit was defined, whether the model and data credibly support that probability, whether the magnitude matters clinically, and whether the conclusion is robust enough to inform the decision.
Check Your Understanding
Answer each question, then compare with the brief solutions.
- Why is a posterior probability of any positive effect not necessarily evidence of a clinically worthwhile effect?
- A posterior probability is reported without identifying the prior. Which assessment dimension is most directly weakened, and why?
- In the first worked example, what assumptions support the beta-binomial update?
- Why are a posterior probability and a frequentist confidence interval not interchangeable?
Solutions. (1) Any positive effect may be smaller than the amount needed to matter clinically; the relevant threshold and outcome must be justified. (2) Transparency is weakened because the result is conditional on the prior, and reviewers cannot assess its role or sensitivity. (3) Outcomes are assumed independent Bernoulli observations with one common response probability, and the prior is \(\operatorname{Beta}(1,1)\). (4) A posterior probability describes uncertainty conditional on the model and prior; a frequentist confidence interval is defined by repeated-sampling properties and does not directly assign probability to the parameter.