What Is the Probability About?
A statement such as “there is a 95% probability of benefit” is incomplete until the event and the source of uncertainty are clear. It might refer to uncertainty about a treatment-effect parameter after observing trial data, the outcome of a future patient, or the frequency of results across repeated trials. These are different probability statements, even when they use the same numerical value.
In Bayesian analysis, a posterior probability describes uncertainty about an unknown parameter or event, conditional on the observed data and the statistical model, including its prior distribution. A posterior predictive probability instead describes uncertainty about unobserved data, conditional on the observed data and model. Neither should be casually translated into a statement about long-run frequencies or a probability attached to an exact null hypothesis.
The conditioning matters. A posterior probability is not a property of the data alone: it is conditional on the data and the model used to analyse them. In clinical reporting, state the event in terms of the treatment effect, give its direction and scale, and specify any clinical threshold. The introductory material on reporting posterior probability of treatment benefit establishes the basic form of these claims; the focus here is avoiding changes in meaning as they are interpreted or paraphrased.
Parameter Uncertainty Is Not a Future Patient Outcome
Suppose \(\delta\) is the treatment-minus-control difference in mean symptom improvement, measured in points, with larger values favouring treatment. The statement \(P(\delta>0\mid y)=0.95\) concerns the unknown population-average treatment effect. It does not say that 95% of treated patients will improve, that 95% of future treated patients will do better than a control patient, or that a particular future patient has a 95% chance of benefiting.
A future outcome also varies because patients differ and because outcomes are stochastic, even if the population parameter were known. To predict future data, those sources of outcome variation must be included. Formally, a posterior predictive probability averages the future-data distribution over the posterior distribution of the parameter:
This identity follows by conditioning on the unknown parameter and integrating over its posterior distribution. The inner probability describes future data conditional on a parameter value; the integral accounts for uncertainty about that value. Therefore, a parameter probability and a predictive probability can differ substantially, even when they concern related clinical questions.
Worked Example: Posterior Probability Versus a Future Response
Suppose, hypothetically, a single-arm trial observes 7 responses among 8 participants. Let \(\theta\) be the response probability for the trial’s target population. Assume independent Bernoulli outcomes with response probability \(\theta\), and a uniform prior \(\theta\sim\mathrm{Beta}(1,1)\). Find the posterior probability that \(\theta\) exceeds 0.60 and the predictive probability that the next participant responds.
The binomial likelihood is proportional to \(\theta^7(1-\theta)^1\), so the beta-binomial updating rule gives \(\theta\mid y\sim\mathrm{Beta}(8,2)\). Its density is \(72\theta^7(1-\theta)\) for \(0<\theta<1\). Thus:
This is a 92.95% posterior probability that the population response probability exceeds 60%, conditional on the stated prior and model. For a future participant’s binary response \(Y_{\mathrm{new}}\), the posterior predictive probability is the posterior mean of \(\theta\):
The two results answer different questions: the first concerns whether a population parameter is above a threshold; the second concerns the response of one future participant. The calculation assumes the future participant comes from the same target population and that the response model remains applicable. Neither result alone establishes the response probability in a different population or under different care conditions.
Worked Example: Individual Outcomes Include More Variation
Suppose, hypothetically, the posterior distribution of a treatment-minus-control mean difference in symptom score is \(\delta\mid y\sim N(2,1)\), in points, with larger scores representing greater improvement. Assume future treated and control outcomes for independently selected patients have a difference distributed as \(N(\delta,200)\), conditional on \(\delta\). This corresponds to independent outcome variation with variance 100 points squared in each arm, treated as known. Compare the posterior probability of a positive mean effect with the predictive probability that a future treated patient has a larger outcome than a future control patient.
The parameter event has posterior probability:
Here \(\Phi\) is the standard normal cumulative distribution function. For the difference \(D_{\mathrm{new}}\) between the two future patients’ outcomes, uncertainty in both the patient outcomes and \(\delta\) must be included. A normal distribution plus an independent normal parameter distribution gives:
Therefore:
The posterior probability of a positive average effect is about 97.72%, whereas the predictive probability that one randomly selected treated patient has a higher outcome than one independently selected control patient is about 55.61%. The latter is close to one half because individual outcome variation is large relative to the mean difference. It is not a probability that the treatment has a positive average effect.
Posterior Probability Is Not Automatically a Long-Run Frequency
A posterior probability is conditional on the observed data. It does not, by itself, assert that the corresponding event will occur in that proportion of repeated trials. For example, a 95% posterior probability that \(\delta>0\) does not mean that 95% of future trials will show a positive estimated treatment effect, nor that 95% of patients benefit. Those are different events with different probability models.
The same distinction applies to credible intervals. A 95% posterior credible interval is constructed to contain 95% posterior probability for the parameter, conditional on the data and model. It is not automatically a procedure with 95% coverage for every fixed parameter value across repeated sampling. Such repeated-sampling coverage is a separate property that depends on the interval procedure and sampling conditions. Reporting the credible interval in its posterior sense does not require claiming that it has a particular frequentist coverage property.
A practical check is to ask what would be repeated in the proposed frequency statement. Is it a trial, a patient, a response, or an interval-producing procedure? If the intended statement is about future observations, specify the predictive event and use a predictive distribution. If it is about repeated trial operating characteristics, those characteristics require an explicit repeated-sampling evaluation; they are not read directly from a posterior probability.
When Can a Hypothesis Have a Posterior Probability?
The phrase “probability that the hypothesis is true” needs special care. A hypothesis that defines a region of parameter values can have a posterior probability. For example, if “benefit” is defined as \(\delta>0\), then \(P(\delta>0\mid y)\) is exactly the posterior probability of that event. If clinically meaningful benefit is defined as \(\delta>m\), where \(m\) is a prespecified positive margin in the units of the outcome, \(P(\delta>m\mid y)\) is the posterior probability of exceeding that margin.
An exact point null, such as \(H_0:\delta=0\), is different under a continuous prior. A continuous posterior assigns probability zero to any single exact value, so \(P(\delta=0\mid y)=0\), even if values very close to zero remain plausible. This is a consequence of assigning probability across a continuum, not evidence that the null has been disproved. A posterior probability for the exact null requires a model that assigns it positive prior probability, such as a model with a point mass at zero.
Worked Example: An Exact Null Under Two Different Models
Consider a hypothetical comparison of two models: \(M_0\), which fixes the treatment effect at \(\delta=0\), and \(M_1\), which allows a nonzero effect under a continuous prior. Suppose each model has prior probability 0.50, and the observed data have a Bayes factor \(BF_{10}=4\), meaning the data are four times as probable under \(M_1\) as under \(M_0\).
The posterior odds for \(M_1\) versus \(M_0\) equal the prior odds multiplied by the Bayes factor. The prior odds are \(0.50/0.50=1\), so the posterior odds are \(1\times4=4\). The posterior model probabilities are therefore:
In this model comparison, the exact null has a posterior probability of 20% because it had positive prior model probability. This is not the same calculation as assigning a continuous prior to \(\delta\) alone, under which \(P(\delta=0\mid y)=0\). The model probabilities also depend on the specified prior model probabilities and the likelihoods under both models. A report should name the model comparison rather than present the 20% as a model-free probability that the null is true.
Comparing the Probability Statements
| Statement type | What is uncertain? | Example of a correctly framed claim |
|---|---|---|
| Posterior parameter probability | An unknown parameter, conditional on data and model | “The posterior probability that the mean treatment effect exceeds 0 points is 97.7%.” |
| Posterior predictive probability | Specified future observations, accounting for parameter uncertainty | “The predictive probability of a response in the next participant is 80.0%.” |
| Repeated-sampling frequency | Results across repetitions of a defined sampling procedure | “In the specified simulation, 92% of repeated trials met the decision criterion.” |
| Posterior probability of a point-null model | Which of the explicitly specified models generated the data | “The posterior probability of \(M_0\) is 20.0% under the stated model comparison.” |
These statements are not interchangeable. A frequentist p-value, for example, is not a posterior probability that a null hypothesis is true; its meaning is defined through the probability of data or more extreme data under a null model. A posterior probability requires a posterior distribution and therefore depends on the Bayesian model and prior. Comparing them as though they answer the same question produces misleading conclusions.
Common Misinterpretations and Checks
- “There is a 95% chance the treatment works.” This leaves “works” undefined and may suggest an individual outcome. State the parameter event instead: “The posterior probability that the average treatment effect is positive is 95%,” or give a clinical threshold if that is the intended event.
- “The posterior probability is the proportion of patients who benefit.” A population-average parameter probability does not describe the fraction of individual patients who benefit. A patient-level claim needs a patient-outcome model and a clearly defined predictive event.
- “There is a 5% probability that the exact null is true.” This does not follow from a 95% posterior probability of benefit under a continuous parameter model. The complement is the posterior probability of the specified complementary parameter region, not the probability of an exact point null.
- “A 95% posterior probability means 95% of repeated trials will reach the same conclusion.” That is a repeated-trial frequency claim. It requires evaluation under a defined repeated-sampling design and should not be inferred from the posterior statement alone.
Before reporting any probability, write down the event in full: the parameter or future observation, the threshold or outcome, the population, and the conditioning information. Then check whether the calculation actually targets that event. If the report’s wording changes “average treatment effect” into “patients benefit,” or “conditional on these data and model” into “in repeated trials,” the interpretation has shifted and should be corrected.
Check Your Understanding
For each question, identify the probability being described or rewrite the claim precisely. Solutions follow.
- A trial reports \(P(\delta>0\mid y)=0.91\). What does this say, and what does it not say?
- A model gives \(P(Y_{\mathrm{new}}=1\mid y)=0.70\) for a future patient. Is this the posterior probability that the population response rate exceeds 70%?
- Under a continuous posterior for \(\delta\), why does \(P(\delta=0\mid y)=0\) not show that the null has been ruled out?
- A report says, “Our 90% posterior probability of benefit means 90% of repeated trials will find benefit.” Correct the interpretation.
Solutions. (1) It says that, conditional on the data and model, the posterior probability that the population treatment-effect parameter is positive is 91%. It does not give the percentage of patients who benefit or the probability of a positive outcome for a particular future patient. (2) No. It is a predictive probability for a future response. The parameter claim would instead be an event such as \(P(\theta>0.70\mid y)\), which must be calculated separately. (3) A continuous distribution assigns zero probability to every exact point while assigning probability to intervals around that point. A point-null model needs positive prior mass if its posterior probability is to be positive. (4) The 90% posterior probability concerns the defined benefit event conditional on the data and model. The proportion of repeated trials that find benefit is a separate operating-characteristic quantity and requires a specified repeated-trial evaluation.