Posterior Probability vs. P-Value: A Comparison

These questions are natural in a Bayesian framework.

Credible Intervals vs. Confidence Intervals

Posterior probabilities are closely related to Bayesian credible intervals.

A 95% posterior credible interval is a region \(C\) satisfying:

This is a probability statement about the parameter conditional on the data.

A frequentist 95% confidence interval has a different interpretation. Its repeated-sampling property is:

where the probability refers to the procedure over repeated samples.

A Second Example: Stronger Evidence

Now suppose 18 responses are observed among 30 patients.

The observed response rate is:

Continue to use:

The exact one-sided binomial p-value is approximately:

Using the same uniform prior:

the posterior is:

The posterior probability that the response rate exceeds 30% is approximately:

Again, the two numbers are answering different questions.

Why a Small P-Value Does Not Mean "Small Probability of the Null"

The classic error becomes clearer mathematically. Suppose someone observes:

They might incorrectly conclude:

But the p-value actually describes:

The desired posterior probability would instead be:

To calculate that probability, one needs additional information about the prior probability of \(H_0\) and the competing hypotheses or parameter values.

Bayes' Theorem Shows Why the Difference Matters

For two competing hypotheses \(H_0\) and \(H_A\):

Notice that the posterior probability depends on:

• The probability of the data under \(H_0\)
• The probability of the data under \(H_A\)
• The prior probability of \(H_0\)
• The prior probability of \(H_A\)

A p-value uses the first component but does not by itself provide the other ingredients needed to calculate a posterior probability.

Prior Odds and Posterior Odds

Bayesian updating can be expressed especially elegantly using odds.

The prior odds are:

The posterior odds are:

Bayes' theorem gives:

The first ratio is the Bayes factor:

Therefore:

Bayes Factors vs. P-Values

A Bayes factor and a p-value are also not interchangeable.

A Bayes factor compares how well two hypotheses predict the observed data:

A p-value instead measures the extremeness of the data under a null reference distribution.

A Simple Discrete-Hypothesis Example

Suppose a trial considers two competing hypotheses:

Suppose 18 responses are observed among 30 patients.

The likelihood under \(H_0\) is:

The likelihood under \(H_A\) is:

The combinatorial factor cancels when calculating the Bayes factor:

The resulting Bayes factor quantifies how much the observed response count changes the relative evidence between these two specific hypotheses.

Posterior Probability Depends on the Prior

Suppose two analysts use the same data but different prior beliefs.

Analyst A assigns:

while Analyst B assigns:

Even if both analysts calculate exactly the same Bayes factor, their posterior probabilities will differ because their prior odds differ.

This is not a computational error. It is a mathematical consequence of Bayesian inference.

Why the Prior Matters Especially With Small Samples

When the sample size is small, the likelihood may contain limited information.

The prior can therefore contribute substantially to the posterior distribution.

As the amount of information in the data increases, the likelihood often dominates a reasonable prior.

Posterior Probability of Treatment Benefit

In clinical research, one of the most useful Bayesian quantities is often:

For example, if \(\theta\) is a treatment effect and zero represents no benefit:

Alternatively, if the clinically meaningful threshold is \(\delta\):

This allows the statistical analysis to focus directly on a clinically meaningful region.

Clinical Significance vs. Statistical Significance

Neither a p-value nor a posterior probability automatically establishes clinical importance.

Suppose a treatment effect is:

with a very small p-value because the study is extremely large.

The result may be statistically convincing but clinically unimportant.

Bayesian analysis can explicitly address this distinction by calculating:

where \(\delta\) is a clinically meaningful threshold.

A Useful Mental Model

Posterior Probability Is Not Just a "Better P-Value"

It is tempting to think of Bayesian posterior probabilities as simply a more intuitive replacement for p-values.

That is not quite correct.

The two approaches are based on different inferential philosophies.

A p-value is a frequentist tail probability derived from a sampling distribution.

A posterior probability is derived from a posterior distribution that combines a prior distribution with the likelihood.

They therefore answer different questions and can legitimately lead to different numerical summaries of the same evidence.

Why P-Values Can Be Difficult to Interpret

The p-value requires conditioning on a null hypothesis that may be scientifically implausible or unrealistic.

For example:

A p-value asks how unusual the data would be if \(\theta\) were exactly zero.

But the scientific question may instead be:

or:

The Bayesian framework can answer these questions directly.

Why Posterior Probabilities Can Also Be Misused

Bayesian inference is not immune to statistical mistakes.

Common problems include:

• Using an unjustified prior
• Ignoring prior sensitivity
• Choosing a prior after seeing the data
• Interpreting a posterior probability without specifying the model
• Ignoring model misspecification
• Treating a posterior probability as objective and prior-free
• Using an arbitrary clinical threshold

A posterior probability is conditional on the entire Bayesian model.

Posterior Probability With a Beta-Binomial Model

For a binary endpoint, one of the simplest Bayesian models is:

with a beta prior:

The posterior is:

This conjugate relationship makes the beta-binomial model especially useful for teaching and simple clinical examples.

Posterior Mean

The posterior mean is:

This combines the prior information and observed responses.

For the uniform prior \(Beta(1,1)\) and \(X=12,n=30\):

Thus the posterior mean is approximately 40.6%.

The observed response rate was exactly 40%. The slight difference arises because the Bayesian posterior incorporates the prior.

Posterior Probability Above a Threshold

The posterior probability that \(p\) exceeds a clinically meaningful threshold \(p^\ast\) is:

where \(F_{\operatorname{Beta}}\) is the beta cumulative distribution function.

For example:

which gives approximately:

R Implementation: Exact P-Value

The exact one-sided binomial p-value can be calculated with binom.test():

x <- 12
n <- 30
p0 <- 0.30

binom.test(
  x = x,
  n = n,
  p = p0,
  alternative = "greater"
)

The resulting p-value is approximately:

p-value
# approximately 0.1593

R Implementation: Posterior Distribution

With a Beta(1,1) prior:

x <- 12
n <- 30

a <- 1
b <- 1

post_a <- a + x
post_b <- b + n - x

post_a
post_b

The posterior is:

# Posterior:
# Beta(13, 19)

R Implementation: Posterior Probability

The posterior probability that the response rate exceeds 30% is:

p_clinical <- 0.30

posterior_prob <-
  1 - pbeta(
    p_clinical,
    shape1 = post_a,
    shape2 = post_b
  )

posterior_prob

which gives approximately:

# approximately 0.8931

R Implementation: Posterior Probability of the Null Region

The posterior probability that the response rate is at or below 30% is:

posterior_null <-
  pbeta(
    p_clinical,
    shape1 = post_a,
    shape2 = post_b
  )

posterior_null

This gives approximately:

# approximately 0.1069

R Implementation: Posterior Mean

posterior_mean <-
  post_a / (post_a + post_b)

posterior_mean

# approximately 0.40625

R Implementation: Credible Interval

A 95% equal-tail posterior credible interval can be calculated using the beta quantile function:

qbeta(
  c(0.025, 0.975),
  shape1 = post_a,
  shape2 = post_b
)

This interval is a direct posterior probability statement about the response rate under the specified Bayesian model.

Frequentist Confidence Interval for the Same Data

The frequentist analysis can also produce a confidence interval:

binom.test(
  x = 12,
  n = 30,
  p = 0.30,
  alternative = "greater",
  conf.level = 0.95
)

The confidence interval and Bayesian credible interval may be numerically similar, particularly with weakly informative priors and moderate sample sizes.

However, their interpretations remain different.

Confidence Interval vs. Credible Interval

The numerical endpoints can sometimes be nearly identical even though the underlying inferential interpretation differs.

A Key Numerical Insight

In the worked example:

while:

At first glance, these numbers may appear contradictory. They are not.

The p-value asks:

The posterior probability asks:

The conditioning is reversed.

The Inverse Probability Fallacy

The error of interpreting a p-value as the probability that the null hypothesis is true is sometimes described as an inverse probability fallacy.

The incorrect reasoning is:

But this inference is not valid without applying Bayes' theorem and specifying the competing hypotheses and their prior probabilities.

Why Prior Probability Can Change the Interpretation

Suppose a new treatment is being tested for a rare disease. If the prior probability that a genuinely effective treatment exists is very low, even a statistically unusual result under the null may not imply a high posterior probability of efficacy.

Conversely, if substantial prior evidence already supports efficacy, the same data may produce a considerably higher posterior probability.

This is a direct consequence of:

Statistical Significance Is Not the Same as Evidence Probability

A conventional significance threshold such as:

It does not mean:

Likewise, a posterior probability of 0.95 is not automatically equivalent to a p-value of 0.05.

There are mathematical relationships between certain Bayesian and frequentist procedures under specific assumptions, but there is no general conversion:

Posterior Probability and Decision Making

Bayesian clinical decisions can be built around posterior probabilities.

For example, a development program might define a decision rule:

This is a Bayesian decision rule.

It directly expresses the probability that the parameter exceeds the clinically relevant threshold.

A frequentist trial might instead use:

These are different decision frameworks.

Posterior Probability Can Incorporate Clinical Thresholds

Suppose a treatment effect of less than 5 percentage points is considered clinically unimportant. Then the relevant Bayesian question might be:

rather than simply:

This distinction is particularly important in large clinical trials, where tiny effects can produce very small p-values.

Multiple Testing and Posterior Probabilities

Bayesian posterior probabilities do not automatically solve every issue associated with multiple comparisons.

If many hypotheses, endpoints, subgroups, or treatments are analyzed, the Bayesian model should account for the multiplicity structure when appropriate.

Hierarchical models, shrinkage priors, multiplicity-adjusted decision criteria, and other approaches may be appropriate depending on the scientific question.

The important point is: switching from p-values to posterior probabilities does not eliminate the need for careful statistical design.

Posterior Probability in Adaptive Clinical Trials

Posterior probabilities are particularly useful in Bayesian adaptive designs.

For example, a trial might monitor:

and use prespecified thresholds for:

• Stopping for futility
• Stopping for efficacy
• Increasing or decreasing allocation probabilities
• Expanding a treatment arm
• Dropping an ineffective treatment

The posterior distribution is updated as new data become available.

Posterior Predictive Probability

Another Bayesian quantity commonly encountered in clinical development is the posterior predictive probability.

This is different from a posterior probability about the parameter.

For example, investigators might ask:

This probability integrates uncertainty in the parameter with uncertainty in future observations.

It can therefore be useful for interim decision making.

Three Probabilities That Should Not Be Confused

These quantities can all be useful, but they answer different questions.

Common Misconceptions

1. "A p-value is the probability that the null hypothesis is true."

   False. A p-value is calculated under a null hypothesis and describes the extremeness of the observed data under that hypothesis.

2. "A p-value of 0.03 means there is a 97% chance the treatment works."

   False. That is not what the p-value represents.

3. "A posterior probability is completely objective."

   Not necessarily. It depends on the specified Bayesian model and prior.

4. "A 95% confidence interval means there is a 95% probability that the parameter is inside it."

   That is not the standard frequentist interpretation.

5. "A 95% credible interval and a 95% confidence interval mean the same thing."

   They can have similar numerical endpoints but have different probability interpretations.

6. "Bayesian and frequentist methods must give the same conclusion."

   Not necessarily. They can produce different numerical evidence summaries and different decisions, particularly with informative priors or small samples.

7. "A Bayesian analysis does not need hypothesis testing."

   Bayesian analyses can use explicit posterior probability decision rules, Bayes factors, posterior predictive probabilities, or other criteria.

8. "A small p-value proves the treatment is clinically important."

   Statistical evidence does not establish clinical relevance.

A Practical Comparison

When a P-Value Is Useful

P-values remain useful for many statistical purposes.

They can provide a standardized measure of how inconsistent observed data are with a specified null model.

They are particularly familiar in frequentist clinical trial designs, where type I error, power, confidence intervals, and prespecified testing procedures form an integrated framework.

The key is to interpret the p-value correctly.

When Posterior Probabilities Are Useful

Posterior probabilities are especially useful when the scientific question is naturally probabilistic.

Examples include:

• Probability that a treatment effect exceeds a clinical threshold
• Probability that a response rate exceeds a target
• Probability that one treatment is better than another
• Probability that a parameter lies within a clinically important range
• Probability of future trial success
• Bayesian adaptive trial decisions

A Bayesian Clinical Trial Statement

Instead of reporting only:

a Bayesian analysis might report:

along with:

• The prior distribution
• The posterior distribution
• A credible interval
• The clinically relevant threshold \(\delta\)
• A sensitivity analysis for alternative priors

This provides a direct probability statement about the parameter of interest.

How to Report a P-Value Correctly

A good frequentist report should connect the p-value to the hypothesis being tested.

For example:

Avoid saying:

How to Report a Posterior Probability Correctly

A Bayesian report should identify the prior and the parameter region.

This statement is interpretable because it identifies:

• The parameter
• The threshold
• The posterior probability
• The prior distribution

Sensitivity Analysis for the Prior

A strong Bayesian clinical analysis should often examine how much the posterior depends on reasonable alternative priors.

For example:

x <- 12
n <- 30
threshold <- 0.30

# Weak / uniform prior
post1 <- pbeta(
  threshold,
  1 + x,
  1 + n - x
)

# Moderately informative prior
post2 <- pbeta(
  threshold,
  3 + x,
  7 + n - x
)

# More concentrated prior centered near 30%
post3 <- pbeta(
  threshold,
  30 + x,
  70 + n - x
)

c(
  uniform = 1 - post1,
  moderate = 1 - post2,
  concentrated = 1 - post3
)

The exact posterior probabilities will differ because the priors encode different amounts of prior information.

If the substantive conclusion is unchanged across scientifically reasonable priors, the result is more robust to prior specification.

Posterior Probability and the Null Hypothesis

There is one additional subtlety.

In a continuous parameter model, the probability of an exact point such as:

Therefore, Bayesian analyses often formulate clinically meaningful regions:

Alternatively, a point null can be assigned positive prior probability using a mixture prior.

Point Nulls and Mixture Priors

Suppose an investigator explicitly wants to assign positive probability to:

A mixture prior can be constructed:

where \(\delta_{0.30}\) represents a point mass at 0.30 and \(g(p)\) represents a continuous alternative distribution.

This permits direct calculation of:

rather than merely calculating the posterior probability of a region such as \(p\le0.30\).

Why This Distinction Matters in Clinical Development

Clinical development decisions frequently involve questions such as:

• Is the treatment sufficiently effective to continue?
• What is the probability that the treatment provides clinically meaningful benefit?
• What is the probability that the response rate exceeds a target?
• How likely is the next study to succeed?
• How much uncertainty remains about the treatment effect?

These questions are not always naturally answered by a p-value.

Bayesian posterior and predictive probabilities can provide direct answers, provided the Bayesian model and prior are appropriate.

The Same Data Can Support Multiple Summaries

For the example with 12 responses among 30 patients:

None of these quantities is "the correct number" independent of context. They summarize different aspects of the evidence.

A Useful Clinical Interpretation Framework

A Direct Side-by-Side Example

Common Reporting Errors

1. Calling the p-value the probability of the null.
2. Calling a 95% confidence interval a 95% probability interval.
3. Reporting a posterior probability without specifying the prior.
4. Ignoring prior sensitivity.
5. Equating posterior probability with clinical importance.
6. Assuming a p-value and posterior probability should numerically agree.
7. Converting a p-value directly into a posterior probability.
8. Using "probability the treatment works" without defining what "works" means.

Practical Workflow for Clinical Statisticians

The Most Important Equations to Remember

The p-value is fundamentally:

A posterior probability is:

For a parameter region:

Bayes' theorem connects prior and posterior probabilities:

And the odds formulation is:

Final Comparison

Worked Example Summary

For 12 responses among 30 patients with a null response rate of 30%:

The p-value and posterior probability are not contradictory. They simply answer different questions.

The Most Important Concept

The single most important concept in this tutorial is:

The first quantity is related to the logic of a p-value. The second is a posterior probability.

Confusing these two probabilities is one of the most common errors in statistical interpretation.

If a clinical statistician remembers only one sentence from this tutorial, it should be:

References

Bayes, T. (1763). An Essay towards solving a Problem in the Doctrine of Chances. Philosophical Transactions of the Royal Society of London, 53, 370–418.
Gelman, A., Carlin, J.B., Stern, H.S., Dunson, D.B., Vehtari, A. & Rubin, D.B. (2013). Bayesian Data Analysis, 3rd ed. CRC Press.
Berry, S.M., Carlin, B.P., Lee, J.J. & Muller, P. (2010). Bayesian Adaptive Methods for Clinical Trials. CRC Press.
Spiegelhalter, D.J., Abrams, K.R. & Myles, J.P. (2004). Bayesian Approaches to Clinical Trials and Health-Care Evaluation. Wiley.
Greenland, S. (2019). Valid P-values behave exactly as they should: Some misleading criticisms of P-values and their resolution. American Statistician, 73(sup1), 106–114.
Wasserstein, R.L., Schirm, A.L. & Lazar, N.A. (2019). Moving to a world beyond "p < 0.05". American Statistician, 73(sup1), 1–19.
Goodman, S.N. (1999). Toward evidence-based medical statistics. 1: The P value fallacy. Annals of Internal Medicine, 130(12), 995–1004.
Goodman, S.N. (1999). Toward evidence-based medical statistics. 2: The Bayes factor. Annals of Internal Medicine, 130(12), 1005–1013.