Credible Intervals vs. Confidence Intervals

patients responding. The observed response rate is:

Suppose we use a Beta prior:

This is a uniform prior on the response probability.

The binomial likelihood is:

and the posterior is:

Therefore:

Bayesian 95% Credible Interval for the Response Rate

The equal-tailed 95% credible interval can be obtained from the 2.5th and 97.5th percentiles of the posterior distribution:

In R:

x <- 18
n <- 50

a <- 1
b <- 1

post_a <- a + x
post_b <- b + n - x

qbeta(
  c(0.025, 0.975),
  post_a,
  post_b
)

The result is a 95% posterior credible interval for the response probability.

Frequentist Confidence Interval for the Same Data

The same data can be analyzed using a frequentist binomial confidence interval. For example, R provides several interval methods through binom.test() and prop.test().

x <- 18
n <- 50

binom.test(
  x,
  n,
  conf.level = 0.95
)

The resulting interval is a frequentist confidence interval.

The numerical endpoints may be close to the Bayesian credible interval, but the interpretation remains different.

Why the Prior Matters

The uniform \(Beta(1,1)\) prior is only one possibility. Suppose instead that historical evidence suggests a response rate near 20%. We could use a prior such as:

This prior has mean:

The corresponding posterior becomes:

or:

The posterior is therefore influenced by both the current trial and the prior information.

Weak vs. Informative Priors

This is one reason Bayesian analyses require transparent specification and justification of prior distributions.

Confidence Intervals Do Not "Ignore" Information

It is sometimes said that confidence intervals are based only on the current data and therefore "ignore prior information."

A more precise statement is that ordinary frequentist confidence intervals do not incorporate a subjective prior probability distribution for the unknown parameter.

Frequentist methods can still use information from previous studies in other ways, such as through study design, covariate adjustment, stratification, meta-analysis, or explicitly developed hierarchical models.

The key distinction is that this information is not interpreted as a prior probability distribution on the parameter in the Bayesian sense.

A Common Misinterpretation

This is not the standard frequentist interpretation.

The difference comes from whether probability is assigned to the unknown parameter after conditioning on the observed data.

A Second Common Misinterpretation

Another common misconception is:

"Credible intervals are always better because they can be interpreted directly."

That conclusion does not follow.

Bayesian intervals require a model and prior distribution. If the prior is poorly chosen, inappropriate, or insufficiently justified, the resulting posterior inference can be misleading.

Likewise, frequentist confidence intervals can have excellent statistical properties when their assumptions and coverage properties are appropriate.

Coverage Probability vs. Posterior Probability

One-Sided Intervals

The same distinction applies to one-sided intervals. Suppose a Bayesian analysis gives:

This is a direct posterior probability statement.

A frequentist one-sided 97.5% confidence bound instead has a repeated-sampling coverage interpretation.

Again, the numerical probability may be similar while the inferential meaning is different.

Credible Intervals in Bayesian Clinical Trials

Credible intervals are particularly common in Bayesian clinical trial applications. For example, suppose a Bayesian model produces a posterior distribution for a response probability \(p\).

The analysis might report:

The analysis can also report:

This can be particularly useful when the clinical decision is based on a threshold.

Credible Intervals in Hierarchical Models

Bayesian credible intervals are also natural outputs from hierarchical models.

Suppose several clinical studies estimate related treatment effects:

A Bayesian hierarchical model produces posterior distributions for:

• Each study-specific effect \(\theta_i\)
• The overall mean effect \(\mu\)
• The between-study heterogeneity \(\tau\)
• Potentially a future-study effect

Credible intervals can then be calculated for each of these quantities.

Why Bayesian Intervals Are Especially Useful for Probabilistic Decisions

Many clinical decisions are naturally probabilistic. For example:

• What is the probability that treatment benefit exceeds zero?
• What is the probability that benefit exceeds a clinically meaningful threshold?
• What is the probability that toxicity exceeds an unacceptable level?
• What is the probability that a future study will meet a target?

Posterior distributions can answer these questions directly.

But Bayesian Probability Is Conditional on the Model

The phrase "95% posterior probability" should not be interpreted as a model-free statement.

The posterior probability depends on:

The model includes assumptions about the likelihood, prior distribution, parameterization, and any hierarchical structure.

R Example: Comparing the Two Approaches

The following example uses the same binomial data for both approaches.

x <- 18
n <- 50

p_hat <- x / n

p_hat

The observed response rate is:

Frequentist Interval

freq_result <- binom.test(
  x,
  n,
  conf.level = 0.95
)

freq_result$conf.int

This produces a 95% confidence interval.

Bayesian Interval

Using a \(Beta(1,1)\) prior:

a <- 1
b <- 1

post_a <- a + x
post_b <- b + n - x

cred_int <- qbeta(
  c(0.025, 0.975),
  post_a,
  post_b
)

cred_int

This produces a 95% equal-tailed credible interval.

Posterior Probability of Exceeding a Clinical Threshold

Suppose the clinically meaningful response rate is 25%. We can calculate:

P_gt_25 <- 1 - pbeta(
  0.25,
  post_a,
  post_b
)

P_gt_25

This directly estimates:

This type of calculation is one of the major practical differences between Bayesian posterior inference and a conventional confidence interval.

Visualizing the Bayesian Posterior

p_grid <- seq(
  0,
  1,
  length.out = 1000
)

posterior_density <- dbeta(
  p_grid,
  post_a,
  post_b
)

plot(
  p_grid,
  posterior_density,
  type = "l",
  xlab = "Response probability",
  ylab = "Posterior density",
  main = "Posterior Distribution"
)

abline(
  v = cred_int,
  lty = 2
)

The posterior distribution makes the Bayesian interpretation visually transparent. The area under the curve between the credible-interval boundaries is 95%.

Confidence Interval vs. Credible Interval: A Mental Model

When the Intervals Agree

The two approaches can converge numerically under certain conditions. For example, with a large sample and a relatively weak prior:

This is one reason that practitioners sometimes overlook the conceptual difference.

The endpoints may look almost identical. The interpretation does not become identical.

When the Intervals Diverge

Greater differences can arise when:

• The sample size is small.
• The prior is informative.
• The posterior is highly skewed.
• The parameter has a constrained range.
• The model is hierarchical.
• The likelihood is non-normal.
• The Bayesian model incorporates external information.

These situations are common in early-phase clinical development, where sample sizes may be limited and prior information can be important.

Common Mistakes

1. Saying that a 95% confidence interval has a 95% probability of containing the parameter. This is not the standard frequentist interpretation.
2. Assuming a credible interval is automatically equivalent to a confidence interval. The numerical values can be similar while the statistical meanings differ.
3. Ignoring the prior. A Bayesian credible interval depends on the prior distribution.
4. Reporting a credible interval without describing the model. Posterior inference is conditional on the Bayesian model.
5. Assuming Bayesian intervals must be symmetric. Posterior distributions can be skewed, producing asymmetric credible intervals.
6. Assuming an informative prior is always beneficial. A poorly justified prior can materially influence inference in undesirable ways.
7. Using "credible interval" and "confidence interval" interchangeably. The distinction should be preserved in statistical reporting.
8. Thinking that Bayesian inference eliminates uncertainty. It does not. It provides a different framework for representing and quantifying uncertainty.

How to Report a Confidence Interval

A frequentist clinical report might say:

The confidence level and interval-generating method should be clearly specified in the statistical analysis plan or statistical methods.

How to Report a Credible Interval

A Bayesian report might say:

The Bayesian analysis should also describe the prior distributions and model used to obtain the posterior.

Clinical Interpretation Side-by-Side

Which One Should a Clinical Statistician Use?

There is no universal answer.

A frequentist confidence interval may be appropriate when the primary analysis is designed around conventional frequentist operating characteristics, regulatory expectations, and repeated-sampling properties.

A Bayesian credible interval may be particularly attractive when:

• Prior information is scientifically relevant.
• Probabilistic statements about treatment effects are desired.
• The analysis is part of a Bayesian adaptive design.
• Hierarchical borrowing is important.
• The parameter has a constrained or skewed posterior distribution.
• Decision-making is naturally expressed through posterior probabilities.

Regulatory and Protocol Considerations

When a Bayesian analysis is used in a clinical trial, the statistical analysis plan should clearly describe:

• The likelihood model
• The prior distributions
• The rationale for the priors
• The posterior distribution
• The definition of the credible interval
• Whether intervals are equal-tailed or HPD
• The decision thresholds
• The sensitivity analyses for alternative priors
• The computational method
• The convergence diagnostics when simulation-based methods are used

This is particularly important when the prior has a meaningful effect on the posterior conclusions.

Prior Sensitivity Analysis

A useful Bayesian sensitivity analysis repeats the analysis using reasonable alternative priors.

For example:

The resulting posterior intervals can then be compared.

If the clinical conclusion remains stable across reasonable priors, the conclusion is less sensitive to prior specification.

A Useful Rule of Thumb

Summary Table

The Most Important Concept

The most important distinction is not the numerical calculation. It is the interpretation of probability.

A confidence interval is constructed from a procedure with a specified repeated-sampling coverage property.

A credible interval is constructed from a posterior distribution and contains a specified amount of posterior probability.

Thus:

The distinction becomes particularly important when communicating results to clinical collaborators.

A statement such as "there is a 95% probability that the treatment effect is between these values" is naturally Bayesian. A conventional frequentist confidence interval does not justify that statement without additional assumptions or a Bayesian reinterpretation.

Practical Workflow for Clinical Statisticians

Quick Reference

References

Gelman, A., Carlin, J.B., Stern, H.S., Dunson, D.B., Vehtari, A. & Rubin, D.B. Bayesian Data Analysis. CRC Press.
Casella, G. & Berger, R.L. Statistical Inference. Duxbury.
Bernardo, J.M. & Smith, A.F.M. Bayesian Theory. Wiley.
Box, G.E.P. & Tiao, G.C. Bayesian Inference in Statistical Analysis. Wiley.
Hobbs, B.P., Carlin, B.P. & Mandrekar, S.J. Hierarchical models and Bayesian inference in clinical trials.