Bayesian Sample Size Determination: A Worked Example

Sample size may then be determined by simulation because the distribution of future events and posterior quantities is substantially more complicated than in the beta-binomial example.

Simulation-Based Bayesian Sample Size Determination

Many realistic Bayesian trial designs cannot be solved analytically.

Simulation is then used.

The general algorithm is:

Monte Carlo Estimate of Assurance

Suppose \(M\) simulated trials are generated for a candidate sample size. Let:

Then the estimated assurance is:

For example, if 8,120 of 10,000 simulated trials satisfy the criterion:

or approximately 81.2% assurance.

Monte Carlo Error Matters

Because simulation produces an estimate rather than an exact probability, Monte Carlo error should be considered.

If the estimated assurance is approximately \(\widehat A\), its Monte Carlo standard error is approximately:

For \(M=10,000\) and \(\widehat A=0.80\):

Thus the simulation estimate has a Monte Carlo standard error of approximately 0.4 percentage points.

Frequentist Operating Characteristics Can Still Be Evaluated

A Bayesian design does not prevent investigators from examining frequentist operating characteristics.

For example, suppose the Bayesian decision rule is:

The investigator can still calculate the probability of declaring success when the true response rate is exactly 20%:

This quantity is sometimes described as a frequentist type I error associated with the Bayesian decision rule.

It is important to distinguish this from the Bayesian posterior probability.

Posterior Probability Is Not Type I Error

Suppose the final analysis produces:

This does not mean that there is a 3% type I error.

It means that, conditional on the specified model, prior, and observed data, the posterior probability that \(p\) exceeds 20% is 97%.

A frequentist operating characteristic such as type I error is instead a long-run probability evaluated across repeated hypothetical datasets under a specified true parameter value.

Sensitivity to the Prior

Bayesian sample size calculations can be sensitive to the prior, particularly when the prior is informative relative to the amount of new clinical data.

Suppose the prior is:

and the study observes \(x\) responses among \(n\) patients.

The posterior is:

A stronger prior therefore contributes more information to the posterior.

This can reduce the amount of new information required to satisfy a posterior criterion.

Prior Sensitivity Analysis

A robust Bayesian sample size assessment should examine multiple plausible priors.

For example:

The sample size should be evaluated under each scientifically plausible prior rather than relying exclusively on the prior that produces the smallest sample size.

Prior-Data Conflict

An informative prior can also conflict with the observed data.

For example, suppose historical studies strongly support a response rate near 40%, but the new trial produces a response rate near 10%.

The posterior will reflect both sources of information.

The degree to which the historical evidence influences the posterior depends on its effective strength and relevance.

Discounting Historical Information

In some Bayesian clinical trial designs, historical information may be discounted.

Conceptually, an effective prior might contribute less information than the raw historical dataset.

For example, a historical dataset containing 100 patients might be represented by a prior with an effective sample size substantially smaller than 100 if there is uncertainty about exchangeability.

This is one reason that prior elicitation and prior-data compatibility assessment are important parts of Bayesian design.

Sample Size and Prior Information

The relationship can be summarized conceptually as:

Therefore, stronger relevant prior information can sometimes reduce the amount of new information needed.

However, this should not be interpreted as a general justification for using strong priors simply to obtain smaller trials.

What Happens If the Prior Is Very Informative?

Suppose the prior effective sample size is very large.

Then the posterior may remain close to the prior even after substantial new data are collected.

This creates an important design consideration: the trial may appear to have high assurance because the prior is doing much of the inferential work.

Investigators should therefore examine how much of the posterior conclusion comes from the new trial versus the prior.

Effective Sample Size of the Prior

For a beta prior:

a simple measure of prior effective sample size is:

After observing \(n\) new patients:

This is a useful conceptual diagnostic, although effective sample size in more complex Bayesian models requires more sophisticated definitions.

Bayesian Sample Size Is Often a Design Optimization Problem

The investigator may not simply want the smallest possible sample size.

There may be competing objectives:

• High assurance of a successful conclusion
• High posterior precision
• Low patient exposure
• High probability of identifying an ineffective treatment
• Robustness to prior uncertainty
• High probability of correctly selecting the best treatment
• Acceptable operational complexity

The final design can therefore be viewed as an optimization problem.

A General Bayesian Sample Size Workflow

Exact Enumeration Versus Simulation

For simple conjugate models, exact enumeration is often possible.

For the beta-binomial example, all possible values of \(X\) can be evaluated:

For each value, calculate the posterior probability and determine whether the Bayesian criterion is satisfied.

The assurance is then obtained by summing the probabilities of the successful response counts.

For complex models, simulation is usually more practical.

R Implementation: Define the Design

p0 <- 0.20
pA <- 0.40

a0 <- 1
b0 <- 1

posterior_cutoff <- 0.95
target_assurance <- 0.80

Here:

• p0 is the clinically relevant threshold.
• pA is the planning true response probability.
• a0 and b0 define the beta prior.
• posterior_cutoff is the required posterior probability.
• target_assurance is the desired probability of satisfying the Bayesian criterion.

R Implementation: Posterior Probability

posterior_prob <- function(x, n,
                            a = a0,
                            b = b0,
                            threshold = p0) {

  1 - pbeta(
    threshold,
    shape1 = a + x,
    shape2 = b + n - x
  )
}

For a given number of responses \(x\), this function calculates:

R Implementation: Determine the Bayesian Success Boundary

success_boundary <- function(n) {

  posterior_values <- sapply(
    0:n,
    function(x)
      posterior_prob(x, n)
  )

  x_success <- which(
    posterior_values >= posterior_cutoff
  ) - 1

  if (length(x_success) == 0) {
    return(NA)
  }

  min(x_success)
}

For the worked example:

success_boundary(24)

# 8

Thus eight responses among 24 patients are sufficient to satisfy the posterior probability criterion.

R Implementation: Calculate Assurance

assurance <- function(n, p_true = pA) {

  x_boundary <- success_boundary(n)

  if (is.na(x_boundary)) {
    return(0)
  }

  pbinom(
    x_boundary - 1,
    size = n,
    prob = p_true,
    lower.tail = FALSE
  )
}

For the worked example:

assurance(24)

# approximately 0.8081

Search for the Required Sample Size

results <- data.frame(
  n = 1:100
)

results$boundary <- sapply(
  results$n,
  success_boundary
)

results$assurance <- sapply(
  results$n,
  assurance
)

results[
  results$assurance >= target_assurance,
][1, ]

The first qualifying design is approximately:

n
# 24

boundary
# 8

assurance
# approximately 0.8081

A Compact Exact Search Function

find_bayesian_n <- function(
    p0,
    pA,
    a0,
    b0,
    posterior_cutoff = 0.95,
    target_assurance = 0.80,
    max_n = 200) {

  for (n in 1:max_n) {

    posterior_values <- sapply(
      0:n,
      function(x) {
        1 - pbeta(
          p0,
          a0 + x,
          b0 + n - x
        )
      }
    )

    successful_x <- which(
      posterior_values >= posterior_cutoff
    ) - 1

    if (length(successful_x) == 0)
      next

    boundary <- min(successful_x)

    assurance <- pbinom(
      boundary - 1,
      size = n,
      prob = pA,
      lower.tail = FALSE
    )

    if (assurance >= target_assurance) {

      return(
        data.frame(
          n = n,
          success_boundary = boundary,
          assurance = assurance
        )
      )
    }
  }

  NULL
}

The design can then be obtained with:

find_bayesian_n(
  p0 = 0.20,
  pA = 0.40,
  a0 = 1,
  b0 = 1,
  posterior_cutoff = 0.95,
  target_assurance = 0.80
)

which returns approximately:

   n   success_boundary   assurance
  24          8             0.8081

Evaluating the Induced Frequentist Type I Error

Although the design is Bayesian, the decision rule can be evaluated under a frequentist null response rate.

For the worked example, the Bayesian success rule is:

when \(n=24\).

If the true response rate is exactly 20%, the probability of falsely declaring success is:

This is approximately:

This quantity is not required to be exactly 5%, because 5% was not used as the primary Bayesian design criterion.

R Code for the Induced Type I Error

n <- 24
boundary <- 8

type1_error <- pbinom(
  boundary - 1,
  size = n,
  prob = p0,
  lower.tail = FALSE
)

type1_error

This evaluates:

Posterior Probability After Different Results

It is useful to examine how the posterior conclusion changes with the observed number of responses.

Observed Response Rate Versus Posterior Probability

An important Bayesian design principle is that the observed response rate and the posterior probability answer different questions.

For eight responses among 24:

The observed response rate is therefore approximately 33.3%.

But the Bayesian decision is based on:

The second quantity incorporates uncertainty about the true response rate and the prior distribution.

Why the Posterior Threshold Is Not the Same as the Clinical Threshold

Another common mistake is to assume that:

That is not necessarily true.

The posterior distribution incorporates uncertainty, so the posterior probability depends on the entire observed dataset and the prior.

Likewise, an observed response rate above 20% does not automatically imply that the posterior probability exceeds 95%.

Bayesian Sample Size Under Different Assumed True Response Rates

The required sample size can be evaluated under several possible true response rates.

For the worked decision rule:

assurance will increase as the true response rate becomes larger.

The sample size calculation should ideally be evaluated across this entire range rather than only at one assumed value.

Assurance Curves

A useful operating characteristic is the assurance curve:

This function shows how the probability of a successful Bayesian conclusion changes as the true treatment effect changes.

For the worked example:

while the assurance will generally be lower when the true response rate is near or below 20% and higher when the true response rate is substantially above 40%.

Assurance Is Not Guaranteed Probability of Clinical Benefit

Suppose the design has 80% assurance at \(p=0.40\).

This does not mean:

nor does it mean that the treatment has an 80% probability of being clinically beneficial.

It means that under repeated hypothetical trials generated with the true response probability set to 40%, approximately 80% would satisfy the prespecified posterior decision criterion.

Bayesian Sample Size With Multiple Scenarios

A strong design analysis often considers several scenarios simultaneously.

For example:

This gives decision-makers a much more complete understanding of how the design behaves.

Bayesian Sample Size and Clinical Interpretation

The most important design quantity should ultimately have a clinical meaning.

For example:

should represent a response rate below which the treatment would not be considered sufficiently promising.

Similarly:

should represent a response rate that would meaningfully change the development decision.

Common Mistakes

1. Calling posterior probability "power." A posterior probability and frequentist power are different quantities.
2. Ignoring the prior. The prior is part of the Bayesian model and can affect the required sample size.
3. Choosing an informative prior only because it reduces sample size. Prior information should be scientifically justified.
4. Reporting assurance without specifying the planning scenario. Assurance depends on the assumed true parameter value or its planning distribution.
5. Confusing assurance with the probability that the treatment works. Assurance is a probability over future datasets under specified planning assumptions.
6. Using the observed effect size as the posterior decision criterion. The posterior distribution, not merely the point estimate, determines the Bayesian decision.
7. Ignoring prior sensitivity. Informative priors can materially affect sample size and posterior conclusions.
8. Failing to evaluate frequentist operating characteristics. Even Bayesian designs can benefit from examining repeated-sampling behavior.
9. Using simulation without enough Monte Carlo replicates. Simulation error can be important when candidate sample sizes are close.
10. Failing to prespecify the Bayesian decision rule. Changing the posterior probability threshold after seeing the data can alter the design's operating characteristics.

Protocol Considerations

A Bayesian sample size calculation should be fully documented before enrollment.

At minimum, the protocol should specify:

• Primary estimand
• Primary endpoint
• Statistical model
• Prior distribution
• Rationale for the prior
• Clinically meaningful threshold
• Posterior decision criterion
• Planning treatment effect
• Target assurance
• Maximum sample size
• Method used to calculate assurance
• Prior sensitivity analyses
• Relevant frequentist operating characteristics
• Handling of missing data
• Rules for unevaluable patients
• Simulation details, when applicable

Interim Analyses in Bayesian Designs

Bayesian trials can naturally accommodate interim analyses because posterior probabilities can be updated as new data accumulate.

For example, a study might evaluate:

after 20, 30, 40, and 50 patients.

The trial could stop for success if:

or stop for futility if:

provided that these rules were prespecified and their operating characteristics were evaluated during design.

Bayesian Monitoring and Sample Size Are Connected

If interim stopping rules are introduced, the design is no longer simply a fixed-sample calculation.

The sample size analysis should then evaluate:

• Probability of early success
• Probability of early futility
• Probability of reaching the maximum sample size
• Expected sample size
• Probability of a final successful conclusion
• Frequentist false-positive behavior

Simulation is often the most practical way to evaluate these quantities.

Bayesian Sample Size With Adaptive Designs

The same principles extend to more complicated adaptive trials.

Examples include:

• Response-adaptive randomization
• Sample size re-estimation
• Adaptive dose selection
• Multiple treatment arms
• Platform trials
• Bayesian hierarchical models
• Borrowing of historical information

In these settings, the probability distribution of future data depends on the adaptive rules themselves.

Consequently, simulation becomes increasingly important.

Decision-Theoretic Bayesian Sample Size

A more advanced Bayesian approach uses expected utility.

Suppose:

is the utility associated with making decision \(d\) when the true parameter is \(\theta\).

The expected utility before observing the data can be written:

where \(C(n)\) represents the cost or burden associated with enrolling additional patients.

The optimal sample size can then be defined as:

This approach explicitly recognizes that larger sample sizes have both benefits and costs.

When Decision-Theoretic Methods Become Useful

Expected-utility approaches can be attractive when:

• Patient exposure has substantial ethical or economic consequences.
• Multiple possible clinical decisions exist.
• The cost of delaying development is important.
• The consequences of false-positive and false-negative decisions are asymmetric.
• There are several competing treatment-development options.

These methods are considerably more complex than the simple assurance approach, but they provide a natural framework for Bayesian decision-making.

Bayesian Versus Frequentist Sample Size: Conceptual Comparison

What Makes Bayesian Sample Size Determination Difficult?

The mathematics of the simple beta-binomial example are relatively easy.

The difficult part is usually the design specification.

Investigators must decide:

• What prior is scientifically appropriate?
• How much historical information should be borrowed?
• What posterior probability constitutes sufficient evidence?
• What effect should be used for planning?
• How much assurance is sufficient?
• What is the cost of an additional patient?
• What operating characteristics are clinically important?

These are scientific and decision-making questions, not merely mathematical questions.

A Practical Bayesian Sample Size Workflow

Worked Example Summary

The complete binary-endpoint example can be summarized as follows.

The Most Important Concept

The most important conceptual point is that Bayesian sample size determination is fundamentally a future-posterior design problem.

The investigator does not know what the future data will look like.

Instead, the investigator considers all plausible future datasets and asks:

For a binary endpoint, this can sometimes be solved exactly by enumerating every possible response count.

For more complex clinical trial designs, the same principle is usually implemented through simulation.

The design therefore connects three distributions:

Bottom Line

References

Berry, S.M., Carlin, B.P., Lee, J.J. & Muller, P. (2010). Bayesian Adaptive Methods for Clinical Trials. CRC Press.
Spiegelhalter, D.J., Abrams, K.R. & Myles, J.P. (2004). Bayesian Approaches to Clinical Trials and Health-Care Evaluation. Wiley.
O'Hagan, A., Stevens, J.W. & Campbell, M.J. (2005). Assurance in clinical trial design. Pharmaceutical Statistics, 4, 187–201.
Berry, S.M. (2006). Bayesian clinical trials. Nature Reviews Drug Discovery, 5, 27–36.
Berry, D.A. (2006). Bayesian clinical trials. Nature Reviews Drug Discovery, 5, 27–36.
Neuenschwander, B., Branson, M. & Gsponer, T. (2008). Critical aspects of the Bayesian approach to phase I cancer trials. Statistics in Medicine, 27, 2420–2439.
Thall, P.F. & Wathen, J.K. (2007). Practical Bayesian adaptive randomisation in clinical trials. European Journal of Cancer, 43, 859–866.
Hobbs, B.P., Carlin, B.P. & Mandrekar, S.J. (2013). Bayesian group sequential designs for clinical trials. Statistical Methods in Medical Research.