Common Mistakes in Bayesian Sample Size Calculations

Why Sample-Size Calculations Go Wrong

A Bayesian sample-size calculation combines a model for future trial data with a rule for declaring success. A numerical result can look precise while answering the wrong design question: for example, the chance of success at one assumed treatment effect rather than the average chance over plausible effects. Other errors arise from averaging over an unintended prior, translating a posterior decision rule incorrectly, or treating a simulation estimate as exact.

The calculations below use a continuous treatment effect to make these distinctions explicit. Let \(\delta\) denote the treatment-minus-control effect, measured in the endpoint’s units. Suppose the trial produces an estimator \(Y\) with known sampling variance \(v\), and use the model \(Y\mid\delta\sim N(\delta,v)\). The analysis prior is \(\delta\sim N(m_0,\tau^2)\), with \(\tau>0\). These assumptions are simplifying: the estimator is unbiased conditional on \(\delta\), its variance is correctly specified and fixed at the proposed sample size, and the normal likelihood and prior adequately represent the analysis. In an actual design, the data model and variance must reflect the endpoint, allocation, and planned analysis.

For a simple posterior decision rule, declare success when \(P(\delta>\delta_*\mid Y)>q\), where \(\delta_*\) is the clinically relevant threshold and \(q\) is the prespecified posterior-probability threshold. The rule, including the direction of the inequality, must be fixed before evaluating the design. A sample-size result is meaningful only with the model, the decision rule, and the distribution over which success is averaged all made explicit.

Deriving the Success Probability

Under the normal model and prior, the posterior for \(\delta\) is normal. Define \(V\) as its variance and \(m(Y)\) as its mean. Completing the square in the product of the normal likelihood and prior gives:

Let \(z_q\) be the \(q\) quantile of the standard normal distribution. The posterior probability rule is equivalent to a cutoff on the posterior mean:

This equivalence follows because the posterior probability is \(\Phi((m(Y)-\delta_*)/\sqrt{V})\), where \(\Phi\) is the standard normal cumulative distribution function. Since \(\Phi\) is increasing, exceeding \(q\) is exactly equivalent to the stated inequality. This derivation depends on a normal posterior with the stated variance; for a different model, the success rule must be evaluated using that model’s posterior rather than borrowing this cutoff.

At a fixed true effect \(\delta\), \(m(Y)\) is normally distributed with mean \(w\delta+(1-w)m_0\) and standard deviation \(w\sqrt{v}\). Thus, under these assumptions, conditional power is:

Assurance averages this probability over the design distribution:

If the design distribution is the same \(N(m_0,\tau^2)\) distribution used as the analysis prior, the marginal distribution of \(m(Y)\) is \(N(m_0,\tau^2-V)\). In that special case, assurance is \(1-\Phi((c-m_0)/\sqrt{\tau^2-V})\). This shortcut does not apply just because an analysis prior exists: it requires the design distribution to match that prior and the model assumptions above to hold.

Average Over the Intended Design Distribution

The analysis prior determines how trial data are translated into a posterior. The design distribution \(g(\delta)\) determines how conditional success probabilities are averaged when calculating assurance. They may be identical, but they serve different roles. A design distribution might instead represent a specified mixture of plausible effects, or a set of scenarios used for sensitivity analysis. The choice should reflect the design question and be stated plainly.

Apply the Posterior Threshold Exactly

A posterior success criterion is a probability statement about the parameter given the observed data. It is not interchangeable with “the estimated effect is positive,” a frequentist significance test, or a posterior interval chosen after seeing results. In the normal example, the exact translation is \(m(Y)>\delta_*+z_q\sqrt V\). Omitting \(\delta_*\), using a variance where a standard deviation is required, or using the wrong tail probability changes the decision rule.

Simulation Error and Design Decisions

When assurance is estimated by simulation, each simulated trial produces an indicator \(I_j\): it is 1 if the trial meets the success rule and 0 otherwise. With \(M\) independent simulated trials, the estimate is \(\hat A=M^{-1}\sum_{j=1}^M I_j\). Its approximate Monte Carlo standard error is \(\sqrt{\hat A(1-\hat A)/M}\). This describes numerical uncertainty from using a finite simulation, not uncertainty in the treatment effect, uncertainty about the model, or uncertainty about the design distribution.

Simulation diagnostics should include the number of replicates, the estimated assurance, its Monte Carlo error, and confirmation that the same success rule and data-generating assumptions were used throughout. If common random numbers or importance sampling are used, their variance calculations must match the simulation method; the simple independent-Bernoulli standard error above should not be applied without justification. A surprisingly stable estimate can still be biased if the simulated model, prior averaging, or decision rule is wrong.

Common Mistakes and Practical Checks

These checks do not replace examining whether the assumed likelihood, variance, analysis prior, and design distribution are reasonable. If the outcome distribution is skewed, the event rate is very low, dropout is substantial, or the planned analysis is more complex than the model used for sizing, a simple normal calculation may not represent the eventual decision process. Simulation can accommodate complexity only when the simulation itself faithfully implements the intended trial and analysis.