Why Simulate Bayesian Power and Assurance?
A Bayesian trial design is evaluated through a decision rule applied to data that have not yet been observed. The rule might declare success when the posterior probability that a treatment effect exceeds a clinically relevant threshold is sufficiently high. To estimate how often that rule will declare success, we must represent both the uncertainty about the parameter that generates the data and the random variation in trial outcomes.
Simulation makes those layers explicit. For conditional power, hold the parameter of interest fixed and repeatedly generate possible trials. For assurance, first draw that parameter from a design distribution, then generate trial data conditional on the draw. In both cases, analyze each simulated trial with the specified analysis model and count whether its posterior meets the success rule.
The data-generation model, design distribution, analysis prior, and success rule must all be specified before simulation. The trial-data model describes outcomes conditional on \(\theta\); the analysis model and analysis prior produce the posterior from each simulated data set. The design distribution represents uncertainty about the value of \(\theta\) across hypothetical trial settings. These components need not be identical, but any difference should be intentional and documented.
The Nested Simulation Procedure
For a candidate sample size, an outer loop represents draws of the design parameter, and an inner loop represents repeated trials conditional on each parameter draw. Conditional power at a fixed \(\theta\) needs only the inner loop. Assurance uses both loops. Each simulated trial must be analyzed using the actual planned posterior decision, not a substitute rule chosen for convenience.
Choose the sample size, data-generating model, design distribution for \(\theta\) if estimating assurance, analysis prior, and precise posterior success criterion.
For assurance, draw \(\theta_j\) from \(\pi_D(\theta)\) for outer replicate \(j\). For conditional power, set \(\theta_j\) to the fixed value of interest.
For each inner replicate \(k\), generate \(Y_{jk}\) from the sampling model conditional on \(\theta_j\), calculate the posterior under the analysis model, and apply the prespecified decision rule.
Record \(D_{jk}=1\) for a trial meeting the rule and 0 otherwise. Average indicators within fixed-parameter runs for conditional power, or across the nested runs for assurance.
With \(M\) outer draws and \(N\) inner trials per draw, the nested assurance estimate is
For fixed \(\theta\), the corresponding conditional power estimate from \(R\) independently generated trials is \(\widehat{P}(\theta)=R^{-1}\sum_{r=1}^{R}D_r\). The nested estimator follows directly from the definition of assurance: conditional on \(\theta_j\), the inner average estimates the success probability at that draw; averaging those conditional estimates over design draws estimates the design-distribution average.
Worked Example: A Posterior Decision and Conditional Power
Worked Example: A Posterior Decision and Conditional Power
Suppose a hypothetical trial observes \(m=80\) independent treatment-effect measurements \(X_i\sim N(\delta,200)\), with known variance \(200\) square units. The analysis prior is \(\delta\sim N(0,100)\). Success is declared when \(P(\delta>2\mid Y)\geq 0.95\), where 2 units is the clinically relevant threshold. Assume the data are generated at the fixed value \(\delta=5\). We first derive the rule, then apply it to a small illustrative simulation.
The sample mean has distribution \(\bar{X}\sim N(\delta,200/80)\). By normal-normal conjugacy, the posterior variance \(V\) and mean are
The posterior probability criterion is equivalent to requiring the posterior mean to exceed \(2+z_{0.95}\sqrt{V}\), where \(z_{0.95}=1.644854\). Thus the success rule in terms of the observed sample mean is
A hypothetical set of ten generated sample means is 5.1, 4.2, 5.0, 3.8, 4.9, 6.0, 4.7, 4.5, 5.4, and 4.0 units. Comparing each value with \(c\approx4.68305\), six meet the criterion: 5.1, 5.0, 4.9, 6.0, 4.7, and 5.4. The estimated conditional power in this small run is \(6/10=0.600\). This is an illustration of the calculation, not a precise design estimate; a real analysis would use many more simulated trials.
As a check on the decision rule and the expected success frequency, under the fixed truth \(\delta=5\), \(\bar{X}\sim N(5,2.5)\). Therefore the exact conditional power for this model and rule is
The small simulation estimate of 0.600 is plausible relative to this value, but its closeness is not guaranteed. This exact calculation is a useful validation when available; simulation is especially valuable when the posterior rule or sampling model makes a direct calculation impractical.
Worked Example: Nested Simulation for Assurance
Worked Example: Nested Simulation for Assurance
Continue with the same hypothetical normal trial, posterior decision rule, and \(m=80\). Now suppose the design distribution is \(\delta\sim N(4,1)\). This distribution is used to average success probabilities; it does not replace the analysis prior \(N(0,100)\). For a compact demonstration, take four outer draws, \(3,4,5,6\) units, and three hypothetical trial means at each draw.
| Outer draw \(\delta_j\) (units) | Inner trial means (units) | Successes | Conditional fraction |
|---|---|---|---|
| 3 | 3.9, 4.7, 5.0 | 2 | 2/3 |
| 4 | 4.2, 4.8, 4.6 | 1 | 1/3 |
| 5 | 4.9, 5.2, 4.4 | 2 | 2/3 |
| 6 | 5.0, 6.1, 4.5 | 2 | 2/3 |
Each inner value is compared with the same success cutoff, \(4.68305\) units. The estimated assurance for this toy run is the average of the four conditional fractions:
Equivalently, there are seven successful trials among the twelve generated trials, so the nested estimate is \(7/12\). The four parameter values in this table are for demonstrating the arithmetic; they are not a representative random sample from \(N(4,1)\), and the resulting estimate should not be interpreted as a useful estimate of assurance. In an actual run, parameter values would be drawn from the specified design distribution.
The reason for drawing parameters as well as outcomes is important: assurance averages conditional success probabilities across plausible parameter values. Simulating outcomes at only one selected value estimates conditional power there, not assurance.
Worked Example: Comparing Sample Sizes
Worked Example: Comparing Sample Sizes
Use the same model, analysis prior, threshold, and success rule, but compare \(m=40\) with \(m=80\). For \(m=40\), the sample-mean variance is \(200/40=5\), so
At the fixed truth \(\delta=5\), the conditional power for \(m=40\) is
For \(m=80\), the cutoff is approximately \(4.68305\) units and the conditional power at \(\delta=5\) is approximately \(0.5795\), as calculated above. In this hypothetical design, the larger sample size increases conditional power at this parameter value.
Assurance can also be checked analytically here. If \(\delta\sim N(4,1)\) and \(\bar{X}\mid\delta\sim N(\delta,v)\), then marginally \(\bar{X}\sim N(4,1+v)\). Since success is equivalent to \(\bar{X}\geq c\), assurance is \(\Phi((4-c)/\sqrt{1+v})\). Thus for \(m=40\), \(v=5\), and assurance is \(\Phi((4-5.86884)/\sqrt{6})\approx0.2229\). For \(m=80\), \(v=2.5\), and assurance is \(\Phi((4-4.68305)/\sqrt{3.5})\approx0.3576\).
A hypothetical nested simulation with \(M=1{,}000\) design draws and \(N=1{,}000\) trials per draw might yield 225,000 successes for \(m=40\) and 359,000 for \(m=80\). The estimates would be \(225{,}000/1{,}000{,}000=0.225\) and \(359{,}000/1{,}000{,}000=0.359\), respectively, consistent with the analytic checks to the displayed precision. Those counts are illustrative, not reported output from a reproducible random-number run.
Assumptions, Diagnostics, and Alternatives
A simulation estimate is meaningful only if the simulated process matches the intended design. Check that the parameter draws follow the stated design distribution, trial outcomes follow the chosen data-generating model conditional on each draw, and the analysis uses the planned likelihood, prior, and posterior decision rule. For example, omitting a nuisance parameter that varies under the design distribution can change the success probability, even if the posterior calculation itself is implemented correctly.
Useful diagnostics include checking that simulated sample sizes and allocations match the design, verifying that generated outcomes have the expected behavior at selected parameter values, and testing the decision code on trial data for which the posterior can be calculated independently. In the normal example, the cutoff calculation is one such check: a sample mean above \(4.68305\) units should meet the rule, and one below it should not. Also inspect whether success rates vary sensibly as the parameter value or sample size changes. Unexpected patterns may indicate a coding error, an incorrectly applied inequality, or an unintended change in the decision rule.
Simulation is not automatically a substitute for exact or numerical integration. When a tractable calculation is available, it can provide a benchmark for validating code. Simulation is more flexible for complicated models and decision rules, but estimates depend on the parameter distribution, data model, and decision implementation supplied to it. A simulation cannot repair a misspecified design distribution or an analysis that does not match the planned trial.
| Approach | What it provides | Main consideration |
|---|---|---|
| Exact calculation or integration | A direct success probability when the model and rule permit calculation. | Can be difficult or unavailable for complex trial models. |
| Nested simulation | Conditional power or assurance under a specified data-generation and analysis procedure. | Requires correct implementation and enough simulations for a stable estimate. |
| Simulation at one fixed parameter | Conditional power at that parameter value. | Does not average over parameter uncertainty and therefore is not assurance. |
A central limitation is that a finite simulation produces an estimate, not the exact probability. Results can vary with the random draws, particularly when few trials are simulated or success is uncommon. The next step in the course treats Monte Carlo error for assurance estimates in detail. Here, the practical point is to retain the simulation counts and settings, use enough replicates for the design decision, and avoid reporting more precision than the estimate supports.
Common Pitfalls
- Using one parameter draw and calling the result assurance. That estimates conditional power at the drawn value. Assurance requires averaging over draws from the design distribution.
- Using the design distribution as the analysis prior without justification. The design distribution generates parameter values across hypothetical settings; the analysis prior determines the posterior within each trial.
- Applying a different decision rule in code. A changed threshold, direction of inequality, or posterior quantity changes the estimand being estimated.
- Reporting a toy run as a stable estimate. A few simulated trials demonstrate the computation, but they do not generally support a precise design conclusion.
- Comparing sample sizes with changed assumptions. To attribute differences to sample size, keep the parameter distribution, data model, analysis, and success rule consistent across candidates.
Check Your Understanding
Answer each question, then compare your reasoning with the brief solutions.
- A simulation fixes \(\theta=3\) for every generated trial and estimates a 0.42 success fraction. What quantity is being estimated?
- In a nested simulation, there are \(M=200\) parameter draws and \(N=500\) trials per draw, with 37,000 successes in total. Calculate the assurance estimate.
- In the normal example with \(m=80\), would a sample mean of 4.60 units meet the posterior success rule? Explain using the cutoff.
- Why should the analysis prior and design distribution be specified separately?
- Give one reason to compare a simulation result with an exact calculation when one is available.
Solutions. (1) It estimates conditional power at the fixed value \(\theta=3\), not assurance. (2) There are \(200\times500=100{,}000\) simulated trials, so \(\widehat{A}=37{,}000/100{,}000=0.370\). (3) No. The cutoff is approximately \(4.68305\) units, and \(4.60<4.68305\), so the posterior probability criterion is not met. (4) The design distribution represents parameter values used to generate hypothetical trial settings, while the analysis prior is part of the posterior calculation for each trial. They serve different purposes and need not be the same. (5) Exact results provide a benchmark that can reveal mistakes in simulation, posterior calculations, or decision-rule code.