Power Is a Function of the Treatment Effect
A Bayesian trial’s success rule may be fixed while its chance of declaring efficacy changes with the true treatment effect. That probability is the rule’s power at that effect. Estimating it across a grid shows how the trial performs over a range of scientifically relevant alternatives, rather than at just one assumed effect.
The success rule and the trial data-generating mechanism must both be specified before simulation. For each grid value, generate data as if that value were the true treatment effect, apply the same trial analysis and fixed rule, and record whether the rule declares success. The resulting estimate is a simulation estimate of power, not a posterior probability that the treatment effect has that value.
The rule is fixed across the grid: it is not re-tuned separately at each treatment effect. The grid points should be chosen in units meaningful for the clinical outcome, including a range of plausible benefits and, where useful, a value near the null. The earlier material on specifying a trial data-generating mechanism and choosing replicate-level outputs supplies the general simulation framework; here the output of interest is the success indicator at each non-null effect.
What the Simulation Estimates
At a grid value \(\theta_j\), each replicate represents a new trial under the same design and the same assumed truth \(\theta_j\). Randomization, outcomes, missingness, interim decisions, and analysis are simulated according to the stipulated mechanism. The rule is applied to the resulting data exactly as it would be in the planned trial. If the rule declares success, record 1; otherwise record 0.
Under independent, identically generated replicates at a fixed \(\theta_j\), these indicators are Bernoulli random variables with success probability \(\pi(\theta_j)\). Their sample mean estimates that probability, and the formula above estimates its Monte Carlo standard error. This standard error measures variation caused by using a finite number of simulation replicates. It does not include uncertainty about whether the assumed data-generating mechanism, treatment effect, or prior is appropriate.
The estimate should be reported with its number of replicates and an uncertainty measure. A Monte Carlo confidence interval describes uncertainty in estimating the simulation probability, under the replicate-generation assumptions. For example, a Wilson interval is often preferable to a simple normal interval near 0 or 1. Given estimate \(\hat{\pi}\), replicate count \(M\), and \(z=1.96\), its center and half-width are
The interval is \([c-h,c+h]\). It is an approximate 95% interval for the probability estimated by the simulation, not a credible interval for \(\theta\) and not an interval that accounts for uncertainty in the trial model.
Worked Example: Deriving an Analytic Benchmark
Suppose, hypothetically, that a continuous endpoint is summarized by an estimated treatment benefit \(Y\), measured in mmHg, with \(Y\mid\theta\sim N(\theta,16)\). Thus the sampling standard deviation is 4 mmHg. The analysis uses the prior \(\theta\sim N(0,100)\), with variance in squared mmHg, and the fixed success rule is \(P(\theta>0\mid Y)\geq 0.975\). Derive the data threshold for success and the power at a true benefit of 8 mmHg.
For this normal likelihood and normal prior, the posterior mean is \((100/116)Y\), and the posterior variance is \(16(100/116)=13.793\) squared mmHg. The posterior standard deviation is therefore 3.714 mmHg. The success rule is equivalent to requiring the posterior mean to be at least \(1.96\) posterior standard deviations above zero:
At true benefit \(\theta=8\) mmHg, the probability of success is therefore
This exact probability is an analytic benchmark for simulations under this simplified mechanism. It says that the fixed rule declares success in about 45.6% of hypothetical trials when the true benefit is 8 mmHg. It is not a guarantee about any particular trial.
Estimating Power at Every Grid Point
A practical grid is a finite set of treatment effects selected before examining the simulation results. At each point, run \(M\) replicates and retain both the count of successful replicates and the total number run. The estimate is the count divided by \(M\); the Monte Carlo standard error follows from the Bernoulli formula. Using the same replicate count at each point makes the simulation precision easier to compare, though it does not make the precision identical when the powers differ.
Worked Example: A Simulated Power Grid
Consider the hypothetical normal trial and fixed success rule in the previous example. Suppose 10,000 independent trials are simulated at each of four true benefits. Imagine the simulation produces the following success counts. These are illustrative hypothetical outputs, not data from a clinical study.
| True benefit (mmHg) | Successful trials | Estimated power | Monte Carlo SE |
|---|---|---|---|
| 0 | 175 | 0.0175 | 0.00131 |
| 4 | 1,325 | 0.1325 | 0.00339 |
| 8 | 4,530 | 0.4530 | 0.00498 |
| 12 | 8,120 | 0.8120 | 0.00391 |
For the 8 mmHg point, the estimate is \(4{,}530/10{,}000=0.4530\). Its Monte Carlo standard error is
Thus, the simulation estimate is 45.3% power, with a Monte Carlo standard error of about 0.50 percentage points. The analytic benchmark in the previous example is 45.6%, a difference of 0.3 percentage points, which is small relative to this simulation’s Monte Carlo uncertainty. Agreement is reassuring for this simple case, though it does not by itself validate every component of a more complex trial simulation.
Showing Monte Carlo Uncertainty
A plotted power curve should not imply that each estimated point is exact. Give the replicate count and either the Monte Carlo standard error or an interval at each grid value. If powers are near 0 or 1, an interval designed for proportions, such as the Wilson interval, avoids some shortcomings of the ordinary estimate plus or minus 1.96 standard errors.
Worked Example: An Interval and a Difference Across the Grid
Use the simulated estimate \(\hat{\pi}=0.4530\) from 10,000 hypothetical replicates at 8 mmHg. With \(z=1.96\), the Wilson center is
to four significant figures, and the half-width is
The approximate 95% Wilson interval is therefore \([0.4432,0.4628]\), or 44.32% to 46.28% power. This interval describes finite-replicate uncertainty in the estimated success probability at 8 mmHg.
The simulated difference between estimates at 12 and 8 mmHg is \(0.8120-0.4530=0.3590\), or 35.90 percentage points. If the simulations at those two effects use independent random streams, the Monte Carlo standard error of this difference is approximately
The increase is much larger than its Monte Carlo uncertainty. If replicates at different effects are deliberately paired using common random numbers, the difference’s standard error must instead account for the covariance between paired success indicators; the independent-stream formula would not apply.
Justification and Diagnostics
The estimate and standard error follow directly from the replicate-level indicators. At a fixed true effect, suppose \(R_1,\ldots,R_M\) are independent Bernoulli outcomes with probability \(\pi(\theta)\). Then \(E[R_b]=\pi(\theta)\) and \(\operatorname{Var}(R_b)=\pi(\theta)[1-\pi(\theta)]\). Independence gives
Replacing the unknown \(\pi(\theta)\) in the standard deviation by its estimate gives the Monte Carlo standard error. This justification depends on the replicates being generated independently from the intended mechanism at that grid point and on each success indicator being correctly calculated. If replicate dependence is introduced by the simulation design, the variance calculation must reflect that dependence.
A power curve will often rise as the true effect becomes more favorable, but monotonicity should not be assumed in every model or decision rule. A small reversal between adjacent estimates may simply be Monte Carlo noise. A large or persistent irregularity can indicate an implementation error, a change in scenario settings across grid points, or a genuinely non-monotone operating characteristic. Check that the same rule, sample size, prior, analysis, and other design choices were used throughout, and verify the outcome coding and replicate counts.
Other useful checks include comparing a simple scenario against an analytic benchmark when available, inspecting whether the reported Monte Carlo standard errors match the replicate counts, and increasing \(M\) at points where the uncertainty is too large for the intended comparison. The earlier material on analytic benchmarks and simulation verification is relevant here. More replicates can reduce Monte Carlo error, but they cannot repair a wrongly specified data-generating mechanism or a rule implemented incorrectly.
Choosing and Interpreting the Grid
Grid values should span effects that matter clinically, rather than being chosen solely to make a smooth-looking curve. Include enough points to reveal important changes, and consider additional points where the curve changes quickly. A grid is still only a collection of evaluated scenarios: interpolation between points is an approximation unless the power function is known to have a form that justifies it.
At each point, state the effect’s units, estimated power, replicate count, and Monte Carlo uncertainty. Keep the distinction clear: power is a probability under the specified true effect and trial mechanism; Monte Carlo uncertainty describes how accurately the simulation estimates that probability. Neither quantity establishes the effect expected in practice. That requires substantive assumptions or evidence beyond this operating-characteristic calculation.
An analytic calculation can be more efficient and exact when the model and rule yield a tractable probability. Simulation is more flexible for complex designs, but brings finite-replicate uncertainty and requires careful implementation. A useful practice is to use analytic results as checks where possible and simulation for the scenarios that lack simple closed forms. The aim is not to replace one approach with the other, but to make the assumptions and uncertainty of the chosen calculation visible.
Common Pitfalls
- Treating estimated power as a posterior probability. A simulated power of 0.45 at a specified true effect is the estimated chance that the rule declares success under that scenario, not a 45% posterior probability that the effect is true.
- Reporting no Monte Carlo uncertainty. A point estimate alone hides how much it could vary with another finite simulation. Report the replicate count and Monte Carlo standard error or interval.
- Changing the rule across the grid. If the success criterion is adjusted at different effects, the estimates no longer describe one fixed rule’s power function.
- Overinterpreting small wiggles. Adjacent estimates may differ because of simulation noise. Compare their uncertainty and investigate substantial irregularities before drawing a substantive conclusion.
- Confusing simulation precision with model certainty. A very small Monte Carlo standard error only indicates a precise calculation under the simulated assumptions; it does not validate those assumptions.
Check Your Understanding
Answer each question, then check the brief solutions.
- At a true effect of 5 units, 2,000 independent hypothetical replicates yield 1,100 successes. What is the estimated power and its Monte Carlo standard error?
- Does a Monte Carlo interval for power describe uncertainty about the true treatment effect? Explain.
- At one grid point, the estimated power is 0.20 from 1,000 replicates. What happens to the Monte Carlo standard error if the replicate count is increased to 4,000, approximately, with the underlying power unchanged?
- Why should the success rule remain fixed across the treatment-effect grid?
- Two independently simulated grid points have estimated powers 0.40 and 0.45, each based on 10,000 replicates. What is the approximate Monte Carlo standard error of their difference?
Solutions. (1) The estimate is \(1{,}100/2{,}000=0.550\); its standard error is \(\sqrt{0.55(0.45)/2{,}000}=0.0111\). (2) No. It quantifies finite-simulation uncertainty in the success probability under the specified mechanism, not uncertainty about the treatment effect. (3) Quadrupling \(M\) approximately halves the standard error because it is proportional to \(1/\sqrt{M}\); the new value is \(\sqrt{0.20(0.80)/4{,}000}=0.00632\), half the original 0.0126. (4) Otherwise, the estimates correspond to different decision rules and do not trace the power function of one fixed rule. (5) For independent simulations, the standard error is \(\sqrt{0.40(0.60)/10{,}000+0.45(0.55)/10{,}000}=0.00698\).