Why Simulate a Trial With a Strong Placebo Response?
When many patients improve in the control group, the treatment must be compared with that substantial background response—not with no improvement. Moreover, the control-group response may differ from trial to trial. A simulation can represent both features, generate hypothetical trials under stated assumptions, and estimate how often a planned analysis detects a clinically relevant treatment effect.
The result is conditional on the model and analysis encoded in the simulation. It is not a guarantee about a future trial. The model must represent the treatment contrast, placebo-response variability, endpoint, randomization and test well enough for the decision being considered.
A Trial Model With Placebo-Response Variability
Consider a continuous endpoint measured in points, where larger values mean greater improvement from baseline. In the hypothetical setting below, the mean placebo improvement is 8 points, a strong response on this scale. The placebo mean also varies across trials. For trial \(j\), let \(B_j\) be a shared trial-level shift, and let \(\epsilon_{ij}\) be the individual residual for participant \(i\). The model is
with the trial-level shifts independent across trials and individual residuals independent across participants and independent of the shifts. In a trial with equal randomization, the outcome for a participant assigned to placebo is \(Y_{ij}=8+B_j+\epsilon_{ij}\); for a participant assigned to treatment it is \(Y_{ij}=8+B_j+\delta+\epsilon_{ij}\). The treatment effect \(\delta\) is an increase in mean improvement, in points. For planning, suppose the clinically relevant effect is \(\delta=4\) points.
The model makes important assumptions: participants are randomized concurrently to treatment and placebo; their residual outcomes are independent and normally distributed with a common variance; the treatment effect is a constant shift of 4 points; and there is no dropout or other missing outcome in this example. The trial shift affects both arms equally. If treatment effects vary across patients or trials, or if the treatment changes outcome variability, that feature must be represented explicitly rather than assumed away.
Worked Example: Translate the Model Into a Trial Simulation
Suppose a proposed trial has 125 participants per arm. For one simulated trial, first draw one \(B_j\) from \(N(0,2.5^2)\). If the draw is \(B_j=3\) points, the placebo mean is \(8+3=11\) points and the treatment mean under the planning effect is \(8+3+4=15\) points. The 4-point treatment-placebo difference is unchanged by the shared shift.
Next, generate 125 placebo outcomes as \(11+\epsilon_{ij}\) and 125 treatment outcomes as \(15+\epsilon_{ij}\), with each residual drawn from \(N(0,9.5^2)\). Apply the planned two-sided pooled two-sample \(t\)-test at \(\alpha=0.05\). Record 1 if the test rejects \(H_0:\delta=0\), and 0 otherwise. One simulated trial gives one such indicator; it cannot by itself estimate power reliably.
Repeat the process for many independent trials, drawing a new \(B_j\) for each. If 9,132 of 10,000 simulated trials reject, the estimated power is \(9{,}132/10{,}000=0.9132\). This estimate describes the probability of rejection under the specified model and 4-point effect, subject to simulation error and model uncertainty.
Why the Shared Trial Shift Cancels
A key justification for this model is that the same trial-level placebo shift enters both randomized group means. Write \(\bar{Y}_{Tj}\) and \(\bar{Y}_{Pj}\) for the treatment and placebo means in trial \(j\), each with \(n\) participants. Their difference is
Thus \(B_j\) cancels from the contrast. Under the stated independence and common-variance assumptions, the variance of the difference is
The cancellation does not mean that placebo variability is irrelevant. Individual variation in placebo outcomes remains part of the residual variance, and differences between trials affect forecasts of absolute outcomes. Cancellation applies specifically to the shared additive shift in a concurrent randomized comparison. It would not automatically apply if the comparator were an independent historical control, the shift differed between arms, or the treatment effect varied with the trial context.
This calculation also explains a common modeling error. Each arm mean varies across hypothetical trials partly because of \(B_j\), but the two arm means are correlated: both contain the same \(B_j\). Adding their variances while treating them as independent would incorrectly count the shared shift twice. The variance of a difference includes the covariance term, \(\operatorname{Var}(A-C)=\operatorname{Var}(A)+\operatorname{Var}(C)-2\operatorname{Cov}(A,C)\), which removes that shared component here.
Worked Example: Approximate Power at 125 Participants per Arm
For a quick check, use the normal approximation with the residual standard deviation treated as known. With \(n=125\) per arm and a 4-point effect, the standard error of the difference under the model is
The effect divided by this standard error is \(4/1.2017=3.329\). For a two-sided test at \(\alpha=0.05\), the standard normal critical value is approximately 1.96. The approximate power is therefore
This is a useful check, not the simulation result: a pooled \(t\)-test estimates the variance from each simulated trial, and finite-sample behavior need not exactly match the known-variance normal approximation. A simulated estimate near 0.91 is plausible for these assumptions.
Compare Candidate Sample Sizes by Simulation
For each candidate \(n\), simulate complete trials from the model, analyze each using the same prespecified pooled \(t\)-test, and estimate power as the fraction of trials with \(p<0.05\). The following hypothetical results illustrate a run with 10,000 replicates per sample size. The estimates are illustrative simulation output, not evidence from an actual drug study.
| Participants per arm | Rejections out of 10,000 | Estimated power | Monte Carlo SE |
|---|---|---|---|
| 80 | 7,580 | 0.7580 | 0.00428 |
| 100 | 8,440 | 0.8440 | 0.00363 |
| 125 | 9,132 | 0.9132 | 0.00282 |
| 150 | 9,530 | 0.9530 | 0.00212 |
For \(R\) simulated trials and estimated power \(\hat{\pi}\), the Monte Carlo standard error (Monte Carlo SE) is the estimated standard deviation of the simulated rejection proportion:
For the 125-per-arm result, this is \(\sqrt{0.9132(1-0.9132)/10{,}000}=\sqrt{0.07926576/10{,}000}=0.002815\), or 0.00282 to three significant figures. This measures simulation noise, not uncertainty about whether the assumed model or effect is correct.
Worked Example: Interpret the Estimated Power and Its Precision
For 125 participants per arm, the estimated power is 0.9132. Under the model with a true 4-point effect, this means about 91.32% of repeated trials analyzed by the specified test would reject the null hypothesis at the 5% two-sided significance level.
Using the Monte Carlo SE of 0.002815, an approximate 95% interval for the underlying rejection probability, accounting only for finite simulation error, is \(0.9132\pm1.96(0.002815)\). The margin is \(0.005517\), giving an interval of approximately \([0.9077,0.9187]\). This is an interval for the simulated rejection probability under the fixed model, not an interval for the treatment effect and not a measure of uncertainty in the model assumptions.
Comparing the table, 100 participants per arm gives estimated power below 0.90, while 125 exceeds it. The 125-per-arm design is therefore the smallest of these candidates to reach an illustrative 90% target. That decision should still account for whether the effect and variability assumptions are credible and whether a different design objective is required.
Diagnostics, Alternatives, and Limitations
A simulation is only as useful as its checks. Before interpreting results, verify that the generated trial data have the intended features: the placebo mean across trials should vary with an SD near 2.5 points; within-trial residual SD should be near 9.5 points; the treatment-placebo mean difference should average 4 points; and both arms should share the same \(B_j\) in each replicate. Also check that the programmed analysis matches the intended test, including its two-sided alternative and significance threshold.
Useful diagnostic comparisons include the simulated power curve against a normal-approximation calculation, the distribution of test statistics under the null, and repeated runs with more simulation replicates. If the null simulation rejects much more or less often than 5%, investigate the implementation, distributional assumptions and test calibration. A simulation that produces implausible endpoint means, variances or treatment contrasts may indicate a coding error or a mismatch between model and design.
The analytical approximation is faster and helps expose how the effect, residual variance and sample size drive power. Simulation is more flexible: it can propagate trial-level variability, apply the exact planned analysis, and accommodate a more complex outcome-generating model. The approximation is a check, not a substitute when important features make its assumptions inappropriate.
The main limitation is that estimated power is conditional on the assumed 4-point effect and variability structure. The model treats the treatment effect as fixed, assumes normal residuals and equal variances, and omits missing outcomes. The shared-shift model also assumes that trial-level placebo response affects both arms equally. If these assumptions are doubtful, modify the simulation and examine how the design conclusion changes. More simulation replicates reduce Monte Carlo error, but they do not repair a misspecified model.
Check Your Understanding
Answer the questions, then compare your reasoning with the solutions.
- In the model, what happens to the shared trial shift when the treatment mean is compared with the placebo mean?
- With 10,000 simulations and estimated power 0.80, calculate the Monte Carlo SE.
- Why does a Monte Carlo SE of 0.00282 not measure uncertainty in the assumed 4-point treatment effect?
- If both arms have a shared trial shift but a simulation draws a separate shift for each arm, what feature of the randomized comparison has been changed?
Solutions. (1) The same \(B_j\) enters both arm means and cancels in their difference. (2) \(SE_{\mathrm{MC}}=\sqrt{0.80(0.20)/10{,}000}=\sqrt{0.000016}=0.00400\). (3) The Monte Carlo SE describes variation in the estimated rejection proportion across finite simulation runs under fixed inputs; it does not account for uncertainty or misspecification in those inputs. (4) The groups no longer share the same trial-level conditions. The simulated contrast then includes variation between independent shifts, rather than the concurrent randomized comparison specified by the model.