Reading a Trade-Off, Not Just a Score
A sample-size re-estimation rule can improve the chance of detecting a target effect, but may also increase expected enrollment, permit a larger maximum enrollment, or alter the probability of a false positive. There is rarely one rule that is best on every measure. The practical task is to understand what each simulated operating characteristic says, decide which constraints are essential, and make the remaining trade-offs explicit.
An operating characteristic describes the long-run behavior of a specified trial design under a specified data-generating scenario. A simulation estimate is not a universal property of a rule: it depends on the assumed treatment effect, variability, recruitment and analysis procedures, and other features of the scenario. Comparisons are most informative when candidate rules are evaluated under the same scenarios and with the same final analysis.
Expected sample size and maximum enrollment answer different questions. Expected enrollment describes average resource use over repeated trials; maximum enrollment describes the upper bound on enrollment in any trial allowed by the rule. A trial that usually stops at a moderate size but occasionally reaches a high cap can have a modest mean and still require substantial recruitment capacity. When enrollment is costly or difficult, both quantities may matter.
What a Simulation Estimate Means
Suppose a simulation generates \(R\) independent trial replications under one fixed rule and one fixed scenario. For each replication, record whether the primary test rejects, and record the final sample size. If \(K\) replications reject, the estimated rejection probability is \(K/R\). Under a null scenario this estimates type I error; under an alternative scenario it estimates power. The average of the \(R\) recorded sample sizes estimates expected sample size.
For the rejection indicator, each replication is a Bernoulli outcome with probability \(\pi\), where \(\pi\) is the true rejection probability under the simulation scenario. Its variance is \(\pi(1-\pi)\). Because \(\pi\) is unknown, the estimated Monte Carlo standard error of \(\hat{\pi}=K/R\) uses \(\hat{\pi}\) in its place:
This standard error measures numerical uncertainty from using a finite number of simulated trials. It is not the standard error of a clinical treatment-effect estimate, and it does not include uncertainty about whether the scenario assumptions are realistic. A rough 95% Monte Carlo interval is \(\hat{\pi}\pm1.96SE_{\mathrm{MC}}\); when the estimate is near zero or one, a more suitable binomial interval may be preferable.
Worked Example: How Precise Is an Estimated Type I Error Rate?
In a hypothetical simulation of \(R=20{,}000\) trials under a null scenario, 982 trials reject \(H_0\). The estimated type I error rate is
The estimated Monte Carlo standard error is
A rough 95% Monte Carlo interval is \(0.0491\pm1.96(0.001528)\), or approximately \(0.0461\) to \(0.0521\). Thus, 4.91% is the estimated rejection rate for this simulated design and scenario; finite simulation variability means the estimate is not exact. The interval describes Monte Carlo uncertainty, not a probability that the design's type I error is within those limits. Since the interval extends above 0.05, these simulation results alone do not establish that the type I error is no greater than 0.05.
Use Constraints Before Preferences
A transparent comparison separates requirements from preferences. A requirement might be that estimated power reaches a target, the maximum enrollment stays within available capacity, and type I error is controlled at the prespecified level. Preferences can then distinguish among the rules that meet those requirements: for example, favoring lower expected enrollment or seeking extra power above the minimum. If a rule fails a genuine constraint, a favorable value on another measure does not automatically make it acceptable.
The following hypothetical results come from 20,000 simulated trials per scenario and rule. Type I error is estimated under a null scenario; power and sample-size summaries are estimated under a prespecified target-effect scenario. All rules use the same primary test and nominal significance level.
| Rule | Estimated type I error | Estimated power | Expected enrollment | Maximum enrollment |
|---|---|---|---|---|
| A | 0.0491 | 0.8014 | 420 | 500 |
| B | 0.0518 | 0.8421 | 456 | 600 |
| C | 0.0476 | 0.8080 | 430 | 550 |
Worked Example: Apply Stated Design Constraints
Suppose the trial team sets these screening requirements: estimated type I error no greater than 0.05, estimated power at least 0.80, expected enrollment no greater than 440, and maximum enrollment no greater than 550. The team prefers the lower expected enrollment among rules that meet every requirement. Which rule is the leading candidate based on the displayed point estimates?
Rule A has type I error 0.0491, power 0.8014, expected enrollment 420, and maximum enrollment 500. It meets all four point-estimate requirements. Rule B fails the type I error, expected enrollment, and maximum enrollment requirements. Rule C meets the type I error, power, and maximum-enrollment requirements, but its expected enrollment of 430 is within the limit; therefore it also meets all four. Between A and C, A has lower expected enrollment (420 rather than 430) and a lower maximum (500 rather than 550), while C has slightly higher estimated power (0.8080 rather than 0.8014). Under the stated preference for lower expected enrollment, A is the leading candidate, provided its power margin is considered adequate.
This is a screening decision, not proof that either rule's true type I error is below 0.05. For Rule A, the Monte Carlo interval calculated in the preceding example extends above 0.05. For Rule C, \(SE_{\mathrm{MC}}=\sqrt{0.0476(0.9524)/20{,}000}\approx0.001506\), giving a rough 95% interval of \(0.0446\) to \(0.0506\), also slightly above 0.05. If error control is a firm requirement, the team should investigate this uncertainty, simulate more replicates, and/or calibrate the rule to provide a suitable margin rather than treating the point estimate as conclusive.
Expected Enrollment and the Upper Bound
The expected sample size is an average over the distribution of final sample sizes generated by the rule. If \(N_r\) is the final enrollment in simulated trial \(r\), its simulation estimate is \(R^{-1}\sum_{r=1}^{R}N_r\). The maximum permitted enrollment is instead a design limit. It does not say how often the rule reaches that limit. The distribution of final sample sizes, including the frequency at the cap, helps explain an average that might otherwise conceal occasional high-enrollment trials.
Worked Example: A Low Average Does Not Remove Capacity Needs
Consider a hypothetical rule simulated in 20,000 trials. In 10,000 trials the final enrollment is 400, in 6,000 it is 500, and in 4,000 it is 600. Its estimated expected enrollment is
The maximum enrollment is 600, and one fifth of the simulated trials reach it: \(4{,}000/20{,}000=0.20\). Reporting only the average of 470 would omit that 20% of trials in the simulation required the full 600 participants. If the trial team cannot recruit more than 500, this rule is unsuitable even if its expected enrollment is affordable. If capacity for 600 exists but resources are limited, the average and the frequency at the cap both inform planning.
Choose According to the Trial's Priorities
There is no universally correct way to rank rules that meet all essential constraints. A team prioritizing a larger chance of detecting the target effect may prefer a rule with higher power, accepting greater average enrollment if feasible. A team facing a strict recruitment ceiling may prefer a lower maximum even when another rule has somewhat higher power. These are design priorities, not conclusions that can be extracted from a single summary statistic.
Worked Example: A Hard Enrollment Ceiling Changes the Choice
In a second hypothetical comparison, Rule L has estimated power 0.840, estimated type I error 0.0490, and simulated final enrollments of 400 in 10,000 trials, 500 in 6,000 trials, and 600 in 4,000 trials. Rule M has estimated power 0.805, estimated type I error 0.0470, and final enrollments of 400 in 16,000 trials and 500 in 4,000 trials. Assume the team requires estimated power of at least 0.80, estimated type I error no greater than 0.05, expected enrollment no greater than 430, and a hard maximum of 500.
Rule L's expected enrollment is \([10{,}000(400)+6{,}000(500)+4{,}000(600)]/20{,}000=470\), and its maximum is 600. It meets the power and point-estimate type I error requirements, but fails both enrollment requirements. Rule M's expected enrollment is
Its maximum is 500, and its estimated power and type I error meet the stated point-estimate requirements. Rule M is therefore the only one of these two that meets every stated requirement. Its lower power than Rule L is not a reason to select L when L violates a hard enrollment ceiling. As before, the type I error estimates should be evaluated for Monte Carlo precision before making a final error-control claim.
Diagnostics, Limitations, and Common Pitfalls
Before selecting a rule, check that the simulated comparisons answer the intended question. At a minimum, inspect whether candidate rules were evaluated under the same null and alternative scenarios, whether the same final test was used, how many replications were run, and whether simulated error rates are precise enough for the decision. Examine more than the primary target scenario: operating characteristics can change when assumptions such as variability or the true effect differ from the central planning values. A rule that looks attractive in one scenario may be less attractive in another.
A useful report includes the estimated characteristics, simulation replication count, relevant Monte Carlo uncertainty, and the assumptions defining each scenario. For sample size, report the expected value and maximum together; when the upper tail matters, also show the distribution or proportion reaching the cap. Averages and point estimates can be useful summaries, but they do not describe every risk relevant to feasibility or error control.
- Pitfall: treating the largest power estimate as the automatic winner. Higher simulated power may come with an unacceptable maximum enrollment or expected cost. First apply the trial's essential constraints, then compare preferences among feasible rules.
- Pitfall: declaring type I error controlled because the estimate is below 0.05. A point estimate such as 0.0491 has Monte Carlo uncertainty. Check its precision and the full simulation assumptions; a finite simulation is not a guarantee about performance under every plausible data-generating process.
- Pitfall: treating expected enrollment as a cap. An average of 470 does not mean every trial will enroll about 470 participants. Report the maximum and, where useful, how often simulations reach it.
- Pitfall: ranking unlike simulations. Comparing results generated under different assumptions, analysis tests, or replication counts can confound the rule comparison with differences in the simulation setup.
The central limitation is that simulation evaluates the specified model, not every possible future trial. More replications reduce Monte Carlo uncertainty, but do not repair unrealistic assumptions. A sound decision therefore combines explicit priorities, careful simulation checks, uncertainty in the reported estimates, and sensitivity to plausible scenarios.
Check Your Understanding
Answer each question, then check the concise solutions below.
- In 10,000 simulated trials under a specified alternative, 7,900 reject \(H_0\). What is the estimated power, and what does it mean?
- Why can a rule with expected enrollment 420 still be infeasible for a trial with a recruitment capacity of 500?
- A rule has estimated type I error 0.048 with a Monte Carlo interval extending above 0.05. What can and cannot be concluded from this result?
- Two rules meet all hard constraints. One has higher power and expected enrollment; the other has lower expected enrollment and maximum enrollment. Is there a universally correct choice?
Solutions. (1) Estimated power is \(7{,}900/10{,}000=0.79\), or 79%. Under the stated alternative and simulation assumptions, about 79% of repeated trials are estimated to reject \(H_0\). (2) Expected enrollment is an average, not a maximum; the rule may permit or frequently require enrollment above 500. The maximum and distribution of final sample sizes must be checked. (3) The point estimate is below 0.05, but the simulation does not establish that the rate is no greater than 0.05 with the desired precision. More simulations or calibration may be needed, and the conclusion remains conditional on the scenario assumptions. (4) No. The choice depends on the trial's priorities and constraints: one team may value extra power, while another may prioritize lower cost or a smaller enrollment cap.