Why Scenario-Specific Reporting Matters
A simulated operating characteristic is not a universal property of a group sequential design. It is a result for a specified design under a specified data-generating scenario. For example, simulated power depends on the effect size and nuisance parameters used to generate trial data; stopping probabilities depend on those assumptions and on the monitoring rules. A single value without its scenario can therefore be difficult to interpret or reproduce.
A useful report lets a reader connect each result to the trial design, the simulated scenario, the number of replications, and the precision of the simulation estimate. The goal is not to restate how to run a simulation, but to make clear what was simulated and what the resulting estimates can support.
Before presenting results, identify the design in terms readers need to interpret the numbers: the hypotheses and test direction, nominal significance level, planned looks or information times, stopping boundaries or rules, maximum information or sample size, and analysis method. State the assumptions that define each scenario, such as the treatment effect, event rate, variability, accrual or follow-up assumptions, and any operational deviations being considered. Include only assumptions relevant to the design and explain how scenarios differ.
The simulation also has its own assumptions. In particular, the \(R\) simulated trials should be generated independently under the stated scenario and analyzed using the same prespecified design procedure. Each trial must contribute to the reported outcome in a defined way. If some runs fail, report the number and reason; do not silently discard them or count only successful runs without explaining the change in denominator.
Reporting Error Rates and Power
For a fixed scenario, let \(I_r=1\) if simulated trial \(r\) rejects \(H_0\), and \(I_r=0\) otherwise, for \(r=1,\ldots,R\). The estimated rejection probability is the proportion of simulated trials that reject. Under a null scenario this estimates type I error; under an alternative scenario it estimates power. The same calculation has a different interpretation because the data-generating scenario differs.
The Monte Carlo standard error estimates the simulation-to-simulation variability of the estimated proportion, assuming independent simulated trials. A rough 95% Monte Carlo interval is \(\hat{\pi}\pm1.96\widehat{SE}_{MC}(\hat{\pi})\) when the normal approximation is adequate. It describes uncertainty due to using a finite number of simulated trials, not uncertainty about the true treatment effect in a real study and not uncertainty about whether the scenario is realistic. When the estimate is near 0 or 1, the simple normal interval can be unreliable; report an appropriate binomial interval or a bound that reflects the number of replications.
Worked Example: Reporting Type I Error and Power
Suppose a hypothetical two-sided group sequential design has nominal \(\alpha=0.025\), four planned analyses, and a specified maximum information. Under the null scenario, \(R=10{,}000\) independent trials are simulated, and 246 reject \(H_0\). Under a separately specified alternative scenario, 7,860 of 10,000 trials reject. Assume the simulation algorithm completed all trials and applied the same planned design in each replicate.
Under the null scenario, the estimated type I error is
Its Monte Carlo standard error is
The rough 95% Monte Carlo interval is \(0.0246\pm1.96(0.00155)\), or approximately 0.0216 to 0.0276. A suitable report would say that the simulated type I error was 0.0246 (246/10,000; Monte Carlo SE 0.00155), under the stated null scenario. This estimate is close to the nominal 0.025; the interval conveys the finite-simulation precision and is not a test that proves exact control.
Under the alternative scenario, the estimated power is
The rough 95% Monte Carlo interval is \(0.786\pm1.96(0.00410)\), approximately 0.778 to 0.794. Report the alternative scenario’s effect size and nuisance assumptions alongside this estimate. Without them, “power was 78.6%” does not identify the condition under which that power applies.
Reporting Stopping and Sample-Size Summaries
Stopping results should be reported as probabilities for clearly defined, mutually exclusive outcomes. Depending on the design, these might be stopping for efficacy at look 1, stopping at look 2, stopping for futility, or reaching the maximum analysis without earlier stopping. Define each category so that every simulated trial belongs to exactly one. Their estimated probabilities should sum to 1, apart from rounding. If outcomes do not cover all trials, explain the unclassified or failed runs rather than hiding the discrepancy.
Also state how sample size is defined: for example, total participants randomized, number of events, or information accrued. Give the maximum and the units. If stopping can occur at different looks, a mean alone may conceal important variation; report the distribution by look or useful quantiles as well. The mean is especially informative when clearly labeled as an expected sample size under the specified scenario, not as a guaranteed requirement.
For simulated total sample sizes \(N_1,\ldots,N_R\), the estimated mean is \(\bar{N}=R^{-1}\sum_{r=1}^R N_r\). If \(s_N\) is their sample standard deviation, the Monte Carlo standard error of that mean is \(s_N/\sqrt{R}\). This quantifies the precision of the simulated mean, not the variation in sample size across future trials; the latter is represented by \(s_N\), quantiles, or the stopping distribution.
Worked Example: Summarizing Stopping and Expected Sample Size
In a hypothetical design with three possible analysis looks, 5,000 independent trials are simulated under one specified alternative scenario. The design stops at total sample sizes of 240, 480, or 720 participants. The respective counts are 1,250, 1,750, and 2,000. Assume these categories include every completed simulated trial and that 720 is the maximum.
The estimated stopping probabilities are \(1{,}250/5{,}000=0.25\), \(1{,}750/5{,}000=0.35\), and \(2{,}000/5{,}000=0.40\). They sum to \(0.25+0.35+0.40=1.00\), confirming that the listed categories cover all trials. The Monte Carlo standard errors for these proportions are, respectively, \(\sqrt{0.25(0.75)/5{,}000}=0.00612\), \(\sqrt{0.35(0.65)/5{,}000}=0.00675\), and \(\sqrt{0.40(0.60)/5{,}000}=0.00693\).
The estimated mean total sample size is
The empirical variance of sample size is
so \(s_N\approx190.1\) participants and the Monte Carlo standard error of the mean is approximately \(190.1/\sqrt{5{,}000}=2.69\) participants. A concise report could state that, under this alternative scenario, 25%, 35%, and 40% of trials stopped at 240, 480, and 720 participants, respectively, with an estimated mean of 516 participants (Monte Carlo SE 2.69; maximum 720). The spread in sample size is substantial, so reporting only the mean would omit useful information.
Comparing Scenarios Without Overstating Differences
Present results in a table when several scenarios are evaluated. Give each row a scenario label and enough information to distinguish it, such as the treatment effect and the nuisance assumptions varied. Include the number of replications, the estimated characteristic, and Monte Carlo precision. Keep design settings fixed or clearly identify which settings also changed. For type I error, identify the null scenario; for power, identify the alternative scenario. For stopping and sample-size summaries, identify the scenario under which their distributions were generated.
| Scenario | Replications | Operating characteristic | Estimate | Monte Carlo SE |
|---|---|---|---|---|
| Null; stated baseline nuisance assumptions | 10,000 | Type I error | 0.0246 | 0.00155 |
| Alternative; stated target effect and baseline nuisance assumptions | 10,000 | Power | 0.786 | 0.00410 |
| Alternative; stated effect with higher event rate | 10,000 | Power | Report the simulated estimate | Report its Monte Carlo SE |
The table is illustrative of the reporting structure; a final report should replace the last row with results actually obtained. When comparing scenario estimates, Monte Carlo error matters. For example, suppose two independent simulations each use \(R=4{,}000\) trials and yield power estimates 0.760 and 0.720. Their difference is 0.040. The Monte Carlo standard error of the difference is approximately
The point estimates differ by about 4.1 Monte Carlo standard errors. This quantifies simulation noise under the independent-simulation assumption, but does not establish that one scenario is more scientifically plausible or that the design performs better in practice. If the simulations use common random numbers or otherwise dependent replicates, the uncertainty of the difference must reflect that dependence; the independent formula should not be applied automatically.
Diagnostics, Alternatives, and Limitations
Several checks make a report more informative and can reveal implementation or communication problems:
- Check that rejection counts divided by the stated \(R\) reproduce the reported error or power estimates.
- Check that stopping counts and probabilities cover all simulated trials and that the probabilities sum to 1 before rounding.
- Verify that sample-size units and the reported maximum match the design, and that a reported mean is compatible with the stopping distribution.
- State failures, exclusions, and any change in the number of usable replications. Unexplained missing runs can bias reported summaries.
- Compare Monte Carlo standard errors with the precision needed for the decision. A small difference between scenarios may be unresolved when simulation error is large.
A single headline number is compact but can hide scenario dependence and stopping behavior. A table of scenario-specific estimates is usually more informative; a supporting description of the simulation assumptions makes the table interpretable. Analytic calculations or benchmark designs can provide useful checks or complementary evidence, but they do not replace reporting simulated results for scenarios that the analytic calculation does not cover.
Simulation remains conditional on its model, implementation, and selected scenarios. Monte Carlo intervals account for finite replication only. They do not account for misspecified event rates, an unexamined operational deviation, a software error shared across all replicates, or uncertainty about which scenario will occur. A report should avoid presenting simulated operating characteristics as guarantees about a future trial.
Check Your Understanding
Answer each question using the definitions and reporting principles above. Brief solutions follow.
- In 2,500 null-scenario simulations, 60 trials reject \(H_0\). What is the estimated type I error?
- For the estimate in question 1, calculate the Monte Carlo standard error using the binomial formula.
- Stopping counts at three looks are 200, 300, and 500 out of 1,000 trials. What probabilities should be reported, and what check should be made?
- Why is the mean simulated sample size not a substitute for reporting stopping probabilities or sample-size variation?
- What uncertainty does a Monte Carlo standard error describe, and what important uncertainty does it not describe?
Solutions. 1. The estimate is \(60/2{,}500=0.024\), or 2.4%, under the specified null scenario. 2. The Monte Carlo standard error is \(\sqrt{0.024(0.976)/2{,}500}=\sqrt{0.0000093696}\approx0.00306\). 3. The probabilities are 0.20, 0.30, and 0.50. They should sum to 1 and the categories should cover every trial exactly once. 4. Different distributions can have the same mean; the mean alone does not show when trials stop or how variable their sample sizes are. 5. The Monte Carlo standard error describes finite-replication variability under the simulation assumptions. It does not measure uncertainty from a misspecified scenario, an unrealistic model, or a software error applied consistently across replicates.