Why Simulation Results Need a Traceable Structure
Simulation evidence can show how an adaptive trial design behaves across specified conditions, including conditions in which analytic calculations are difficult. But a table of percentages is not enough for review. A reviewer needs to be able to follow the chain from the assumed data-generating model, through each simulated trial and its prespecified decision rules, to the reported operating characteristic. If any link is unclear, a result may be impossible to interpret or reproduce.
The purpose of the report is not to imply that simulated trials predict exactly what will happen in a future study. Rather, it is to describe the design’s behavior under stated assumptions and scenarios, quantify simulation error in the estimates, and make the implementation auditable. This requires separating assumptions about trial data from rules governing the adaptive design and from summaries of simulation outcomes.
What a Reviewable Simulation Report Specifies
A useful report presents information in an order that lets a reviewer reconstruct what was evaluated. First describe the data-generating assumptions: the outcome distribution, parameter values, allocation, accrual or follow-up assumptions when relevant, and treatment of missing observations or other features included in the simulator. These are assumptions defining simulated conditions, not facts established about a future trial.
Next specify the design and analysis implemented in every replicate. State the timing and information cutoff for each interim analysis, the quantities used to make decisions, the possible actions, and the rule for the final analysis. Include the hypotheses, testing procedure, significance level, and any multiplicity control relevant to the reported decision. The rules must be sufficiently precise to distinguish permitted adaptations from discretionary choices. For example, “stop for futility if the interim result is unfavorable” is not executable; a numerical boundary, the statistic to which it applies, and the action triggered by crossing it are needed.
Then define the scenarios. Each row or scenario identifier should state the parameter configuration and any other condition that differentiates it. For error-rate evaluation, include the null configurations relevant to the claims being assessed; for power, state the alternative configuration and the endpoint on which it is defined. Avoid labeling a scenario only “null” or “target effect” when several parameter values or configurations are possible.
Finally, define each reported summary and its denominator. For example, “early-stop probability” should specify whether its numerator is all simulated trials that stopped at any interim and whether its denominator is all replicates in that scenario. For expected sample size, clarify whether the value is randomized across trial paths and whether it counts enrolled, randomized, or evaluable participants. A concise scenario table can make those links visible.
| Report element | Information to make explicit |
|---|---|
| Scenario | Parameter values, distributions, and conditions defining the simulated truth |
| Design and analysis | Interim timing, decision boundaries, actions, final test, and error-control procedure |
| Simulation implementation | Number of replicates, software and version, random-number procedures, and validation checks |
| Operating characteristic | Event or quantity summarized, numerator and denominator where applicable, estimate, and units |
| Simulation uncertainty | Monte Carlo standard error or interval, clearly distinguished from uncertainty about model assumptions |
Monte Carlo Error in Simulated Rates
Suppose the simulation generates \(B\) independent trial replicates under one fixed scenario. For a binary event such as rejection of \(H_0\), let \(I_b=1\) if the event occurs in replicate \(b\), and \(I_b=0\) otherwise. If the event probability under the simulated scenario is \(\pi\), then \(I_b\) has a Bernoulli distribution with variance \(\pi(1-\pi)\). The estimated event probability is \(\hat{\pi}=B^{-1}\sum_{b=1}^{B} I_b\).
Because the replicates are independent under this simulation setup, the variance of their average is the variance of one indicator divided by \(B\). Replacing the unknown \(\pi\) with its estimate gives the usual estimated Monte Carlo standard error:
This standard error measures simulation noise: how much the estimate would vary across repeated sets of \(B\) simulated trials under the same assumptions. It does not measure uncertainty about whether the data-generating assumptions are realistic. A normal-approximation interval, \(\hat{\pi}\pm1.96SE(\hat{\pi})\), can summarize Monte Carlo precision when the event is not too rare and the number of simulated events and non-events is adequate. For very rare events or estimates near zero or one, this approximation may be unreliable; use a suitable binomial interval or increase the number of replicates.
Worked Example: Reporting Simulated Power and Its Precision
Consider a hypothetical two-arm trial simulation with a continuous endpoint. In each arm, outcomes are generated independently from normal distributions with common standard deviation 10 points; the simulated treatment-group means differ by 3 points. Allocation is equal, and the adaptive design and final two-sided test at \(\alpha=0.05\) are fixed in advance. Suppose 10,000 independent trial replicates are generated under this alternative, and the prespecified final analysis rejects \(H_0\) in 7,420 replicates.
The estimated power under this particular scenario is \(\hat{\pi}=7{,}420/10{,}000=0.742\), or 74.2%. The Monte Carlo standard error is
Thus the estimated Monte Carlo standard error is about 0.438 percentage points. A rough 95% Monte Carlo interval is \(0.742\pm1.96(0.004375\ldots)\), or approximately 0.733 to 0.751. This describes the precision of the simulation estimate conditional on the stated model and design; it is not an interval for the treatment effect and does not account for uncertainty in the model assumptions. A suitable report would state the scenario, replicate count, rejection definition, estimate, and Monte Carlo precision together.
Worked Examples of Reviewable Reporting
Worked Example: Estimating a Null Rejection Probability
Suppose a hypothetical simulation evaluates a prespecified adaptive design under a scenario in which the two treatment means are equal. Outcomes are generated independently from normal distributions with common standard deviation 10 points, and each simulated trial uses the planned interim rule and multiplicity-adjusted final analysis. Of \(B=10{,}000\) replicates, 489 reject the primary null hypothesis.
The estimated rejection probability is \(489/10{,}000=0.0489\), or 4.89%. Under this null scenario, this is an estimate of the design’s Type I error probability for that hypothesis and configuration. The estimated Monte Carlo standard error is
The approximate 95% Monte Carlo interval is \(0.0489\pm1.96(0.0021566\ldots)\), or about 0.0447 to 0.0531. The estimate is near 0.05, but finite simulation error means this result alone does not establish that the underlying rejection probability is at most 0.05. The report should identify this as one stated null configuration, not generalize it to all configurations without evaluating those relevant to the error-control claim.
Worked Example: Making an Early-Stopping Summary Reproducible
Suppose a hypothetical design has a maximum total sample size of 240 participants and conducts an interim analysis after 120 have been randomized. Its prespecified rule stops for futility if the interim statistic \(Z_{\text{int}}<0.2\); otherwise the trial continues to a total of 240. For illustration, the final analysis rejects in the prespecified direction if its final statistic exceeds 1.96. Assume the outcome model and scenario are specified elsewhere in the report and the simulator applies these rules identically in each of 1,200 independent replicates. The simulation records 780 early stops and 420 continuations.
The estimated early-stop probability is \(780/1{,}200=0.650\), or 65.0%. Since each early stop enrolls 120 participants and each continuation enrolls 240, the simulated mean total sample size is
The calculation is traceable because the decision boundary, interim sample size, maximum sample size, recorded paths, and denominator are stated. A submission should also report the distribution of sample size or relevant path counts when they help explain the mean. The mean alone does not show whether most trials stop at 120 or whether a smaller number follow a more varied set of paths.
From Simulation Runs to Evidence
A reproducible report treats each simulated replicate as a complete execution of the planned design, not as a convenient source of a final test statistic. Record the interim data used, the decision reached, any adaptation applied, the resulting analysis, and the contribution to each summary. This makes it possible to check whether the specified rule was implemented and whether denominators match the estimand being reported.
Record the scenario parameters, distributions, and other assumptions that determine data generation.
Apply the stated interim cutoffs, adaptation rules, and final analysis to every replicate.
Calculate each operating characteristic from its explicitly defined event or quantity and denominator.
Report replicate counts, Monte Carlo uncertainty, software details, and checks supporting reproducibility.
Useful implementation checks include confirming that every replicate follows a permitted path, that no decision is made using data beyond its cutoff, and that counts of mutually exclusive pathways sum to the total number of replicates. For rates, show event counts as well as percentages. For continuous summaries such as sample size, state the unit and method of calculation. Report rounding consistently so that displayed percentages and counts agree.
A simulation report also benefits from distinguishing a simulation estimate from an analytic result. If a known special case has an analytic benchmark, compare the simulator with that benchmark as a validation check. Where no such result is available, document other checks, such as independently checking decision logic or verifying results across separate runs. These checks support confidence in implementation, but they do not establish that the chosen statistical model represents every future trial condition.
Comparisons, Limitations, and Common Mistakes
Analytic calculations can provide exact or approximate operating characteristics under tractable models, often without Monte Carlo error. Simulation is useful when adaptation rules, discrete outcomes, or multiple decision stages make a direct derivation difficult. The approaches are complementary: where an analytic calculation is available, it can serve as a benchmark; where simulation is used, its assumptions and finite-replicate precision must be reported.
A single aggregate result is less informative than scenario-specific results because different parameter configurations can produce different operating characteristics. Conversely, a very large table without clear definitions can obscure the design being evaluated. The report should prioritize the scenarios and outcomes connected to the stated design objectives and identify how each result was computed.
The main limitation is conditionality: a simulation result applies to the model and rules implemented. It cannot prove performance under unmodeled conditions, and a small Monte Carlo standard error cannot compensate for an unrealistic model. Keep these two issues separate in the interpretation: the Monte Carlo error concerns numerical precision under the assumed setup; assumptions concern what setup is relevant.
- Mistake: treating the simulated rate as exact. Reporting 4.89% as proof that error is controlled at 5% ignores finite Monte Carlo error. Report the estimate and its simulation precision, and evaluate the null configurations needed for the claim.
- Mistake: omitting the scenario behind a percentage. “Power was 74%” is incomplete without the assumed effect, variability, design, and analysis. Tie every operating characteristic to its scenario and decision rule.
- Mistake: treating a small Monte Carlo error as evidence that assumptions are correct. More replicates can stabilize an estimate under a specified model, but cannot validate the model’s relevance. State simulation uncertainty separately from model assumptions.
- Mistake: reporting only a final outcome. A final rejection rate may conceal an incorrectly implemented interim rule. Record and summarize decision pathways so the adaptive algorithm can be audited.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- A simulation reports 62% power but gives no effect size or outcome variability. What essential information is missing?
- In 5,000 independent simulated trials, an event occurs 250 times. Calculate the estimated event probability and its approximate Monte Carlo standard error.
- Why does a narrow Monte Carlo interval not show that the simulation assumptions are realistic?
- In 800 simulated trials, 300 stop at an interim sample size of 100 and 500 continue to a total sample size of 200. Calculate the estimated early-stop probability and mean sample size.
Solutions. (1) The scenario is not identifiable: at minimum, the assumed treatment effect and outcome variability are needed, along with the design and analysis conditions that define power. (2) \(\hat{\pi}=250/5{,}000=0.05\). Its estimated standard error is \(\sqrt{0.05(0.95)/5{,}000}=\sqrt{0.0000095}\approx0.00308\), or 0.308 percentage points. (3) Monte Carlo precision measures variation from using a finite number of replicates conditional on the implemented model; it does not assess whether that model represents the trial conditions of interest. (4) The early-stop estimate is \(300/800=0.375\), or 37.5%. The mean sample size is \([300(100)+500(200)]/800=130{,}000/800=162.5\) participants.