Simulation Plans That Support Design Review

Why a Simulation Plan Must Be Reviewable

A Bayesian trial design can depend on more than its sample size. The prior, likelihood, interim or final decision rule, and assumptions about the data-generating process all affect what the design will do. Simulation evaluates those interactions by repeatedly generating hypothetical trial data under specified conditions and applying the proposed analysis and decision rule. Its value for design review depends on whether another statistician can see what was assumed, what was simulated, and how success or failure was judged.

A simulation plan is therefore more than a statement that “many trials will be simulated.” It specifies the design to be evaluated, the scenarios, the analysis to be applied in each replicate, the performance measures, the criteria for judging the design, and checks on computational accuracy. The plan should be fixed before inspecting the simulation results. This makes the evaluation interpretable and helps distinguish a genuine design weakness from an unexamined modeling or coding choice.

Specify the Design Before Generating Data

The plan should first identify the exact design being evaluated. State the target population and clinical question, the planned enrollment and follow-up, the endpoint and its timing, the analysis model and prior, and the decision rule. If the design includes interim analyses, adaptation, or early stopping, describe the schedule and rules sufficiently to reproduce them. Simulation must apply the proposed rules as written; it must not quietly replace them with a simpler analysis.

Next distinguish the analysis assumptions from the data-generating assumptions. The analysis model is the method applied to each simulated data set. The data-generating assumptions describe how those data arise in a particular scenario. They need not match. Indeed, scenarios that depart from the analysis assumptions can reveal sensitivity to misspecification. State which features are deliberately varied and which are held fixed.

For example, in a single-arm trial with a binary endpoint, a simple analysis might assume independent Bernoulli outcomes with response probability \(p\). A scenario may generate each participant’s outcome independently with a specified true response probability. This setup assumes a fixed sample size, complete endpoint ascertainment, identical response probability across participants, and no dependence between outcomes. If the proposed trial has staggered enrollment, missing outcomes, or multiple interim looks, those features must either be represented in the simulation or explicitly excluded with a rationale. A model that omits a design feature cannot evaluate its consequences.

Scenarios should be chosen to answer the regulatory and clinical question, not merely to make simulation convenient. Include conditions under which the design should not declare success, conditions representing a clinically meaningful benefit, and plausible variations that could undermine performance. For a decision threshold, a scenario at or near the threshold is often especially informative. Scenario assumptions should be clinically interpretable and traceable to stated evidence or judgments; a large number of arbitrary parameter combinations is not a substitute for a justified scenario set.

Choose Scenarios and Performance Measures

A scenario table makes the evaluation auditable. For every scenario, give the values or distributions used to generate relevant variables, the reason the scenario is included, and the design feature it tests. Keep a clear distinction between a parameter value used to generate data and a prior distribution used in the analysis. For example, generating data at a fixed \(p=0.20\) does not mean that the analysis treats \(p\) as known to equal \(0.20\).

Performance measures must match the question being asked. For a binary success decision, one measure is the probability of declaring success under a specified scenario. Under a clinically inadequate response rate, that probability describes false declarations of success relative to the chosen scenario; under a clinically meaningful response rate, it describes the probability of success under that scenario. Other designs may require measures such as probability of stopping early, expected enrollment, or probability of an undesirable safety outcome. Define the event and denominator precisely. A measure called “power” or “false-positive rate” is not self-explanatory unless its scenario and decision event are specified.

The plan should also state design acceptance criteria. These are prespecified standards for judging whether simulated performance is acceptable, such as a maximum tolerated probability of success in an inadequate-response scenario or a minimum probability of success in a target-benefit scenario. Such criteria are contextual design choices, not universal statistical constants. Explain their clinical or regulatory rationale and identify which scenarios each criterion governs. A point estimate near a criterion may be too uncertain to support a firm judgment, so criteria for Monte Carlo precision should be stated as well.

Justify the Simulation Error Calculation

The usual Monte Carlo standard error follows directly from the replicate-level indicators. Suppose the simulation uses independent replicates under one fixed scenario, and \(Y_r\) equals 1 when a defined event occurs and 0 otherwise. Then \(E(Y_r)=\theta\), the event probability, and \(\operatorname{Var}(Y_r)=\theta(1-\theta)\). The simulation estimate \(\widehat{\theta}=R^{-1}\sum_{r=1}^R Y_r\) has variance \(\theta(1-\theta)/R\), because the replicates are independent. Replacing the unknown \(\theta\) with \(\widehat{\theta}\) gives the estimated Monte Carlo standard error used above:

The bound \(\theta(1-\theta)\leq0.25\) follows from \(\theta(1-\theta)=\tfrac14-(\theta-\tfrac12)^2\), so it can be used when the performance probability is not known in advance. This calculation assumes independent simulation replicates generated under the same scenario. If replicates are dependent, or if the simulation uses a more complex sampling scheme, the simple formula may not apply without adjustment.

Diagnostics, Reproducibility, and Alternatives

A reviewable plan includes checks capable of finding implementation errors. Specify how the analysis code will be checked against independently calculated cases, such as the posterior update in the first example. Check that simulated outcomes have the intended distributions, that the decision rule is coded with the correct inequality and boundary, and that event counts cannot exceed the number of replicates. For adaptive designs, check that each simulated path follows the stated enrollment, interim, and stopping rules. Run code on small, inspectable test cases before the full simulation.

Record the software and relevant versions, random-number generator, seed or seed-management approach, replicate count, and any numerical integration or convergence settings. A seed helps reproduce a run, but it does not by itself validate the code or guarantee agreement across software environments. Where computation is stochastic, independent reruns or carefully selected test cases can help detect instability. The plan should say what happens if a computation fails, including how failed replicates are identified and handled; silently dropping them can bias an estimate.

Simulation is often preferable to a simple analytic calculation when the trial has complex decision rules, repeated looks, adaptation, or a posterior calculation that makes an exact operating probability difficult to obtain. When a reliable analytic calculation is available, it can be a useful check or a more efficient alternative for the corresponding performance quantity. Simulation complements, rather than replaces, justification of the prior, clinical assumptions, and decision threshold. Sensitivity Analyses for Regulatory Robustness addresses variation in assumptions; the simulation plan should make clear which such variations are represented as scenarios and how they enter the data-generating process.

Common warning signs include a scenario set that omits the decision boundary, a success measure with no defined event, results reported without a replicate count, unexpectedly different results across reruns, and implementation code that does not reproduce simple known cases. Another failure is to generate data from exactly the analysis model in every scenario and then treat that agreement as evidence of robustness. A well-matched scenario tests basic behavior, but deliberately challenging, plausible scenarios are needed to examine departures from the model.