Reporting Simulation Assumptions and Results

Why Simulation Results Need an Audit Trail

A simulation result is an estimate of how a specified design behaves under specified conditions. A reported value such as “power was 82%” is not sufficient on its own: readers need to know what counted as success, how the trial was simulated, which scenario assumptions were used, and how many simulated trials supported the estimate. Without those details, neither the design nor the precision of the reported result can be evaluated.

This tutorial focuses on reporting those elements. It does not treat simulation output as a substitute for the statistical design or for clinical justification of its assumptions. Instead, the goal is to make the relationship between assumptions, simulation algorithm, and reported operating characteristics explicit. An operating characteristic is a probability describing a design’s performance under a stated scenario, such as the probability of rejecting \(H_0\) when a specified alternative is true.

What a Transparent Report Specifies

A useful report lets a reviewer reconstruct the logic of the simulation without guessing what the analyst meant. It identifies the simulated design and its decision rules, the data-generating assumptions, the event counted for each operating characteristic, and the Monte Carlo procedure used to estimate that probability. If the design is adaptive, the description must make clear how interim data determine actions such as stopping, continuing, dropping an arm, or changing allocation.

For each scenario, state the assumptions that affect the simulated data and decisions. Depending on the design, these may include treatment effects, outcome distributions, recruitment and accrual patterns, missingness, adherence, subgroup prevalence, and operational deviations. Give values, ranges, or distributions and units where relevant. State which inputs are held fixed within a scenario and which, if any, vary across replications. If inputs are drawn from a distribution, name that distribution and its parameters. A result averaged over such draws answers a different question from a result under one fixed scenario.

The algorithm description should be sufficiently specific to distinguish the actual procedure from plausible alternatives. For example, state the order in which enrollment, outcomes, interim analyses, and decision rules are applied. Define what happens at boundaries, when information is missing, or when a simulated path cannot proceed as planned. For each reported probability, define the event in operational terms: for example, “the final test rejects \(H_0\) at the prespecified two-sided \(\alpha=0.05\),” rather than simply “success.”

This sequence is an outline, not a complete algorithm for every design. The report must fill in the trial-specific details, including the statistical test or interval procedure, timing of analyses, and treatment of exceptional paths. A statement such as “we simulated the adaptive design” leaves too many consequential choices unspecified.

Monte Carlo Uncertainty and a Short Derivation

For a fixed scenario, suppose each independent simulated trial has probability \(q\) of producing event \(E\). Let \(I_b=1\) if event \(E\) occurs in replication \(b\), and \(I_b=0\) otherwise. Then \(K=\sum_{b=1}^{B}I_b\), and \(\hat q=K/B\). Under independent replications with the same event probability, \(K\) has a binomial distribution with parameters \(B\) and \(q\).

Because a Bernoulli variable has variance \(q(1-q)\), independence gives \(\mathrm{Var}(K)=Bq(1-q)\). Dividing by \(B^2\) yields \(\mathrm{Var}(\hat q)=q(1-q)/B\). Since \(q\) is unknown, estimate the Monte Carlo standard error (SE) by substituting \(\hat q\):

When the number of simulated events and non-events is not small, a normal-approximation interval is \(\hat q\pm1.96SE(\hat q)\), an approximate 95% Monte Carlo interval. It describes uncertainty due to the finite number of simulation replications, conditional on the simulation model and algorithm. It is not a 95% probability statement about the unknown clinical effect, and it does not account for uncertainty or misspecification in the scenario assumptions. If the estimated probability is near 0 or 1, or event counts are small, the normal approximation can perform poorly; use and identify a suitable binomial interval, such as a Wilson or exact interval.

The replication count should be selected in light of the precision needed for the decision. Since the standard error is approximately proportional to \(1/\sqrt{B}\), reducing it by half requires about four times as many replications, other things being equal. More replications reduce Monte Carlo error but do not correct an incorrect model, an incorrectly implemented decision rule, or an unrepresentative scenario.

Worked Reporting Under a Challenging Scenario

An operating-characteristic estimate is interpretable only alongside the assumptions that define its scenario. The same design can have substantially different performance when recruitment, missingness, adherence, or treatment effects differ. Reporting a scenario label alone, such as “high dropout,” is inadequate unless the label is defined numerically or by a specified model.

Reporting Averages Across Uncertain Inputs

Some design assessments vary uncertain inputs across replications rather than evaluating only fixed scenarios. In that case, say how the inputs are varied and what probability is being estimated. For example, if each replication first draws a treatment effect from a stated distribution and then simulates a trial, the resulting estimate averages performance over that input distribution. Its Monte Carlo interval measures finite-replication precision for that averaged quantity; it does not establish that the distribution represents clinical knowledge correctly.

The same care is needed when reporting a joint event, such as meeting both a power target and a safety target. Define the event as a conjunction of explicit conditions, and count a replication only when all conditions hold. If the analysis uses importance sampling rather than ordinary independent simulation, report that method, the weighting procedure, and the uncertainty calculation appropriate to the weighted estimate. The simple binomial standard error above applies to ordinary independent Bernoulli replications, not automatically to weighted observations.

Diagnostics, Alternatives, and Limitations

Before interpreting a precise-looking estimate, check whether the simulation itself behaved as intended. Useful checks include confirming that the number of completed replications equals the reported \(B\), verifying that event counts are possible and internally consistent, and examining whether adaptive decisions follow the stated rules in representative trial paths. Compare simple special cases with known calculations where possible. Unexpectedly extreme operating characteristics, large differences between nearly identical scenarios, or estimates that change materially when the replication count is increased deserve investigation.

Report failed or incomplete replications and how they were handled. Excluding paths because they are inconvenient can bias an estimate if those paths have different event probabilities. Also distinguish a true design failure, such as inability to reach a decision under a specified rule, from a software failure. If a computational approximation is used, document it and assess whether it can affect the target event.

For ordinary Monte Carlo, the direct estimate \(K/B\) and binomial uncertainty are natural when replications are independent and identically distributed under the stated scenario. Other approaches may be useful in different settings: exact or Wilson intervals can improve inference for sparse event counts; larger \(B\) can reduce Monte Carlo error; and importance sampling can improve estimation of rare probabilities when its weighting and uncertainty calculations are valid. These alternatives do not make assumptions disappear. In particular, a weighted estimate cannot be reported as if it were an unweighted event proportion.

A Monte Carlo interval is conditional on the programmed algorithm and assumed data-generating process. It cannot reveal that a treatment effect is implausible, a missingness assumption is wrong, or a decision rule was coded incorrectly. Report scenario-specific estimates rather than relying only on an average when important scenarios may have poor performance. Finally, do not imply that a small Monte Carlo standard error guarantees sound design assurance: it guarantees only that the simulated quantity was estimated with limited simulation noise.

Common Reporting Mistakes

• Reporting a percentage without its event definition. “Success was 82%” is ambiguous. State exactly which decision or trial outcome counted as success.
• Omitting scenario assumptions. A scenario name is not a model. Report the numerical inputs or distributions and the mechanisms used to generate data.
• Giving no replication count or uncertainty. A reader cannot tell whether an estimate is stable. Report \(B\), the event count \(K\), the estimate, and an appropriate Monte Carlo uncertainty measure.
• Confusing Monte Carlo error with model uncertainty. More replications narrow Monte Carlo intervals; they do not validate the assumptions.
• Using the binomial standard error for dependent or weighted results. Check that the stated interval matches how the estimate was produced.