Why Reproducible Reporting Matters
A design-comparison simulation is useful only if readers can tell what was simulated, how each design was implemented, and how much numerical noise remains in the reported results. A table of power estimates alone cannot answer those questions. It may omit a response model, allocation rule, analysis method, or handling of failed runs that materially changes the comparison.
A reproducible report connects the scientific question to the computational experiment. It states the assumptions and scenarios, defines each comparator, describes the simulation and analysis procedures, reports Monte Carlo precision, and records enough software detail for an independent analyst to rerun or audit the work. Reproducibility does not mean that every reader will obtain identical results on every machine; it means the computational choices and sources of variation are explicit.
What Must Be Specified
Report the simulation as a set of scenarios and competing designs. For each scenario, give the trial conditions that determine the generated data: for example, outcome distributions, treatment effects, prognostic factors, enrollment size, outcome timing, and any response-model departures under study. State which conditions are fixed and which vary across the scenario grid. Earlier material on matching trial conditions across design comparators is relevant here: the report should make clear which features are held constant so that a contrast can be attributed to the designs rather than to different trial conditions.
For each design, describe the allocation rule as an executable sequence of decisions, not merely by a name. Include the initial allocation, the information available when each decision is made, the response or covariate model used, any burn-in, allocation-probability formula and bounds, update schedule, tie-breaking rule, and stopping or fallback behavior. State how randomization is performed. If the design uses an estimated response model, explain how its parameters are initialized and updated. If allocation is constrained, describe the constraint and what happens when it cannot be met.
Then specify the analysis applied to each simulated trial. Identify the estimator, test or interval procedure, significance level where relevant, covariate adjustment, and treatment of missing or undefined estimates. A design can change the data collected, while an analysis procedure determines how those data become a reported estimate or decision; readers need both. Name the performance measures and define their denominators and direction of preference. For instance, say whether a rejection indicator is evaluated under the null or an alternative, and whether allocation imbalance is measured by a difference or a ratio.
List scenarios, trial conditions, design comparators, and the data-generating assumptions for every scenario.
Give the allocation algorithms, simulated trial procedure, analysis methods, and rules for failures or undefined results.
Report the number of replicates and Monte Carlo standard errors for estimates and design contrasts.
Record software settings, random-number details, code or access instructions, and diagnostics that show whether runs completed as intended.
Monte Carlo Precision Belongs Beside the Results
Let \(Y_{rj}\) denote a performance outcome from replicate \(r\) under scenario \(j\), where \(r=1,\ldots,R\). For a binary outcome such as rejection, the estimated operating characteristic is \(\hat{p}_j=R^{-1}\sum_{r=1}^{R}Y_{rj}\). If replicates are independent and generated under the specified scenario, its Monte Carlo standard error is estimated by \(\sqrt{\hat{p}_j(1-\hat{p}_j)/R}\). This describes simulation variability in the estimated operating characteristic; it is not the standard error from an individual trial and does not measure uncertainty about whether the scenario represents reality.
For a numerical performance measure, report the replicate mean and the Monte Carlo standard error \(s_Y/\sqrt{R}\), where \(s_Y\) is the sample standard deviation across replicates. When designs are paired within replicate, define the replicate-level contrast \(D_r=Y_{r,A}-Y_{r,B}\), and report \(\bar{D}\) with Monte Carlo standard error \(s_D/\sqrt{R}\). This uses the covariance induced by pairing. State the sign convention for \(D_r\), because a positive difference may favor one design for one metric and disfavor it for another.
Worked Example: Reporting Precision for a Type I Error Estimate
Suppose a hypothetical null scenario is simulated for \(R=1{,}000\) replicates. A test rejects in 80 replicates, so the estimated Type I error rate is \(\hat{p}=80/1{,}000=0.0800\). Assume replicates are independently generated and the same prespecified analysis is applied in every replicate.
The estimated Monte Carlo standard error is
Thus the estimated simulation error is about 0.858 percentage points. A normal-approximation range of estimate plus or minus \(1.96SE_{\mathrm{MC}}\) is \(0.0800\pm1.96(0.00858)\), or approximately \(0.0632\) to \(0.0968\). A report could state: “The estimated Type I error rate was 8.00% (1,000 replicates; Monte Carlo standard error 0.858 percentage points).” The approximate range describes Monte Carlo variability around the estimated rate, not uncertainty about the null hypothesis or the true error rate of an unspecified population of trials.
Worked Example: A Paired Design Contrast
In a hypothetical comparison, both designs are run on the same simulated replicate conditions for \(R=1{,}000\) replicates. Define \(D_r\) as the treatment-estimation error under design A minus the error under design B, with lower absolute error preferred. Suppose the mean signed contrast is \(\bar{D}=-0.025\) percentage points and the sample standard deviation of the replicate contrasts is \(s_D=0.180\) percentage points.
The estimated contrast is \(-0.025\) percentage points, so the signed error under A is lower on average by 0.025 percentage points under this definition. Its Monte Carlo standard error is
For orientation, a normal-approximation Monte Carlo range is \(-0.025\pm1.96(0.00569)\), or approximately \(-0.0362\) to \(-0.0138\) percentage points. The result supports a negative average signed contrast in this simulation, but it does not by itself establish that A has smaller absolute error: signed errors can cancel. The report should therefore define the metric carefully and, if the scientific comparison concerns absolute error, report the paired contrast in absolute error instead. Independent design runs would require a different contrast calculation and would generally lose the precision benefit of pairing.
Algorithms, Software, and Audit Trail
A useful computational description has two layers. The main text gives the design logic and analysis clearly enough to understand the comparison. A supplement, repository, or archived package can provide executable code, configuration files, and machine-readable results. State the software and version, relevant package versions, operating environment when material, and any nondefault numerical settings. Include the random-number generator and seed or seed-management scheme. For parallel computation, explain how independent streams or substreams are assigned so that parallel runs do not inadvertently reuse random numbers.
A single seed is not always a complete reproducibility record. Results may also depend on code revisions, package updates, parallel execution, or platform-specific numerical behavior. Record a code version or archived release, and make scenario inputs and tuning constants explicit rather than embedding undocumented values in code. If access restrictions prevent sharing code or data, say so and describe what can be provided for independent checking.
Document the run protocol: replicates per scenario, whether replicate conditions are paired across designs, how runs are split across batches or processors, and how completion was checked. Report the number of attempted, completed, and excluded replicates if these differ. Give failure categories and prespecified handling rules. Do not silently discard a failed fit or an invalid estimate; such events can be informative and their omission can distort performance summaries.
Worked Example: Making a Scenario Record Reproducible
Consider a fully hypothetical scenario with 240 participants, two treatment arms, a binary outcome, and a true outcome probability of 0.30 in each arm. The report compares equal allocation with a response-adaptive rule that begins with 40 participants allocated equally, then updates allocation probabilities after each block of 20 participants using the prespecified fitted response model. Probabilities are restricted to the interval 0.20 to 0.80. The analysis is a two-sided test at \(\alpha=0.05\), and the performance outcomes include rejection under the null and the arm allocation counts.
A reproducible scenario record would also state whether outcomes are generated independently conditional on treatment, whether covariates enter the response model, exactly how model estimates are updated, and what happens if a fit fails. Suppose 10,000 paired replicates are run. The report identifies the random-number generator, software and package versions, code revision, and parallel stream procedure; states that a replicate uses the same underlying generated covariate and outcome random inputs where the coupling is defined; and reports attempted, completed, and failed runs by reason. Merely saying “we used a response-adaptive design” would not specify enough to reconstruct this algorithm.
The scenario record separates assumptions from implementation choices. Equal outcome probabilities define the null data-generating condition; the burn-in, block size, probability bounds, and update rule define the adaptive algorithm. This distinction allows a reader to identify whether a changed result arises from a different scientific scenario or from a different implementation.
Presenting Results and Diagnosing Problems
Organize results by scenario and performance measure, not only by a single overall ranking. A compact table should identify scenario, design, replicate count, estimate, Monte Carlo standard error, and any relevant paired contrast. Use units or percentage-point labels. For example, “0.08” should be identified as an 8% rejection rate, not left for readers to infer. If a direction is better only under a particular objective, state that direction in the table note or surrounding text.
| Scenario and measure | Design A | Design B | Paired contrast or precision |
|---|---|---|---|
| Null scenario: rejection rate | Estimate, MCSE, \(R\) | Estimate, MCSE, \(R\) | Difference in rates, MCSE, sign convention |
| Alternative scenario: power | Estimate, MCSE, \(R\) | Estimate, MCSE, \(R\) | Difference in power, MCSE, sign convention |
| Allocation: mean participants on preferred arm | Mean, MCSE, units | Mean, MCSE, units | Paired mean difference, MCSE |
Check that replicate counts agree across tables and that estimates use the stated denominator. Investigate unexpectedly large Monte Carlo errors, missing cells, boundary values, or abrupt changes across similar scenarios. Examine replicate-level failures, convergence warnings, and extreme values rather than reporting only the final aggregate. If a simulation uses paired replicates, confirm that pairing was preserved through analysis and that the contrast uses the same replicate IDs. Diagnostics can reveal coding or data-management errors; they do not prove that the scientific assumptions are adequate.
The most common reporting mistake is to interpret a small difference between two simulated estimates as a meaningful design advantage without considering Monte Carlo error. The correction is to report the contrast and its Monte Carlo standard error, then distinguish numerical precision from the substantive importance of the difference. A second mistake is to omit failed simulations because they seem inconvenient. Instead, state the failure rule, quantify failures, and assess whether their occurrence could change the comparison.
Choosing a Reporting Format and Stating Limitations
A narrative-only report is readable but often too vague to reproduce. A results table alone is concise but cannot communicate algorithm details or assumptions. A practical alternative is a layered report: a clear main-text account, tables covering all prespecified scenarios, and an accessible supplement or code archive containing implementation details. These formats complement rather than replace one another. A reader should not have to infer core assumptions from code, and code should not be the only place where the analysis procedure is defined.
No amount of computational detail removes limitations in the simulation assumptions. A report should identify which response mechanisms, outcome distributions, trial sizes, and operational conditions were not examined. Simulation results apply to the specified scenarios and procedures; they do not guarantee the same performance in a real trial or in unstudied conditions. Computational reproducibility also does not imply independent scientific validation. Separate the limits of the scenario set from Monte Carlo uncertainty, and explain both.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions below.
- A report gives a rejection rate of 0.06 but no replicate count or Monte Carlo standard error. Name two details needed to assess its precision.
- Two designs are run on paired replicate conditions. What quantity should be used to estimate the Monte Carlo standard error of their mean difference?
- Why is it not sufficient to report only the name of an adaptive allocation rule?
- A simulation has 500 attempted runs, 490 completed runs, and 10 failed model fits. What should the report disclose?
Solutions. (1) Report the number of replicates \(R\) and the Monte Carlo standard error, along with the estimate’s definition and denominator; for a binary outcome, the standard error depends on both \(\hat{p}\) and \(R\). (2) Calculate the replicate-level paired differences \(D_r\), then use their sample standard deviation \(s_D\) divided by \(\sqrt{R}\). (3) The name does not determine initialization, information timing, model updating, probability bounds, update schedule, or failure behavior; those implementation details can alter the design. (4) State the attempted, completed, and failed counts, explain the failure category and prespecified handling rule, and discuss whether excluding those fits could affect the reported performance.