Preparing a Reproducible Design Report

From Design Decisions to a Reviewable Record

A group sequential design is not fully reviewable just because a table gives its planned looks and maximum sample size. A reviewer also needs to know why those choices were made, which assumptions and inputs produced the reported operating characteristics, what software performed the calculations, and how to inspect or rerun the work. A reproducible design report connects these pieces so that the conclusions can be checked rather than simply accepted.

The report complements, rather than replaces, the protocol and statistical analysis plan. Those documents state what will be done in the study and how study data will be analyzed. The design report records how the proposed design was developed and evaluated. It should be detailed enough to reconstruct the design evaluation, but it should not silently introduce operational rules that conflict with the governing study documents.

The earlier material on operating characteristics provides the performance summaries a report may present. Here the emphasis is on making the design rationale, inputs, computational process, results, and limitations traceable as a single record.

Definition: A reproducible design report is a structured account of a proposed design and its evaluation that identifies the design rationale, assumptions and inputs, computational methods and software, results, limitations, and records or access instructions needed for an independent reviewer to inspect or rerun the analysis. Reproducibility means the work is sufficiently documented and accessible to be checked; it does not mean that every computer or software environment must produce identical output in every circumstance.

What the Report Needs to Make Explicit

A useful report lets a reader follow a chain: the scientific and operational question motivates the design; assumptions and inputs define the evaluated scenarios; the analysis method and decision rules determine what counts as success; software implements those rules; and results support a qualified design recommendation. When any link is missing, even plausible results can be difficult to interpret or verify.

Rationale and estimand: State the scientific question, treatment contrast or other target, outcome, target population, and why a group sequential design is being considered. Explain the reasons for the number and timing of looks, the error-control approach, and the maximum information or enrollment target.
Design specification: Record the hypotheses, significance level \(\alpha\), sidedness, planned analyses, stopping boundaries or boundary-generating method, information fractions, and any futility rule. State how information translates into enrollment and how allocation, event timing, missing outcomes, and operational delays are handled when relevant.
Inputs and assumptions: Give values, units, and sources or rationale for the assumed effect, variability or event rate, accrual pattern, dropout, nuisance parameters, and other quantities used. Separate the primary planning assumptions from sensitivity and stress-test scenarios. State the model assumptions on which the design evaluation relies.
Computational method and results: Identify whether the reported quantities came from analytic calculation, simulation, or both. For simulation, provide the number of replicates, scenario definitions, endpoint and decision rule used in each replicate, and the estimates with Monte Carlo uncertainty where appropriate.
Implementation and reproducibility records: Identify software, version, relevant packages or custom code, computational settings, code access instructions, and files needed to reproduce tables or figures. Record enough information to distinguish the evaluated analysis from later edits.

The level of detail should fit the design. A report need not reproduce general statistical theory or paste every line of a long codebase into the narrative. It should, however, identify the specific algorithms and rules used, provide working access to the code or a complete executable specification, and describe how to connect the code outputs to the reported results.

Worked Example: Turning a Design Rationale into Reportable Inputs

Suppose a hypothetical two-arm study will compare a continuous outcome between groups. The proposed group sequential design uses two-sided \(\alpha=0.05\), targets 90% power for a treatment contrast of 4 outcome units, and assumes a common standard deviation of 10 units. Three interim or final analyses are planned at information fractions 0.50, 0.75, and 1.00. These values are illustrative planning inputs, not findings from a real study.

A report should not state only that the design has “three looks.” It should identify the outcome and contrast, specify the assumed effect and variability with units, and state that the target standardized contrast under these assumptions is \(4/10=0.40\). It should also name the boundary method and give the actual boundaries or the exact method and inputs needed to generate them. An information fraction describes information accrued relative to the planned maximum; it should not be presented as an enrollment fraction unless the design justifies that equivalence.

The report should then explain how its stated assumptions support the evaluation. For example, if simulations generate normally distributed outcomes with a common variance, state that model and any assumed allocation ratio. If enrollment is used to determine when information targets are reached, describe the conversion and any allowance for incomplete outcome data. The reported operating characteristics apply to the specified model and scenarios; they are not automatically guaranteed under a different outcome distribution, dropout pattern, or analysis rule.

Finally, distinguish a target from a result. “Target power is 90% at a 4-unit contrast” describes a design objective. A simulated estimate of power is an output of the evaluation and should be reported separately, with its scenario, replicate count, and simulation uncertainty.

Presenting Simulation Evidence with Its Uncertainty

When an operating characteristic is estimated by simulation, the report should identify the probability being estimated, the scenario under which it is estimated, and the number of independent simulated trials. If \(B\) simulated trials produce \(K\) occurrences of an event, such as rejection under \(H_0\), the estimated probability is \(\hat{p}=K/B\). Under independent Bernoulli simulation outcomes, its estimated Monte Carlo standard error is \(\sqrt{\hat{p}(1-\hat{p})/B}\). This standard error describes simulation precision, not uncertainty in the scientific assumptions or the true treatment effect.

For instance, with \(B=100{,}000\), the estimated standard error is smaller than with \(B=10{,}000\), but no increase in replicate count repairs a misspecified model or incorrect implementation. A report should therefore pair numerical precision with diagnostics and sensitivity analyses, rather than treating a small Monte Carlo error as evidence that the design is robust.

Worked Example: Reporting Simulated Type I Error and Power

In a hypothetical evaluation, 100,000 simulated trials under the null hypothesis produce 4,980 rejections. Under a specified alternative scenario, 90,120 of 100,000 trials meet the design’s success criterion. Assume the simulation replicates are independent within each scenario and that the same decision rules are applied in every replicate.

Under the null, the estimated Type I error probability is:

\hat{p}_0=\frac{4{,}980}{100{,}000}=0.04980.

Its estimated Monte Carlo standard error is:

SE_{\mathrm{MC}}=\sqrt{\frac{0.04980(1-0.04980)}{100{,}000}} =\sqrt{0.0000004732}\approx 0.000688.

A rough 95% interval for simulation error alone is \(0.04980\pm1.96(0.000688)\), or approximately 0.04845 to 0.05115. This interval includes 0.05, so the simulation estimate does not resolve whether the underlying simulated rejection probability is slightly below or above 0.05. It does not show that \(H_0\) is true, nor does it account for uncertainty about the model inputs.

Under the alternative, the estimated power is:

\hat{p}_1=\frac{90{,}120}{100{,}000}=0.9012,\qquad SE_{\mathrm{MC}}=\sqrt{\frac{0.9012(1-0.9012)}{100{,}000}} \approx 0.000944.

The corresponding rough 95% interval for Monte Carlo error alone is approximately \(0.9012\pm1.96(0.000944)\), or 0.8993 to 0.9031. A report could say that estimated power was 90.12% under the stated alternative, with an estimated Monte Carlo standard error of 0.094 percentage points. It should not imply that power is 90.12% under every plausible effect or nuisance assumption.

Derivation: Why the Replicate Count and Event Count Matter

For a fixed scenario, define \(I_b=1\) if simulated trial \(b\) meets the event of interest and \(I_b=0\) otherwise, for \(b=1,\ldots,B\). If the simulations are independent and each has event probability \(p\), then \(E(I_b)=p\) and \(\operatorname{Var}(I_b)=p(1-p)\). The estimated probability is the sample mean \(\hat{p}=B^{-1}\sum_{b=1}^{B}I_b\). Independence gives:

\operatorname{Var}(\hat{p}) =\operatorname{Var}\left(\frac{1}{B}\sum_{b=1}^{B}I_b\right) =\frac{1}{B^2}\sum_{b=1}^{B}\operatorname{Var}(I_b) =\frac{p(1-p)}{B}.

Replacing the unknown \(p\) by \(\hat{p}\) gives the estimated Monte Carlo standard error above. This calculation explains why a report must give both \(K\) and \(B\), or equivalently \(\hat{p}\) and \(B\): the estimate alone does not reveal its simulation precision. The calculation also relies on the stated independence and common-probability setup. If simulations use a different sampling scheme, the uncertainty calculation must match that scheme.

Software, Access, and Reproducibility Records

A reproducibility record should let a reviewer identify the exact computational work behind the report. At minimum, record software and version, packages and versions when used, the operating environment where it affects execution, random-number generator and seed for stochastic work, and any parallelization or other settings relevant to the simulation. Identify the source files and the specific code revision used. A seed can help reproduce a random stream in a specified environment; it is not a complete specification of the software, inputs, algorithms, and execution settings.

Give practical access instructions. For example, state where the code and input files are archived, which file generates each reported table, and the command or documented steps to run it. Include a README, dependency information, and expected outputs or checks when useful. If code cannot be made public, explain the restriction and provide a controlled-access route or sufficient alternative materials for review. Do not expose confidential data or credentials in an archive.

Worked Example: A Compact Computational Manifest

Consider a hypothetical report whose simulation used custom base R code. A concise manifest might state: “Generated with R 4.3.2 on Ubuntu 22.04; base R only; RNG kind Mersenne-Twister; seed 271828; 100,000 replicates per scenario; no parallel execution. The evaluated scripts and scenario-input file are archived together as release `design-report-v1.0`, commit `a1b2c3d4`. Run `Rscript run_all.R`; the script writes the scenario summary table and a session-information file to the output directory.”

The accompanying README should define the input fields, identify the output files that match each report table, and say how to check that a completed run produced the expected replicate counts. If the report instead includes a manually edited table, it should say so and describe how the table was checked against program output. That link matters: an archived script is not evidence that a particular reported number was generated by that script.

This manifest is useful but does not promise bit-for-bit identical results in every environment. Different software versions, numerical libraries, parallel execution, or code changes may affect outputs. Where exact reruns matter, the archive should preserve the relevant environment and execution instructions; regardless, the report should retain the inputs and outputs used for the submitted results.

Review, Alternatives, and Failure Modes

A reproducible design report is more than a software log and less than a substitute for the protocol or statistical analysis plan. The report explains and supports the design evaluation; the protocol and plan govern study conduct and analysis; code comments and a README explain implementation details. These records can refer to one another, but important design decisions should not be left to inference from code alone.

Freeze the design specification.
Record the hypotheses, design rules, planned looks, assumptions, inputs, and scenarios that the evaluation is intended to represent.

Connect methods to implementation.
Identify analytic calculations or simulation procedures, code revision, software environment, and run settings.

Reconcile outputs.
Check that counts, estimates, uncertainty summaries, and figures in the report match archived outputs and that the stated scenarios match the executed inputs.

Qualify the recommendation.
Explain what the evidence supports, where sensitivity or operational checks raise concern, and which assumptions limit interpretation.

Warning signs include missing units or scenario definitions; different replicate counts in the narrative and output; a seed without software or code details; unexplained manual changes; and performance claims that omit Monte Carlo uncertainty. Also check for mismatches between the proposed design and its evaluation, such as a stopping rule described in the report that is absent from the simulation code. Such mismatches can make otherwise precise results irrelevant to the design being reviewed.

The report’s limitations should distinguish at least two sources of qualification. First, model and operational assumptions may not capture the study setting; sensitivity analyses can show how results change under specified alternatives, but cannot cover every possibility. Second, simulation estimates have Monte Carlo error; increasing \(B\) can reduce that error but cannot remove assumption uncertainty or implementation bias. A recommendation should therefore be proportionate to the evidence and state what changes could require reassessment.

Common pitfall: “We reported the random seed, so the simulation is reproducible.” A seed alone does not identify the code revision, inputs, software and relevant versions, random-number settings, or how outputs map to the report. Record those elements and preserve the actual inputs and outputs; then describe any environmental differences that may prevent an exact rerun.

Check Your Understanding

Answer each question, then compare your reasoning with the brief solutions.

A report gives estimated power of 0.90 but omits the simulated scenario and number of replicates. Name two pieces of information a reviewer cannot determine from that estimate alone.
In 20,000 independent simulated trials, a null event occurs 1,000 times. Calculate the estimated probability and its estimated Monte Carlo standard error.
Why is a seed insufficient as the only reproducibility record?
State one reason a design report and a protocol have different roles, even if they refer to the same design.
A report’s table does not match the archived simulation output. Give a useful first review action and explain why it matters.

Solutions. (1) The reviewer cannot identify the alternative assumptions and success criterion behind the estimate, or assess its simulation precision without the replicate count. (2) \(\hat{p}=1{,}000/20{,}000=0.0500\); \(SE_{\mathrm{MC}}=\sqrt{0.05(0.95)/20{,}000}\approx0.001541\). (3) A seed does not identify the code, inputs, software environment, or execution settings that determine the analysis and its outputs. (4) The report records the rationale and evaluation supporting the design, while the protocol specifies study conduct; the report should not silently replace the governing study document. (5) Trace the table to its generating script and archived inputs, then rerun or inspect the output-generation steps. This can reveal a changed input, code revision, or manual transcription and establishes which values were actually evaluated.