Reproducible Design Assurance Reports

Why a Design Assurance Report Must Be Reproducible

A design assurance conclusion is only as reviewable as the chain of evidence behind it. A statement such as “the design meets the target” is incomplete unless a reviewer can identify the target, the simulated scenarios, the algorithm used, the amount of Monte Carlo error, and the checks supporting the computation. A reproducible report makes that chain explicit and allows another analyst to inspect the choices and reproduce the results.

The report is not merely a collection of simulation output. It connects the question (what design property is being assessed?), the implementation (what code and assumptions produced the output?), the evidence (what checks and uncertainty summaries support it?), and the decision (how do the results relate to the stated acceptance rule?). The earlier material on reporting simulation assumptions and results and design assurance tables and visualizations provides useful foundations; here the emphasis is on preserving traceability across the whole analysis.

Definition: A reproducible design assurance report is a documented, versioned account of a simulation-based design assessment that identifies the design and target quantities, defines the scenarios and assumptions, links results to the code and computational environment that produced them, records validation evidence and Monte Carlo uncertainty, and explains the decision using prespecified criteria.

What the Report Needs to Specify

A reader should be able to answer “what was simulated?” before interpreting any result. State the trial design and its relevant operating rules, the population and outcome assumptions, the scenarios considered, and which quantity is estimated in each scenario. For example, if the target is the probability that a design declares success under a specified scenario, define success precisely: identify the simulated event and the rule that determines whether it occurs. Name the denominator: usually the number of completed simulation replications for that scenario.

Separate assumptions that define the scenario from choices that define the computation. Scenario assumptions might include the treatment effect, outcome variability, accrual pattern, or missingness mechanism. Computational choices include the number of replications, random-number generator and seed, software versions, and how results are summarized. Explain any scenario weights if an overall average is reported; do not leave a reader to infer how the average was formed.

The code and report should also identify their versions. A practical record includes a version-control commit or immutable code archive, software and package versions, run date, input-file identifiers or checksums, and the configuration used for the run. A random seed is helpful, but is not a complete reproducibility guarantee: software or generator changes can alter the sequence of draws, and parallel computation can change how draws are assigned. Record these details and, where needed, the computational environment or container.

Monte Carlo Uncertainty Belongs Beside the Estimate

Suppose a fixed scenario is simulated \(N\) times, and replication \(i\) has an indicator \(I_i=1\) if the prespecified event occurs and \(I_i=0\) otherwise. If the replications are independent and generated under the same scenario, then the indicators are independent Bernoulli random variables with event probability \(\theta\). The simulation estimate is the observed event fraction, and its estimated Monte Carlo standard error quantifies the sampling variability due to using a finite number of simulation replications.

\hat{\theta}=\frac{K}{N}, \qquad SE(\hat{\theta})=\sqrt{\frac{\hat{\theta}(1-\hat{\theta})}{N}},

Here \(K=\sum_{i=1}^{N} I_i\) is the number of replications in which the event occurred. The formula estimates the standard deviation of the simulated fraction under independent Bernoulli replications. It measures Monte Carlo error, not uncertainty about the true clinical treatment effect or uncertainty about whether the scenario is a realistic description of a future trial.

A usual approximate 95% Monte Carlo interval is \(\hat{\theta}\pm1.96SE(\hat{\theta})\). This normal approximation is most suitable when the expected numbers of events and non-events are not small. For rare events or estimates near 0 or 1, a symmetric interval can be misleading; use an appropriate binomial interval or increase the simulation size. State the interval method, not just its endpoints.

Worked Example: Reporting a Simulated Success Probability

In a hypothetical design assessment, a prespecified scenario is simulated for \(N=5{,}000\) independent replications. The design meets its success rule in \(K=4{,}100\) replications. Assume each replication uses the same scenario and the same success definition.

The estimated probability of meeting the rule is

\hat{\theta}=\frac{4{,}100}{5{,}000}=0.8200.

The Monte Carlo standard error is

SE(\hat{\theta})= \sqrt{\frac{0.8200(1-0.8200)}{5{,}000}} =\sqrt{\frac{0.1476}{5{,}000}} =\sqrt{0.00002952} \approx 0.005433.

The approximate 95% Monte Carlo interval is

0.8200 \pm 1.96(0.005433) =0.8200\pm0.01065 =(0.8094,\ 0.8306).

A report can state that the estimated success probability in this hypothetical scenario is 0.820, with approximate 95% Monte Carlo interval 0.809 to 0.831. If the prespecified criterion is that the lower endpoint exceed 0.800, this estimate clears that criterion. That conclusion concerns Monte Carlo precision for this scenario; it does not establish that the scenario assumptions themselves are correct.

Validation Evidence and a Short Derivation

Validation should test more than whether the program finishes without an error. Begin with code-level checks of inputs and outputs: scenario values are within their permitted ranges, sample sizes and allocation totals are as intended, event indicators are 0 or 1, and summary denominators equal the number of completed replications. Add targeted checks against simple cases with known or independently calculable behavior. For example, a rule that always declares success should yield an estimated probability of 1, while a rule that can never declare success should yield 0. These tests do not prove the full implementation correct, but they can reveal coding and bookkeeping errors.

For an important result, an independent implementation or rerun is stronger evidence than repeatedly inspecting the same output. The rerun should use the same scenario definitions and decision rule, while documenting whether it uses identical code, a separate implementation, or an independent random stream. With different random streams, close agreement rather than exact equality is expected.

The Monte Carlo standard error formula follows directly from the variance of a sample average. Under the stated independent Bernoulli model, \(E(I_i)=\theta\) and \(\operatorname{Var}(I_i)=\theta(1-\theta)\). The estimate is the average of \(N\) indicators; independence makes the variance of that average equal to the individual variance divided by \(N\). Replacing the unknown \(\theta\) by \(\hat{\theta}\) gives the estimated standard error.

\operatorname{Var}(\hat{\theta}) =\operatorname{Var}\left(\frac{1}{N}\sum_{i=1}^{N}I_i\right) =\frac{1}{N^2}\sum_{i=1}^{N}\operatorname{Var}(I_i) =\frac{\theta(1-\theta)}{N}, \qquad \widehat{SE}(\hat{\theta}) =\sqrt{\frac{\hat{\theta}(1-\hat{\theta})}{N}}.

The independence condition matters. If replications are correlated, for example because of a shared random component that induces dependence, the binomial standard error above need not apply. The report should describe the simulation unit and dependence structure, and use an uncertainty calculation suited to that structure. It should not label a standard error “Monte Carlo” without stating how it was obtained.

Worked Example: A Zero-Event Result Is Not Proof of Zero Risk

In a hypothetical simulation of a rare failure event, \(N=20{,}000\) independent replications produce \(K=0\) failures. The point estimate is \(0/20{,}000=0\), but the true simulated failure probability is not thereby shown to equal zero.

For zero events, a useful one-sided 95% exact binomial upper bound \(u\) solves \(P(K=0\mid\theta=u)=0.05\). Since \(P(K=0\mid\theta)=(1-\theta)^N\), solving gives

(1-u)^{20{,}000}=0.05, \qquad u=1-0.05^{1/20{,}000} \approx 0.0001498.

Thus, under the independent, identically distributed replication assumption, the data are compatible at the one-sided 95% level with a failure probability as large as about 0.000150, or 0.0150%. The zero-event estimate has not established that the probability is zero. A report should show the number of events and replications, give an interval or bound appropriate to the event rate, and consider whether more replications are needed for the assurance target.

From Output to a Reviewable Decision

A results section should connect each estimate to its scenario and criterion. Keep the estimate, its Monte Carlo uncertainty, the target, and the decision rationale together. If the conclusion depends on several conditions, say whether the rule requires every condition to pass or uses a different combination. Do not replace the prespecified rule with a favorable interpretation after viewing the results. The earlier material on thresholds for design acceptance and go-no-go assurance under uncertain inputs addresses the construction of criteria; the report’s task is to show how the stated criterion was applied.

The following sequence is a practical structure for assembling the evidence:

Define the assessment.
Name the design, estimand or operating characteristic, event definition, target, and acceptance rule before presenting results.

Freeze and identify the inputs.
Record the scenario specifications, weights if applicable, code version, computational environment, run configuration, and input provenance.

Show validation and uncertainty.
Report checks, replication counts, Monte Carlo standard errors or suitable intervals, and any independent reproduction evidence.

Apply the criterion transparently.
State which results pass or fail the prespecified rule, explain any unresolved limitations, and retain enough detail to trace each conclusion to its output.

Worked Example: Comparing an Independent Rerun

An analyst independently reruns the hypothetical success-probability calculation using a separate random stream, with \(N=5{,}000\) replications, and observes \(K=4{,}082\) successes. The first run gave 4,100 successes in 5,000 replications. Assume both runs implement the same event definition and scenario.

The rerun estimate is

\hat{\theta}_2=\frac{4{,}082}{5{,}000}=0.8164.

Its standard error is

SE(\hat{\theta}_2) =\sqrt{\frac{0.8164(1-0.8164)}{5{,}000}} =\sqrt{\frac{0.14989104}{5{,}000}} =\sqrt{0.000029978208} \approx 0.005475.

The observed difference between runs is \(0.8200-0.8164=0.0036\). If the two runs use independent random streams, the estimated standard error of their difference is

\sqrt{0.005433^2+0.005475^2} \approx 0.007713, \qquad \frac{0.0036}{0.007713}\approx 0.467.

The difference is less than one estimated standard error, so it is consistent with ordinary Monte Carlo variation between independent runs. This is a useful agreement check, not proof that the code is correct: both implementations could share the same mistaken scenario definition or algorithmic error. The report should therefore pair this comparison with input checks and targeted validation tests.

Diagnostics, Alternatives, and Limitations

Several warning signs deserve explicit investigation. An estimate with more displayed digits than its Monte Carlo precision can support suggests false precision. A reported interval that excludes the estimate, or a denominator inconsistent with the replication count, signals a reporting or calculation error. A zero-event result with no upper bound can conceal inadequate simulation size. Large discrepancies across independent runs may indicate ordinary Monte Carlo variability when few replications were used, but can also reveal differences in code, random-number handling, scenario inputs, or event definitions. Diagnose the source before treating the runs as interchangeable.

A narrative report is flexible and can explain why assumptions and decisions were chosen, but can be hard to compare across scenarios if results are buried in prose. Tables and visualizations make patterns easier to scan, while a machine-readable results file supports downstream checking. None is a substitute for the others: a useful assurance package combines concise presentation with a detailed specification and archived executable materials. A notebook can combine code and explanation, but its rendered output alone may not identify the software environment or preserve a reliable execution record.

Reproducibility also has limits. Exact reruns may not be possible across different hardware, software libraries, or parallel execution strategies. Independent implementations can agree while sharing incorrect assumptions. Simulation results assess the specified model and scenarios; they cannot, by themselves, establish that those assumptions represent the future trial or that an unexamined scenario is harmless. State these limits alongside the decision rather than implying that a reproducible computation guarantees a valid design.

Common Mistake

Common mistake: “The estimated success probability is 0.820, which exceeds 0.800, so the design is assured to meet the target.”

Why it is wrong: The statement treats a finite simulation estimate as exact and can also blur Monte Carlo uncertainty with uncertainty about the scenario assumptions. In the worked example, a prespecified lower-bound criterion is met because the approximate Monte Carlo interval’s lower endpoint is about 0.809. That conclusion is conditional on the simulated scenario, the implementation, and the validity of the uncertainty calculation. Report those conditions and the evidence supporting them.

A compact, defensible conclusion identifies the criterion, the result relevant to it, the uncertainty, and the limits of the inference. It should allow a reviewer to trace the decision back through the scenario, code version, checks, and simulation output without relying on undocumented analyst memory.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions.

A simulation has 300 successes in 400 independent replications. What is the estimated success probability and its Monte Carlo standard error?
Why does observing no failures in a finite simulation not establish a failure probability of zero?
List two reproducibility details, beyond the random seed, that should be recorded.
Two independent runs give estimates that differ by less than one estimated standard error of their difference. What does that suggest, and what does it not prove?
A report gives a success estimate but omits the event definition and scenario assumptions. What is missing for independent review?

Solutions. (1) \(\hat{\theta}=300/400=0.7500\). Its Monte Carlo standard error is \(\sqrt{0.75(0.25)/400}=\sqrt{0.00046875}\approx0.02165\). (2) A finite number of replications can miss a rare event; report an appropriate upper bound or interval and the replication count. (3) Examples include code and package versions, a version-control identifier, the input-file checksum, the random-number generator, and parallelization settings. (4) The difference is compatible with Monte Carlo variation under the assumed independent-run setup; it does not validate shared assumptions or rule out a common coding error. (5) The reader needs the precise simulated event and the scenario inputs, as well as the link from those definitions to the reported estimate, to reproduce and interpret the result.