Reporting a Reproducible Design Simulation Study

Why Reporting Is Part of the Simulation

A design simulation can be carefully coded and still be difficult to evaluate if its report omits what generated the data, which designs were compared, or how failed analyses were handled. A reproducible report makes those decisions inspectable. It allows a reader to understand what the reported performance means, assess whether the assumptions fit the intended application, and rerun the calculation when the software and computational details are available.

Reproducibility does not mean that a simulation result is certain or that its assumptions are true. It means that the study describes its computational and statistical choices sufficiently for another analyst to implement the same procedure, and that it reports the remaining Monte Carlo error and any departures from the planned procedure. The report should distinguish the question answered by the simulation from the evidence needed to apply its results to practice.

What Another Analyst Needs to Reconstruct

The report should make the simulation a sequence of specified operations rather than a general description. State what was generated for each replicate, how each candidate design determined the observations, what analysis was applied, and how the resulting performance measure was calculated. Include fixed inputs and decision rules, not merely the name of a model or software package.

• Data-generating assumptions: name the model, parameter values, distributions, dependence structures, covariates, residual error, and any transformations. Identify which quantities are fixed and which vary across individuals or replicates. State how missingness, censoring, measurement limits, or other relevant features were generated—or state that they were absent.
• Scenarios: define each scenario by its changed assumptions and list the values used. Explain whether scenarios represent alternative plausible settings, stress tests, or other specified conditions. Avoid leaving “low,” “typical,” or “high” undefined.
• Candidate designs: give the planned sampling times, sample sizes, dose or observation schedule, allocation, and other features that distinguish the designs. If a design is adaptive, state the decision rule, the information available at each decision, and any constraints.
• Analysis procedure: specify the model fitted, estimation method, initialization, bounds, convergence criteria, and how estimates or performance measures were obtained. Report relevant software and version information and any custom implementation choices.
• Simulation execution: report the planned and completed replicate counts for each scenario and design, random-number generator and seed or stream strategy, code availability, and relevant computational environment. These details help readers understand the implementation; a seed alone does not describe the study.
• Results and exclusions: define each performance measure, its units, its denominator, and how failed fits or invalid outputs were handled. Present results by scenario and design, with an appropriate measure of Monte Carlo error.

For a pharmacometric illustration, consider an explicitly hypothetical comparison of two concentration-sampling schedules following an intravenous bolus. A report might state that concentrations were generated from a one-compartment model, \(C_i(t)=D/V_i\exp[-(CL_i/V_i)t]\), with dose \(D=100\) mg, typical clearance \(CL=5\) L/h, and typical volume \(V=50\) L. It would then specify any between-individual variability, residual error, and their distributions and parameter values. For example, if individual clearance and volume were lognormally distributed, the report would give the standard deviations of their log-scale random effects and their correlation. It would list the two schedules as actual sampling times in hours, state the number of individuals assigned to each design, and name the fitted model and estimation procedure. Reporting only “a population PK model with typical variability” would not permit reconstruction.

Replicate Counts and Monte Carlo Error

A simulated performance estimate is itself subject to Monte Carlo error because it is calculated from a finite number of random replicates. If a performance measure is a probability, such as coverage or successful estimation, define \(I_r=1\) when replicate \(r\) meets the stated criterion and \(I_r=0\) otherwise. With \(R\) independent replicates, the estimated probability is \(\hat p=R^{-1}\sum_{r=1}^{R}I_r\). Its estimated Monte Carlo standard error (MCSE) is

This expression follows because each \(I_r\) has estimated variance \(\hat p(1-\hat p)\), and the variance of the mean of \(R\) independent indicators is that variance divided by \(R\). Taking the square root gives the estimated standard deviation of the simulation estimate across repeated runs of the same simulation. Independence here refers to the replicates used for that estimate; the report should describe any design that induces dependence, such as pairing candidate designs within replicate.

For a continuous performance measure \(Y_r\), report its replicate mean and the sample standard deviation \(s_Y\) across replicates. The estimated MCSE of the mean is \(s_Y/\sqrt{R}\), when the replicate-level values are independent and have finite variance. For paired design comparisons, define the replicate-level difference first and report its mean and MCSE across replicate differences; this preserves the pairing in the uncertainty calculation.

An MCSE quantifies numerical uncertainty from using finitely many simulation replicates. It does not quantify uncertainty about whether the data-generating assumptions are correct, nor does it establish that simulated performance will match performance in a real population. If a normal approximation is reasonable, an estimate plus or minus about 1.96 MCSEs can summarize its Monte Carlo precision, but the report should identify this as an approximation and not present it as a general-purpose confidence interval for real-world performance.

Failures, Exclusions, and Denominators

A report should distinguish simulations attempted, simulations completed, and simulations included in each summary. State failure categories and counts—for example, optimizer nonconvergence, an invalid parameter estimate, or a software error—and whether each was included, excluded, or assigned a defined outcome. Apply pre-specified rules consistently across designs and scenarios. If rules changed after examining results, describe the change and its consequences rather than presenting the final analysis as if it had been planned from the outset.

Different denominators can answer different questions. A performance summary among successful fits estimates performance conditional on obtaining a successful fit. A summary that treats failed fits as not meeting an estimation criterion includes operational failure in the reported rate. Neither is automatically correct for every purpose; the report must define the target and show enough information for readers to see how failures affect it. Never silently discard failures while reporting a denominator that suggests all attempted replicates were analyzed.

Scenario-Specific Results and Reporting Choices

Present results at the level needed to evaluate the comparisons. A table organized by scenario and candidate design should identify the performance measure, units, replicate count, estimate, MCSE, and any failures or exclusions. State clearly which direction of a measure is preferred. If an overall summary combines scenarios, document its weighting rule and retain scenario-specific results so that the aggregate cannot conceal a setting where performance differs substantially.

A compact table is not a substitute for methods. Explain how measures were computed, how uncertain or invalid outputs were classified, and which results correspond to each scenario. Report relevant deviations from the planned analysis. When several metrics are presented, define each one in the same terms used in the simulations; a label such as “precision” needs an operational definition and units or scale.

Common Reporting Pitfalls and Limits

• Giving a seed but not the simulation specification. A seed cannot reveal the model, candidate schedules, analysis settings, or failure rules. Report the statistical specification and implementation details as well as the random-number strategy.
• Reporting only successful fits. A success-only estimate can overstate overall operational performance if failures are common. Give all relevant counts and label conditional summaries.
• Using a large replicate count as a quality claim. More replicates can reduce MCSE, but they cannot repair misspecified assumptions, an incorrectly implemented model, or an analysis that does not match the intended question.
• Reporting an aggregate without its components. An overall result can obscure scenario-specific behavior. Show the component results and explain any aggregation rule.
• Interpreting MCSE as real-world uncertainty. MCSE describes finite simulation noise under the specified setup. It does not measure uncertainty in the model assumptions or guarantee transportability.

The report is limited by the choices it describes: a perfectly reproducible simulation can still address an unrealistic scenario or omit an important source of variation. Transparency makes those limitations assessable; it does not remove them. A useful final check is whether a reader could write down the generating distributions, implement each candidate design and analysis, determine the denominator of every result, and understand the precision and limitations of each reported estimate.