Reporting Model-Based Simulations With Assumptions and Limits

Why Simulation Reports Need More Than a Prediction

A statement such as “the simulated concentration was adequate” leaves essential questions unanswered: adequate for which outcome, at what time, in which population, and under what parameter assumptions? A useful report lets readers trace a prediction back to the objective, model, inputs, and simulation procedure that produced it. It also distinguishes uncertainty in the model-based result from variability among simulated individuals and from the numerical error of using a finite number of simulations.

This tutorial concerns how to report those elements, not how to establish that a model is true. A simulated prediction is conditional on the model and inputs used. Reporting the conditions makes the result interpretable and reproducible; it does not turn the prediction into an observation or remove the assumptions behind it.

Specify the Question and the Simulation Inputs

State the objective as a question that can be answered by the simulation. Name the population or simulated unit, the intervention or regimen, the time horizon, the outcome, and any threshold used to classify an outcome. “Compare predicted trough concentration at steady state for two regimens in adults with the stated covariate distribution” is more informative than “simulate exposure.”

Parameter reporting should make clear which values are fixed, which vary between simulated individuals, and which are varied to represent parameter uncertainty. Give sources and scales: for example, a clearance estimate from the fitted model in L/h, a between-subject variability estimate on the log scale, or a parameter distribution obtained from a stated uncertainty procedure. If values come from different sources, identify each source rather than presenting the input set as if it were a single observed data set.

A scenario is a complete, interpretable set of simulation conditions. List the regimen, relevant covariates or population characteristics, model structure, and other inputs that differ across scenarios. State what remains the same. That distinction matters: if two scenarios differ in dose and renal function, a difference in predictions cannot be attributed to dose alone.

State Which Uncertainty the Simulation Represents

“Uncertainty” can refer to different quantities, so state what was varied and what a reported interval summarizes. A conditional prediction holds the model and parameters fixed. A population predictive distribution may represent between-subject variability and, if specified, residual variability. Parameter uncertainty represents uncertainty in estimated model parameters, commonly by drawing parameter sets from a stated distribution or procedure. These components can be included together, but their inclusion should not be assumed from the word “simulation.”

A separate issue is Monte Carlo error: finite simulation runs approximate the result implied by the simulation mechanism. For a target-attainment proportion, this numerical error can be quantified from the number of simulated outcomes. That interval describes precision of the Monte Carlo estimate under the specified simulation mechanism; it is not automatically an interval for the true clinical target-attainment probability, nor does it incorporate omitted model uncertainty.

For a proportion, the Monte Carlo calculation follows from the simulated indicators. Let \(I_j=1\) if simulation \(j\) meets the target and \(I_j=0\) otherwise. If the \(M\) simulations are independent draws under the same specified mechanism, then each indicator has modeled success probability \(q\), and the estimated proportion is \(\hat{q}=\sum_{j=1}^M I_j/M\). Since a Bernoulli indicator has variance \(q(1-q)\), independence gives:

This standard error estimates Monte Carlo error for the simulated proportion. A Wilson interval is often preferable to a simple normal interval, especially when proportions are near zero or one or the number of simulations is modest. For a 95% Wilson interval, use \(z=1.96\), \(M\) simulations, and the observed proportion \(\hat{q}\):

The independent, identically distributed indicator assumption is important for this calculation. If simulations are weighted, clustered, or dependent, this binomial standard error and interval may not be appropriate without adapting the calculation to the simulation design.

Compare Scenarios Without Hiding Their Differences

A scenario comparison is informative when the reader can see both the inputs and the outputs. Use a table or parallel prose to show the common assumptions, the inputs that differ, and the same outcome summary for each scenario. Report differences in the outcome’s natural units: for proportions, percentage points; for concentrations, the concentration unit; for time-to-event predictions, a time unit.

When comparing scenarios, using the same simulated individuals or parameter draws can make the comparison more focused. This is often called a paired or common-random-number approach: each simulated unit is evaluated under both scenarios, so the contrast is based on within-unit differences. If scenarios are simulated independently, say so. Marginal scenario proportions alone do not reveal the uncertainty of a paired difference; that calculation requires the paired outcomes or their differences.

Reproducibility, Diagnostics, and Common Reporting Failures

A reader should be able to identify enough detail to reproduce the simulation or understand why exact reproduction is unavailable. Report the software and relevant version, simulation code or algorithm, number of replicates, random-number seed when applicable, parameter inputs and sources, scenario definitions, and summaries calculated. A seed alone is not sufficient if the parameter inputs, code, or random-number implementation are missing.

Check both the model output and the simulation process. Confirm that units and time scales agree, generated values are plausible under the model, and calculated outcomes follow the stated endpoint definition. Look for unexpected missing or non-finite outputs, implausible parameter draws, and a target proportion that changes materially when the number of replicates increases. Where parameter uncertainty is sampled, inspect whether draws obey the specified parameterization and constraints. If a simulation fails or requires an unplanned change, report the deviation and its effect on the result rather than silently replacing the planned procedure.

A useful comparison is between a single deterministic prediction and a simulation distribution. The deterministic result is simpler and can isolate how a specified input set maps through the model, as in the concentration calculation above. It does not describe a distribution across individuals or parameter values. Simulation can represent those distributions, but only when the relevant variation is explicitly encoded and adequately summarized. Neither approach removes dependence on the model assumptions.

• Pitfall: “The 95% interval means there is a 95% probability the true clinical rate is in this range.” If the interval is a Wilson interval from simulated indicators, it describes Monte Carlo uncertainty for the modeled proportion under the simulation mechanism. It does not automatically quantify clinical sampling uncertainty or all model uncertainty.
• Pitfall: “The higher simulated proportion proves the regimen is better.” The comparison establishes a difference between model-based predictions under specified inputs. It does not establish a causal or clinical conclusion beyond those assumptions.
• Pitfall: “We varied all important parameters.” This is not reproducible unless the report names the parameters, their sources and distributions, and how they were sampled.
• Pitfall: reporting only a mean prediction. A mean can conceal the spread, tail behavior, or fraction crossing a decision threshold. Choose summaries that answer the stated objective and report their units.