Simulation Code and Operating Characteristic Review

Why Review Simulation Code and Operating Characteristics?

A trial simulation is evidence about how a specified design and analysis procedure may perform under specified assumptions. Its reported success probability, type I error, bias, or coverage is only informative if the code implements the design that was described and the simulated scenarios are relevant to the decision. A plausible-looking summary table cannot establish either point by itself.

The review therefore has two connected tasks: trace the written trial rules into the code, and assess whether the reported simulation summaries are precise and appropriately scoped. The target is not to prove that a simulation predicts a future trial exactly. It is to determine whether the computational experiment answers its stated question, and whether its limitations are visible.

Definition: An operating characteristic is a performance measure of a specified design and analysis procedure under a specified data-generating scenario. Examples include the probability of meeting a success criterion, the probability of a false positive when a null hypothesis is true, the bias of an estimator, and the coverage probability of an interval procedure. The scenario, design, endpoint, analysis, and performance measure together define what is being evaluated.

For example, a reported probability of trial success is not a property of the drug alone. It depends on such features as the assumed treatment effect, patient variability, dose assignment, dropout, endpoint definition, analysis method, and success rule. A review should make those dependencies explicit before interpreting the reported number.

Trace the Trial Rules into the Simulation

Start from the protocol or design description, not from a code summary. For each rule, identify where it is implemented and how a reviewer could verify it. A clear audit trail connects the rule, code component, and observed check. The following sequence is useful:

Confirm the scenario and design.
Check population size, randomization ratio, treatment arms, stratification, visit schedule, and duration against the stated design. Verify that simulated participants are assigned and followed as intended.

Trace dosing and exposure inputs.
Check dose amount, route, frequency, titration, interruptions, adherence, and dose changes. Confirm that the dosing history passed to any exposure or outcome model follows those rules and uses consistent units.

Check dropout and endpoint generation.
Compare dropout timing and determinants with the stated assumptions. Verify that the endpoint is generated at the specified time and on the specified scale, with the intended handling of intercurrent events and missing values.

Match the analysis and decision rule.
Confirm that the analysis uses the intended population, endpoint, covariates, missing-data approach, test or estimator, and threshold for success. Check that the simulation records success using that same rule.

Reconcile summaries and quantify simulation error.
Recompute selected trial-level results and operating-characteristic summaries independently. Check replicate counts, Monte Carlo standard errors, and whether conclusions are stable enough for their stated purpose.

The checks should include simple invariants that do not depend on a model fit: number randomized, allocation totals, number of scheduled doses, follow-up time, dropout counts, and analysis denominators. These checks can expose errors that aggregate performance summaries conceal.

Worked Example: Detecting a Dosing Mismatch

Suppose a hypothetical parallel-group trial is described as randomizing 240 participants equally to placebo or 100 mg of an oral drug once daily for 84 days. A reviewer checks the simulation's dosing function and finds that it creates two 100 mg administrations per day in the active arm.

The intended scheduled active dose per participant is \(100\text{ mg/day}\times84\text{ days}=8{,}400\text{ mg}\). The code schedules \(200\text{ mg/day}\times84\text{ days}=16{,}800\text{ mg}\), twice the intended total. With 120 active-arm participants and no interruptions, the corresponding total scheduled amounts are \(120\times8{,}400=1{,}008{,}000\text{ mg}\) under the design and \(120\times16{,}800=2{,}016{,}000\text{ mg}\) under the code.

This is a design-implementation discrepancy, not a small numerical difference. Any exposure or outcome results that depend on dose may therefore be based on a different trial from the one described. The reviewer should request correction or a justified design clarification, then rerun the simulation and verify the dosing totals with a simple test case. The reported operating characteristics from the mismatched version do not answer the stated design question.

Operating Characteristics and Monte Carlo Precision

A simulation estimates an operating characteristic by repeatedly generating trial data under a scenario and applying the specified analysis. For a binary event such as “the trial meets its success criterion,” let \(I_r=1\) if replicate \(r\) meets that criterion and \(I_r=0\) otherwise. With \(R\) independent replicates, the estimated probability is the proportion of successful replicates.

\hat{p}_{\mathrm{sim}}=\frac{1}{R}\sum_{r=1}^{R}I_r

If the true probability under the simulation scenario is \(p_{\mathrm{sim}}\), the replicate indicators are Bernoulli outcomes with variance \(p_{\mathrm{sim}}(1-p_{\mathrm{sim}})\). The standard error of their average is therefore approximately estimated by the Monte Carlo standard error (MCSE):

\widehat{\mathrm{MCSE}}(\hat{p}_{\mathrm{sim}}) = \sqrt{\frac{\hat{p}_{\mathrm{sim}}(1-\hat{p}_{\mathrm{sim}})}{R}}

This formula assumes independent replicates generated under the same scenario and evaluated using the same event definition. It measures simulation error in estimating the scenario-specific probability; it does not measure uncertainty about whether the scenario assumptions are correct. For an estimated mean from replicate-level values with sample standard deviation \(s\), the corresponding MCSE is \(s/\sqrt{R}\), provided the replicate values are independent and identically distributed with finite variance.

A Short Justification for Replicate Counts

The replicate-count calculation follows from the variance of an average. For independent indicators, \(\mathrm{Var}(I_r)=p_{\mathrm{sim}}(1-p_{\mathrm{sim}})\). The variance of their average is the variance of one indicator divided by \(R\), so its standard error is \(\sqrt{p_{\mathrm{sim}}(1-p_{\mathrm{sim}})/R}\). Replacing the unknown probability by a planning value \(p_0\), and requiring an approximate MCSE no larger than a target \(\delta\), gives:

\sqrt{\frac{p_0(1-p_0)}{R}}\leq\delta \quad\Longrightarrow\quad R\geq\frac{p_0(1-p_0)}{\delta^2}

This is a planning approximation, not a guarantee of precision for every possible result. When the probability is unknown, using \(p_0=0.5\) gives the largest Bernoulli variance and is conservative for a fixed \(R\). If the event is rare, the observed count of events matters as well as the nominal replicate count: a large number of replicates can still yield few successes and relatively imprecise tail estimates.

Worked Example: Reviewing a Simulated Success Probability

In a hypothetical review, 94 of 2,000 independent simulated trials meet a prespecified success rule when the scenario has no treatment effect. The estimated probability of meeting the rule is \(94/2{,}000=0.0470\), or 4.70%. If meeting the rule constitutes a false positive under this scenario, this estimates the procedure's type I error for that scenario.

The estimated MCSE is:

\sqrt{\frac{0.0470(1-0.0470)}{2{,}000}} = \sqrt{\frac{0.0470(0.953)}{2{,}000}} = \sqrt{0.0000223955} \approx 0.004732

Thus, the Monte Carlo standard error is about 0.0047, or 0.47 percentage points. A rough 95% Monte Carlo interval, using the normal approximation \(\hat{p}_{\mathrm{sim}}\pm1.96\,\widehat{\mathrm{MCSE}}\), is \(0.0470\pm1.96(0.004732)\), or approximately 0.0377 to 0.0563. This interval describes simulation error around the probability for the specified scenario; it is not a range of plausible treatment effects.

If the review target is an MCSE of at most 0.002 and the planning probability is 0.0470, then:

R\geq\frac{0.0470(0.953)}{(0.002)^2} =\frac{0.044791}{0.000004} =11{,}197.75

The replicate count must be a whole number, so at least 11,198 replicates are needed by this approximation; about 11,200 is a practical rounded target. Reporting 2,000 replicates without discussing its MCSE would not show whether the result is precise enough for the intended comparison.

Check Dropout, Endpoint, and Analysis Rules Together

Dropout generation and missing-data analysis cannot be audited as unrelated code blocks. The data-generating process determines which endpoint values are observed, while the analysis determines how those observations contribute to the decision. The simulation must reflect the stated estimand and analysis rule, including any rule for participants who discontinue treatment or lack a scheduled measurement. A mismatch can bias an operating characteristic even when each code component appears reasonable in isolation.

Useful diagnostics include tabulating dropout by arm and visit, checking the number of endpoint records and analysis-set members in each replicate, and comparing the recorded success flag with an independent implementation of the success rule. Review a small number of trial-level datasets, not only aggregated summaries. A simulation that reports the right average number of dropouts could still place them at the wrong visits or use the wrong denominator.

Worked Example: Verifying a Dropout Endpoint Rule

Consider a hypothetical trial with 10 randomized participants per arm. Its stated primary endpoint is a binary week-12 responder outcome, and the rule is that a participant who drops out before week 12 is counted as a nonresponder. In one simulated replicate, the active arm has two early dropouts and six responders among the eight participants who complete follow-up. In the control arm, one participant drops out and four of the nine completers respond.

Under the stated rule, all randomized participants remain in the denominator. The active-arm response proportion is \(6/10=0.60\), and the control-arm response proportion is \(4/10=0.40\). The observed difference is \(0.60-0.40=0.20\), or 20 percentage points in favor of active treatment for this replicate.

If code instead excludes dropouts, it reports \(6/8=0.75\) and \(4/9\approx0.444\), giving a difference of approximately \(0.306\), or 30.6 percentage points. That is not the stated endpoint rule: the changed denominators produce a different analysis. The reviewer should inspect the participant-level endpoint flags, confirm the denominators in the analysis, and verify that the simulated success indicator uses the specified procedure. This single replicate is a code check, not evidence about the average treatment effect or overall trial performance.

Are the Scenarios and Replicate Counts Adequate?

A large \(R\) improves Monte Carlo precision only for the scenarios that were actually simulated. It cannot compensate for scenarios that omit a decision-relevant condition. Reviewers should ask whether scenarios cover the plausible range of treatment effects, patient characteristics, variability, dropout behavior, and other assumptions that materially affect the operating characteristic. The scenario set should also include cases directly tied to the claim—for example, a null case when evaluating false-positive control and clinically relevant effects when evaluating the chance of success.

For each reported operating characteristic, record the scenario, replicate count, replicate-level outcome, estimator, MCSE or another suitable precision summary, and any decision threshold used. Confirm that scenario labels and result summaries correspond. When performance is compared across scenarios, precision should be adequate for the comparison being made; an MCSE small enough for a broad descriptive summary may be too large to distinguish two close values.

There are alternatives to Monte Carlo estimation in some settings. An exact calculation or analytic approximation can be more efficient when the design and outcome distribution allow it. Simulation is more flexible for complex designs, nonlinear analyses, dropout, and decision rules, but introduces Monte Carlo error and code-validation needs. These approaches can be complementary: an analytic result or a small hand-calculated test case can serve as a benchmark for part of the code, while simulation evaluates the full procedure.

Diagnostics, Failure Modes, and Limits

Common warning signs include unexplained differences between intended and simulated allocation; impossible values or units; unexpected dose totals; dropout rates or timing inconsistent with the scenario; endpoint values recorded after dropout when they should be missing; analysis denominators that do not match the stated rule; and success counts that cannot be reproduced from replicate-level outputs. Also check that all declared scenarios and replicates were completed, and that failed fits or numerical errors are handled and reported consistently rather than silently discarded.

An audit should distinguish three questions. First, is the code doing what the specification says? Second, is the specification itself a defensible representation of the trial and analysis? Third, are the reported estimates precise enough for the conclusion? Code review can address the first and third directly, but cannot establish that assumptions about future patients or treatment effects are true. A simulation’s conclusion must remain conditional on its scenarios and model assumptions.

Wrong reasoning: “The simulation used 10,000 replicates, so its conclusions are reliable.”
Correction: A high replicate count can reduce Monte Carlo error, but it does not repair a dosing mismatch, an incorrect endpoint rule, or an unrepresentative scenario set. Review implementation and scenario relevance as well as MCSE.
Wrong reasoning: “The estimated success probability is 0.047, so the true probability is exactly 4.7%.”
Correction: The estimate has Monte Carlo error and is conditional on the simulated scenario. Report its precision and explain the conditions under which it applies.

Key takeaway: A credible operating-characteristic result requires traceable implementation of the stated trial and analysis, scenarios that address the decision question, and enough independent replicates to quantify simulation error at a useful level. None of these checks substitutes for the others.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions below.

A simulation reports 40 successes in 1,000 replicates. What is the estimated success probability, and what is its approximate MCSE?
A protocol specifies 80 mg once daily for 30 days, while code schedules 80 mg twice daily. What scheduled total dose per participant does each imply, and what should a reviewer conclude?
Under a stated rule that early dropout is nonresponse, 12 of 20 randomized participants respond, including no early dropouts counted as responders. What response proportion should be used?
Can increasing the number of replicates correct a scenario set that omits a relevant high-dropout condition? Explain.

Solutions. 1. The estimate is \(40/1{,}000=0.0400\). Its approximate MCSE is \(\sqrt{0.0400(0.960)/1{,}000}=\sqrt{0.0000384}\approx0.00620\), or 0.62 percentage points. 2. The stated schedule gives \(80\times30=2{,}400\) mg; the code gives \(160\times30=4{,}800\) mg. The code does not implement the stated design, so results from it should not be presented as performance of that design until the discrepancy is resolved and the simulation rerun. 3. The denominator is all 20 randomized participants, so the proportion is \(12/20=0.60\), or 60%. 4. No. More replicates can reduce Monte Carlo error within the simulated scenarios, but cannot provide evidence for a condition that was not simulated. The scenario set must be reconsidered separately.