Reporting Simulation-Based Calibration Results

What a Simulation-Based Calibration Result Describes

A reported operating characteristic is meaningful only relative to the setting that generated the simulated trials and the rule used to classify each trial. For example, a statement that a design has a 70% success rate is incomplete unless the reader knows which treatment effect, event rate, sample size, analysis procedure, and success criterion produced that rate. Reporting should make those ingredients visible, as well as the precision with which the simulation estimated the rate.

Here, simulation-based calibration means evaluating a trial design by generating many hypothetical trials under specified scenarios, applying a specified analysis and decision rule to each trial, and summarizing the resulting decisions. It does not mean that the scenarios are known to represent every future trial. The reported results describe performance under the stated simulation assumptions.

Definition: For a fixed scenario \(s\), let \(Y_{sj}\) be the simulated data from replicate \(j\), and let \(D(Y_{sj})\) equal 1 when the prespecified decision rule declares the event of interest and 0 otherwise. The operating characteristic for that event under scenario \(s\) is \(\theta_s=P\{D(Y)=1\mid s\}\). With \(M_s\) simulation replicates, its estimate is \(\hat{\theta}_s=M_s^{-1}\sum_{j=1}^{M_s}D(Y_{sj})\).

The event can be a beneficial decision, a harmful decision, or another defined outcome of the trial process. Examples include declaring efficacy, declaring efficacy when the treatment effect is null, or meeting a prespecified joint efficacy-and-safety criterion. The label must describe the event actually counted; “success” alone is not sufficiently precise.

What to Specify Before Presenting Estimates

A clear report first identifies the scenario set. For each scenario, describe the data-generating assumptions and parameter values that materially determine the simulated outcomes. Depending on the trial, these may include control and treatment event rates, treatment-effect structure, outcome dispersion, missingness, subgroup composition, accrual, or other design features. If a scenario represents a null or clinically important alternative, label it explicitly and state its defining values. Avoid relying on labels such as “base case” without explaining what they mean.

Next, specify the trial and analysis procedure being evaluated: planned sample size or stopping rule, randomization features, analysis model, prior distributions when relevant, and the decision rule. State the cutoff and inequality exactly. For instance, a rule might declare efficacy when the posterior probability that the treatment effect exceeds zero is greater than 0.975. A rule using “greater than or equal to” 0.975 is a different rule at the boundary and should be reported as such.

For every scenario and every reported event, give the number of replicates used, the number meeting the event definition, the estimated rate, and a measure of Monte Carlo uncertainty. If some simulated trials fail to return an analysis result or have missing outcomes, state how they were handled and give the relevant counts. A denominator that silently excludes difficult or unsuccessful replicates can change the interpretation of the estimate.

Report element	What the reader needs to know
Scenario	Parameter values and data-generating assumptions defining the simulated setting
Trial and analysis	Design features, analysis procedure, and any relevant prior specification
Decision rule	The event counted, its threshold, and whether equality at the threshold qualifies
Replicates and denominator	Planned and analyzed replicate counts, including failures or exclusions
Estimate and precision	Event count, estimated operating characteristic, and Monte Carlo uncertainty
Summary across scenarios	Scenario-specific results plus any averaging weights or worst-case summary

Monte Carlo Uncertainty for a Simulated Rate

Under a fixed scenario, suppose the replicate trials are generated independently and each is assessed using the same prespecified decision rule. Then the indicators \(D(Y_{sj})\) are independent Bernoulli random variables with success probability \(\theta_s\). Their average estimates that probability. The standard error below describes simulation noise: how much the estimate would vary if the simulation were repeated with a new set of replicates under the same assumptions.

\operatorname{Var}(\hat{\theta}_s\mid s)=\frac{\theta_s(1-\theta_s)}{M_s}, \qquad \widehat{SE}_{MC}(\hat{\theta}_s)= \sqrt{\frac{\hat{\theta}_s(1-\hat{\theta}_s)}{M_s}}.

The variance follows because each indicator has variance \(\theta_s(1-\theta_s)\), and the variance of the mean of \(M_s\) independent indicators is that variance divided by \(M_s\). Substituting the observed rate for the unknown \(\theta_s\) gives the estimated Monte Carlo standard error. A rough 95% Monte Carlo interval is \(\hat{\theta}_s\pm1.96\widehat{SE}_{MC}\), but this normal approximation may be inaccurate when the event is rare, very common, or based on few replicates. In those cases, report a Wilson or exact binomial interval instead, or increase the number of replicates.

These intervals quantify simulation error conditional on the scenario and the simulation procedure. They do not include uncertainty about whether the scenario is realistic, whether the analysis model is appropriate, or whether the trial will follow the simulated design. Those are separate limitations and should not be hidden by reporting a narrow Monte Carlo interval.

Worked Example: Efficacy Rate and Monte Carlo Precision

Suppose 2,000 independent hypothetical trials are simulated under a prespecified alternative scenario. In each trial, efficacy is declared if the posterior probability that the treatment effect is greater than zero exceeds 0.975. The rule is applied to all 2,000 trials, and 1,260 meet it. Estimate the efficacy-decision rate and its Monte Carlo standard error.

The decision rule and scenario are fixed, and each replicate contributes one binary indicator. Thus \(M=2{,}000\), and the estimated rate is

\hat{\theta}=\frac{1{,}260}{2{,}000}=0.630.

The estimated Monte Carlo standard error is

\widehat{SE}_{MC} =\sqrt{\frac{0.630(1-0.630)}{2{,}000}} =\sqrt{0.00011655} =0.0108.

A rough 95% Monte Carlo interval is \(0.630\pm1.96(0.0108)\), or approximately \(0.609\) to \(0.651\). The report can say that the simulated efficacy-decision rate under this stated alternative scenario was 63.0%, with Monte Carlo standard error 1.08 percentage points. The interval describes simulation precision under the scenario; it is not a probability statement that the scenario’s true operating characteristic lies in this particular interval.

Rare Events, Denominators, and Failed Replicates

The normal-approximation interval is especially unreliable when the estimated rate is near zero or one. For a binomial event count \(K\) from \(M\) replicates, a Wilson interval is one practical alternative. It avoids the possibility of an interval extending below zero or above one, which can occur with the simple normal approximation. A zero observed count also does not establish that the event has probability zero: it may reflect a rare event that was not seen in the available replicates.

Worked Example: Reporting a Rare False-Positive Decision

Under a null scenario, suppose a trial is simulated 500 times using a fixed rule that declares efficacy when the posterior probability of a positive treatment effect exceeds 0.975. Nine trials meet that rule. The false-positive decision rate is the fraction of simulated null trials meeting the efficacy rule.

The point estimate and estimated Monte Carlo standard error are

\hat{\theta}=\frac{9}{500}=0.018, \qquad \widehat{SE}_{MC} =\sqrt{\frac{0.018(1-0.018)}{500}} =0.00595.

A normal interval would be \(0.018\pm1.96(0.00595)\), with a lower endpoint below zero, so it is not a sensible interval for this rate. Using a 95% Wilson interval with \(z=1.96\), the interval is approximately 0.0095 to 0.0338. The appropriate contextual report is that 9 of 500 simulated trials under the stated null scenario triggered the efficacy rule, an estimated false-positive decision rate of 1.8%; the Wilson interval expresses the associated binomial Monte Carlo uncertainty. The result does not show that the rate is exactly 1.8%, nor does it establish performance under null scenarios that were not simulated.

The denominator deserves equal attention. If, for example, 12 of 500 simulated analyses fail to converge and the report uses only the 488 successful analyses, then \(9/488\) estimates the event rate among successful analyses, not necessarily among all simulated trials. Report the failures and specify whether they count as non-events, are treated as a separate outcome, or are excluded for a justified reason. If failure is related to the simulated data or decision, exclusion may distort the operating characteristic.

Scenario Summaries and What They Mean

Scenario-specific estimates should generally remain visible even when an overall summary is supplied. A weighted average has an interpretation only when its weights are stated and the target scenario mixture is clear. If simulations in separate scenarios are independent, the estimated Monte Carlo variance of a weighted average is the sum of the scenario variances multiplied by the squared weights. This calculation describes simulation precision for that average, not uncertainty in the choice of weights or scenarios.

Worked Example: Reporting a Weighted Rate Across Two Scenarios

Suppose two prespecified scenarios have target weights 0.70 and 0.30. In independent simulations, the first has 720 efficacy decisions in 1,000 replicates, and the second has 540 in 1,000. The overall summary is intended to represent this particular weighted mixture, not an equal-weight average.

The scenario-specific estimates are \(0.720\) and \(0.540\). The weighted estimate is

\hat{\theta}_{w}=0.70(0.720)+0.30(0.540)=0.666.

Under independent simulation across scenarios, its estimated Monte Carlo variance is

\widehat{\operatorname{Var}}_{MC}(\hat{\theta}_{w}) =0.70^2\frac{0.720(0.280)}{1{,}000} +0.30^2\frac{0.540(0.460)}{1{,}000} =0.00012114.

Therefore, the Monte Carlo standard error is \(\sqrt{0.00012114}=0.0110\). A rough 95% Monte Carlo interval is \(0.666\pm1.96(0.0110)\), or about 0.644 to 0.688. A transparent report gives all three scenario figures—the two component rates and the weighted summary—and states the 0.70 and 0.30 weights. The unweighted average, \((0.720+0.540)/2=0.630\), answers a different question and should not be substituted without justification.

Diagnostics, Alternatives, and Common Mistakes

Before interpreting a calibration table, check that every estimate has a defined event, scenario, and denominator. Confirm that the stated decision rule matches the rule implemented in the simulation, including treatment of boundary values. Check replicate counts for every row, inspect whether failures or missing outcomes differ across scenarios, and compare Monte Carlo standard errors with the precision needed for the decision being informed. If two candidate designs have similar estimated rates but uncertainty is large relative to their difference, the simulations may not support a confident ranking.

A scenario-by-scenario table is more informative about variation across settings than a single aggregate number. An aggregate can be useful when it answers a clearly defined target question, but it can mask poor performance in one scenario. A worst-case summary and a weighted average also answer different questions; neither should replace disclosure of the underlying scenario results. Likewise, a simple normal Monte Carlo interval is convenient for moderate rates and adequate replicate counts, while Wilson or exact binomial intervals are preferable for sparse events. These are alternative ways to express simulation precision, not alternative decision rules for the trial itself.

Reporting a rate without its scenario. A 70% rate has no stable interpretation without the conditions under which it was estimated. Name the scenario and its relevant parameters.
Reporting only the point estimate. An estimate based on 100 replicates can be much noisier than the same estimate based on 10,000. Give the replicate count and Monte Carlo uncertainty.
Calling Monte Carlo uncertainty total uncertainty. A precise simulation can still rely on an unrealistic data-generating model. Discuss limitations of the scenarios separately.
Silently dropping failed fits. The resulting rate describes only the retained analyses unless the treatment of failures is explained. Report their number and handling.
Giving an aggregate without its weights. The summary then has no reproducible target interpretation. State the weights and retain scenario-specific results.

Key takeaway: A transparent calibration result links each operating-characteristic estimate to its data-generating scenario, replicate denominator, implemented decision rule, and Monte Carlo precision. It also distinguishes simulation error from uncertainty about the assumptions used to simulate trials.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions below.

In 1,000 independent simulated trials, 400 meet a prespecified decision rule. What are the estimated rate and Monte Carlo standard error?
Why should a report state whether equality at a decision threshold qualifies?
Under a null scenario, no false-positive decisions occur in 200 replicates. What can and cannot be concluded?
Two scenario-specific rates are 0.80 and 0.50. Their target weights are 0.25 and 0.75. Calculate the weighted rate and explain why the equal-weight mean is not the requested summary.
What should be reported if some simulated trial analyses fail and the failures are excluded from the rate denominator?

Solutions. (1) The estimated rate is \(400/1{,}000=0.400\). Its Monte Carlo standard error is \(\sqrt{0.400(0.600)/1{,}000}=0.0155\), or 1.55 percentage points. (2) Rules using \(>\) and \(\geq\) classify trials exactly at the threshold differently, so the reported rule must match the implemented rule. (3) The observed rate is 0/200, but the true operating characteristic is not proven to be zero; the finite simulation may simply not have observed a rare event. Report the denominator and use an interval suited to a zero count. (4) The weighted rate is \(0.25(0.80)+0.75(0.50)=0.575\). The equal-weight mean, 0.650, uses different weights and therefore answers a different question. (5) Report the number and handling of failed analyses, identify the denominator used, and clarify that the resulting estimate is conditional on inclusion if failures were excluded. Discuss whether exclusion could affect interpretation.