Why the Display Is Part of the Analysis
A design assurance simulation may produce many operating-characteristic estimates across scenarios: for example, the probability of detecting an effect, the probability of a false positive, or the expected sample size. A summary that reports only one overall number can hide a design that performs poorly in an important scenario. Conversely, a large table of numbers can make important contrasts difficult to see. Good displays make the scenario structure, performance targets, and precision of the simulation estimates visible together.
The display should distinguish two kinds of variation. Scenario variation is the difference in performance across the assumptions being examined. Monte Carlo uncertainty is the simulation error in estimating performance within a scenario from a finite number of simulated trials. Neither should be mistaken for uncertainty in the clinical effect or other scenario inputs themselves. The earlier material on reporting simulation assumptions and results provides the context for presenting the assumptions alongside these displays.
What a Scenario-Level Table Should Show
A scenario-level table is often the most reliable reference display because readers can retrieve exact values. Use one row per scenario, or one row per scenario and design when comparisons are needed. Label scenarios with meaningful, concise descriptions; avoid relying on opaque codes unless a key is immediately available. Include the quantities needed to interpret both the estimate and its precision.
- Scenario assumptions: identify the features that distinguish rows, such as event rate or degree of treatment-effect dilution.
- Performance estimate: state the measure and units, such as power as a proportion or expected sample size in participants.
- Simulation information: give the number of simulated trials and a Monte Carlo standard error or interval.
- Benchmark: state the target and whether it is a minimum, maximum, or reference value.
- Unfavorable cases: retain scenarios with poor performance in the main display rather than relegating them to a footnote or omitting them.
A table should not imply more precision than the simulation supports. Match decimal places to the Monte Carlo uncertainty, and report the count behind a proportion when it helps readers understand the estimate. If scenario-weighted averages or worst-case summaries are useful, show them as additional summaries, not substitutes for the scenario-level results.
Worked Example: Tabulating Simulated Power Across Scenarios
Suppose a hypothetical design is evaluated in four scenarios, with 2,000 independently simulated trials in each. The number meeting the prespecified success rule is 1,640, 1,540, 1,280, and 980, respectively. The target power is 0.80. For a simulated proportion \(\hat{p}\) based on \(M\) independent trials, estimate its Monte Carlo standard error by \(SE=\sqrt{\hat{p}(1-\hat{p})/M}\). A large-sample 95% Monte Carlo interval is \(\hat{p}\pm1.96SE\).
For the first scenario, \(\hat{p}=1640/2000=0.8200\). Thus \(SE=\sqrt{0.8200(0.1800)/2000}=\sqrt{0.0000738}=0.008591\). The interval is \(0.8200\pm1.96(0.008591)=0.8200\pm0.01684\), or approximately \([0.8032,0.8368]\).
Applying the same calculation to all four scenarios gives the following report. The intervals are approximate Monte Carlo intervals, not intervals for clinical efficacy in future patients.
| Scenario | Successes / simulations | Estimated power | Monte Carlo SE | Approximate 95% interval | Target 0.80 |
|---|---|---|---|---|---|
| Reference assumptions | 1,640 / 2,000 | 0.8200 | 0.008591 | 0.8032 to 0.8368 | Above |
| Moderate dilution | 1,540 / 2,000 | 0.7700 | 0.009410 | 0.7516 to 0.7884 | Below |
| Lower event rate | 1,280 / 2,000 | 0.6400 | 0.01073 | 0.6190 to 0.6610 | Below |
| Severe dilution | 980 / 2,000 | 0.4900 | 0.01118 | 0.4681 to 0.5119 | Below |
The first scenario's estimate is above target, while the other three estimates are below it. The table makes the unfavorable scenarios as visible as the reference case. The interval widths also show that the simulation has estimated each proportion with finite precision. They do not account for uncertainty about whether these four scenarios adequately represent possible trial conditions.
Why the Monte Carlo Standard Error Has This Form
For a binary success indicator in one simulated trial, let \(Y_i=1\) if the trial meets the success rule and \(Y_i=0\) otherwise. Under a fixed scenario, assume \(M\) independent simulations are generated by the same design and \(P(Y_i=1)=p\). Then \(\hat{p}=\sum_{i=1}^{M}Y_i/M\). A Bernoulli variable has variance \(p(1-p)\); independence therefore gives the variance of the average:
Because \(p\) is unknown, replace it with \(\hat{p}\) to estimate the standard deviation of \(\hat{p}\), giving \(SE=\sqrt{\hat{p}(1-\hat{p})/M}\). When the simulated success and failure counts are both sufficiently large, the sampling distribution of \(\hat{p}\) is approximately normal, which motivates the interval \(\hat{p}\pm1.96SE\). This is a measure of simulation precision under the stated scenario and simulation procedure. It is not a probability that the true operating characteristic lies in the reported interval.
The approximation can be unreliable when success is very rare or nearly certain, particularly when the expected count of either outcome is small. In those settings, do not rely mechanically on the symmetric normal interval: use a suitable binomial interval, such as a Wilson interval, or increase simulation effort. For weighted simulation estimates, the unweighted binomial standard-error formula does not apply; use an uncertainty calculation appropriate to the weighting method.
Choosing a Plot That Preserves the Scenario Story
A plot should make the principal comparison easier to see than a table while keeping the values interpretable. For proportions or other point estimates, a dot-and-interval plot is often effective: place the estimate on a labeled numerical axis, show its Monte Carlo interval, and draw a clearly labeled line for the performance target. Keep scenario labels beside the points and order scenarios in a substantively meaningful way, such as from least to most adverse assumptions. A target line is a benchmark, not another estimate.
A heatmap can help when there are many scenarios defined by combinations of two inputs. Label both axes, state what each cell's color represents, and include a numerical color scale. A heatmap is useful for locating patterns, but color alone is a poor way to communicate precise values or uncertainty; pair it with a table or another display when those details matter. For simulated quantities that vary from trial to trial, such as sample size or duration, use a distribution plot or quantile summary to show that spread. An interval around the simulated mean does not describe the range of individual trial outcomes.
Worked Example: Showing an Estimate Near a Performance Target
In a separate hypothetical scenario, 792 of 1,000 simulated trials meet a power criterion. The estimate is \(\hat{p}=792/1000=0.7920\). Both counts, 792 successes and 208 failures, are large enough for the normal approximation to be reasonable here. The Monte Carlo standard error is
The approximate 95% interval is \(0.7920\pm1.96(0.01284)=0.7920\pm0.02516\), giving \([0.7668,0.8172]\) to four decimal places. A dot-and-interval plot should place the point at 0.7920, show this interval, and mark the target 0.80 with a separate vertical line. The estimate is below the target, and the Monte Carlo interval includes the target. That inclusion indicates limited simulation precision for distinguishing the estimate from the target; it does not establish that the underlying performance equals the target or that the design meets it.
If a decision rule requires more simulation precision near a threshold, the display helps identify where additional simulation could be informative. It does not itself determine the decision rule. Keep the target and the estimated value visually distinct, and state plainly whether the point estimate is above or below the target.
Worked Example: Plotting Expected Sample Size Without Hiding Trial-to-Trial Spread
Suppose 2,500 independent simulated trials under one hypothetical scenario yield a mean total sample size of 412 participants and a sample standard deviation of 58 participants. The Monte Carlo standard error for the estimated mean is \(s/\sqrt{M}\), where \(s\) is the standard deviation across simulated trials. Here,
Using an approximately 1.96 multiplier for this large simulation count gives \(412\pm1.96(1.16)=412\pm2.2736\), or an approximate 95% Monte Carlo interval of 409.7 to 414.3 participants for the expected sample size. A point-and-interval display can report this mean and its simulation precision. However, a dot at 412 with that narrow interval would not show that individual simulated trials varied substantially: their sample-size standard deviation was 58 participants. Add a distribution plot or quantiles if readers need to understand the sample sizes a trial might actually attain. The interval around the mean is not a prediction interval for a future trial's sample size.
Diagnostics, Alternatives, and Limitations
Before finalizing a table or figure, check that it tells the same substantive story as the underlying results. Warning signs include a pooled summary that conceals a poor scenario, intervals omitted for estimates based on few simulations, axis limits that exaggerate small differences, a color scale with no labeled values, or rounding that changes whether a value appears to meet a target. Also verify that labels make the direction of favorable performance clear: a higher value may be desirable for power but undesirable for false-positive probability or sample size.
Tables and plots serve different purposes. Tables give exact estimates, counts, units, and scenario details. Dot-and-interval plots emphasize comparisons against targets and differences between scenarios. Heatmaps help scan many combinations of assumptions, while distribution plots show trial-to-trial variability. These are complements rather than mutually exclusive alternatives. A concise plot can lead the reader to the important pattern; a table can provide the exact values needed to inspect it.
Every display has limits. Monte Carlo intervals quantify simulation error conditional on the simulation setup; they do not measure uncertainty in scenario weights, parameter assumptions, model misspecification, or unexamined operational conditions. A clean plot cannot compensate for missing or poorly chosen scenarios. Nor should apparent small differences be overinterpreted when their simulation uncertainty is large. Report the scenario definitions and simulation counts close enough to the results that the displays remain understandable when viewed separately.
Common Mistake
Check Your Understanding
Use the calculations and display principles above to answer the questions; concise solutions follow.
- A scenario has 450 successes in 500 independent simulated trials. Calculate the estimated proportion and its Monte Carlo standard error.
- Why can a table of only scenario-weighted average power be inadequate when one scenario has very low power?
- A simulated mean sample size is 300 participants with a standard error of 2 participants. Does an interval of about 296 to 304 describe the range of sample sizes in individual trials?
- When might a heatmap be useful, and what should accompany it to avoid obscuring values or uncertainty?
Solutions. (1) \(\hat{p}=450/500=0.9000\). The estimated standard error is \(\sqrt{0.9000(0.1000)/500}=\sqrt{0.00018}=0.01342\). (2) An average can conceal a scenario with poor performance; show scenario-level results and label any average as an additional summary. (3) No. The interval describes Monte Carlo uncertainty in the estimated mean, not the spread of sample sizes across individual trials; use a distribution plot or quantiles for that spread. (4) A heatmap is useful for scanning many combinations of two scenario inputs. Label the axes and color scale, and accompany color patterns with exact values and uncertainty when those are important.