Why Visual Choices Matter
Simulation results are often summarized by many performance measures across multiple designs and scenarios. A single overall ranking or table can hide the pattern that matters for a design decision: a design may perform well under the typical scenario and poorly under a plausible stress scenario, or appear to meet a target when Monte Carlo uncertainty is substantial. A useful figure should make the comparison visible while preserving the identity of the scenario, the units of each measure, and the uncertainty due to using a finite number of simulation replicates.
The goal is not to make every result fit into one graphic. Instead, choose a display for the question being asked. Comparisons of estimates across scenarios, distributions across replicates, binary outcomes such as interval coverage, and trade-offs between performance and cost generally call for different visual encodings. Visual summaries should complement the simulation summaries and diagnostics established in the earlier material on simulation analysis, not replace them.
Start With the Comparison and the Scale
Before selecting a plot, specify the quantity being displayed, its units, the designs being compared, and whether the comparison is within a scenario or across scenarios. For example, bias is measured in the units of the parameter being estimated; relative bias is a percentage; coverage and failure rates are proportions; and computational cost might be reported in seconds per replicate or total CPU-hours. A plot that mixes these quantities on one numerical axis is difficult to interpret.
For estimates of a continuous performance measure, such as bias or root mean squared error, a point-and-interval plot is often effective. Put the measure on a labeled numerical axis and use separate points or symbols for the designs. Include Monte Carlo intervals when they are informative. For binary outcomes estimated as proportions, such as confidence interval coverage or convergence success, display the proportion on a common scale, often from 0 to 1, and identify any decision target with a reference line. A reference line is a criterion, not evidence that the estimate is known exactly.
When the distribution across simulation replicates is itself relevant, use a distribution plot such as a histogram, density display, or box plot. These show features that a mean and interval may conceal, including skewness, multiple modes, or a tail of severe estimation errors. Keep replicate-level distributions distinct from Monte Carlo intervals for a summary: they answer different questions.
Displaying Monte Carlo Uncertainty
For \(R\) independent simulation replicates, let \(I_r\) equal 1 if a prespecified event occurs on replicate \(r\), and 0 otherwise. The estimated event probability is \(\hat{p}=\sum_{r=1}^{R} I_r/R\). Under independent, identically generated replicates for a fixed design and scenario, the approximate Monte Carlo standard error is \(\sqrt{\hat{p}(1-\hat{p})/R}\). A point estimate without this uncertainty can make small differences look more decisive than the simulation supports.
For a replicate-level continuous summary \(Y_r\), the estimated mean is \(\bar{Y}\), and its Monte Carlo standard error is \(s/\sqrt{R}\), where \(s\) is the sample standard deviation of the replicate-level values. An approximate 95% Monte Carlo interval is \(\bar{Y}\pm 1.96s/\sqrt{R}\) when the normal approximation is reasonable. The approximation concerns uncertainty in the estimated simulation mean; it does not claim that 95% of replicate values lie in that interval.
Worked Example: Showing Coverage and Its Monte Carlo Uncertainty
Suppose a hypothetical simulation evaluates a confidence interval for a parameter under one fixed scenario. In \(R=1{,}000\) independent replicates, 940 intervals contain the true parameter. The estimated coverage is \(\hat{p}=940/1{,}000=0.940\). Suppose the design criterion is coverage of 0.950 or greater.
The estimated Monte Carlo standard error is
An approximate 95% Monte Carlo interval is
A point-and-interval plot should show coverage at 0.940, its interval from about 0.925 to 0.955, and a vertical reference line at 0.950. The estimate is below the target, but the Monte Carlo interval includes the target. This result does not establish that true coverage is at least 0.950; more replicates may be needed to distinguish a small shortfall from Monte Carlo noise. The interval also says nothing by itself about coverage in other scenarios.
Preserve Scenario-Specific Performance
When a study includes several scenarios, show them explicitly. A faceted point-and-interval plot can give each scenario its own panel while keeping the performance axis and design symbols consistent. This makes it easier to see whether a design's behavior changes with sample size, dropout, variability, or model misspecification. If scenario labels are long, concise labels plus a clear legend or caption are preferable to unexplained codes.
A heatmap can provide a compact overview when there are many design-scenario combinations. Put scenarios on one axis, designs on the other, and encode one performance measure by color. State the measure and units, use a labeled color scale, and choose its limits deliberately. For measures with a meaningful target, consider displaying distance from the target rather than raw values, but explain that transformation. If colors are scaled separately within each panel, similar colors may represent different values; if a common scale is used, small but important differences can be hard to see. Do not let a heatmap replace a more precise display when uncertainty around the estimates drives the decision.
Worked Example: Finding a Scenario-Specific Coverage Problem
Consider hypothetical coverage estimates from 1,000 replicates per cell for three designs and two scenarios. The target is 0.950.
| Design | Routine scenario | High-dropout scenario |
|---|---|---|
| A | 0.952 | 0.918 |
| B | 0.947 | 0.944 |
| C | 0.951 | 0.949 |
A single average across the two scenarios would give Design A \((0.952+0.918)/2=0.935\), Design B \((0.947+0.944)/2=0.9455\), and Design C \((0.951+0.949)/2=0.950\). Those averages conceal the main pattern: Design A has a marked shortfall in the high-dropout scenario, while Design C is close to the target in both.
A faceted point plot should therefore show two panels, one per scenario, with all three designs on the same coverage scale and a reference line at 0.950. To judge whether small differences are resolved by the simulation, add Monte Carlo intervals. For example, Design A's high-dropout estimate has approximate standard error
Its approximate 95% Monte Carlo interval is \(0.918\pm1.96(0.00868)\), or \((0.901,0.935)\), entirely below the target. The plot communicates a clear scenario-specific concern that the pooled averages obscure. These intervals quantify simulation error for each estimated coverage; they do not account for uncertainty about which scenarios are relevant to practice.
Show Trade-Offs Without Hiding Their Structure
A scatterplot is useful when two design criteria are to be compared directly. For example, place computational cost on the horizontal axis and absolute bias on the vertical axis, with one labeled point per design. Add uncertainty intervals where the corresponding estimates have Monte Carlo uncertainty. Label the units and indicate which direction is preferred for each measure. If the plot is faceted by scenario, use consistent axes when direct scenario comparisons are intended.
Do not combine unlike quantities on a single axis simply to make a compact graphic. A design's cost in hours and its coverage as a proportion do not share a natural scale. A dual-axis plot can also encourage misleading visual comparisons because the apparent distance and slope depend on independently chosen scales. Separate panels or a carefully defined, decision-justified transformation are usually clearer. The earlier material on Pareto fronts addresses how to identify non-dominated trade-offs; here, a plot should make the underlying performance and cost values interpretable rather than replace them with a ranking alone.
For paired design comparisons using common random numbers, the replicate-level difference is often more informative than plotting two independent-looking sets of estimates. Display the estimated paired difference and its Monte Carlo interval, label the subtraction order, and identify the units. Pairing can reduce uncertainty when replicate outcomes are positively correlated, but it does not justify treating the paired observations as independent. The earlier material on uncertainty in paired design differences provides the relevant inferential calculation.
Worked Example: Plotting a Paired Difference Rather Than Two Separate Means
Suppose a hypothetical comparison uses 100 paired replicates to compare the absolute estimation error of Design A with Design B under the same simulated data conditions. Define the replicate-level difference as \(D_r=|E_{A,r}|-|E_{B,r}|\), measured in mg/L. Assume the 100 differences have mean \(\bar{D}=0.012\) mg/L and sample standard deviation \(s_D=0.120\) mg/L.
The estimated Monte Carlo standard error of the mean paired difference is
The approximate 95% Monte Carlo interval is \(0.012\pm1.96(0.0120)\), or \((-0.0115,0.0355)\) mg/L. Because the difference is defined as A minus B, a positive value means A has greater absolute error. A paired-difference plot should show the estimate, interval, and a zero reference line. Since the interval includes zero, this simulation does not resolve which design has lower mean absolute error at this Monte Carlo precision. Plotting only the two marginal means would omit the pairing that determines the uncertainty of their difference.
Diagnostics, Failure Modes, and Limitations
Inspect the figure for features that can signal either a real performance issue or a display problem. Check that every design-scenario combination is present, that axes start and end at defensible values, that units and legends are unambiguous, and that a color scale has not exaggerated negligible differences. Look for estimates near boundaries, wide Monte Carlo intervals, outlying replicate results, and scenario-specific failures. If the number of failed fits differs by design, make the failure rate visible; silently plotting only successful fits can make a design look better than it is.
Monte Carlo intervals based on the usual formulas can be unreliable with very few replicates or proportions near 0 or 1. In those settings, use an interval method appropriate for binomial proportions rather than relying uncritically on a symmetric normal approximation. For continuous summaries, strong skewness or extreme outliers can make a symmetric mean interval a poor description; show the replicate distribution and consider whether the target summary itself is appropriate. If replicates are dependent, the independent-replicate standard error formulas above do not apply as written.
No plot can correct an incomplete scenario set, an unsuitable performance measure, or an unrepresentative weighting scheme. A visually clean overall summary may still conceal a scenario that matters clinically. Present enough detail to trace an overall conclusion back to its scenario-specific components, and use a table when exact values are important. Conversely, a dense table alone may make broad patterns difficult to see. Figures and tables serve complementary purposes.
Common Mistakes
- Treating Monte Carlo intervals as patient-level intervals. They quantify simulation error in an estimated performance measure, not the spread of individual outcomes.
- Pooling scenarios before inspecting them. An average can hide an important failure in one scenario. Show scenario-specific results before presenting an overall summary.
- Showing only point estimates. Small apparent differences may be simulation noise. Include uncertainty when it is relevant to the decision.
- Using incomparable scales. Do not visually equate cost, bias, and coverage by placing them on an unexplained shared or dual axis.
- Dropping failed replicates without disclosure. Report failures as performance results and state how any conditional summaries were calculated.
Check Your Understanding
For each question, consider what should be plotted and what the display can legitimately support. Short solutions follow.
- A fixed scenario has estimated convergence probability 0.980 from 500 independent replicates. Calculate its approximate Monte Carlo standard error.
- Two designs have coverage 0.960 and 0.950 in one scenario, but 0.900 and 0.948 in a high-dropout scenario. What display best reveals the difference?
- A plot shows the distribution of estimation errors across replicates. Does its central 95% of values represent a 95% Monte Carlo interval for mean error?
- What information should accompany a scatterplot of computational cost against absolute bias?
Solutions. (1) \(SE_{\mathrm{MC}}=\sqrt{0.980(0.020)/500}=\sqrt{0.0000392}=0.00626\). (2) Use scenario-specific panels or a heatmap with a common labeled coverage scale; panels make the high-dropout shortfall for the first design immediately visible. (3) No. The distribution describes replicate-level errors; a Monte Carlo interval for mean error instead uses the standard error of the replicate mean, such as \(s/\sqrt{R}\). (4) Label both axes and units, identify each design, state which direction is preferred, and include Monte Carlo uncertainty for estimated bias when relevant. Facet by scenario if the comparison spans scenarios.