Why Visualize Operating Characteristics?
A simulated MAMS design does not have one error rate, one power value, or one expected sample size for every possible setting. These operating characteristics depend on the scenario: for example, the treatment effects, nuisance parameters, accrual conditions, and other assumptions used to generate a trial replicate. A single summary can therefore conceal a setting in which a design performs poorly. A plot across a planned scenario grid makes that dependence visible.
This tutorial focuses on how to construct and read those displays. It does not replace careful scenario specification or simulation validation. The earlier material on scenario grids and sensitivity analysis provides the basis for choosing settings; here the goal is to display the results so that patterns, trade-offs, and concerning regions can be recognized without overinterpreting simulation noise.
The measures need precise definitions before they can be meaningfully plotted. Under a scenario in which all treatment effects are null, the family-wise type I error rate is the probability that the design rejects at least one true null hypothesis. In a scenario with some null and some non-null effects, the family-wise error rate is the probability of rejecting at least one null hypothesis that is true in that scenario. Power must also be labelled precisely: it might be the probability of rejecting the null for a specified effective arm, or the probability of detecting at least one of a specified set of effective arms. These are different outcomes.
For a replicate, total sample size can be defined as the total number of participants randomized across all arms, including any planned control participants. Expected sample size is the expectation of that total under the scenario. Trial duration likewise needs an operational definition, such as time from first enrolment to the prespecified primary analysis, including accrual and required follow-up. A duration measured only to the end of accrual is a different quantity.
From Replicates to a Plot
Suppose \(R\) valid simulated replicates are generated under a fixed scenario. Let \(I_r=1\) if replicate \(r\) has the event defining an error rate or power outcome, and \(I_r=0\) otherwise. The estimated probability is the fraction of replicates with the event. If \(N_r\) is the total sample size in replicate \(r\), its estimated expectation is the average across replicates. The same averaging principle applies to a numerical duration measure.
Each scenario gives one estimate for each chosen characteristic. The resulting estimates can be organized as a line plot when scenarios vary along an ordered numerical input, as a heatmap when two inputs vary, or as a scatterplot when the aim is to show a relationship between two characteristics such as power and expected sample size. The plot should preserve the meaning of each scenario: connecting points is sensible for an ordered, continuous input, but can imply a smooth progression that is not justified for unrelated categorical scenarios.
Simulation estimates are themselves noisy. For a simulated event proportion \(\hat{p}\) based on \(R\) independent replicates, its approximate Monte Carlo standard error is \(\sqrt{\hat{p}(1-\hat{p})/R}\). A confidence interval can be displayed when the uncertainty matters for interpretation; earlier material on confidence intervals for simulated error rates covers interval construction. The interval describes uncertainty from finite simulation, not uncertainty about which scenario is appropriate or whether the statistical model is correct.
Worked Example: Error Rate and Power Across a Nuisance Parameter
Consider a hypothetical MAMS design evaluated at four values of a nuisance parameter \(q\), with 10,000 valid replicates at each value. The parameter is expressed on a scale where larger \(q\) means greater outcome variability. Under the global null, the estimated family-wise type I error rates are 0.021, 0.036, 0.049, and 0.061 for \(q=0.5, 0.8, 1.0,\) and \(1.2\), respectively. Under a separately simulated target-effect scenario, the corresponding powers for a specified treatment arm are 0.82, 0.80, 0.77, and 0.73.
Plot \(q\) on the horizontal axis. Use separate panels for family-wise type I error and arm-specific power, or use clearly distinguishable series with separate vertical scales. On the error-rate panel, add a horizontal reference line at the prespecified \(\alpha=0.05\). The estimate at \(q=1.0\) is 0.049, just below that reference; at \(q=1.2\), the estimate is 0.061, above it. The pattern flags the high-\(q\) scenario for investigation, but the plot alone does not establish that the design violates error control: simulation uncertainty and the intended control requirement must also be considered.
For the estimate 0.049 with \(R=10{,}000\), the approximate Monte Carlo standard error is
This scale of Monte Carlo error helps explain why a value close to 0.05 should not be treated as meaningfully different from the reference just because it falls slightly below it. Meanwhile, power declines from 0.82 to 0.73 across the displayed scenarios. Because the power values come from the target-effect scenario, they must not be interpreted as power under the global null used for the error-rate estimates.
Choosing a Display for the Scenario Grid
A line plot is useful when one scenario input changes in a natural order while other inputs are held fixed. Plotting a family-wise error rate against a nuisance parameter, for example, can reveal a trend or a local rise that a single worst-case summary would not show. When other factors also vary, use separate panels or line styles and make their values explicit. Avoid an unreadable plot in which many lines obscure the scenario identities.
A heatmap is often clearer when two inputs vary across a grid. Put one input on each axis and encode the operating characteristic by colour. A colour scale should be labelled with the characteristic and its units or range; it should not be allowed to exaggerate small differences by silently changing scale between panels. If comparing heatmaps for different characteristics, use separate scales and state that their colours are not directly comparable.
A scatterplot can show a trade-off between characteristics, such as power and expected sample size, for the same scenario. Each point must identify its scenario, and the axes must make clear whether larger or smaller values are desirable. Such a plot shows the joint pattern but can hide how the values change across a particular input; a scenario-indexed line plot or labels may be needed as a companion. None of these displays determines an acceptable design by itself. Requirements such as a maximum error rate or a minimum target power come from the design objectives, not from the plotting method.
Worked Example: Reading an Expected-Sample-Size Heatmap
Suppose a hypothetical design is simulated under three effect scenarios and two accrual-rate scenarios. The entries below are estimated expected total sample sizes, in participants. The effects are indexed by a prespecified scenario label; they are not estimates from an observed trial.
| Effect scenario | Slow accrual | Fast accrual |
|---|---|---|
| Null effects | 420 | 420 |
| One moderate effect | 360 | 360 |
| Two substantial effects | 290 | 290 |
A heatmap would place effect scenario on one axis and accrual scenario on the other, with colour representing expected total sample size. The null-effects row has the largest expected sample size, 420 participants, while the two-substantial-effects row has the smallest, 290 participants. Relative to the null-effects scenario, the reduction in the latter scenario is
Thus, under the stated simulation assumptions, the expected total sample size is about 31.0% lower in the two-substantial-effects scenario than in the null-effects scenario. The equal values for slow and fast accrual do not imply that duration is also equal: the table reports participants, not calendar time. A separate duration characteristic is needed to display the operational effect of accrual speed.
Showing Duration and Multiple Trade-Offs
Expected sample size and expected duration are related but not interchangeable. Sample size counts participants; duration is measured in time. A design can have similar expected sample sizes under two accrual scenarios yet take longer under slower accrual. Duration may also depend on when arms stop, follow-up requirements, and the rule defining the end of the trial. A useful display keeps these definitions fixed across the plotted scenarios.
Worked Example: Separating Sample Size From Trial Duration
In a hypothetical simulation, two scenarios have expected total sample sizes of 360 participants. Under slow accrual, the estimated mean time from first enrolment to primary analysis is 32 months; under fast accrual, it is 25 months. The duration reduction in the fast-accrual scenario is
A plot with expected sample size on the horizontal axis and expected duration on the vertical axis would place these scenarios at \((360,32)\) and \((360,25)\), with duration in months. They have the same plotted expected sample size but differ by 7 months in expected duration. Reporting only sample size would miss this operational distinction. These are averages: the point does not say that every simulated trial in either scenario has exactly that sample size or duration.
Checks, Failure Modes, and Interpretation
A visually striking pattern is useful only if the plotted data are trustworthy and the axes preserve their meaning. Before interpreting curves or colour patterns, check that scenarios are correctly labelled, that the same outcome definition is used throughout each panel, and that every estimate uses the intended set of valid replicates. Invalid replicates and numerical failures should be investigated and reported according to the simulation plan; silently dropping difficult cases can make a design look better than it is.
Display Monte Carlo uncertainty when differences are small relative to simulation error or when a plot is used to assess proximity to a prespecified threshold. Point estimates alone can make random fluctuation look like a sharp boundary between acceptable and unacceptable regions. Conversely, uncertainty bars that are too crowded can obscure the pattern; separate panels or a focused display of the most relevant scenarios may be clearer. The number of replicates should be adequate for the precision needed in the region being assessed, especially for rare error events.
- Mixing distinct scenarios: A power estimate under a target-effect scenario is not the power corresponding to a global-null error-rate point. Label the scenario for each plotted quantity.
- Using a misleading axis: A truncated vertical scale can make small differences look large. Show the scale clearly and explain any deliberate restriction.
- Confusing a sample count with time: Participants and months have different units and must be plotted and interpreted separately.
- Overreading a heatmap: Colour highlights patterns but does not establish that a difference is important or precisely estimated. Inspect the values and simulation uncertainty.
- Plotting only a favourable average: An overall summary can conceal a region with unacceptable error or inadequate power. Show relevant scenario-specific behaviour as well.
A common mistake is to see an estimated error rate of 0.049 against a 0.05 reference line and conclude that the design definitively satisfies the error requirement. The correction is to treat the plotted value as a simulation estimate with finite Monte Carlo precision, then assess its uncertainty and the prespecified requirement. The plot is a diagnostic and communication tool, not a substitute for that assessment.
Compared with a table, plots make trends, interactions, and localized adverse regions easier to detect, while tables preserve exact numerical values. A compact summary, such as the worst-case error rate across a specified grid, can communicate a decision-relevant extreme but may hide where it occurs and how performance changes nearby. A strong report therefore often pairs a plot with a small set of exact values and a clearly defined summary. It also states which scenarios were examined: no finite grid can show performance in scenarios that were not simulated.
Specify the event or numerical outcome, including the exact error or power definition and units for sample size or duration.
Record the scenario settings and the number of valid replicates used for every estimate; do not merge different data-generating conditions without explanation.
Use a line plot for an ordered input, a heatmap for two varying inputs, or a trade-off plot for two characteristics. Label axes, units, scenario groups, and reference thresholds.
Inspect uncertainty, missing or invalid replicates, scales, and boundary regions. Describe what the simulated pattern suggests without claiming more than the examined scenarios support.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- In a scenario where all treatment effects are null, what event defines the family-wise type I error rate?
- A simulated event occurs in 240 of 6,000 valid replicates. Calculate its estimated probability.
- Why should expected sample size and expected trial duration generally be shown as different operating characteristics?
- A heatmap colours error-rate estimates from 0.01 to 0.10. Name two details needed to interpret it responsibly.
- A power curve declines across a nuisance-parameter range, but its confidence intervals overlap substantially. What can and cannot be concluded from the plot?
Solutions. (1) The event is at least one rejection of a true null hypothesis. Under the global null, every rejected null is true. (2) The estimate is \(240/6{,}000=0.0400\), or 4.00%. (3) Sample size is a count of participants, whereas duration is calendar time and also depends on accrual, follow-up, stopping, and the chosen end-of-trial definition. (4) At minimum, identify the scenario represented by each cell and label the colour scale as an error rate; also state the error-rate definition and, where relevant, the simulation uncertainty or replicate count. (5) The plotted estimates suggest a downward pattern over the examined scenarios, but overlapping intervals mean the visual evidence may not distinguish local differences clearly. The plot alone does not prove that power is constant, establish a cause for the pattern, or support conclusions outside the simulated range.