Why Plot an Operating-Characteristic Curve?
A single operating-characteristic result describes one simulated scenario. A curve shows how a decision probability changes as an input changes, such as the true treatment effect or a nuisance parameter like the control event rate. This makes it easier to see whether a trial decision rule behaves as intended across plausible scenarios, rather than only at a selected point.
A plotted curve is an estimate, not a direct view of an exact probability. Each point is calculated from a finite set of simulated trials, so it has Monte Carlo error. A useful display therefore makes the scenario definition, decision rule, axis units, simulation precision, and relevant reference values clear.
For example, \(D=1\) could indicate that a Bayesian decision rule declares the treatment successful. The probability \(g(\theta)\) is then the chance that this rule reaches that decision under the specified scenario. It is not the posterior probability of treatment benefit in an individual trial. The plotted values also depend on the trial design and analysis choices held fixed while \(\theta\) varies.
Constructing the Curve and Its Monte Carlo Uncertainty
Choose a set of scenario values \(\theta_1,\ldots,\theta_K\). At each value, simulate \(R\) trials under the same prespecified design and decision rule, changing only the input intended for that curve. Record \(D_r\) for each simulated trial and plot \(\hat{g}(\theta_j)\) against \(\theta_j\). If \(D_r\) is binary and the simulated trials at that grid value are independent and identically generated, then the number of decisions \(S=\sum_{r=1}^{R}D_r\) has a binomial distribution with probability \(g(\theta_j)\). This gives the estimated Monte Carlo standard error.
This standard error describes simulation error in estimating \(g(\theta_j)\). It does not describe variability in outcomes among the patients in a future trial, uncertainty about the unknown treatment effect, or uncertainty in the model assumptions. A pointwise Monte Carlo interval is one way to show this precision. When \(R\) is large and \(\hat{g}\) is not close to 0 or 1, a normal approximation is often adequate: \(\hat{g}\pm1.96\widehat{SE}_{MC}\). Near 0 or 1, that approximation can extend outside the probability range or perform poorly; use a binomial interval such as the Wilson interval instead.
Worked Example: A Treatment-Effect Curve With Monte Carlo Error Bars
Suppose a hypothetical Bayesian trial design uses the same analysis and success rule in every scenario: declare success when the posterior probability that the treatment effect is beneficial exceeds 0.95. For each true treatment effect, 1,000 independent trials are simulated. The results are 180 success decisions at an effect of 0 units, 310 at 0.2 units, and 620 at 0.5 units. These effects are on a hypothetical continuous outcome scale.
The estimated decision probabilities are \(180/1000=0.180\), \(310/1000=0.310\), and \(620/1000=0.620\). For the middle point, the Monte Carlo standard error is
A normal-approximation 95% Monte Carlo interval at this point is \(0.310\pm1.96(0.01463)\), or approximately \((0.2813,0.3387)\). Thus, with this simulation size, the estimated decision probability at a true effect of 0.2 units has a Monte Carlo margin of about 0.0287. This interval reflects finite simulation error, not uncertainty about the true effect in a real trial.
Plot the three estimates against treatment effect and show pointwise error bars. The increasing estimates suggest that the rule is more likely to declare success at larger beneficial effects. Three points alone, however, do not establish that the underlying curve is smooth or exactly monotone. A denser grid and adequate simulation at each point are needed to inspect the shape reliably.
Varying a Nuisance Parameter
A curve need not be indexed by the treatment effect. It can instead show how a decision probability changes over a nuisance parameter while the treatment effect is held fixed. For example, a binary-outcome trial may be evaluated across different control event rates. This reveals whether the apparent performance at one assumed baseline rate is representative of other plausible rates.
When plotting such a curve, identify what is being held fixed and what changes. If both the treatment effect and the control event rate change, the display no longer represents a one-dimensional sensitivity curve unless that relationship is explicitly part of the scenario definition. A grid over two changing inputs is usually better shown as separate curves or a carefully labelled surface.
Worked Example: Decision Probability Across Control Event Rates
Consider a hypothetical binary-outcome trial evaluated at a fixed treatment effect. At each control event rate, 2,000 independent trials are simulated using the same design, analysis, and success rule. At control event rates of 0.10 and 0.30, the simulation yields 130 and 210 success decisions, respectively.
The estimated decision probabilities are \(130/2000=0.0650\) and \(210/2000=0.1050\). Their Monte Carlo standard errors are
Plot 0.0650 at a control event rate of 0.10 and 0.1050 at a rate of 0.30, with pointwise Monte Carlo error bars. The estimates indicate a higher success-decision probability at the higher baseline rate under this particular fixed-effect scenario. The curve does not establish that changing the baseline rate causes this difference in an actual trial; it describes simulated performance under the stated model and design.
Reference Targets and Curve Crossings
Reference lines help readers judge a curve against a prespecified benchmark. A horizontal line may show a desired decision probability, while a vertical line may mark a null effect, a clinically meaningful effect, or another scenario boundary. Label each line in the figure and explain its role. A target is a criterion for interpretation, not evidence that the trial design meets the criterion at every scenario.
A crossing can be estimated from simulated grid points, but its precision depends on both Monte Carlo error and how densely the scenario axis was sampled. Linear interpolation between adjacent estimates is a descriptive approximation; it does not prove the underlying curve is linear between them. Avoid reporting an interpolated crossing with more precision than the simulation and grid support.
Worked Example: Locating an Approximate Target Crossing
In a separate hypothetical simulation, 1,000 trials are run at each treatment effect. There are 720 success decisions at 0.4 units and 860 at 0.6 units, giving estimated probabilities 0.720 and 0.860. A horizontal reference line marks a target probability of 0.800.
Assuming a straight line between the two plotted points, the fraction of the probability increase needed to reach the target is \((0.800-0.720)/(0.860-0.720)=0.080/0.140=0.5714\). The corresponding fraction of the effect interval is \(0.5714(0.6-0.4)=0.1143\) units. The interpolated crossing is therefore \(0.4+0.1143=0.5143\) units.
This is an approximate location of the target crossing, not an exact property established by the simulation. The Monte Carlo standard errors at the two points are \(\sqrt{0.720(0.280)/1000}\approx0.01420\) and \(\sqrt{0.860(0.140)/1000}\approx0.01097\). These uncertainties, and the spacing of the grid, should temper any interpretation of the crossing.
How to Read and Diagnose a Curve
A curve is useful only if its construction and display permit a fair reading. Use labelled axes with units, show the scenario range, state the number of simulations per point, and identify the decision rule. Points should represent simulated estimates; lines connecting them are visual guides unless a model-based curve was explicitly fitted. Reference targets must be distinguishable from the estimated curve.
A short justification explains why the Monte Carlo standard error has the form used above. At a fixed scenario value, each independent simulation produces a Bernoulli decision with probability \(g(\theta)\). The variance of a Bernoulli variable is \(g(\theta)\{1-g(\theta)\}\). The sample mean of \(R\) independent decisions has variance equal to that quantity divided by \(R\). Replacing the unknown \(g(\theta)\) with \(\hat{g}(\theta)\) and taking the square root gives the estimated standard error.
Inspect the curve for signs that simulation or design choices may be obscuring its meaning. Large point-to-point fluctuations may reflect Monte Carlo noise, especially when error bars are wide. A surprising nonmonotone pattern should prompt checks of the simulation code, scenario inputs, and number of replications before it is smoothed away. At probabilities near 0 or 1, use an interval method that respects the probability bounds. If the curve changes sharply between sparse grid points, add points in that region rather than relying on distant interpolation.
If the same random numbers are deliberately used across scenario values, estimates at different points may be correlated. This can be useful for comparing scenarios, but it does not make the pointwise standard error formula a measure of uncertainty in differences between points. Likewise, pointwise error bars are not a simultaneous band guaranteeing coverage of the entire curve. State what uncertainty display is shown and do not make a whole-curve claim from pointwise intervals alone.
| Display | Best use | Important limitation |
|---|---|---|
| Curve with pointwise Monte Carlo intervals | Showing how one decision probability changes across an ordered scenario range | Intervals quantify simulation error at points, not all sources of uncertainty |
| Table of scenario results | Giving exact numerical values for a small set of prespecified scenarios | Patterns between listed scenarios are difficult to see |
| Separate curves or a surface for multiple inputs | Showing how results vary over more than one scenario dimension | Can become difficult to read and requires clear labels and scales |
A curve complements, rather than replaces, scenario-specific values. A table may be preferable when exact estimates are central and there are few scenarios. A curve is more informative when the ordered pattern matters. Averaging across scenarios or reporting a worst-case value answers a different summary question and can conceal how performance varies across the input range; such summaries should not be inferred from a plotted average without saying how it was formed.
Common Pitfalls and Limitations
- Confusing Monte Carlo error with trial uncertainty. More simulation replications narrow Monte Carlo intervals, but do not reduce uncertainty in the treatment effect in an actual trial.
- Connecting points as if the line were exact. A line between grid estimates is generally a visual aid. Sparse points cannot reveal every bend or crossing.
- Ignoring scenario definitions. A curve is interpretable only when the varied parameter, fixed inputs, analysis, and decision rule are specified.
- Overreading an apparent crossing. A target crossing is approximate when it is interpolated, and may be sensitive to simulation error and grid spacing.
- Smoothing away a warning. A jagged or unexpected pattern may be Monte Carlo noise, but it may also indicate a coding error or a real feature of the decision rule. Diagnose before smoothing.
Check Your Understanding
Answer each question, then compare with the brief solutions.
- At a scenario value, 400 of 2,000 independent simulated trials meet the decision rule. Find the estimated operating characteristic and its Monte Carlo standard error.
- What does a pointwise Monte Carlo interval around a curve estimate, and what does it not estimate?
- A plotted curve crosses a target between two widely spaced grid points. Name one reason not to report the crossing as an exact threshold.
- Why can a nonmonotone sequence of simulated estimates deserve investigation before smoothing?
Solutions. (1) \(\hat{g}=400/2000=0.200\), and \(\widehat{SE}_{MC}=\sqrt{0.200(0.800)/2000}=\sqrt{0.000080}=0.008944\). (2) It estimates finite-replication error in the simulated decision probability at that scenario; it does not estimate patient-outcome variability or uncertainty about the true treatment effect. (3) The crossing depends on interpolation between sparse points, and the estimates themselves have Monte Carlo error. (4) The pattern may be simulation noise, but it may also reveal a coding problem or genuine behavior of the specified decision rule; inspect the inputs and simulation before altering the display.