Comparing Rule-Based Escalation Operating Characteristics

Why Compare Designs Under the Same Scenarios?

A rule-based escalation design is a sequence of decisions: treat participants at a dose, observe dose-limiting toxicities (DLTs), and use the observed counts to decide whether to escalate, expand, or stop. A design’s operating characteristics describe what those rules tend to produce across repeated hypothetical trials. Useful summaries include the probability of recommending each dose, the number of participants treated, and the proportion treated above a prespecified toxicity threshold.

A comparison is meaningful only when the designs face the same underlying dose-toxicity scenarios. Otherwise, a design might appear to select better or expose fewer participants to high-risk doses simply because it was tested under more favorable toxicity rates. The comparison also depends on fully specified rules: “3+3+3” is not a unique algorithm, and rolling-six procedures depend on how pending outcomes and concurrent enrollment are handled.

For a transparent comparison, consider a fixed ordered dose grid with true DLT probabilities \(p_1,\ldots,p_K\). A simulation generates a trial according to one of these scenarios, applies each design’s decision rules, and repeats this process many times. The same scenario means the same dose grid, \(p_1,\ldots,p_K\), DLT-outcome model, follow-up assumptions, and eligibility rules for every design.

Specify the Rules Before Comparing Them

The following operational definitions make the comparison precise. They are examples of protocol rules, not a claim that every protocol using these design names follows the same details. Assume complete DLT evaluation before a conventional cohort decision, no unevaluable participants, and no dose skipping. At each dose, individual DLT outcomes are independent Bernoulli outcomes with that dose’s true probability \(p_k\). The dose grid and \(p_k\) values are fixed within a scenario.

• Conventional 3+3: Treat 3 participants. With 0 DLTs, escalate. With 1 DLT, treat 3 more at that dose; escalate if at most 1 of 6 has a DLT, and stop escalation if at least 2 of 6 do. With at least 2 DLTs in the first 3, stop escalation at that dose.
• 3+3+3 variant used here: Treat 3. With 0 DLTs, escalate; with 1 DLT, add 3. If at least 2 of the first 3 have DLTs, add participants up to 6 unless all 3 had DLTs. If at least 2 of 6 have DLTs, add a final cohort of 3, stopping earlier if the total reaches 3 DLTs. Escalate from 9 only if no more than 2 of 9 have DLTs. At a total of 6, escalate if there is at most 1 DLT. This variant therefore allows additional observation in some cases where conventional 3+3 would stop.
• Rolling-six variant used here: Apply the conventional 3+3 DLT thresholds, but allow enrollment to continue while outcomes are pending, up to 6 participants at a dose. Pending participants do not count as DLT-free. Escalation requires the relevant evaluable-patient criterion to be met; the design may open the next dose while participants at the preceding dose are still awaiting outcomes, subject to the protocol’s concurrency limit.

For all three rules, define the recommended dose as the highest dose the algorithm clears for escalation, subject to its stopping rules. If the lowest dose is rejected, record “no dose recommended.” This is an algorithmic recommendation, not a guarantee that the dose has a particular true DLT probability. In simulation, an overdose is a participant treated at a dose with \(p_k\) above the scenario’s prespecified toxicity threshold. That threshold must be set before reviewing the results.

The rolling-six rule requires an additional assumption about time: how quickly DLTs arise relative to enrollment and assessment. For a comparison of selection and enrollment, simulate those times or explicitly assume a timing pattern. If all outcomes are observed before any next-dose enrollment, rolling six may have little or no advantage in accrual speed and may behave much like a cohort-based rule.

Estimating Operating Characteristics

For \(R\) simulated trials under one scenario, the estimated probability of selecting dose \(k\) is the number of trials selecting that dose divided by \(R\). The estimates for all doses plus “no dose recommended” should sum to 1, apart from rounding. The mean participant count is total participants across replicates divided by \(R\). A pooled patient-level overdose rate is total participants treated at doses above the threshold divided by total participants treated across all replicates. State the denominator: this pooled rate weights trials with more participants more heavily than trials with fewer participants.

Here \(N_r\) is the number enrolled in replicate \(r\), and \(O_r\) is the number of those participants treated above the overdose threshold. These summaries answer different questions. Selection probability measures how often the algorithm recommends a dose; participant count measures enrollment burden; overdose rate measures exposure among enrolled participants. None alone gives a complete ranking.

Simulation Uncertainty and a Short Derivation

A simulation estimate is itself subject to Monte Carlo error because it is calculated from a finite number of random replicates. For a selection indicator \(I_r\), let \(I_r=1\) when replicate \(r\) selects a specified dose and \(I_r=0\) otherwise. If replicates are independent and the probability of selecting that dose under the scenario is \(\pi\), then \(I_r\) is Bernoulli with variance \(\pi(1-\pi)\). The estimate \(\hat{\pi}=R^{-1}\sum I_r\) has variance \(\pi(1-\pi)/R\), because independent variances add and the sum is divided by \(R^2\). Replacing the unknown \(\pi\) with \(\hat{\pi}\) gives the estimated Monte Carlo standard error.

This standard error quantifies simulation precision, not clinical uncertainty about the true dose-toxicity probabilities. It also assumes independent simulation replicates generated under the same scenario and rules.

Compare Patterns Across Scenarios

A single scenario can favor a design by chance or reveal only one part of its behavior. A useful comparison includes scenarios where the dose grid is mostly below the threshold, crosses the threshold gradually, or becomes unsafe quickly. The table below gives a second hypothetical set of simulation summaries for three scenarios. For simplicity, each entry is an illustrative pooled result from the same number of replicates per design and scenario.

Assume for these summaries that the prespecified target dose is dose 3 in the gradual and steep scenarios, while the overdose threshold is \(p_k>0.25\). In the mostly-low scenario, use \(p_k>0.25\) as the threshold, so the pooled overdose rate is zero for every design because no dose exceeds it. A low target-selection probability in that scenario is not automatically evidence of poor safety: the target label is a fixed comparison benchmark, while the designs may select another dose or recommend no dose.

Diagnostics, Alternatives, and Limitations

Before interpreting a simulation, verify that the implementation matches the protocol. Check that the dose grid is ordered, the same random outcome scenario is used for each design, escalation and stopping boundaries are coded correctly, and the recommendation rule is applied consistently. For the rolling-six design, audit enrollment while outcomes are pending and ensure that its concurrency cap is never exceeded. Compare simulated branch frequencies with simple checks: for example, if all \(p_k=0\), a rule that stops at the first dose should trigger an audit.

A paired simulation can sharpen a comparison: generate the same underlying participant-level potential DLT outcomes for each design, then apply each set of allocation rules to those outcomes. This does not make the designs identical, because they may assign participants to different doses; it can nevertheless reduce noise from using unrelated random streams. Report the number of replicates, scenario probabilities, timing assumptions, dose-selection definition, overdose threshold, and Monte Carlo uncertainty. If those are missing, operating-characteristic numbers are difficult to reproduce or interpret.

Simulation is often more flexible than exact enumeration, especially when rules include pending outcomes, varying cohort sizes, pauses, or complex stopping conditions. Exact enumeration can be a useful independent check when the tree of possible outcomes is small and the outcome model is simple. Neither approach corrects a poorly specified protocol or an unrealistic scenario. A rule-based design also has limitations: observed DLT counts can be sparse, the recommendation may be sensitive to one event, and the algorithmic dose recommendation does not by itself quantify uncertainty in the dose-toxicity curve.

Comparisons should therefore cover several plausible toxicity scenarios rather than optimize performance under one assumed curve. They should also separate safety, target-dose selection, enrollment burden, and operational speed. A design that enrolls more participants may provide more observations, but that alone does not establish better dose selection; a design that selects a dose often may still select doses that exceed the prespecified threshold. In particular, rolling-six accrual advantages depend on DLT timing and recruitment assumptions, not merely on the name of the rule.

Common Mistakes

• Using different scenarios for different designs. The correction is to apply every design to the same toxicity probabilities and comparable timing assumptions.
• Calling a design “3+3+3” without stating its rule. Variants differ in when an extra cohort is added and what count permits escalation. Define the exact branches before simulating.
• Reporting overdose rate without its denominator. State whether the rate is pooled across participants, averaged across trial-specific rates, or a probability of any overdose exposure. Those quantities are not interchangeable.
• Treating Monte Carlo error as clinical uncertainty. It reflects finite simulation replicates. Uncertainty about actual toxicity probabilities is a separate issue.
• Interpreting selection probability as proof of safety. A recommendation is the output of a rule under observed data; it does not ensure the selected dose’s true DLT probability is below the threshold.