Comparing Adaptive Dose-Selection Designs by Simulation

Why Compare Dose-Selection Designs by Simulation?

A dose-ranging design can influence both which dose is selected and what patients receive while evidence is being collected. A fixed design allocates participants according to a prespecified schedule. An adaptive design updates allocation as trial outcomes accumulate. The adaptive design may direct more participants toward promising doses, but early random variation can also steer allocation away from the dose that is truly best. A simulation comparison makes these trade-offs measurable under explicit assumptions.

The comparison is not a claim that one design is universally superior. Its conclusions are conditional on the simulated dose-response relationships, outcome variability, allocation rules, decision criterion, sample size, and other design details. The goal is to estimate relevant operating characteristics—repeated-trial performance measures—under scenarios that matter for the development decision.

Definition: An operating characteristic is a performance measure calculated across repeated trials generated under specified assumptions. Dose-selection probability is the proportion of trials selecting a dose or dose set of interest. False-selection probability is the proportion selecting a dose that does not meet the prespecified definition of a correct selection. The definition of “correct” must be set before comparing designs.

Specify the Simulation Before Comparing Designs

Consider a hypothetical study with four candidate doses: 5, 10, 20, and 40 mg. Each participant has a binary efficacy outcome and a binary adverse-event outcome. For this illustration, the true probabilities are assumed known to the simulation: efficacy probabilities are 0.36, 0.52, 0.60, and 0.66, respectively; adverse-event probabilities are 0.03, 0.05, 0.12, and 0.24. Outcomes are generated independently across participants and, conditional on dose, efficacy and adverse-event outcomes are independent.

Define a simple utility as the efficacy probability minus twice the adverse-event probability. This particular weight is a hypothetical decision rule, not a universal clinical value judgment. It makes the selection target explicit: choose the candidate dose with the largest true utility. Since the four true utilities are 0.30, 0.42, 0.36, and 0.18, the unique best dose under this rule is 10 mg.

U_d = p_{E,d} - 2p_{A,d}

Here \(p_{E,d}\) and \(p_{A,d}\) are the true efficacy and adverse-event probabilities at dose \(d\). In each simulated trial, the corresponding estimated utility is \(\hat U_d=\hat p_{E,d}-2\hat p_{A,d}\), where the proportions are calculated from participants assigned to that dose. The trial selects the dose with the greatest estimated utility; ties are resolved by random selection among tied doses. A correct selection is selection of 10 mg. Any other selected dose counts as a false selection for this example.

The fixed design assigns 30 of 120 participants to each dose. The adaptive design assigns 10 participants to each dose initially, then allocates the remaining 80 in eight cohorts of 10. After each cohort, it recalculates estimated utility. In the next cohort, each participant has probability 0.70 of assignment to the currently highest-estimated-utility dose and probability 0.10 for each other dose. This probability floor preserves some allocation to every candidate. At the end, the adaptive design uses the same estimated-utility selection rule as the fixed design.

These rules define the algorithms to compare. Other allocation probabilities, cohort sizes, endpoint definitions, or selection rules would define different designs and require their own comparison. The two designs use the same total sample size, candidate doses, outcome-generating assumptions, and final selection criterion, so differences in their simulated results can be attributed to their allocation strategies, subject to simulation error.

Worked Example: Finding the True Best Dose

Use the hypothetical probabilities above to calculate the utility of each dose and identify the dose that defines a correct selection.

For 5 mg, \(U_5=0.36-2(0.03)=0.36-0.06=0.30\). For 10 mg, \(U_{10}=0.52-2(0.05)=0.52-0.10=0.42\). For 20 mg, \(U_{20}=0.60-2(0.12)=0.60-0.24=0.36\). For 40 mg, \(U_{40}=0.66-2(0.24)=0.66-0.48=0.18\).

Because \(0.42\) is the largest utility, 10 mg is the unique best dose under the stated rule. A trial selecting 5, 20, or 40 mg is a false selection in this scenario. The definition is specific to this utility rule: a different clinical constraint or benefit-risk weighting could yield a different target.

Estimate Operating Characteristics Across Replicates

For each design, simulate many complete trials. In every replicate, generate participant outcomes at their assigned doses, apply the allocation rule where relevant, and record the final selected dose and the aggregate dose assigned. If \(R\) trials are simulated and \(I_r\) equals 1 when replicate \(r\) selects 10 mg and 0 otherwise, the estimated probability of correct selection is the sample mean of these indicators. With one unique best dose, the estimated false-selection probability is one minus this estimate.

\widehat{P}(\text{correct selection}) = \frac{1}{R}\sum_{r=1}^{R} I_r \qquad \widehat{P}(\text{false selection}) = 1-\widehat{P}(\text{correct selection})

In a hypothetical run of 10,000 replicates, suppose the adaptive design selects 10 mg in 6,180 trials and the fixed design does so in 5,620. The estimated correct-selection probabilities are therefore 0.618 and 0.562. Their false-selection probabilities are 0.382 and 0.438. These are simulated estimates under the specified scenario, not probabilities that the selected dose is correct in an individual real trial.

Worked Example: Calculating Selection Performance

Suppose a simulation of 10,000 trials gives 6,180 correct selections for the adaptive design and 5,620 for the fixed design. Calculate each estimated success and false-selection probability, then compare them.

For the adaptive design, \(6{,}180/10{,}000=0.618\), or 61.8% correct selection. Its false-selection estimate is \(1-0.618=0.382\), or 38.2%.

For the fixed design, \(5{,}620/10{,}000=0.562\), or 56.2% correct selection. Its false-selection estimate is \(1-0.562=0.438\), or 43.8%.

The adaptive design's estimated correct-selection rate is higher by \(0.618-0.562=0.056\), or 5.6 percentage points, in this simulation scenario. That difference alone does not show whether the designs differ reliably; Monte Carlo error must also be considered.

Quantify Monte Carlo Error

A simulation estimate is itself based on a finite number of random replicates. If the true probability of correct selection under a design is \(q\), the indicator \(I_r\) has variance \(q(1-q)\). The average of \(R\) independent replicate indicators has variance \(q(1-q)/R\), so its standard error is estimated by substituting the simulated proportion \(\hat q\) for \(q\). This standard error describes simulation noise in estimating the operating characteristic; it is not the standard error of a clinical treatment effect.

SE(\hat q) \approx \sqrt{\frac{\hat q(1-\hat q)}{R}}

An approximate 95% Monte Carlo interval is \(\hat q\pm1.96SE(\hat q)\). It describes uncertainty from using a finite number of replicates to estimate the simulation probability, under the simulation setup. It does not include uncertainty about whether the assumed dose-response model is realistic.

Worked Example: Checking Whether a Simulated Difference Exceeds Monte Carlo Noise

Using the 10,000-replicate results above, calculate approximate Monte Carlo intervals for the correct-selection probabilities. Treat the two sets of replicates as independent for the comparison.

For the adaptive design, \(SE=\sqrt{0.618(1-0.618)/10{,}000}=\sqrt{0.00002361}=0.00486\). Its approximate interval is \(0.618\pm1.96(0.00486)\), or 0.608 to 0.628.

For the fixed design, \(SE=\sqrt{0.562(1-0.562)/10{,}000}=\sqrt{0.00002462}=0.00496\). Its approximate interval is \(0.562\pm1.96(0.00496)\), or 0.552 to 0.572.

The estimated difference is 0.056. Under the stated independent-replicate comparison, its approximate standard error is \(\sqrt{0.00486^2+0.00496^2}=0.00694\), giving an approximate interval \(0.056\pm1.96(0.00694)\), or 0.042 to 0.070. This interval reflects Monte Carlo error only. It does not establish that the adaptive design will perform better under real-world conditions or under other plausible scenarios.

Measure Patient Dose Exposure Alongside Selection

A selection rate is not the only relevant outcome. In this example, define aggregate assigned dose as the sum of the assigned dose levels across the 120 participants, in mg per trial. It is a simple measure of dose exposure during the study, not a pharmacokinetic measure or a complete measure of clinical benefit and harm. For the fixed design, the total is \(30(5+10+20+40)=2{,}250\) mg per trial. The adaptive total varies with allocations, so summarize it across replicates using its mean and spread.

Worked Example: Comparing Aggregate Assigned Dose

Suppose the 10,000 adaptive-design replicates have a mean aggregate assigned dose of 1,842 mg per trial and a standard deviation of 360 mg. The fixed design has 2,250 mg per trial in every replicate because its allocation is fixed. Calculate the difference and relative reduction.

The mean difference is \(2{,}250-1{,}842=408\) mg per trial. Relative to the fixed design, this is \(408/2{,}250=0.1813\), or an estimated 18.1% lower aggregate assigned dose under the adaptive design in this scenario.

For the adaptive mean, the Monte Carlo standard error is \(360/\sqrt{10{,}000}=3.6\) mg per trial. An approximate 95% Monte Carlo interval is \(1{,}842\pm1.96(3.6)\), or 1,834.9 to 1,849.1 mg per trial. The fixed-design total has no replicate-to-replicate variation under the specified allocation rule, though that does not mean the fixed design has no uncertainty in its clinical outcomes.

This exposure comparison should be interpreted narrowly. It does not imply lower exposure for every participant, and a smaller aggregate dose is not automatically better: the clinical value of a dose depends on efficacy, safety, treatment duration, and the decision being made.

Diagnostics, Alternatives, and Limitations

Before relying on a result, inspect whether the simulation is behaving as specified. Confirm that every replicate has 120 participants, that fixed allocations total 30 per dose, and that adaptive probabilities sum to 1 at every update. Check that all candidate doses continue to receive participants as intended, that the selection rule is applied identically, and that reported counts and proportions agree. For adaptive allocation, also examine the distribution of dose-specific sample sizes: a favorable average operating characteristic can conceal frequent runs with very little information at one or more doses.

An adaptive design can be sensitive to early outcomes. A chance run of unusually favorable responses at a nonoptimal dose may draw participants there, reinforcing an initial impression before enough evidence accumulates. The allocation floor reduces the chance that a dose is nearly abandoned, but it does not eliminate instability or guarantee reliable selection. Compare results across plausible efficacy and adverse-event scenarios, including cases where dose utilities are close, the efficacy curve is shallow, or the highest dose has substantial toxicity. If the design ranking changes across plausible scenarios, report that dependence rather than presenting one scenario as decisive.

The fixed design is a useful comparator because its allocation is predictable and provides balanced information across candidate doses. Adaptive allocation may change who receives which dose and can improve the allocation of participants toward promising candidates, but it makes sample sizes outcome-dependent. Neither feature alone determines which design is preferable. The design decision depends on how much value is placed on selection accuracy, information across doses, and participants’ assigned exposure.

The simulation also inherits its assumptions. Here, outcomes are binary, probabilities are known to the data-generating mechanism, and the utility rule assigns a fixed penalty to adverse events. The simple aggregate dose measure ignores drug concentrations, adherence, treatment duration, and individual differences. A more realistic assessment would represent the outcomes and clinical decision criteria relevant to the intended study and test uncertainty in the assumed model. A simulation cannot repair a poorly chosen decision criterion, and Monte Carlo precision cannot compensate for an implausible scenario.

Common pitfall: “The adaptive design selected the best dose more often in the simulation, so it is the better design.” This reasoning treats one scenario and one estimated operating characteristic as a universal conclusion. The correction is to report the result as conditional on the specified assumptions, quantify Monte Carlo error, compare relevant outcomes such as false selection and assigned exposure, and assess whether the conclusion persists across plausible scenarios.

Check Your Understanding

Try the questions first, then check the solutions below.

Under the utility rule in this tutorial, what is the true best dose, and what makes a selection “false”?
A design selects the best dose in 7,250 of 10,000 simulated trials. Calculate its estimated correct-selection and false-selection probabilities.
For an estimated correct-selection probability of 0.50 from 10,000 replicates, calculate its approximate Monte Carlo standard error.
Why is a reduction in aggregate assigned dose not, by itself, proof that a design is clinically preferable?
Name one diagnostic specific to adaptive allocation and one scenario feature that should be varied in a sensitivity analysis.

Solutions. (1) The true best dose is 10 mg because its utility is 0.42, the largest of the four. A selection is false if it selects 5, 20, or 40 mg. (2) Correct selection is \(7{,}250/10{,}000=0.725\), or 72.5%; false selection is \(1-0.725=0.275\), or 27.5%. (3) \(SE=\sqrt{0.50(1-0.50)/10{,}000}=\sqrt{0.000025}=0.00500\). (4) Aggregate assigned dose does not capture the full balance of efficacy, safety, treatment duration, or individual exposure, and lower dose is not automatically greater clinical value. (5) For example, inspect the distribution of per-dose sample sizes across adaptive replicates; vary the separation between candidate-dose utilities or the adverse-event probabilities.