Why Calibrate a Joint Dose-Finding Algorithm?
A joint dose-finding algorithm combines evidence about benefit and toxicity to decide where to assign patients, when to stop, and which dose to recommend. Its rules contain choices: a posterior-probability threshold for declaring a dose acceptably safe, a boundary for stopping when benefit appears insufficient, or a prior concentration that determines how strongly the model is regularized. These choices affect patient allocation and the final recommendation. Calibration uses repeated simulated trials to assess whether candidate choices achieve prespecified goals under a specified joint outcome model.
The earlier material on operating characteristics defines the quantities used to describe a design’s performance. Calibration takes the next step: it uses those quantities to select algorithm settings. The goal is not to find one setting that looks best in one simulated situation. It is to find a setting that meets required safety and stopping constraints across prespecified scenarios, then compare the remaining candidates on dose-selection performance, sample size, or another stated objective.
Specify the Model, Algorithm, and Goals
For clarity, consider binary efficacy \(E\) and binary toxicity \(T\), measured for each patient at an assigned dose \(d\). Write \(p_{et,d}=P(E=e,T=t\mid d)\), where \(e,t\in\{0,1\}\). The four probabilities sum to one. In particular, the marginal efficacy and toxicity probabilities at dose \(d\) are \(p_{10,d}+p_{11,d}\) and \(p_{01,d}+p_{11,d}\). Modeling the four joint probabilities allows efficacy and toxicity to be associated; it does not assume independence between them.
A calibration exercise must state its assumptions. In the examples below, patients’ joint outcomes are independent conditional on their assigned doses, and each dose has fixed joint probabilities throughout a trial. Outcomes are fully observed before the next allocation decision. The algorithm and simulations use the same outcome definitions and follow-up rules. These assumptions are illustrative; a real calibration must represent the design’s actual outcome timing, allocation rules, and joint model.
Suppose the protocol defines a dose as suitable when its efficacy probability is at least \(\pi_E\) and its toxicity probability is no more than \(\pi_T\). The algorithm might exclude doses unless their posterior probability of being below the toxicity limit exceeds a threshold \(c\), stop if posterior evidence of any suitable dose falls below a boundary \(\gamma\), and select the highest dose that passes its final rules. These are design choices, not universal constants. The calibration target should state, for example, a maximum probability of recommending an unsafe dose and a minimum probability of selecting a suitable dose when one exists.
A candidate setting is therefore a complete set of choices, not an isolated number. For each candidate, simulate many trials under every prespecified scenario, apply the algorithm exactly as written, and record outcomes such as whether the recommended dose is suitable, whether an unsafe dose is recommended, whether the trial stops, and how many patients are enrolled. Compare candidates against criteria declared before examining their simulated performance.
Monte Carlo Estimation and Its Precision
For a given candidate setting and scenario, let \(R\) denote the number of simulated trials, and let \(Y_r=1\) if replicate \(r\) meets a specified event, such as recommending an unsafe dose. The estimated operating characteristic is the fraction of replicates meeting that event. Because it is estimated from a finite simulation, it has Monte Carlo error.
This standard error describes uncertainty from using a finite number of simulated replicates. It is not the standard error of a treatment-effect estimate in a trial, and it does not represent uncertainty about the model’s true probabilities. If a candidate is close to a calibration limit, increase \(R\) before deciding whether the limit is met. A small simulated difference between candidates may be Monte Carlo noise rather than a meaningful design difference.
Worked Example: Selecting a Safety Threshold
Consider three doses with hypothetical joint probabilities shown below, in the order \((p_{00,d},p_{10,d},p_{01,d},p_{11,d})\): dose 1 has \((0.45,0.40,0.08,0.07)\), dose 2 has \((0.25,0.45,0.15,0.15)\), and dose 3 has \((0.12,0.43,0.25,0.20)\). Let \(\pi_E=0.50\) and \(\pi_T=0.30\). Dose 1 has efficacy \(0.40+0.07=0.47\) and toxicity \(0.08+0.07=0.15\); dose 2 has efficacy \(0.60\) and toxicity \(0.30\); dose 3 has efficacy \(0.63\) and toxicity \(0.45\). Thus dose 2 is the only suitable dose.
Suppose the algorithm selects the highest dose that passes its posterior safety and efficacy rules. The candidate safety threshold \(c\) is the required posterior probability that a dose’s toxicity probability is no greater than \(0.30\). Simulate \(R=5{,}000\) trials under these truths for each candidate. The following are hypothetical simulation results:
| Safety threshold \(c\) | Suitable recommendation | Unsafe recommendation | No recommendation |
|---|---|---|---|
| 0.80 | 3,250/5,000 = 65% | 400/5,000 = 8% | 1,350/5,000 = 27% |
| 0.90 | 3,050/5,000 = 61% | 200/5,000 = 4% | 1,750/5,000 = 35% |
| 0.95 | 2,700/5,000 = 54% | 100/5,000 = 2% | 2,200/5,000 = 44% |
Assume the design goals are an unsafe-recommendation probability no greater than \(5\%\), a no-recommendation probability no greater than \(40\%\), and then the largest possible suitable-recommendation probability. The \(0.80\) candidate fails the safety limit because \(400/5{,}000=0.08\). The \(0.95\) candidate meets the safety limit but fails the no-recommendation limit because \(2{,}200/5{,}000=0.44\). The \(0.90\) candidate meets both constraints and has a suitable-recommendation rate of \(3{,}050/5{,}000=0.61\).
For the selected candidate, the Monte Carlo standard error of the unsafe-recommendation estimate is approximately \(\sqrt{0.04(1-0.04)/5{,}000}=0.00277\), or 0.277 percentage points. The estimated \(4\%\) is therefore not exact: its approximate 95% Monte Carlo margin is \(1.96(0.277)=0.543\) percentage points. This precision is informative for simulation error, but does not guarantee performance outside the specified scenario.
Tuning Stopping Boundaries
Stopping rules also require calibration. An aggressive boundary can end a trial quickly when no dose is suitable, but may prematurely stop a trial in which a suitable dose exists. A useful pair of performance measures is the probability of stopping under a scenario with no suitable dose and the probability of stopping before identifying a suitable dose under a scenario where one exists. The acceptable balance is a design goal; it cannot be determined from the boundary’s numerical value alone.
Worked Example: Choosing an Early-Stopping Boundary
Suppose the algorithm stops for futility when the posterior probability that at least one dose is both sufficiently efficacious and acceptably safe falls below \(\gamma\). Candidate values \(\gamma=0.10\) and \(\gamma=0.20\) are evaluated with 4,000 hypothetical trial replicates per scenario. The no-suitable-dose scenario has no dose meeting both targets; the suitable-dose scenario has one dose meeting both.
For \(\gamma=0.10\), the algorithm stops in 3,320 of 4,000 trials under the no-suitable-dose scenario, so the estimated desired stopping probability is \(3{,}320/4{,}000=0.83\). It prematurely stops in 240 of 4,000 trials under the suitable-dose scenario, an estimated probability of \(240/4{,}000=0.06\). For \(\gamma=0.20\), the corresponding estimates are \(3{,}600/4{,}000=0.90\) and \(520/4{,}000=0.13\).
If the goals are to stop in at least \(80\%\) of trials when no suitable dose exists and to prematurely stop in no more than \(10\%\) when one does exist, \(\gamma=0.10\) meets both goals. The \(0.20\) setting meets the first but not the second. The choice follows from the stated trade-off, not from a general claim that one boundary is always preferable.
Calibrating Prior Strength
Prior distributions affect how quickly the algorithm’s posterior probabilities change as outcomes accumulate. For a four-category joint model at a dose, a common example is a Dirichlet prior with parameters \(\kappa q_{00},\kappa q_{10},\kappa q_{01},\kappa q_{11}\), where the \(q\)'s sum to one. The prior mean for each category is \(q_{et}\); \(\kappa\) is the total prior concentration. Larger \(\kappa\) gives the same prior mean more influence relative to the observed outcomes. A prior should be scientifically defensible; calibration can assess the operational consequences of its strength, but cannot make an implausible prior appropriate.
Worked Example: Comparing Prior Concentrations
Suppose the prior mean joint probabilities at each dose are fixed, while \(\kappa\) is varied across 1, 4, and 12. Under the same hypothetical joint outcome scenario and algorithm thresholds, 5,000 simulated trials produce the following results:
| Prior concentration \(\kappa\) | Suitable recommendation | Unsafe recommendation | No recommendation |
|---|---|---|---|
| 1 | 2,900/5,000 = 58% | 200/5,000 = 4% | 1,900/5,000 = 38% |
| 4 | 3,200/5,000 = 64% | 350/5,000 = 7% | 1,450/5,000 = 29% |
| 12 | 3,450/5,000 = 69% | 550/5,000 = 11% | 1,000/5,000 = 20% |
If the unsafe-recommendation limit is \(5\%\), only \(\kappa=1\) meets that constraint: \(200/5{,}000=0.04\). Although \(\kappa=12\) has the largest suitable-recommendation rate, \(11\%\) unsafe recommendations violates the goal. Thus, among these candidates, \(\kappa=1\) is selected under the stated constraint. This result is specific to the prior mean, scenario, and algorithm; it is not a universal argument for weak priors.
A Practical Calibration Sequence
State the safety, efficacy, selection, stopping, and resource goals, and specify the true joint probabilities used to evaluate them.
Vary only prespecified quantities, such as posterior thresholds, prior concentration, or stopping boundaries. Keep the remaining algorithm rules fixed so comparisons are interpretable.
For every setting and scenario, run the complete algorithm and estimate each target operating characteristic, along with its Monte Carlo standard error.
Discard settings that fail required goals. Rank only those that remain using a stated objective, and increase the number of replicates when Monte Carlo error could change the decision.
Diagnostics, Alternatives, and Limitations
Calibration is going wrong if a reported operating characteristic cannot be traced to a defined event, a specified scenario, and an implemented algorithm. Check that the four joint probabilities sum to one at every dose, that marginal efficacy and toxicity are calculated from the correct cells, and that the simulation applies allocation, stopping, and recommendation rules in their intended order. Confirm that event counts add up to the total number of replicates. Re-run close comparisons with more replicates, and examine every prespecified scenario rather than reporting only the most favorable one.
A table of candidate settings is often more useful than a single “optimal” setting. It makes the safety-selection trade-off visible and shows whether several candidates meet the constraints. Analytic calculations can be useful for simple fixed designs, but adaptive allocation, posterior decisions, and early stopping usually make whole-trial simulation the more direct way to evaluate the algorithm. Expert judgment can propose sensible threshold ranges, but selecting a number by convention alone does not establish that the design meets its goals.
Calibration has important limits. It establishes performance only for the simulated scenarios, assumptions, and implementation evaluated. Monte Carlo precision cannot compensate for an inadequate set of scenarios, and a high selection rate under one assumed model is not evidence that the model is correct. Thresholds must also be chosen before observing the trial’s accumulating outcomes; adjusting them in response to emerging results changes the design being evaluated. Calibration is a design decision supported by simulation, not a guarantee of safety or benefit in every possible setting.
Check Your Understanding
Answer each question using the stated calibration principles.
- At a dose, \(p_{01}=0.12\) and \(p_{11}=0.08\). What is the marginal toxicity probability?
- In 2,500 simulated trials, 125 recommend an unsafe dose. Calculate the estimated unsafe-recommendation probability and its approximate Monte Carlo standard error.
- One candidate selects a suitable dose in 70% of trials but recommends an unsafe dose in 8%. Another selects a suitable dose in 62% and recommends an unsafe dose in 3%. If the unsafe-recommendation limit is 5%, which candidate remains eligible for comparison?
- Why should calibration compare candidates across prespecified scenarios rather than rely on one scenario?
Solutions. (1) Toxicity occurs when \(T=1\), whether efficacy occurs or not, so \(P(T=1)=p_{01}+p_{11}=0.12+0.08=0.20\), or 20%. (2) The estimate is \(125/2{,}500=0.05\), or 5%. Its approximate Monte Carlo standard error is \(\sqrt{0.05(1-0.05)/2{,}500}=\sqrt{0.000019}=0.00436\), or 0.436 percentage points. (3) The first candidate fails the 5% constraint because 8% exceeds it. The second candidate remains eligible because 3% is below the limit; its 62% selection rate can then be compared with other eligible candidates. (4) A candidate can perform well under one assumed pattern of joint outcomes and poorly under another. Prespecified scenarios reveal whether the design meets its goals across the situations it was intended to address.