Operating Characteristics Of Joint Dose-Finding Designs

Why Operating Characteristics Matter

A joint dose-finding design is a sequence of decisions: it uses accumulating efficacy and toxicity information to assign patients, possibly stop early, and eventually recommend a dose. A model can describe the outcomes expected at each dose, but that description alone does not tell us how the complete design behaves when applied repeatedly. A design may often recommend a suitable dose yet expose many patients to toxicity on the way there; another may limit toxicity but frequently stop without a recommendation. Operating characteristics summarize these repeated-trial consequences under specified assumptions.

The key is to evaluate the whole decision process, not just the model's fit or a final posterior probability. For each hypothetical trial, the prespecified allocation and stopping rules are applied to the outcomes that would arise under a particular data-generating scenario. Repeating this process produces distributions of final recommendations, allocation patterns, and patient outcomes. These summaries describe performance under the chosen scenario; they are not guarantees about an actual trial.

Define the Target and the Trial-Level Summaries

Suppose there are \(J\) ordered doses. For patient \(i\), let \(E_i=1\) indicate efficacy and \(Z_i=1\) indicate toxicity within the protocol-defined assessment windows. The joint outcome is the pair \((E_i,Z_i)\); its probabilities need not be obtained by treating efficacy and toxicity as independent. A dose-finding algorithm assigns patients using accumulating data and produces either a final recommended dose \(\hat d\) or no dose recommendation.

Selection accuracy requires a target defined independently of the algorithm's observed results. One possible clinical target is the dose with greatest efficacy among doses satisfying a toxicity constraint. In the examples below, the target is the dose with the largest true efficacy probability among doses whose true toxicity probability is at most 0.30; ties go to the lower dose. Other clinical objectives can define a different target, but the criterion must be explicit before interpreting accuracy.

Let \(R\) denote the number of simulated trial replicates. In replicate \(r\), write \(\hat d_r\) for the final recommendation, \(N_{jr}\) for the number of patients assigned to dose \(j\), and \(E_{ir}\) and \(Z_{ir}\) for patient outcomes. If no dose is recommended, the replicate is retained as a no-recommendation outcome; it must not silently disappear from a selection-accuracy denominator.

Here \(d^*\) is the prespecified target and \(I(\cdot)\) equals 1 when its condition is true and 0 otherwise. This estimates the probability that a trial recommends the target dose. The probability of any recommendation can also be reported separately, as can the recommendation probabilities for each individual dose. Those additional summaries reveal whether apparent inaccuracy is mostly due to wrong-dose recommendations or to stopping without a recommendation.

Allocation is summarized by dose-specific counts or proportions. A trial-level proportion is \(N_{jr}/N_r\), where \(N_r\) is the number of patients actually treated in replicate \(r\). Averaging these proportions describes the average share of treated patients assigned to each dose. If trial sizes vary because of early stopping, also report the distribution of \(N_r\): the average share and the total number of patients answer different questions.

The first quantity estimates the average allocation proportion at dose \(j\). The second estimates the mean total number of toxicities per trial. A complementary toxicity rate is the total toxicity count divided by the total number treated across replicates. State which summary is used: mean counts emphasize burden per trial, whereas a pooled rate weights trials with more patients more heavily.

Efficacy outcomes can be summarized analogously, but the precise endpoint matters. Report, for example, the mean number of patients with efficacy per trial, or the pooled efficacy proportion. A joint summary such as the probability of efficacy without toxicity, \(P(E=1,Z=0)\), retains information that separate marginal efficacy and toxicity rates cannot provide. If a utility score is used, define its values and interpretation in advance rather than selecting weights after seeing results.

Worked Examples

Monte Carlo Precision and a Short Justification

Operating characteristics are usually estimated by simulation because adaptive assignments make the distribution of a complete trial difficult to derive directly. If independent simulated replicates each produce an indicator \(I_r=I(\hat d_r=d^*)\), then \(I_r\) is Bernoulli with success probability \(\pi\), the true selection accuracy under the specified scenario and design. The estimator is the sample mean \(\hat\pi=R^{-1}\sum I_r\).

Independence across replicates gives \(\mathrm{Var}(\hat\pi)=\mathrm{Var}(I_r)/R\). Since a Bernoulli variable has variance \(\pi(1-\pi)\), replacing the unknown \(\pi\) by \(\hat\pi\) gives the estimated Monte Carlo standard error:

For the first example, this is \(\sqrt{0.660(0.340)/200}=0.0335\), or about 3.35 percentage points. This quantifies simulation error in estimating the scenario-specific selection probability; it does not describe uncertainty about the real-world outcome rates or the model assumptions. For an estimated mean count or rate, use the sample standard deviation of the replicate-level values divided by \(\sqrt R\). Replicate-level calculations are important when outcomes within a trial are related through adaptive decisions.

Assumptions, Diagnostics, and Alternatives

A simulation estimate describes the design only under the conditions used to generate and analyze its replicates. At minimum, the design rules must be fully specified, the scenario must define the outcomes that can occur at each dose, and the outcome-generation process must represent the relevant dependence between efficacy and toxicity. The simulated trial must also apply the protocol's handling of assessment timing, missing or unevaluable outcomes, and stopping. If these features are omitted or coded differently from the intended design, a precise numerical estimate can still answer the wrong question.

Check that dose allocation counts sum to the number treated, allocation proportions sum to 1 within a complete trial, and joint outcome probabilities are nonnegative and sum to 1 at each dose. Confirm that the target is calculated from the stated criterion, not inferred from which dose the algorithm most often selects. Inspect the full recommendation distribution, the no-recommendation frequency, trial sizes, and dose-specific exposure, rather than relying on one overall score. Large Monte Carlo standard errors, rare recommendations, or substantial variability between replicates signal that a point estimate alone is inadequate.

Operating characteristics should not be confused with posterior summaries from one observed trial. A posterior probability describes uncertainty about model parameters conditional on that trial's data and model; a repeated-trial selection probability describes how often the design makes a specified decision over hypothetical repetitions under a scenario. Neither quantity substitutes for the other. Likewise, a single average can conceal poor performance in an important scenario, so operating characteristics should be interpreted alongside the scenario and endpoint definitions that produced them.

The main limitation is dependence on the assumed scenarios and implementation. Results do not establish performance under unexamined outcome distributions, and a favorable average can mask unsafe allocations or low accuracy in a clinically important case. A broad set of plausible scenarios gives a more informative performance picture than one convenient scenario. Separate patient-level safety, efficacy, and joint outcome summaries are often more interpretable than an arbitrarily weighted composite. If a composite is clinically justified, its weights and decision meaning must be prespecified.

Common Pitfall

A common error is to report the fraction of correct recommendations only among trials that recommended a dose and call it selection accuracy. In the first example, this gives \(132/186=71.0\%\), while the probability of recommending the target among all trials is \(132/200=66.0\%\). Excluding stopped trials can make performance look better by removing a meaningful design outcome. Report unconditional selection accuracy and the probability of no recommendation; give conditional accuracy only as a clearly labelled supplementary summary.