Escalation Boundaries From Joint Outcome Probabilities

Why define boundaries on joint outcomes?

At a dose-finding decision, efficacy and toxicity must be considered together. A dose can produce responses and also produce unacceptable toxicity; counting either outcome alone can obscure that tradeoff. Joint outcome probabilities make the relationship explicit while allowing a design to state its clinical criteria in interpretable terms.

A boundary is a prespecified rule that maps the estimated outcome probabilities at the current dose to an action: escalate to a higher dose, retain the current dose, or de-escalate. There is no single set of numerical boundaries appropriate for every trial. They depend on the clinical thresholds and the decision rule chosen before outcomes are examined. The construction below is one transparent probability-scale rule, not a universal standard.

The four joint probabilities are nonnegative and sum to one. No independence between efficacy and toxicity is assumed: \(\theta_{11,d}\) is estimated as its own cell probability, not constructed as the product of two marginal probabilities. The rule below uses the marginal probabilities \(r_d\) and \(q_d\), each calculated by summing the relevant joint cells.

Specify the clinical thresholds and derive the regions

Let \(\phi_T\) be the largest acceptable toxicity probability and \(\psi_E\) the minimum efficacy probability considered adequate at the current dose. These cutoffs are clinical design inputs. For a chosen estimate \(\tilde q_d\) of toxicity probability and \(\tilde r_d\) of efficacy probability, define the following rule:

• De-escalate if \(\tilde q_d>\phi_T\).
• Escalate if \(\tilde q_d\leq\phi_T\) and \(\tilde r_d<\psi_E\).
• Retain if \(\tilde q_d\leq\phi_T\) and \(\tilde r_d\geq\psi_E\).

Toxicity takes priority: a dose above the toxicity boundary is not retained or escalated merely because its efficacy estimate is high. On the boundary \(\tilde q_d=\phi_T\), the dose is treated as meeting the toxicity criterion; on \(\tilde r_d=\psi_E\), it meets the efficacy criterion.

These conditions follow directly from the clinical criteria. First, acceptable toxicity requires \(\tilde q_d\leq\phi_T\); violating that inequality gives the de-escalation region. Within the toxicity-acceptable region, efficacy below its minimum gives escalation, while efficacy meeting or exceeding the minimum gives retention. In terms of joint cells, the boundaries are \(\tilde\theta_{01,d}+\tilde\theta_{11,d}=\phi_T\) and \(\tilde\theta_{10,d}+\tilde\theta_{11,d}=\psi_E\). They partition the probability simplex into the three action regions.

Estimating the joint probabilities in small cohorts

For a simple illustration, assume that participants treated at dose \(d\) have independent outcomes from the same four-category multinomial distribution, and that each participant has a fully assessed efficacy and toxicity outcome. Let \(n_{et,d}\) be the number in joint category \((E=e,T=t)\), and let \(N_d\) be the total number assessed at that dose. A Dirichlet prior with parameters \(a_{00},a_{10},a_{01},a_{11}\) gives a Dirichlet posterior with parameters \(a_{et}+n_{et,d}\). Its posterior mean for each cell probability is

For the examples, use the symmetric prior \(a_{et}=1\) in all four cells. It adds one prior count to each category, so the posterior denominator is \(N_d+4\). The estimates used in the boundary rule are then \(\tilde q_d=\tilde\theta_{01,d}+\tilde\theta_{11,d}\) and \(\tilde r_d=\tilde\theta_{10,d}+\tilde\theta_{11,d}\). This small prior avoids zero estimates in an unobserved cell, but it does not make a small cohort informative by itself.

What the boundaries do—and do not—establish

The calculation above gives a clear action for each pair of probability estimates, but it does not establish that those estimates are precise. Posterior means can be close to a threshold even when considerable posterior probability lies on both sides of it. A probability-uncertainty rule could instead require, for example, a prespecified posterior probability of exceeding the toxicity limit before de-escalating, or a prespecified probability of meeting both clinical criteria before retaining. Such rules use posterior tail probabilities rather than only posterior means; their probability cutoffs need clinical justification and design calibration.

The Dirichlet-multinomial illustration assumes a common outcome distribution for participants at a dose and fully observed outcomes. If assessment windows differ, outcomes are delayed, or participants have not completed follow-up, treating unobserved outcomes as absent can distort all four cells. Likewise, a model that imposes dose-response structure can borrow information across doses, but its estimates depend on that structure. The earlier material on joint dose models provides context for model-based estimates; the probability-scale boundaries themselves do not require a particular dose-response model.

The marginal efficacy criterion also needs careful clinical interpretation. In this rule, \(r_d\) counts efficacy whether or not toxicity occurred, while \(q_d\) counts toxicity whether or not efficacy occurred. If the clinical requirement is specifically for efficacy without toxicity, the relevant probability is \(\theta_{10,d}\), and the boundary should be defined on that joint event instead. Substituting one criterion for another after seeing the data changes the decision rule.

A further limitation is that a local boundary does not, by itself, guarantee desirable trial-wide behavior. It does not establish how often the design will recommend an unsafe dose, how many participants will be treated at each dose, or how often it will select an effective dose. Those properties depend on the whole design and should be examined under plausible joint outcome patterns. The present rule is a decision map, not a complete assessment of design performance.

Diagnostics and common pitfalls

• Check cell totals. Confirm that all four counts refer to the same dose, outcome definitions, and assessment window, and that \(n_{00}+n_{10}+n_{01}+n_{11}=N_d\).
• Check the marginal sums. Toxicity includes both \((0,1)\) and \((1,1)\); efficacy includes both \((1,0)\) and \((1,1)\). Omitting the both-outcomes cell underestimates both margins.
• Inspect uncertainty near a boundary. A point estimate just below \(\phi_T\) is not strong evidence that the true toxicity probability is acceptable. With very small cohorts, report or evaluate uncertainty rather than treating a threshold crossing as precise.
• Do not multiply marginal probabilities. Replacing \(\theta_{11,d}\) by \(q_dr_d\) assumes independence. Efficacy and toxicity may be associated, so estimate or model their joint distribution directly.
• Respect dose and safety constraints. The local rule cannot authorize a dose outside the permitted range or override a separately prespecified safety action.

A common mistake is to say, “The efficacy estimate is high, so we should stay at this dose,” even when toxicity exceeds its limit. The correction is to apply the toxicity condition first: if \(\tilde q_d>\phi_T\), this rule de-escalates regardless of \(\tilde r_d\). Another mistake is to interpret a boundary crossing as proof that the true probability is on one side. It is an action under a prespecified rule, not certainty about an unknown parameter.

How this rule compares with alternatives

A raw empirical-proportion rule replaces \(\tilde\theta_{et,d}\) with \(n_{et,d}/N_d\). It is easy to calculate but can yield zero or extreme probabilities in a small cohort. The Dirichlet posterior mean used here smooths those estimates, while remaining explicit about all four outcome categories. A posterior-tail rule goes further by basing actions on the probability that a clinical criterion is violated or met; this represents uncertainty more directly but requires choices about posterior probability cutoffs.

The BOIN12 approach uses a different decision construction for joint benefit and toxicity. The probability-scale rule here makes its marginal clinical cutoffs visible and yields simple inequalities, but it does not supply a utility-based ranking of doses or account automatically for uncertainty in its point estimates. These methods should not be treated as interchangeable: the estimand, thresholds, and intended action logic should match the trial’s clinical question.