Tutorials › Adaptive Clinical Trial Design › Diagnosing Nonmonotone DLT Patterns

Rule-Based Escalation Designs · Tutorial 534 of 799

Diagnosing Nonmonotone DLT Patterns

Learn to diagnose decreasing observed DLT proportions, audit possible inconsistencies, and apply the planned 3+3 decisions without treating a noisy pattern as a reason to rewrite the rules.

Intermediate 10 min read

What You'll Learn

  • Define a nonmonotone observed pattern using proportions at successive dose levels.
  • Distinguish a downturn in observed proportions from a decrease in raw DLT counts.
  • Explain how sparse cohorts can produce a downturn even when the underlying risks are equal.
  • Audit DLT definitions, observation windows, denominators, and source records.
  • Apply the prespecified 3+3 decision at each dose without changing the algorithm after seeing results.
  • Recognize when unresolved data or protocol discrepancies require clarification before a decision.

Why a DLT Pattern Can Turn Down

In dose escalation, it is clinically plausible for the probability of a dose-limiting toxicity (DLT) to rise as dose increases. Yet the observed DLT results need not rise smoothly. Early cohorts are small, so a higher dose may show fewer DLTs than a lower one simply by chance. Alternatively, a downturn may flag a problem with the data, the dose assignment, or how the protocol’s DLT definition was applied. The pattern is a reason to check carefully, not by itself a diagnosis or permission to change the escalation rule.

The relevant comparison is usually the observed proportion with DLT at each dose, rather than the number of DLTs alone. This matters whenever cohort sizes differ: a larger cohort can have more DLTs but a smaller DLT proportion.

Definition: Let \(x_i\) be the number of participants with a DLT among \(n_i\) evaluable participants at dose level \(i\), using the protocol’s DLT definition and observation window. The observed DLT proportion is \(\hat{p}_i=x_i/n_i\). The sequence of observed proportions is nonmonotone if it decreases at one or more successive dose levels; that is, if \(\hat{p}_{i+1}<\hat{p}_i\) for at least one \(i\). Equality does not constitute a decrease. This definition describes observed proportions, not a requirement that raw counts decrease.

For example, proportions of 0/3, 0/3, and 0/3 are monotone nondecreasing: none decreases. Proportions of 2/6 and 1/3 are equal, even though the raw counts differ. In contrast, proportions of 1/3 and 0/3 show a decrease from the first dose to the next. This is an observed downturn; it does not establish that the true DLT risk is lower at the higher dose.

What the Prespecified Rule Does

The conventional 3+3 rule makes a decision from the DLT outcomes at the dose currently being evaluated. In brief, 0 DLTs among 3 participants leads to escalation; 1 among 3 leads to expansion to 6 at that dose; and 2 or more among 3 leads to stopping escalation at that dose. After expansion, 1 or fewer DLTs among 6 permits escalation, while 2 or more among 6 leads to stopping escalation at that dose. The protocol specifies the study’s exact implementation, including how the maximum tolerated dose (MTD) is identified.

A nonmonotone sequence across dose levels does not, on its own, replace these dose-specific decisions. The design does not call for averaging adjacent cohorts, changing a DLT count because a higher dose looks safer, or skipping a rule step unless the protocol explicitly prespecifies that action. If an apparent downturn reveals a possible error, first investigate and resolve it according to documented procedures; then apply the prespecified rule to the valid, protocol-defined outcomes.

Worked Example: A Downturn After Cohort Expansion

Hypothetical data. At dose 1, 0 of 3 participants have a DLT. At dose 2, the first 3 participants include 1 DLT; 3 more are treated at dose 2, with 0 additional DLTs. At dose 3, 0 of 3 participants have a DLT. Assume all outcomes are complete and classified under the same protocol definition and observation window.

At dose 2, the first-cohort proportion is \(1/3=0.3333\). Under the 3+3 rule, 1 DLT among 3 calls for expansion at dose 2, not immediate escalation. After expansion, the pooled dose-2 proportion is:

$$ \hat{p}_2=\frac{1+0}{3+3}=\frac{1}{6}=0.1667. $$

Dose 1 has \(\hat{p}_1=0/3=0\), dose 2 has \(\hat{p}_2=1/6\), and dose 3 has \(\hat{p}_3=0/3=0\). The observed proportions decrease from dose 2 to dose 3, since \(0<1/6\). The sequence is therefore nonmonotone, even though the final higher-dose cohort had no DLTs.

The rule-based decisions still follow the results at each dose: dose 1 permits escalation; the initial result at dose 2 calls for expansion; and 1 DLT among the 6 participants at dose 2 permits escalation. At dose 3, 0 of 3 permits escalation to the next level if one is available and the protocol allows it. The downturn is worth documenting, but it does not erase the dose-2 DLT or change the prescribed decision.

Sampling Variation: A Short Justification

A downturn can occur without any data error or reversal in underlying risk. To see why, consider two successive dose levels with the same true DLT probability \(p\), and assume each has 3 independent participants whose outcomes are classified consistently. Under these assumptions, the DLT count at each level follows a binomial distribution, \(X\sim\operatorname{Binomial}(3,p)\). This is a simple illustration of sampling variation, not a claim that equal risks are the usual dose-response pattern.

If both levels have the same denominator, a decrease in observed proportions occurs exactly when the lower-dose count exceeds the higher-dose count. For \(p=0.20\), the probabilities of 0, 1, 2, and 3 DLTs in a cohort of 3 are 0.512, 0.384, 0.096, and 0.008. Independence gives:

$$ \begin{aligned} P(X_{\mathrm{low}}>X_{\mathrm{high}}) &=P(1,0)+P(2,0)+P(2,1)\\ &\quad+P(3,0)+P(3,1)+P(3,2)\\ &=(0.384)(0.512)+(0.096)(0.512)+(0.096)(0.384)\\ &\quad+(0.008)(0.512)+(0.008)(0.384)+(0.008)(0.096)\\ &=0.29056. \end{aligned} $$

Thus, under this illustrative equal-risk model, there is about a 29.1% probability that two independent cohorts of 3 will show a strictly lower observed proportion at the higher level. Small cohorts make irregular patterns unsurprising. This probability is not a test of the study’s dose-response relationship and does not establish that a particular downturn is harmless.

Justification: With independent cohorts and common \(p\), each pair of counts \((a,b)\) has probability \(P(X_{\mathrm{low}}=a)P(X_{\mathrm{high}}=b)\). A strict observed downturn for equal cohort sizes occurs for \(a>b\). Summing those six possible pairs for counts from 0 to 3 gives the probability calculated above. If cohort sizes differ, a downturn must instead be assessed by comparing \(a/n_{\mathrm{low}}\) with \(b/n_{\mathrm{high}}\); comparing counts alone is not valid.

Worked Example: More DLTs but a Lower Proportion

Hypothetical data. At a lower dose, 2 of 3 evaluable participants have DLTs. At the next dose, 3 of 6 have DLTs. The raw count rises from 2 to 3, but the observed proportions are:

$$ \hat{p}_{\mathrm{lower}}=\frac{2}{3}=0.6667, \qquad \hat{p}_{\mathrm{higher}}=\frac{3}{6}=0.5000. $$

Because \(0.5000<0.6667\), the proportions decrease at the higher dose, so this pair is nonmonotone by the definition above. Looking only at counts would miss the decrease because the higher-dose cohort is twice as large. The 3+3 decisions must nevertheless be read from the counts and denominators at each dose according to the rule: 2 of 3 at the lower dose meets the stopping criterion there, while 3 of 6 at the higher dose meets the stopping criterion there as well. The descriptive comparison does not itself determine a new dose-selection rule.

Audit Before Interpreting a Downturn

When a pattern looks surprising, verify that the quantities being compared are genuinely comparable. A short, documented review should check the following:

  • Dose level: Confirm the assigned dose and the dose actually administered against source records and the protocol’s dose-level definitions.
  • DLT definition: Check that each event was assessed using the protocol’s stated criteria, including any required severity, attribution, or event-type rules.
  • Observation window: Confirm that the same protocol-defined follow-up window was applied at each level and that outcomes were classified consistently within it.
  • Numerators and denominators: Reconcile the number of DLTs, the number of evaluable participants, and any exclusions against the trial records and prespecified evaluability rules.
  • Timing and data status: Verify that the data are sufficiently complete for the planned decision and that provisional or corrected values are clearly identified.

An inconsistency should be resolved through the trial’s documented data review and decision process, not by informally altering a count to make the dose pattern look plausible. If a discrepancy remains unresolved and could change the rule-based action, the appropriate response is to obtain clarification under the protocol’s governance procedures before acting on the disputed value. The audit does not authorize a new DLT definition, a new observation window, or a retrospective change to the escalation algorithm.

Worked Example: A DLT Classification Discrepancy

Hypothetical data. The recorded results are 0 of 3 DLTs at dose 1 and 0 of 3 at dose 2. A review of the source records finds that one dose-2 participant had an event within the protocol’s stated observation window. The event meets the protocol’s written DLT criteria, but the record had been entered as “no DLT.” Assume all three dose-2 participants have complete follow-up.

Before correction, the recorded proportions are both 0, and there is no downturn. The review does not justify counting every adverse event as a DLT; it identifies a specific event that meets the existing criteria. After the classification is corrected through the documented process, dose 2 has:

$$ \hat{p}_2=\frac{1}{3}=0.3333, \qquad \hat{p}_1=\frac{0}{3}=0. $$

The corrected sequence increases from dose 1 to dose 2. More importantly, the rule-based action at dose 2 is now the action for 1 DLT among 3: expand that dose to 6, rather than treating the original entry as 0 of 3 and escalating immediately. The correction is based on the prespecified definition and documented evidence, not on a desire to produce a monotone pattern. If the event did not meet the protocol criteria, the count would not be changed merely because it seemed clinically concerning.

Use the Pattern as a Diagnostic, Not a New Rule

The main alternatives are not competing ways to “fix” the sequence. One is to follow the prespecified rule at each dose, after verifying the data. Another is to inspect the pattern descriptively and report the downturn, its denominators, and the outcome of any audit. A pooled trend or a smoothed curve might summarize dose-response information differently, but it is not a substitute for the planned 3+3 decision unless such an analysis and its role were specified in advance. Choosing a new rule only after seeing an inconvenient pattern risks making the decision depend on the observed results in an unplanned way.

There are important limits to what a small escalation study can establish. A few cohorts cannot precisely characterize the true DLT risk at each dose. A monotone observed sequence does not prove that risk rises with dose, and a nonmonotone sequence does not prove that it falls. Even after data checks, sampling variation remains a plausible explanation. These observations should be reported with their denominators and interpreted in the context of the design, rather than presented as a definitive dose-response conclusion.

Key takeaway: Define the observed pattern using \(x_i/n_i\): a strict decrease between successive dose levels is nonmonotone; equality is not. Check comparable definitions, windows, denominators, and records, then apply the prespecified rule to the valid data. Do not invent a post hoc adjustment to make the sequence look orderly.

Common Mistake

Wrong reasoning: “The higher dose has fewer DLTs, so its true risk must be lower. We should ignore the lower-dose DLT and continue using the higher dose.” This treats noisy observed proportions as known risks and overrides the protocol’s dose-specific rule.

Correction: A downturn is an observed feature, not proof of a true reversal. Compare proportions rather than raw counts, check the data and protocol classifications, and make the decision prescribed for the valid outcomes at each dose. If a potentially decision-changing discrepancy is unresolved, use the documented clarification process instead of improvising.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions.

  1. Are observed proportions of 0/3, 0/3, and 1/6 nonmonotone? Explain.
  2. At a lower dose there are 2 DLTs among 6 participants; at the next dose there are 1 among 3. Is this a decrease in observed proportion?
  3. Under the illustrative equal-risk model with \(p=0.20\) and two independent cohorts of 3, what is the probability that the lower-dose cohort has a strictly larger observed proportion?
  4. A dose has 1 DLT among 3, and the next dose has 0 among 3. What should the team do before interpreting this as evidence of a true decrease?

Solutions. (1) The proportions are \(0\), \(0\), and \(1/6=0.1667\). There is no strict decrease, so the sequence is monotone nondecreasing. (2) Both proportions are \(2/6=1/3\) and \(1/3\); there is equality, not a decrease. (3) The probability is 0.29056, or about 29.1%, by summing the six independent binomial outcome pairs in which the lower-dose count exceeds the higher-dose count. (4) First verify dose assignment, DLT criteria, observation windows, denominators, and source records. If the data are valid, report the downturn as an observation and apply the prespecified rule; do not infer a true risk decrease or alter the algorithm from these cohorts alone.

Clinical Biostats