Reconciling Nonmonotone Observed DLT Rates

Monotone probabilities do not require monotone observed rates

A BLRM represents the underlying probability of dose-limiting toxicity (DLT) as a function of dose. In the monotone version considered here, that probability cannot decrease as dose increases. The observed DLT rate at each dose, however, is calculated from a finite number of patients and can rise or fall because of sampling variability. A decrease in observed rates is therefore not, by itself, a contradiction of the model.

The important question is whether the observed pattern is reasonably compatible with the sampling variation expected under the fitted curve. Small, isolated reversals may be unsurprising when cohorts are small. A large or repeated reversal, especially with substantial data at the affected doses, can instead signal that the model fits poorly or that the data need review. The model assumption should be examined, not enforced by altering the observations.

Definition: Let \(d_j\) be dose level \(j\), \(n_j\) the number of patients assessed for DLT at that dose, and \(y_j\) the number with a DLT. The observed rate is \(\hat p_j=y_j/n_j\). A rate sequence is nonmonotone if it decreases at one or more successive doses. This describes the observed data; it does not establish that the underlying DLT probabilities decrease.

The BLRM assumption and the data likelihood

Write \(x_j=\log(d_j/d_{\mathrm{ref}})\), where \(d_{\mathrm{ref}}\) is a fixed reference dose. A common two-parameter BLRM specifies

\operatorname{logit}(p_j)=\alpha+\beta x_j, \qquad \operatorname{logit}(p)=\log\!\left(\frac{p}{1-p}\right), \qquad p_j=\Pr(\mathrm{DLT}\mid d_j).

For a monotone increasing dose-toxicity curve, this version constrains \(\beta>0\). Indeed, because \(d_j\) increases with \(x_j\), the derivative of the toxicity probability with respect to log dose is

\frac{dp}{dx}=\beta p(1-p)>0 \quad\text{when }0<p<1\text{ and }\beta>0.

This is a restriction on the probabilities \(p_j\), not on the realized proportions \(\hat p_j\). Under the usual working likelihood, conditional on the dose assignments and model parameters, DLT counts at different dose levels are independent and

Y_j\mid \alpha,\beta\sim\operatorname{Binomial}(n_j,p_j), \qquad L(\alpha,\beta)=\prod_j {n_j\choose y_j}p_j^{y_j}(1-p_j)^{n_j-y_j}.

The posterior is proportional to this likelihood times the specified prior for \(\alpha\) and \(\beta\), with prior support consistent with the monotonicity restriction. This analysis assumes that each patient’s DLT status is assessed using the defined DLT window and that the binomial model is an adequate description of the counts. Dependence between outcomes, materially different follow-up, data errors, or important variation in patient risk can undermine that working model.

The likelihood permits any sequence of observed counts, including one in which a higher dose has fewer observed DLTs. It gives such a sequence its probability under the increasing probabilities predicted by the parameters; it does not require the realized counts to be ordered.

Worked examples

Worked Example: A Small Reversal Under an Increasing Curve

Suppose six hypothetical patients are assessed at each of two doses. The lower-dose group has \(y_2=2\) DLTs and the higher-dose group has \(y_4=1\). The observed rates are \(2/6=0.3333\) and \(1/6=0.1667\), respectively. Consider one illustrative increasing-curve parameter value giving \(p_2=0.3648\) and \(p_4=0.5689\). These probabilities are illustrative fixed values for this calculation, not posterior summaries.

Under the binomial model, the probability of exactly two DLTs among six patients at the lower dose is

\Pr(Y_2=2) ={6\choose2}(0.3648)^2(1-0.3648)^4 \approx 0.32497.

At the higher dose, the probability of exactly one DLT is

\Pr(Y_4=1) ={6\choose1}(0.5689)(1-0.5689)^5 \approx 0.05082.

If the two counts are conditionally independent at these parameter values, the probability of this particular pair of counts is approximately

0.32497\times0.05082\approx0.01652.

Thus, the exact pair is possible even under an increasing curve, although its probability at these particular parameter values is modest. This calculation is not the probability of any rate decrease, nor is it a posterior predictive probability: it conditions on fixed probabilities and considers one exact pair of counts. To assess the broader pattern, one must define an event or discrepancy that includes other possible count pairs and account for parameter uncertainty.

Worked Example: Comparing a Monotone Fit With a Saturated Fit

Use the same hypothetical counts: 2 DLTs among 6 patients at the lower dose and 1 among 6 at the higher dose. A saturated model assigns a separate probability to each dose, so its maximum-likelihood estimates are \(1/3\) and \(1/6\). A two-dose logistic model with \(\beta\geq0\) cannot fit a decreasing probability. Its maximum occurs at the boundary \(\beta=0\), where both dose probabilities are equal to the pooled rate \(3/12=0.25\).

For comparing these fits, omit the binomial coefficients, which are identical in both likelihoods. The maximized log likelihood under the saturated fit is

\ell_{\mathrm{sat}} =2\log(1/3)+4\log(2/3) +1\log(1/6)+5\log(5/6) \approx -6.52245.

Under the monotone model’s boundary fit, it is

\ell_{\mathrm{mono}} =3\log(0.25)+9\log(0.75) \approx -6.74802.

The saturated model improves the maximized log likelihood by about \(0.22557\). The corresponding descriptive deviance is

2(\ell_{\mathrm{sat}}-\ell_{\mathrm{mono}}) \approx 2(0.22557)=0.4511.

This small difference indicates that, for these data, allowing the two probabilities to decrease does not improve the maximized fit much. It is not automatically a calibrated hypothesis-test result: the models are being compared under an inequality constraint, and the counts are small. A formal reference distribution or a Bayesian comparison would require its own justification. Here, the calculation is a transparent measure of the fit cost imposed by monotonicity, not a rule for accepting or rejecting the BLRM.

Worked Example: A Larger Reversal as a Diagnostic Signal

Consider hypothetical counts at three increasing doses: \(4/40\), \(24/40\), and \(8/40\). The observed rates are \(0.10\), \(0.60\), and \(0.20\). The decrease from the second to the third dose is \(0.60-0.20=0.40\). As a rough diagnostic, under independent binomial sampling, the estimated standard error of the difference between those two observed rates is

\widehat{SE} =\sqrt{\frac{0.60(1-0.60)}{40} +\frac{0.20(1-0.20)}{40}} =\sqrt{0.006+0.004} =0.100.

The observed drop is therefore \(0.40/0.100=4.00\) of these estimated standard errors. This is a conspicuous reversal and should prompt careful investigation. The calculation is only a rough descriptive signal: it uses observed rates in place of unknown probabilities and is not a formal test of the BLRM. A posterior predictive check under the fitted model can assess the whole pattern while incorporating uncertainty in the parameters.

Assessing whether a reversal challenges the model

A useful diagnostic starts with the question: how often would the fitted model generate a discrepancy at least as pronounced as the one observed? For example, define a maximum adjacent drop by

T(\mathbf y)= \max_{j}\left(\frac{y_j}{n_j}-\frac{y_{j+1}}{n_{j+1}}\right).

A positive value identifies at least one observed decrease. To conduct a posterior predictive check, draw \((\alpha,\beta)\) from the posterior, generate replicate counts \(Y_j^{\mathrm{rep}}\) from the binomial model at the observed dose levels and sample sizes, and calculate \(T(\mathbf Y^{\mathrm{rep}})\) for each replicate. The proportion of replicates with \(T(\mathbf Y^{\mathrm{rep}})\geq T(\mathbf y)\) summarizes how often the fitted model reproduces a drop at least this large. The comparison is meaningful only if the discrepancy is chosen to reflect the feature being investigated and the replicate data are generated under the stated sampling model.

This posterior predictive tail proportion is a diagnostic, not the probability that the model is true or a conventional p-value. Because the same observations inform the posterior and the check, posterior predictive checks can also be conservative. Look at the pattern, not only one summary: a check based on the maximum drop may miss a sustained sequence of reversals or other ways the curve can fail to fit.

When a reversal appears, useful checks include verifying dose labels and DLT classifications, confirming that each patient had the required follow-up, and checking that cohorts are comparable in assessment and recording. Then inspect the posterior curve and predictive distributions alongside observed counts. A single irregularity in small cohorts can arise through sampling variation; repeated or large discrepancies deserve more scrutiny. These observations are diagnostic clues, not proof of a particular data problem or alternative dose-toxicity relationship.

Common mistake, alternatives, and limitations

Mistake: “The higher-dose rate is lower, so the monotone model is impossible.” This confuses observed proportions with underlying probabilities. Binomial sampling can produce a lower observed rate at a higher dose even when its true probability is larger.
Mistake: changing the counts to make them increase with dose. This alters the data and invalidates the likelihood. Keep the recorded counts; investigate their quality and assess the model’s fit.
Mistake: treating one calculated probability or deviance as a verdict. An exact count-pair probability depends on the parameter values, while the interpretation of a deviance depends on its reference distribution. Neither alone establishes that the monotonicity assumption is adequate or inadequate.

The monotone BLRM is a structured, parsimonious way to estimate a smooth increasing dose-toxicity relationship, particularly when data are sparse. A saturated model is more flexible because it estimates each dose probability separately, but it does not impose a dose-response structure and can be unstable with small cohorts. Order-restricted fits can enforce nondecreasing probabilities without the BLRM’s specific logistic shape, but they too constrain the probabilities and do not make observed rates monotone. These alternatives answer different modeling questions; none makes data checks unnecessary.

A monotone curve may fit poorly if the true relationship is not adequately represented by the model, if relevant patient groups differ across doses, or if the working likelihood assumptions fail. With sparse data, a posterior predictive check may have limited ability to distinguish these possibilities. The appropriate conclusion is calibrated: the observations may be compatible with ordinary sampling variation, or they may reveal a discrepancy requiring investigation. A model-based dose decision should not be replaced by an informal reading of the raw rates, but neither should a clear lack of fit be ignored.

Key takeaway: Monotonicity constrains the BLRM’s underlying DLT probabilities, not the observed DLT rates. Preserve the counts, quantify the size of reversals, and use model-based predictive diagnostics to judge whether sampling variation is a plausible explanation.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions below.

At two increasing doses, the observed DLT rates are \(3/10\) and \(2/10\). Does this alone show that the underlying dose-toxicity curve is decreasing?
For an increasing logistic BLRM with \(0<p_j<1\), what condition on \(\beta\) makes the toxicity probability increase with log dose?
At one posterior draw, suppose two dose probabilities are \(0.30\) and \(0.50\), with 5 patients at each dose. Write the binomial expression for the probability of exactly one DLT at the lower dose and zero at the higher dose, conditional on this draw and independent counts.
Why is the posterior predictive proportion of replicates with a drop at least as large as observed not the probability that the BLRM is true?

Solutions. (1) No. The observed proportions are estimates from finite samples; a lower rate at the higher dose can occur through sampling variation under increasing probabilities. (2) \(\beta>0\), since \(dp/dx=\beta p(1-p)>0\). (3) Conditional independence gives \({5\choose1}(0.30)(0.70)^4\times {5\choose0}(0.50)^0(0.50)^5\). The expression is approximately \(5(0.30)(0.70)^4(0.50)^5=0.0112546875\approx0.01125\). (4) It is a model-based diagnostic computed using data that also informed the posterior. It describes how often replicated data under the fitted model produce the chosen discrepancy; it does not assign a probability to the model being true.