Dose-Response Models For Toxicity Within Joint Designs

Why Model Toxicity by Dose?

In a joint efficacy-toxicity design, a dose is judged using more than its chance of producing an efficacy outcome. The marginal probability of a clinically defined toxicity is another essential input: it describes the chance of toxicity at each dose, averaged over the patients represented by the model. A dose-response model can borrow information across dose levels and provide estimates even when some dose cohorts are small.

The goal here is to model that marginal toxicity probability. A marginal model does not say which patients experience both efficacy and toxicity, nor does it specify dependence between the outcomes. Those questions require additional assumptions. For toxicity-based decisions, it is also important to make the clinical meaning of the event and the observation period explicit: for example, dose-limiting toxicity within 28 days.

Definition: Let $Y_T=1$ if a patient experiences the prespecified toxicity within the specified observation window, and $Y_T=0$ otherwise. At dose $d$, the marginal toxicity probability is $p_T(d)=P(Y_T=1\mid D=d)$, where $D$ is the assigned dose. A toxicity threshold $p_T^*$ is a prespecified maximum acceptable value of $p_T(d)$, chosen to reflect clinical priorities.

A simple starting model treats outcomes as independent Bernoulli observations conditional on assigned dose, with a common toxicity probability for patients at the same dose. The toxicity definition and follow-up window must be consistent across patients, and outcomes used as complete binary responses must have adequate follow-up. Randomization or a suitable design and analysis must also address differences in patient characteristics across dose groups. The model describes the population represented by the trial; it does not guarantee that this population represents every future treatment setting.

Parametric Models and Monotonicity

A logistic model is a common way to link dose to toxicity. It keeps predicted probabilities between zero and one and gives a direct condition for a nondecreasing dose-response curve.

Definition: A logistic dose-response model is $$ \operatorname{logit}\{p_T(d)\}=\log\left\{\frac{p_T(d)}{1-p_T(d)}\right\}=\alpha+\beta z(d), $$ where $z(d)$ is a specified dose transformation, such as dose centered and divided by a scale, and $\alpha,\beta$ are model parameters. If $z(d)$ increases with dose and $\beta\geq 0$, the model constrains toxicity probability to be nondecreasing with dose.

The transformation determines how to interpret the slope. If $z(d)=(d-d_0)/s_d$, increasing dose by $s_d$ units multiplies the odds of toxicity by $\exp(\beta)$. If $z(d)=\log(d/d_0)$, multiplying dose by $c$ multiplies the odds by $\exp\{\beta\log(c)\}$. Odds ratios are not probability differences: the change in probability depends on the starting probability.

The monotonicity condition follows directly by differentiating. For a differentiable increasing transformation $z(d)$,

\frac{d p_T(d)}{d d} =\beta z'(d)p_T(d)\{1-p_T(d)\}.

Since a probability lies between zero and one, $p_T(d)\{1-p_T(d)\}>0$. Thus the derivative is nonnegative when $\beta\geq0$ and $z'(d)>0$. A negative slope would instead imply decreasing toxicity with dose; allowing it can be appropriate when scientifically plausible, but it does not encode the usual increasing-toxicity assumption.

Worked Example: Interpreting a Fitted Logistic Model

Suppose a hypothetical trial observes complete toxicity outcomes for 20 patients at each of 50, 100, 150, and 200 mg; the respective toxicity counts are 1, 3, 7, and 11. A logistic model using $z(d)=(d-100)/50$ has hypothetical fitted estimates $\hat{\alpha}=-1.80$ and $\hat{\beta}=0.90$. The coefficient estimates are treated here as the output of fitting the model; the calculations below show how to interpret its predictions.

At 150 mg, $z(150)=(150-100)/50=1$. The fitted log odds are $-1.80+0.90(1)=-0.90$, so $$ \hat{p}_T(150)=\frac{\exp(-0.90)}{1+\exp(-0.90)}=0.2891. $$ At 100 mg, $z(100)=0$, and $\hat{p}_T(100)=\exp(-1.80)/\{1+\exp(-1.80)\}=0.1419$. The fitted toxicity probability is therefore about 14.2% at 100 mg and 28.9% at 150 mg under this model.

Because $\hat{\beta}=0.90>0$, fitted toxicity increases with dose. A 50 mg increase multiplies the fitted odds by $\exp(0.90)=2.460$. It does not multiply the probability by 2.460; the predicted probabilities above illustrate the distinction. The monotonicity is a model constraint, not proof that every patient's toxicity risk rises in exactly this fashion.

Monotone Estimates Without a Logistic Curve

A parametric curve can be efficient when its shape is plausible, but a particular logistic curve may not fit well. An alternative is isotonic regression: estimate a nondecreasing toxicity probability at each observed dose without requiring one smooth curve. For binomial outcomes, the fitted values maximize the binomial likelihood subject to the ordering constraint. The pool-adjacent-violators algorithm (PAVA) implements this fit by pooling adjacent dose groups whenever their observed proportions violate the required order.

For two adjacent groups pooled into one block, the fitted probability is the total number of toxicities divided by the total number of patients in the block. This weighted average, rather than the unweighted average of the two proportions, is appropriate because it maximizes the binomial likelihood for a shared probability. Further pooling may be necessary until the resulting block probabilities are nondecreasing.

Worked Example: Enforcing Monotonicity With PAVA

Consider hypothetical complete outcomes at four increasing doses. Each dose has 10 patients, with toxicity counts 2, 5, 4, and 8. The observed proportions are 0.20, 0.50, 0.40, and 0.80. The reversal from 0.50 to 0.40 violates a nondecreasing constraint.

Pool the second and third dose groups. The shared estimate is $$ \frac{5+4}{10+10}=\frac{9}{20}=0.45. $$ The resulting fitted proportions are 0.20, 0.45, 0.45, and 0.80, which are nondecreasing. No further pooling is required because $0.20\leq0.45$ and $0.45\leq0.80$.

The fit assigns the same estimated probability, 45%, to the second and third doses even though their observed proportions differed. This does not claim their true risks are identical; it is the likelihood-maximizing fit under the specified monotonicity constraint. If the groups had different sample sizes, their pooled estimate would still be total toxicities divided by total patients, not the simple average of their observed proportions.

Monotonicity is a scientific assumption to assess, not a rule that makes all data credible. Random variation can create a local reversal even when the underlying probabilities increase. Conversely, a consistent or clinically meaningful reversal may signal a data problem, an unmodeled feature of treatment, or a failure of the monotonicity assumption. Isotonic regression can conceal the reversal in the fitted values, so the unpooled observations must remain visible during review.

Translating a Clinical Threshold Into a Dose

A fitted toxicity model can translate a clinically selected maximum acceptable toxicity probability $p_T^*$ into a corresponding dose. For the logistic model, solve the model equation at the threshold. This yields a model-based boundary, not an automatically justified dose recommendation: clinical priorities determine the threshold, and the data determine how well the dose-response curve is estimated.

Result: If $z(d)$ is increasing and $\beta>0$, the dose at which the fitted toxicity probability equals $p_T^*$ satisfies $$ z(d^*)=\frac{\operatorname{logit}(p_T^*)-\alpha}{\beta}. $$ If the transformation can be inverted, this equation gives $d^*$. Under the fitted model, doses below $d^*$ have toxicity probabilities below $p_T^*$, and doses above it have probabilities above $p_T^*$. This interpretation applies only within a range where the model is credible.

The identity follows by substituting $p_T(d^*)=p_T^*$ into the logistic model: $\operatorname{logit}(p_T^*)=\alpha+\beta z(d^*)$. Subtracting $\alpha$ and dividing by $\beta$ gives the displayed expression. When $\beta=0$, the model is flat: there may be no dose at which the probability equals the threshold, or every dose may equal it. When the solution lies outside the studied range, it is extrapolation and should not be treated as an observed, supported dose boundary.

Worked Example: Solving for the Dose at a Toxicity Cap

Suppose a hypothetical analysis uses $\operatorname{logit}\{p_T(d)\}=-2.00+0.70\log(d/100)$, with dose $d$ measured in mg. A clinical team has specified a maximum acceptable toxicity probability of $p_T^*=0.25$. The model estimates the threshold dose by solving $$ \log(d^*/100)=\frac{\log\{0.25/(1-0.25)\}-(-2.00)}{0.70} =\frac{-1.0986+2.00}{0.70}=1.288. $$

Therefore $d^*=100\exp(1.288)=362.4$ mg. For context, the fitted probabilities at 100 and 200 mg are, respectively, $$ \hat{p}_T(100)=\frac{\exp(-2.00)}{1+\exp(-2.00)}=0.1192, \qquad \hat{p}_T(200)=\frac{\exp\{-2.00+0.70\log(2)\}}{1+\exp\{-2.00+0.70\log(2)\}}=0.1803. $$

Both predictions are below 0.25, consistent with the model-based threshold at 362.4 mg. But if the hypothetical trial studied doses only up to 300 mg, the threshold lies beyond the observed range. The model does not provide direct trial evidence that 362.4 mg is tolerable; the estimate depends on extending the fitted curve beyond observed doses. A decision should respect the studied dose range and uncertainty, rather than treating this extrapolated crossing as established.

Using Toxicity Models Within a Joint Design

A toxicity model supplies dose-specific marginal estimates that can be considered alongside dose-specific efficacy estimates and prespecified clinical priorities. A simple use is to flag doses whose estimated toxicity probability exceeds a clinical cap; a more cautious approach accounts for uncertainty in that probability. The threshold itself should be justified clinically rather than selected because it makes a favored dose appear acceptable.

The logistic and isotonic approaches make different commitments. A logistic model imposes a smooth functional shape and can estimate a curve using relatively few parameters, but predictions may be distorted by the wrong link or dose transformation. Isotonic regression imposes only nondecreasing order at the observed doses, avoiding a specific curve shape, but may produce flat blocks and offers no natural smooth extrapolation. Earlier material on model-based dose finding, including the continual reassessment method (CRM), is relevant when a design uses a dose-toxicity model to guide assignments; the present focus is how to represent toxicity probabilities for joint decisions, not how to derive a particular allocation rule.

Neither marginal approach identifies the probability of a joint efficacy-toxicity outcome. For example, knowing $p_T(d)$ and the marginal efficacy probability at dose $d$ does not by itself determine the probability that a patient experiences both outcomes. Combining the margins as if efficacy and toxicity were independent would add an assumption that must be justified separately.

Diagnostics, Limitations, and Common Mistakes

Review dose-level counts and observed proportions alongside fitted predictions. Large systematic discrepancies, influential dose groups, sparse data, and complete separation (for example, no observed toxicities at low doses and all toxicities at high doses) can make estimates unstable or poorly determined. A monotone fit should also be checked against the raw pattern: pooling may be statistically required under the constraint while still obscuring a potentially important reversal. Sensitivity to the dose transformation, link, and monotonicity assumption is useful when conclusions change across plausible models.

The probability being modeled must match the clinical endpoint. Different toxicity definitions, grading thresholds, or observation windows do not estimate the same $p_T(d)$. Incomplete follow-up can make a simple binary analysis misleading if patients without a fully observed window are treated as non-toxic. Finally, predictions at doses far outside the data range are especially dependent on parametric assumptions. These limitations affect how a dose threshold should be interpreted and communicated.

Common mistake: “The observed toxicity proportion at the third dose is lower than at the second, so toxicity must decrease at that dose; alternatively, monotonicity proves the higher dose is safer because its fitted rate was pooled downward.” Both conclusions overstate the data. An observed reversal can be sampling variation or a real warning, while a monotone fit is an assumption-constrained summary. Inspect the counts, assess the assumption, and avoid treating constrained fitted values as proof of individual or population risk ordering.

Check Your Understanding

Answer each question, then compare your reasoning with the brief solutions.

In a logistic model with $z(d)$ increasing, what condition on $\beta$ gives a nondecreasing toxicity probability, and why?
Two adjacent dose groups have 3 toxicities among 8 patients and 4 among 12 patients. If PAVA pools them, what is the pooled fitted probability?
A fitted dose-response curve crosses a clinical toxicity cap above the highest studied dose. What can and cannot be concluded?
At a fixed dose, both the marginal efficacy probability and marginal toxicity probability are known. Is the probability of both outcomes known without an additional assumption?

Solutions. (1) $\beta\geq0$, because $dp_T(d)/dd=\beta z'(d)p_T(d)\{1-p_T(d)\}$, with the other factors positive when $z'(d)>0$. (2) The pooled estimate is $(3+4)/(8+12)=7/20=0.35$. (3) The fitted model predicts where the crossing would occur under extrapolation, but the trial has not directly established toxicity risk at that dose; conclusions must be limited by the observed range and model assumptions. (4) No. The marginal probabilities do not specify dependence, so they do not determine the joint probability without an additional model or assumption.