The Cohort Likelihood for Dose-Limiting Toxicities

Why the Cohort Likelihood Matters

In a dose-escalation trial, the observed toxicity data are often recorded as a number treated and a number with a dose-limiting toxicity (DLT) for each cohort. The cohort likelihood translates those counts into evidence about the dose-toxicity model. It is the data contribution used for Bayesian updating: the likelihood is combined with a prior distribution to obtain a posterior distribution for the model parameters.

The calculation is compact, but its assumptions matter. The binomial likelihood is appropriate when each treated patient has a binary DLT outcome defined over a common assessment period, patients in a cohort share the modeled toxicity probability, and outcomes are conditionally independent given the model. Incomplete follow-up, differing outcome definitions, or unmodeled dependence can make a simple count-based likelihood misleading.

Definition: For cohort $k$, let $d_k$ be its dose, $n_k$ the number of treated patients with evaluable DLT outcomes, and $y_k$ the number of those patients who experienced a DLT. Let $p_k=p(d_k;\alpha,\beta)$ denote the model probability of a DLT at that dose. Under the cohort binomial model, $Y_k\mid d_k,\alpha,\beta\sim\operatorname{Binomial}(n_k,p_k)$, and the likelihood contribution is $$ L_k(\alpha,\beta)=\binom{n_k}{y_k}p_k^{y_k}(1-p_k)^{n_k-y_k}. $$

From Individual Outcomes to a Cohort Contribution

Write the DLT outcome for patient $i$ in cohort $k$ as $Z_{ki}$, where $Z_{ki}=1$ indicates a DLT and $Z_{ki}=0$ indicates no DLT during the specified assessment period. Conditional on the model parameters and the cohort dose, the binomial model assumes that the $n_k$ outcomes are independent Bernoulli variables with the same probability $p_k$. The observed count is $y_k=\sum_{i=1}^{n_k}Z_{ki}$.

For one particular ordered sequence of patient outcomes containing $y_k$ ones and $n_k-y_k$ zeros, independence gives probability $p_k^{y_k}(1-p_k)^{n_k-y_k}$. The cohort count does not retain the order of those outcomes. There are $\binom{n_k}{y_k}$ distinct ordered sequences with exactly $y_k$ DLTs, and each has the same probability. Adding their probabilities yields the binomial likelihood in the definition.

For the two-parameter Bayesian logistic regression model, use the dose-scaled covariate $x(d)=\log(d/d_{\mathrm{ref}})$, where $d_{\mathrm{ref}}$ is a positive reference dose. In the usual BLRM parameterization, the DLT probability is

\operatorname{logit}\{p(d;\alpha,\beta)\} =\log(\alpha)+\beta\log(d/d_{\mathrm{ref}}), \qquad p(d;\alpha,\beta) =\frac{\alpha(d/d_{\mathrm{ref}})^\beta} {1+\alpha(d/d_{\mathrm{ref}})^\beta}, \quad \alpha>0.

Here $d/d_{\mathrm{ref}}$ is dimensionless. The likelihood expression is evaluated at the probability implied by the model for each cohort’s dose. Any restrictions on the parameter space, including restrictions on $\beta$, come from the specified model and prior; they do not change the binomial count formula.

Worked Example: One Cohort With Two DLTs

Suppose a hypothetical cohort has $n_1=6$ evaluable patients and $y_1=2$ observed DLTs. At this dose, suppose a particular parameter value in the model implies $p_1=0.25$. Calculate the cohort likelihood contribution at those parameter values.

The binomial probability, viewed as a function of the model parameters with the observed data fixed, is

L_1=\binom{6}{2}(0.25)^2(1-0.25)^{6-2} =15(0.0625)(0.75)^4.

Since $(0.75)^4=0.31640625$, the arithmetic is

L_1=15(0.0625)(0.31640625)=0.296630859375\approx0.2966.

This is the probability of observing exactly two DLTs among six independent patients if the DLT probability is 0.25. As a likelihood, it tells us how compatible this observed count is with parameter values that imply $p_1=0.25$; it is not a probability distribution over parameter values.

Combining Cohorts Across Dose Levels

Suppose the trial has $K$ cohorts, with counts $n_k,y_k$ and dose $d_k$ for cohort $k$. Conditional on the doses and on $(\alpha,\beta)$, assume the patient outcomes in different cohorts are independent as well as independent within each cohort. The joint likelihood is then the product of the cohort contributions:

L(\alpha,\beta) =\prod_{k=1}^{K} \binom{n_k}{y_k} \{p(d_k;\alpha,\beta)\}^{y_k} \{1-p(d_k;\alpha,\beta)\}^{n_k-y_k}.

The same model parameters determine probabilities at all dose levels, while each cohort has its own probability through its dose. Thus the combined likelihood uses all cohorts to constrain one dose-toxicity curve. It does not assume that the cohorts have equal toxicity probabilities when their doses differ.

For computation, the log likelihood is often more stable than multiplying many small terms. Taking the logarithm converts products into sums:

\ell(\alpha,\beta) =\sum_{k=1}^{K} \left[ \log\binom{n_k}{y_k} +y_k\log p(d_k;\alpha,\beta) +(n_k-y_k)\log\{1-p(d_k;\alpha,\beta)\} \right].

The full likelihood includes the binomial coefficients. For inference about $(\alpha,\beta)$ from fixed observed counts, those coefficients do not depend on the parameters, so a likelihood kernel proportional to the full likelihood may omit them. They should not be omitted if the goal is the actual probability of the observed count data or a predictive calculation involving those counts.

Worked Example: Combining Three Hypothetical Cohorts

Consider three hypothetical cohorts with $(n_1,y_1)=(3,0)$, $(n_2,y_2)=(3,1)$, and $(n_3,y_3)=(1,1)$. Suppose the dose-toxicity model, at one illustrative parameter value, implies probabilities $p_1=0.10$, $p_2=0.25$, and $p_3=0.50$ at their respective doses. For example, these probabilities are consistent with the BLRM when the doses have $d_k/d_{\mathrm{ref}}=1,e,e^2$, and $\alpha=1/9,\ \beta=\log(3)$.

For cohort 1, no DLTs are observed:

L_1=\binom{3}{0}(0.10)^0(0.90)^3=0.729.

For cohort 2, one DLT is observed:

L_2=\binom{3}{1}(0.25)^1(0.75)^2 =3(0.25)(0.5625)=0.421875.

For cohort 3, its one patient experiences a DLT:

L_3=\binom{1}{1}(0.50)^1(0.50)^0=0.500.

Under the conditional independence assumption, the joint likelihood at these parameter values is

L(\alpha,\beta)=L_1L_2L_3 =(0.729)(0.421875)(0.500) =0.15380859375\approx0.1538086.

The value is the probability of this particular combination of cohort counts under the stated probabilities. Other parameter values will generally imply different probabilities at the three doses and therefore a different likelihood.

Repeated Cohorts at the Same Dose

When two cohorts receive the same dose under the same model, they have the same DLT probability. Their separate likelihood contributions can be multiplied directly. If their counts are $(n_1,y_1)$ and $(n_2,y_2)$, the product has parameter-dependent part $p^{y_1+y_2}(1-p)^{n_1+n_2-y_1-y_2}$. This is proportional to the binomial likelihood for the combined count $y_1+y_2$ among $n_1+n_2$ patients.

The full probabilities need not be equal: the separate-cohort probability includes $\binom{n_1}{y_1}\binom{n_2}{y_2}$, whereas the combined-count probability includes $\binom{n_1+n_2}{y_1+y_2}$. The difference is a constant with respect to the model parameters, so it does not change posterior inference based on the likelihood and a fixed prior. But it does matter when describing the probability of the observed data at the resolution actually recorded.

Worked Example: Separate Versus Pooled Counts at One Dose

Suppose two hypothetical cohorts at the same dose have counts $(n_1,y_1)=(4,1)$ and $(n_2,y_2)=(3,0)$. For an illustrative common DLT probability $p=0.20$, the probability of the observed separate cohort counts is

L_{\mathrm{separate}} =\binom{4}{1}(0.20)(0.80)^3 \binom{3}{0}(0.80)^3 =(0.4096)(0.512)=0.2097152.

If only the total count is retained, there is one DLT among seven patients. The probability of that aggregate count is

L_{\mathrm{pooled}} =\binom{7}{1}(0.20)(0.80)^6 =7(0.20)(0.262144)=0.3670016.

These are different probabilities because they refer to different events: the first specifies which cohort had the DLT, while the second specifies only the total. Their parameter-dependent factors are both $p(1-p)^6$, so they give the same likelihood shape for $p$. Pooling is therefore harmless for inference about a common probability under these assumptions, but it discards the cohort allocation of the observed DLTs.

Assumptions, Diagnostics, and Alternatives

The cohort likelihood is justified when the following conditions are reasonable:

Binary, consistently defined outcome. Each patient has a DLT outcome defined over the same clinically specified assessment window.
Evaluable follow-up. The count $n_k$ includes patients whose DLT status is known for that window. A patient with incomplete follow-up should not automatically be counted as a non-DLT.
Common probability within a cohort. Conditional on the modeled dose and parameters, all patients in a cohort share $p(d_k;\alpha,\beta)$, or any relevant differences are represented in the model.
Conditional independence. Given the model, one patient’s DLT outcome does not alter another patient’s outcome. Strong clustering or shared unmodeled risk factors may violate this assumption.
Correct dose and model inputs. The assigned dose and the reference-dose scaling are recorded correctly, and the chosen model is an adequate representation for the dose range being analyzed.

Diagnostics begin with checking the data structure, not with a formula. Verify that $0\le y_k\le n_k$, that $n_k$ counts evaluable patients rather than everyone assigned, and that the DLT definition and follow-up window are consistent. Then compare observed cohort proportions $y_k/n_k$ with probabilities implied by the fitted model, while remembering that small cohorts can produce proportions of zero or one even when the underlying probability is not at an endpoint.

Warning signs include unexpected differences among cohorts at the same dose, substantial variation not plausibly explained by sampling, and a pattern of counts poorly represented by the model across dose levels. These can arise from patient heterogeneity, changing assessment practices, delayed toxicity, or dependence. Such findings do not by themselves identify the cause, but they call for checking the data and the assumptions rather than mechanically trusting the product likelihood.

The individual Bernoulli likelihood and the cohort binomial likelihood are equivalent for inference when the individual outcomes are independent with the same probability within each cohort: grouping outcomes into $y_k$ and $n_k-y_k$ changes the likelihood only by the binomial coefficient, which is parameter-independent. The cohort form is more concise. By contrast, a model that allows extra-binomial variation, such as a beta-binomial model, addresses overdispersion by changing the assumed distribution; it is not the same likelihood with a different way of writing the counts. A time-to-event analysis may also be needed when patients have incomplete or unequal follow-up, rather than treating unobserved outcomes as non-DLTs.

Key takeaway: For each cohort, use its evaluable count and DLT count in the binomial contribution at the dose-specific model probability. Multiply the contributions across cohorts only under the stated conditional independence and outcome assumptions, and preserve cohort-level counts when their allocation may be informative.

Check Your Understanding

Answer each question, then compare with the solutions below.

A hypothetical cohort has $n=5$, $y=0$, and model-implied DLT probability $p=0.10$. What is its binomial likelihood contribution?
For $n=4$, $y=2$, and $p=0.30$, write the full likelihood contribution, including its binomial coefficient.
Why does multiplying cohort likelihoods require a conditional independence assumption?
Two cohorts at one dose are pooled. What is lost by retaining only the total number of DLTs?

Solutions. (1) $\binom{5}{0}(0.10)^0(0.90)^5=0.59049$, approximately 0.5905. (2) $\binom{4}{2}(0.30)^2(0.70)^2=6(0.09)(0.49)=0.2646$. (3) Without conditional independence, the probability of all cohort outcomes need not equal the product of their separate probabilities, so the product may not be the joint likelihood. (4) Pooling removes information about which cohort contributed the DLTs; at a common modeled probability, the parameter-dependent likelihood shape is unchanged, but the recorded event is less specific.