Why Elicitation Matters
A CRM skeleton is a starting set of dose-specific toxicity probabilities, ordered to reflect increasing doses. It is not a set of observed toxicity rates: it encodes prespecified clinical judgment about the relative risk profile across the dose grid. If those starting values are poorly elicited, the model can begin with a pattern that is clinically implausible, too vague to be useful, or inconsistent with the endpoint the trial will actually measure.
Elicitation is therefore more than asking, “What is the risk at each dose?” It requires a precise event definition, a consistent time window, a clear way to express probabilities, and a transparent method for handling uncertainty and disagreement. The aim here is to create a defensible monotone skeleton, not to choose a target dose or tune the skeleton around a target rate.
Set the Question Before Eliciting Probabilities
The clinician’s answer depends on what “toxicity” means. State the dose levels and the dose-limiting toxicity (DLT) definition consistently, including the assessment window. For example, ask about the probability that a patient experiences a protocol-defined DLT within the first 28 days after starting treatment at a specified dose. Do not mix a 28-day endpoint at one dose with cumulative risk over several cycles at another.
A useful question names one patient’s treatment at one dose and asks for a probability: “Among 100 otherwise similar eligible patients treated at this dose under this schedule, how many would you expect to experience at least one DLT within 28 days?” The frequency framing can make a probability more concrete, but the answer remains a judgment—not a sample proportion from 100 observed patients. Ask the same question at every dose, and make explicit that uncertainty is expected.
For each dose, it is useful to record a central estimate and a plausible range, or lower, central, and upper probability judgments. Ask what evidence or clinical mechanism informs the estimate, which dose comparisons are especially uncertain, and whether the clinician is extrapolating beyond familiar experience. When multiple clinicians contribute, preserve their individual answers before seeking consensus; otherwise, early discussion can conceal meaningful disagreement.
Specify the DLT definition, assessment window, treatment schedule, and patient population for every question.
Request a central probability at each ordered dose, using the same patient-level question and probability scale throughout.
Record plausible ranges, confidence, and the clinical basis for estimates; identify where evidence is sparse or extrapolated.
Inspect the raw values for decreases with dose, then apply a stated reconciliation rule and retain the raw and final values.
Monotone Reconciliation by Weighted Isotonic Regression
Even thoughtful elicitation can produce a local decrease: a clinician might judge a middle dose riskier than the next dose because the estimates are uncertain or because the clinical evidence is uneven. The monotonicity assumption is that, over the prespecified ordered dose grid and for the same endpoint and population, toxicity probability does not decrease as dose increases. It does not require strictly increasing risks; adjacent doses may have equal skeleton values.
One transparent reconciliation is weighted isotonic regression. Let \(r_j\) be the raw central judgment at dose \(d_j\), and let \(w_j>0\) represent the relative confidence assigned to that judgment. Choose adjusted values \(q_j\) that minimize the weighted squared changes from the raw judgments while satisfying the required order.
If two adjacent estimates violate the order, the pooling calculation follows directly from minimizing their weighted squared deviations. For a pooled common value \(q\), differentiate \(w_a(q-r_a)^2+w_b(q-r_b)^2\) with respect to \(q\), set the derivative to zero, and solve:
Thus, the pooled value is a weighted mean, not an arbitrary midpoint. The pool-adjacent-violators algorithm applies this calculation: begin with one block per dose, merge neighboring blocks whenever their means decrease, and continue until the block means are nondecreasing. Every dose in a merged block receives that block’s weighted mean. For this squared-error objective, the resulting blocks satisfy the optimality conditions: their means are ordered, each pooled block’s weighted residuals have nonpositive cumulative sums up to the block’s end, and the total residual within each block is zero. These conditions ensure no order-preserving change can lower the objective. This is a mathematical reconciliation rule, not proof that the adjusted values are the true probabilities.
Weights deserve particular care. They may encode relative confidence in elicited judgments, but they are not automatically sample sizes, inverse variances estimated from data, or objective measures of expertise. If weights are used, define how they were assigned and examine whether reasonable alternatives materially change the adjusted values. With equal weights, all dose judgments have equal influence in the least-squares adjustment.
Worked Example: Repairing One Local Decrease
Suppose a hypothetical clinician gives central DLT probability judgments of 0.05, 0.18, 0.15, and 0.42 at four increasing doses, using the same endpoint and assessment window. The estimates are otherwise treated as equally reliable.
The raw sequence decreases from 0.18 to 0.15, so it is not monotone. Pool those adjacent estimates using equal weights:
The adjusted skeleton is \((0.05,0.165,0.165,0.42)\). It is nondecreasing, and the pooled probabilities are both between their original values. The other two estimates remain unchanged because the resulting blocks are already ordered. The adjustment states that the two inconsistent judgments are represented by a shared value under the chosen equal-weight squared-error rule; it does not establish that the risk at either dose is exactly 16.5%.
Worked Example: Using Unequal Confidence Weights
In a second hypothetical elicitation, central judgments at four ordered doses are 0.08, 0.20, 0.16, and 0.40. A panel assigns relative confidence weights of 1, 4, 1, and 2, respectively. These are documented judgment weights, not patient counts.
The second and third estimates violate monotonicity. Pool them with their assigned weights:
The adjusted values are \((0.08,0.192,0.192,0.40)\), which are ordered. The more heavily weighted second-dose judgment pulls the pooled value closer to 0.20 than to 0.16. This influence is justified only if the panel’s confidence assessment is itself defensible. For comparison, equal weights would give \((0.20+0.16)/2=0.18\). Recording both the weight rationale and this sensitivity check makes the reconciliation auditable.
Handling Wide Uncertainty and Clinician Disagreement
A monotone sequence can still be weakly elicited. For instance, all estimates may be presented as single numbers even though clinicians consider a wide range plausible, or different clinicians may give very different risks at the same dose. Isotonic regression repairs order violations; it does not resolve these underlying uncertainties. Follow-up questions should investigate the disagreement: Are clinicians using the same endpoint? Do they have different assumptions about the population, schedule, or supportive care? Is one estimate extrapolated from a different treatment context?
When there are multiple clinicians, one practical approach is to summarize their separate central judgments and uncertainty ranges dose by dose, discuss substantive differences, and then document how the panel arrives at a final set of raw central values. If a summary such as the median is chosen, state that rule in advance and apply it consistently. Then check monotonicity and, if needed, apply the stated isotonic rule. Do not average away important disagreement without recording it.
Worked Example: Separating Disagreement From Order Repair
Three hypothetical clinicians provide central DLT judgments at three increasing doses. Their answers are: at dose 1, 0.06, 0.08, and 0.10; at dose 2, 0.15, 0.18, and 0.25; at dose 3, 0.20, 0.30, and 0.35. For this illustration, the panel chooses the median clinician judgment at each dose as its summary rule.
The medians are 0.08, 0.18, and 0.30. They are already nondecreasing, so isotonic adjustment makes no change. However, dose 2 has a much wider spread of answers (0.15 to 0.25) than dose 1 (0.06 to 0.10). The monotone median sequence does not erase that uncertainty. The panel should document the range and ask why dose-2 judgments differ before treating 0.18 as a precise clinical fact. The median is a summary of this hypothetical panel’s judgments, not an estimate computed from trial outcomes.
Diagnostics, Alternatives, and Limitations
Before accepting a skeleton, check that all elicited probabilities refer to the same event, time window, population, and treatment conditions. Inspect both the raw and adjusted sequences; verify that adjusted values are nondecreasing and lie strictly between zero and one; and confirm that rounding has not introduced a decrease or changed an equality into an apparent violation. Review which values were pooled, how the weights were assigned, and whether the adjustment is large enough to require renewed clinical discussion.
Isotonic regression is not the only option. Its advantage is a reproducible, minimum weighted squared-change adjustment under a stated order constraint. Equal-weight isotonic regression avoids subjective confidence weights but treats every elicited value equally. A clinical panel can instead revise inconsistent judgments through structured discussion; this may incorporate clinical reasoning that a numerical projection cannot, but it needs explicit documentation to remain reproducible. Choosing a monotone sequence directly during elicitation may avoid some corrections, yet can hide initial disagreement and give a false impression of certainty. These approaches can be combined: discuss the raw judgments, apply a transparent summary and adjustment rule, then return substantial changes to the panel for review.
| Approach | Strength | Limitation |
|---|---|---|
| Equal-weight isotonic regression | Simple, reproducible order correction | Does not distinguish confidence across doses |
| Weighted isotonic regression | Allows documented relative confidence to affect the adjustment | Results depend on defensible weights |
| Structured panel reconciliation | Can incorporate clinical context and clarify different assumptions | May be difficult to reproduce unless the process is recorded |
No elicitation rule guarantees that the skeleton matches the unknown toxicity curve. The probabilities depend on clinical judgment and available evidence, and a monotone sequence can still be inaccurate or too uncertain for the intended use. Retain the raw judgments, ranges, rationale, weights, and final values so that later evaluation can distinguish uncertainty in the clinical inputs from evidence accumulated during the trial.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- Why must the DLT definition and assessment window be fixed before asking clinicians for dose-specific probabilities?
- Raw equal-weight judgments at three increasing doses are 0.10, 0.24, and 0.18. What is the isotonic adjustment?
- For two violating judgments 0.30 and 0.20 with weights 1 and 3, calculate the pooled value.
- Does a monotone set of elicited values establish that the true toxicity risks are known precisely? Explain.
- Name one useful diagnostic besides checking that the final values are nondecreasing.
Solutions. (1) Otherwise the probabilities may refer to different events or periods and cannot be compared consistently across doses. (2) Pool the last two values: \((0.24+0.18)/2=0.21\), giving \((0.10,0.21,0.21)\). (3) The weighted pool is \(\{1(0.30)+3(0.20)\}/(1+3)=0.90/4=0.225\). (4) No. Monotonicity is an ordering constraint on the elicited inputs, not evidence that they equal the unknown risks or have little uncertainty. (5) Check that every judgment uses the same endpoint and assessment window, review the rationale for confidence weights, or compare adjusted values with the original judgments to identify large changes.