Why Examine Model Sensitivity?
A model-based escalation recommendation depends on more than the observed dose-limiting toxicities (DLTs). It also depends on how the model translates dose into toxicity risk. A shallow curve, a steep curve, and different prior expectations about the curve can agree near the doses already studied yet imply different risks at an untried dose. Sensitivity analysis makes that dependence visible before it drives a decision.
The aim is not to choose whichever model recommends the preferred dose. Instead, specify scientifically plausible alternatives, apply the same decision rule to each, and report whether the resulting actions agree. Agreement supports a robust recommendation relative to the alternatives examined; disagreement signals that the decision depends on assumptions and may warrant caution, further data, or a narrower set of candidate doses.
What Is Being Varied?
A dose-toxicity model specifies how the probability of a DLT within a defined assessment window changes with dose. For a simple binary-outcome setup, let \(Y_i=1\) denote a DLT for patient \(i\), and let \(p(d)\) be the probability of a DLT at dose \(d\). Conditional on the model and its parameters, a common assumption is that patient outcomes are independent Bernoulli observations with probabilities determined by their doses. The examples below use complete DLT follow-up and no patient covariates.
Sensitivity analysis can vary several related but distinct assumptions:
- Functional form or shape: for example, a model with a gradual increase in risk versus one that permits a steeper increase across a dose interval.
- Dose skeleton: a set of prespecified toxicity probabilities at ordered doses, often used to calibrate a model such as the continual reassessment method (CRM). Different skeletons represent different prior expectations about how risk changes across the dose levels.
- Prior assumptions: these include priors on model parameters and, when a finite set of models is used, prior weights on the models themselves.
These are not always interchangeable. A skeleton changes the dose-risk assumptions within a model; a prior on model parameters changes how strongly the data can move the fitted curve; and a prior model weight changes the starting relative plausibility assigned to competing models. A useful analysis states which feature is being changed and holds the others fixed where possible.
Comparing Candidate Models
For a finite collection of candidate curves \(M_1,\ldots,M_K\), let \(\pi_k=P(M_k)\) be the prior probability assigned to model \(M_k\). Given observed dose and DLT data \(D\), Bayes' rule gives the posterior model probability:
For fixed model-specific probabilities \(p_{ik}\) at the doses received by patients \(i=1,\ldots,n\), and conditionally independent binary outcomes, the likelihood is
In a model with unknown parameters, \(P(D\mid M_k)\) is the marginal likelihood: the likelihood integrated over that model's parameter prior. Once posterior model weights are available, a posterior predictive DLT risk at a candidate dose \(d\) is the weighted average of the model-specific risks:
For the fixed curves used below, the within-model predictive risk is simply the curve's specified probability at that dose. Model averaging is one way to summarize the alternatives, not a substitute for showing what each model recommends. A decision rule must also be stated: for example, select the highest dose whose posterior probability of exceeding a toxicity-risk threshold is no greater than a specified bound.
Worked Example: Updating Weights Across Three Toxicity Curves
Hypothetical data. Consider four ordered doses, \(D_1\) through \(D_4\), and three fixed, monotone candidate curves. Their DLT probabilities are shown below. Three patients receive \(D_1\), with no DLTs; three receive \(D_2\), with one DLT and two patients without DLT. Assume equal prior probability for each curve, complete follow-up, and conditionally independent outcomes.
| Curve | \(D_1\) | \(D_2\) | \(D_3\) | \(D_4\) |
|---|---|---|---|---|
| A | 0.05 | 0.15 | 0.30 | 0.50 |
| B | 0.10 | 0.25 | 0.40 | 0.65 |
| C | 0.15 | 0.35 | 0.50 | 0.75 |
The likelihood for each curve is the probability of three non-DLT outcomes at \(D_1\), multiplied by the probability of one DLT and two non-DLT outcomes at \(D_2\). For curve A:
For curves B and C, the same calculation gives
The likelihood sum is \(0.0929180+0.1025156+0.0908137=0.2862473\) (using unrounded values, \(0.2862474\)). With equal prior weights, the normalized posterior weights are approximately \(0.3246\), \(0.3581\), and \(0.3173\) for A, B, and C. The posterior predictive risk at \(D_2\) is
At \(D_3\), the corresponding average is approximately \((0.3246)(0.30)+(0.3581)(0.40)+(0.3173)(0.50)=0.3993\). The average alone hides meaningful disagreement: the model-specific risks at \(D_3\) range from 0.30 to 0.50. If the target is 0.30 and the rule is to choose the highest dose with model-specific risk no greater than 0.30, curve A selects \(D_3\), curve B selects \(D_2\), and curve C selects \(D_1\). Thus the action is sensitive to the assumed curve, even though the posterior model weights are not dramatically different.
Prior Sensitivity Can Change an Escalation Action
Posterior weights reflect both the likelihood and the prior model weights. If different plausible prior weights produce different actions, the recommendation should not be presented as though it were determined by the data alone. The following example uses the same hypothetical curves and observed data, so only the prior model weights change.
Worked Example: Changing Prior Model Weights
In the preceding example, replace equal prior weights with \(0.15\), \(0.25\), and \(0.60\) for curves A, B, and C. Multiplying each likelihood by its prior weight gives unnormalized posterior weights:
Their sum is \(0.0940548\). Dividing each unnormalized weight by this sum gives posterior weights approximately \(0.1482\), \(0.2725\), and \(0.5793\). The increased weight on curve C raises the posterior probability of a DLT risk above 0.30 at \(D_2\) from \(0.3173\) under equal priors to \(0.5793\), because only curve C assigns \(D_2\) a risk greater than 0.30.
For a concrete decision rule, suppose a dose is eligible only if the posterior probability that its DLT risk exceeds 0.30 is at most 0.40, and the trial selects the highest eligible dose. At \(D_3\), curves B and C exceed 0.30, so the probability is \(0.3581+0.3173=0.6754\) under equal priors and \(0.2725+0.5793=0.8518\) under the alternative priors. Thus \(D_3\) is ineligible under either prior specification. At \(D_2\), only curve C exceeds 0.30: the probability is \(0.3173\) under equal priors, making \(D_2\) eligible, but \(0.5793\) under the alternative priors, making it ineligible. Since \(D_1\) is below the threshold under all three curves, it remains eligible. The rule therefore selects \(D_2\) under equal priors and \(D_1\) under the alternative priors. This is a prior-sensitive escalation decision, not evidence that either prior set is automatically correct.
Checking Whether New Outcomes Alter the Comparison
A model comparison should be revisited as outcomes accumulate. If a candidate model repeatedly assigns low probability to observed data, its posterior weight may fall. But a change in weights is not, by itself, proof that the remaining model is correctly specified. All candidate curves can fit poorly, particularly when risk rises sharply between tested doses or when toxicity is not monotone.
Worked Example: Updating After No DLTs at the Next Dose
Continue the hypothetical example. Suppose three additional patients are treated at \(D_3\), with no DLTs. Under curves A, B, and C, the likelihood contributions from these outcomes are \((1-0.30)^3=0.343\), \((1-0.40)^3=0.216\), and \((1-0.50)^3=0.125\), respectively. Multiplying these by the earlier likelihoods gives:
The updated likelihood sum is \(0.0653660\). With equal prior weights, the updated posterior model weights are approximately \(0.4876\), \(0.3388\), and \(0.1737\). The posterior predictive risk at \(D_4\) becomes
The no-DLT outcomes at \(D_3\) increase the relative weight of the less steep curve A. Nevertheless, the predicted risk at \(D_4\) remains substantial because every candidate curve assigns appreciable risk there. The update illustrates both the information gained from outcomes at a new dose and the limits of relying on one model's extrapolation.
Diagnostics, Alternatives, and Limitations
Useful sensitivity checks begin with a decision table: list each plausible model or prior specification, its relevant posterior risk summaries, and the action from the same prespecified decision rule. Inspect risk estimates at both studied and unstudied doses. A model that agrees with others at studied doses but diverges sharply at the next dose is a warning that escalation depends on extrapolation. Also check whether the direction of disagreement is clinically plausible: for example, whether a curve implies an abrupt risk increase unsupported by the dose spacing or available evidence.
The main alternatives are complementary rather than interchangeable:
| Approach | What it contributes | Main caution |
|---|---|---|
| Single prespecified model | A clear recommendation under one stated set of assumptions. | Does not show dependence on alternatives that may also be plausible. |
| Separate fits under alternative shapes or skeletons | Makes the range of model-specific recommendations visible. | Does not itself assign probabilities to the models or combine them. |
| Bayesian model averaging | Combines model-specific risks using posterior model weights. | Results depend on the candidate set and prior model weights; averaging can conceal disagreement. |
Several assumptions deserve particular attention. The simple likelihood above presumes the binary DLT outcome is assessed over a common, defined window and is known for every included patient. It also presumes conditional independence given the model and dose. If follow-up is incomplete, outcomes are clustered, or patient characteristics materially change toxicity risk, this simple likelihood may not be appropriate without extensions. In an adaptive design, conditioning on the doses actually assigned is generally reasonable when the allocation mechanism depends on observed history and known design rules rather than on unknown toxicity-model parameters; the assumptions of the outcome model still need scrutiny.
Sensitivity analysis has limits. It cannot protect against every possible misspecification: an unexamined curve, nonmonotone toxicity, or an unmodeled patient subgroup may still matter. Candidate models and prior weights should therefore be justified before inspecting which one yields the preferred action. Posterior model probabilities are conditional on the candidate set; a large weight for one model does not establish that it is true. Finally, a stable recommendation across several nearby assumptions is evidence of robustness only to those assumptions, not a guarantee of safety.
Common Pitfalls
- Reporting only the model-averaged risk. An average can look decisive even when individual models recommend different actions. Show the model-specific risks or recommendations alongside the average.
- Changing several assumptions at once. If shape, skeleton, and prior weights all change together, it is difficult to identify what caused a changed decision. Vary one feature at a time where feasible, then examine justified combinations.
- Treating a posterior model weight as proof of correctness. The weight is relative to the candidate models and their priors. It does not measure the probability that the model set contains the true dose-toxicity relationship.
- Choosing the model after seeing its recommendation. This makes the sensitivity exercise a way to justify a preferred action rather than assess robustness. Define plausible alternatives and a common decision rule first.
Check Your Understanding
Use the three curves and hypothetical data from the worked examples. Solutions are included after the questions.
- With equal prior weights and the initial data, what is the posterior weight for curve B?
- With prior weights \(0.15\), \(0.25\), and \(0.60\), which curve has the largest posterior weight, and what is that weight approximately?
- After observing no DLTs in three patients at \(D_3\), what is the approximate posterior weight for curve A under equal prior weights?
- Under the decision rule in the prior-sensitivity example, why does the recommended dose change when the prior weights change?
Solutions. (1) The likelihood sum is \(0.2862474\), so curve B's posterior weight is \(0.1025156/0.2862474\approx 0.3581\). (2) Curve C has the largest weight: \(0.0544882/0.0940548\approx 0.5793\). (3) The updated likelihood sum is \(0.0653660\), so A's weight is \(0.0318709/0.0653660\approx 0.4876\). (4) At \(D_2\), only curve C assigns risk greater than 0.30. Its posterior weight is \(0.3173\) with equal priors, below the 0.40 eligibility bound, but \(0.5793\) with the alternative priors, above the bound. Thus \(D_2\) is eligible only under equal priors; \(D_1\) is the highest eligible dose under the alternative priors.