Why Model Structure Matters for Adaptation
At an interim analysis, several pharmacometric models may fit the available data and remain scientifically plausible. One model might describe a saturating exposure–response relation; another might allow a delayed response or a different toxicity pattern. If those structures recommend different dose regimens, selecting the single model with the best interim fit can make the adaptation depend sharply on a limited data set. Robust adaptation makes that structural uncertainty part of the decision rather than hiding it behind one selected model.
The decision target is a dose or regimen that performs well for the trial’s prespecified clinical objective, such as balancing expected benefit against toxicity. The objective must be defined on a common scale across models. The calculations below concern the adaptation decision; they do not, by themselves, guarantee valid error rates for a later confirmatory treatment-effect test.
From Interim Data to Model-Averaged Decisions
Let \(D\) denote the interim data, \(a\) a candidate action (for example, assigning a particular dose regimen to future participants), and \(U(a,\theta_k)\) the utility of that action under model \(M_k\) and its parameters \(\theta_k\). Utility should represent the trial’s decision target, including relevant benefits and harms. Its scale and direction must be specified before comparing actions; higher utility will mean preferable outcomes here.
Within each model, average over parameter uncertainty given the interim data. Then average across models using their posterior probabilities. The resulting quantity is the model-averaged expected utility of action \(a\).
For Bayesian model probabilities, let \(P(M_k)\) be the prior probability assigned to model \(M_k\), and let \(p(D\mid M_k)\) be its marginal likelihood, which averages the data likelihood over the model’s parameter prior. Bayes’ rule gives:
The basic model-averaged rule selects an action maximizing \(\bar Q(a)\), subject to any prespecified safety or operational constraints. This rule integrates model-conditional parameter uncertainty and uncertainty over the candidate structures. It does not mean that the models are interchangeable: each model must describe the same decision target and supply a meaningful utility for each action.
Worked Calculations
Worked Example: Updating Probabilities for Two Structures
Suppose an interim analysis compares two hypothetical exposure–response structures. Before observing the interim data, the team assigns \(P(M_1)=0.60\) and \(P(M_2)=0.40\). For the observed data, suppose the marginal likelihoods are \(p(D\mid M_1)=0.20\) and \(p(D\mid M_2)=0.05\). Find the posterior model probabilities.
The unnormalized posterior weights are prior probability times marginal likelihood:
Their sum is \(0.12+0.02=0.14\). Normalizing gives:
Thus the data and prior assign about 85.7% posterior probability to \(M_1\), and 14.3% to \(M_2\). This does not prove that \(M_1\) is true. It quantifies relative support within the two-model set and under the specified priors. In particular, if an important plausible structure was omitted, these two probabilities do not account for it.
Worked Example: Choosing a Dose by Model-Averaged Utility
Consider two hypothetical models with interim posterior probabilities \(w_1=0.65\) and \(w_2=0.35\), and three candidate regimens. The table gives each regimen’s expected net utility within each model, after averaging over that model’s parameter uncertainty. Utility is in prespecified abstract decision points; it is not a clinical measurement unit.
| Candidate regimen | \(Q_1(a)\), points | \(Q_2(a)\), points |
|---|---|---|
| 50 mg | 4 | 5 |
| 100 mg | 8 | 3 |
| 150 mg | 9 | -2 |
For 50 mg, the model-averaged utility is \(0.65(4)+0.35(5)=2.60+1.75=4.35\) points. For 100 mg, it is \(0.65(8)+0.35(3)=5.20+1.05=6.25\) points. For 150 mg, it is \(0.65(9)+0.35(-2)=5.85-0.70=5.15\) points. Therefore the model-averaged rule selects 100 mg, with the largest weighted expected utility, 6.25 points. The calculation does not establish that 100 mg is safe: a separately specified safety constraint would still need to be checked.
Why Averaging Produces This Rule
The justification follows from conditioning on the unknown model structure. Suppose the utility \(U\) is evaluated for a proposed action after observing \(D\). The expected utility given the interim data is an average over all possible structures and, within each structure, all possible parameter values. Applying the law of total expectation gives:
If the decision criterion is to maximize expected utility, the action with the largest value of this conditional expectation is the maximizing action. This establishes the model-averaged rule; it does not establish that expected utility is the only defensible decision criterion. In particular, averaging can favor an action whose performance is poor under a lower-weight model, which may be unacceptable when that model represents a serious safety concern.
Worked Example: Protecting Against Uncertain Model Weights
Suppose two models give the following expected utilities for two hypothetical regimens. The interim evidence does not justify a precise posterior weight, but the team considers the weight \(w=P(M_1\mid D)\) plausibly between 0.40 and 0.70. The weight on \(M_2\) is \(1-w\).
| Regimen | Utility under \(M_1\), points | Utility under \(M_2\), points |
|---|---|---|
| 80 mg | 7 | 5 |
| 120 mg | 10 | 1 |
A model-averaged calculation at the nominal weight \(w=0.65\) gives 80 mg an expected utility of \(0.65(7)+0.35(5)=4.55+1.75=6.30\) points. For 120 mg, it gives \(0.65(10)+0.35(1)=6.50+0.35=6.85\) points, so this nominal calculation selects 120 mg.
A maximin rule instead evaluates the lowest average utility over the plausible weight range. For 80 mg, the weighted utility is \(7w+5(1-w)=5+2w\), which increases with \(w\); its minimum over \(0.40\leq w\leq0.70\) is \(5+2(0.40)=5.80\) points. For 120 mg, it is \(10w+1(1-w)=1+9w\), also increasing with \(w\); its minimum is \(1+9(0.40)=4.60\) points. The maximin rule therefore selects 80 mg. This is more protective against low support for \(M_1\), but sacrifices some utility at the nominal weights. The choice depends on how the trial’s decision makers value protection from structural uncertainty relative to expected performance.
Checking the Recommendation and Choosing an Approach
A useful robustness diagnostic is to examine whether the recommended action changes as model weights vary over defensible values. A recommendation that remains the same across a broad range is less dependent on the exact weighting than one that switches near the nominal weights. Also inspect the model-specific utilities and any safety constraints: a favorable average can conceal an unfavorable prediction under one plausible structure. These checks assess decision sensitivity; they do not show that the candidate models are correctly specified.
Several decision approaches answer different questions. None removes the need to define the adaptation target and constraints.
| Approach | Decision principle | Main limitation |
|---|---|---|
| Select one model | Use the preferred model’s expected utility. | Can make the decision sensitive to model selection at interim. |
| Model averaging | Maximize utility averaged over posterior model weights. | Depends on model priors, candidate structures, and the adequacy of the utility scale. |
| Maximin over weights | Maximize the lowest expected utility over an allowed set of model weights. | Can be conservative and depends on how the allowed set is chosen. |
| Model-specific constraint | Optimize an average while requiring a safety condition under every specified model. | May rule out otherwise attractive actions and cannot protect against omitted structures. |
A related option is minimax regret: choose the action minimizing the largest loss relative to the best action for each possible model weighting. Unlike maximin utility, regret focuses on avoiding a large shortfall from the best model-specific decision. It can be useful when utility levels are difficult to compare directly, but it still requires a defensible set of models and weights. The rule should be chosen according to the clinical objective and prespecified rather than selected after seeing which method favors a preferred regimen.
Common Pitfalls and Limitations
- Confusing model weights with certainty. A posterior probability of 0.86 for one candidate model is conditional on the model set and priors; it is not an 86% guarantee that the structure is correct.
- Ignoring parameter uncertainty. Plugging in one parameter estimate for each structure can make \(Q_k(a)\) too concentrated. The within-model expectation should reflect relevant parameter uncertainty.
- Comparing incompatible utilities. Averaging model-specific scores is not meaningful if they represent different endpoints, scales, or definitions of benefit and harm. Define a common decision target first.
- Treating a model-robust dose as automatically safe or confirmatory. Model averaging does not replace prespecified safety constraints, trial monitoring, or evaluation of the operating characteristics of any later inferential procedure.
- Assuming the candidate set is complete. Robustness to weights among included models cannot protect against a scientifically plausible structure that was left out.
Because interim adaptation can affect who receives which regimen and what data are observed, decision rules should be specified before examining the interim results. The rule should state the candidate structures, how model weights are obtained or bounded, the utility, any safety constraints, and what happens if no action meets those constraints. If the recommendation is highly sensitive to reasonable weight choices, that sensitivity is itself important information for the adaptation decision, not a reason to report only the most favorable weighting.
Check Your Understanding
Use the definitions and decision rules above. Utility values below are hypothetical decision points, not clinical measurements.
- Two models have posterior probabilities 0.25 and 0.75. Action A has model-specific expected utilities 6 and 2; action B has utilities 3 and 8. Which action maximizes model-averaged utility?
- Why does a posterior probability of 0.90 for one structure not mean there is a 90% chance that the structure is correct without qualification?
- For two models, an action has utilities 4 and 10 points. Write its model-averaged utility as a function of \(w=P(M_1\mid D)\). Is it increasing or decreasing in \(w\)?
- Give one reason a maximin rule might be preferred to model averaging and one cost of using it.
Solutions. 1. Action A has \(0.25(6)+0.75(2)=1.50+1.50=3.00\) points. Action B has \(0.25(3)+0.75(8)=0.75+6.00=6.75\) points, so B is selected. 2. The probability is conditional on the included candidate models, their prior probabilities, and the likelihood assumptions; an omitted structure is not represented. 3. The utility is \(4w+10(1-w)=10-6w\) points, which decreases as \(w\) increases. 4. Maximin can be preferred when poor performance under a plausible model is especially concerning. Its cost is that it can select a conservative action with lower expected utility under the nominal model weights.