Diagnosing Unstable Dose Recommendations

Why Diagnose Stability?

A dose recommendation is the output of a chain of choices: data, a fitted model, a decision criterion, and a set of candidate doses. A recommendation can appear precise while depending on a small change in any part of that chain. For example, two doses may have nearly identical estimated benefit-risk utility, so a small shift in the fitted curve changes which one ranks first. Diagnosing this instability is different from calculating an uncertainty interval: the goal is to learn whether the selected dose is robust to specified changes in inputs and assumptions.

Let $\mathcal{D}$ be the finite set of candidate doses and let $\widehat{U}(d)$ be the estimated decision utility at dose $d$, calculated using a stated model and decision rule. The recommendation is

\widehat{d}=\arg\max_{d\in\mathcal{D}}\widehat{U}(d).

If feasibility constraints are part of the rule, the maximization is over the doses that satisfy those constraints. The diagnostic question is not simply whether $\widehat{d}$ is an optimum for the original analysis; it is whether the recommendation changes under reasonable perturbations, and whether any changes are consequential.

Definition: A dose recommendation is stable with respect to a specified set of perturbations when applying each perturbation and rerunning the complete stated dose-selection procedure leaves the recommendation unchanged, or changes it only within a prespecified acceptable range. Stability is always relative to the perturbations, decision rule, candidate doses, and tolerance chosen for the analysis.

Designing a Perturbation Analysis

A perturbation analysis changes one input or assumption at a time, or explores a deliberately specified set of joint changes, then repeats the full decision calculation. Plausible perturbations can include small changes to influential observations, alternative defensible parameter values, reasonable model specifications, or assumptions about the population used for prediction. Earlier material on sensitivity of optimal dose to prior assumptions and structural model uncertainty addresses particular sources of uncertainty; here the emphasis is on diagnosing whether such changes alter the selected dose.

The perturbations should be chosen before inspecting which ones produce a preferred answer. They should be scientifically defensible, sufficiently small or plausible for the question being asked, and clearly documented. For each scenario $s$, calculate $\widehat{U}_s(d)$ for every candidate dose and select

\widehat{d}_s=\arg\max_{d\in\mathcal{D}}\widehat{U}_s(d).

Record both the selected dose and the utility profile. A changed winner can be practically unimportant if neighboring doses perform almost equally; conversely, a small change in dose can be important if it crosses a safety constraint or leads to materially different expected outcomes. Report the magnitude and context of the change, not just whether the winner changed.

Worked Example: A Model Perturbation Changes the Winner

Suppose a hypothetical analysis compares four candidate doses. Utility is a dimensionless, higher-is-better summary of the stated efficacy-safety decision rule. The table shows the original fitted model and a defensible alternative model specification.

Dose (mg)	Original utility	Alternative-model utility
50	0.42	0.40
75	0.58	0.54
100	0.61	0.59
125	0.55	0.62

Under the original model, the maximum utility is $0.61$ at $100$ mg, so $\widehat{d}=100$ mg. Under the alternative, the maximum is $0.62$ at $125$ mg, so $\widehat{d}_{s}=125$ mg. The selected dose changes by $25$ mg. The original utility advantage of $100$ mg over $75$ mg is $0.61-0.58=0.03$; in the alternative, $125$ mg exceeds $100$ mg by $0.62-0.59=0.03$.

This result flags model sensitivity, but it does not establish that either model is correct or that the change is clinically important. The analyst should examine why the models differ, whether both are scientifically credible, and what the utility differences mean under the decision rule. If a $25$-mg change materially affects risk or treatment burden, the instability deserves particular attention.

Bootstrap Selection Frequencies

A bootstrap selection-frequency analysis evaluates how often each candidate dose is selected across resampled data sets. For each bootstrap replicate, resample the observational units, refit the model, recalculate utility at all candidate doses, and apply the same selection rule. If the resulting selected doses are $\widehat{d}^{(1)},\ldots,\widehat{d}^{(B)}$, the selection frequency for dose $d$ is

\widehat{f}(d)=\frac{1}{B}\sum_{b=1}^{B}I\!\left(\widehat{d}^{(b)}=d\right),

where $I(\cdot)$ equals 1 when the statement is true and 0 otherwise. Frequencies across the candidate doses sum to 1 when every replicate produces exactly one selected dose. The resampling unit must match the data structure: for repeated observations from the same participant, resample participants with their observations together rather than treating those observations as independent rows. Stratification or other design features may also need to be preserved.

These frequencies describe selection variability under the chosen bootstrap procedure, model-fitting method, candidate grid, and decision rule. They are not probabilities that the doses are truly optimal, and they do not automatically account for structural model uncertainty or uncertainty in the decision criterion. A high frequency is evidence of stability under the resampling procedure, not a guarantee of good calibration or a correct model.

Worked Example: Calculating Bootstrap Selection Frequencies

In a hypothetical analysis, $B=200$ participant-level bootstrap replicates are fitted and the same four-dose decision rule is applied each time. The selected dose is $50$ mg in 20 replicates, $75$ mg in 100 replicates, $100$ mg in 80 replicates, and $125$ mg in none.

The frequencies are $20/200=0.10$ for $50$ mg, $100/200=0.50$ for $75$ mg, $80/200=0.40$ for $100$ mg, and $0/200=0$ for $125$ mg. They sum to $0.10+0.50+0.40+0=1.00$. The original-data recommendation is not supplied by these counts; the counts show that $75$ mg is selected most often across these resamples, but it is selected in only half of them.

The appropriate conclusion is that the choice between $75$ and $100$ mg is sensitive to sampling variation as represented by this bootstrap. It would be incorrect to say there is a 50% probability that $75$ mg is the true optimal dose. The frequency is conditional on the observed data and the specified resampling and decision procedures.

Candidate-Dose Plots and the Utility Margin

A candidate-dose plot displays estimated utility across the entire dose grid, often showing uncertainty bands or bootstrap distributions at each dose. It helps distinguish a sharply preferred dose from a broad, nearly flat optimum. A plot of the selected dose alone hides that distinction. When utility is an expected net benefit or another decision scale, include its units or state that it is dimensionless, and label the candidate doses and any constraints.

A useful numerical summary is the utility margin between the best and second-best candidate:

\widehat{\Delta} =\widehat{U}(\widehat{d})-\max_{d\in\mathcal{D},\,d\ne\widehat{d}}\widehat{U}(d).

A small margin means that small changes in estimated utilities may change the ranking. It does not, by itself, prove instability: the scale and uncertainty of utility estimates matter. Conversely, a large margin in one fitted model does not establish stability under alternative models or data perturbations.

Justification: Suppose the estimated utility at every candidate dose can change by at most $\epsilon$ under a perturbation: $\left|\widehat{U}_{s}(d)-\widehat{U}(d)\right|\leq\epsilon$. Let $\widehat{d}$ be the original winner and let $\widehat{\Delta}$ be its margin over the runner-up. For any competitor $d\ne\widehat{d}$, the perturbed difference is bounded below by $$ \widehat{U}_{s}(\widehat{d})-\widehat{U}_{s}(d) \geq \widehat{U}(\widehat{d})-\epsilon-\{\widehat{U}(d)+\epsilon\} \geq \widehat{\Delta}-2\epsilon. $$ Thus, if $\widehat{\Delta}>2\epsilon$, the original winner remains strictly better than every competitor under that perturbation. If the margin is no larger than $2\epsilon$, this bound cannot guarantee that the winner is preserved; it does not imply that the winner must change.

Worked Example: Reading a Near-Tied Candidate-Dose Profile

Consider hypothetical utilities at $50$, $75$, $100$, and $125$ mg of $0.50$, $0.60$, $0.62$, and $0.54$, respectively. The highest value is $0.62$, so the point-estimate recommendation is $100$ mg. The runner-up is $75$ mg with utility $0.60$, giving a margin of $0.62-0.60=0.02$.

Suppose a prespecified perturbation analysis indicates that utility at each dose can shift by as much as $0.015$. Here $2\epsilon=0.030$, which exceeds the $0.020$ margin. The bound therefore cannot certify that $100$ mg will remain the winner. This does not show that the recommendation will change; it indicates that the observed separation is too small to guarantee stability under the stated bound. The plot should make the near-tie visible, and the analyst should report the perturbation results and assess whether the dose difference matters clinically.

Diagnostics, Alternatives, and Limitations

Useful diagnostics include checking whether the bootstrap fit succeeds in nearly all replicates, whether selected doses cluster at a boundary of the candidate grid, and whether instability is driven by a small number of observations or a narrow model feature. Inspect utility profiles for discontinuities or unexpected reversals. Verify that constraints are applied identically in every replicate, and that ties are handled by a stated rule. If a replicate has no feasible dose or the model fails to fit, do not silently discard it: report its frequency and investigate the cause.

These tools answer related but different questions. An uncertainty interval for a dose summarizes uncertainty under its stated construction; a decision curve describes decisions or consequences across thresholds. A perturbation analysis asks how results change under chosen modifications. Bootstrap selection frequencies summarize how often the rule selects each dose under a resampling scheme. Robust or minimax approaches instead formulate a decision rule intended to perform acceptably across specified uncertainty sets. None of these summaries replaces the others, and the appropriate method depends on the decision question.

Stability is not the same as validity. A misspecified model can recommend the same dose under every tested perturbation. A narrow candidate grid can make the recommendation appear stable merely because nearby doses were not considered. Bootstrap results can also be unreliable with sparse data, influential clusters, boundary estimates, or a poorly specified resampling scheme. Finally, a dose change that is statistically frequent may have negligible practical effect, while a rare switch across a binding safety threshold may be consequential. State the scope of the perturbations, the decision rule, the candidate-dose grid, and the practical meaning of changes.

Common Mistakes

Calling selection frequencies probabilities of truth. A bootstrap frequency is a proportion of resampled analyses selecting a dose; it is not the probability that dose is truly optimal.
Changing only the final dose label. A meaningful diagnostic reruns the model and decision rule, not just the last maximization step.
Equating a changed winner with a consequential change. Compare utility differences and clinical implications, not just dose labels.
Choosing perturbations after seeing their results. Specify plausible changes and tolerances transparently to avoid a selective sensitivity analysis.
Treating stability as proof of correctness. Repeatedly selecting one dose cannot rule out shared model misspecification or an incomplete candidate grid.

Key takeaway: Diagnose stability by rerunning the complete dose-selection procedure under clearly stated perturbations, examining bootstrap selection frequencies, and inspecting utility across candidate doses. Interpret all three in relation to practical consequences and their explicit assumptions.

Check Your Understanding

Answer each question using the definitions and calculations in this tutorial.

A bootstrap analysis selects $60$ mg in 45 of 150 replicates and $80$ mg in 105. What are the selection frequencies, and what do they describe?
The estimated utility margin is $0.04$, and a perturbation changes every candidate utility by no more than $0.01$. Does the bound guarantee the original winner remains best?
Why should repeated observations from one participant generally be resampled together in a participant-level bootstrap?
A dose is selected in every tested perturbation. Give one reason this does not establish that it is the correct dose.

Solutions. (1) The frequencies are $45/150=0.30$ for $60$ mg and $105/150=0.70$ for $80$ mg. They describe how often each dose is selected under that bootstrap procedure, not probabilities that either is truly optimal. (2) Yes. Here $2\epsilon=0.02$, and the margin $0.04$ is greater than $0.02$, so the stated bound guarantees the original winner remains strictly better than its competitors. (3) Repeated observations within a participant are dependent; resampling them as independent rows would fail to preserve the data structure. (4) All tested analyses could share a misspecified model, an incomplete candidate grid, or an inadequate set of perturbations, so stability does not establish correctness.)