Why Diagnose a Utility-Based Recommendation?
A dose can maximize an overall utility score while performing poorly on one outcome and well on another. The overall score identifies the preferred dose under specified utility assumptions; it does not, by itself, explain the trade-offs that produced the choice. A diagnostic analysis makes those trade-offs visible and asks whether reasonable changes in preferences would change the recommendation.
This tutorial focuses on interpreting component plots and dose-wise utility tables, then using counterfactual weight changes to assess a recommendation. These are diagnostics of the decision calculation, not tests of whether a clinical effect is statistically significant. The component values and weights used below are hypothetical.
For this calculation to be interpretable, the component scores must be on scales whose directions and clinical meanings are understood, and the weights must represent the intended trade-offs across those scales. The additive form also assumes that the stated weighted sum is an appropriate way to combine the components. If scores are expected utilities already incorporating outcome consequences, their weighting must reflect the intended preference structure rather than count the same consequence twice. A utility diagnostic cannot repair a poorly specified utility model.
A component plot commonly displays \(C_k(d)\) for each component and dose, using stacked bars or grouped marks. The plot makes the sources of a dose’s total score easier to see: a component can add to the score, contribute little, or place the dose at a disadvantage relative to another dose. The dose-wise utility table provides the corresponding numbers, which are needed to verify the ranking and calculate exact differences. A plot alone may obscure a small difference, especially when axes are truncated or component contributions are shown on different scales.
Decomposing a Dose Comparison
The most useful diagnostic is often not the absolute utility at a dose, but the difference between two doses. Subtracting their utilities separates the overall advantage into component-level contributions. This explains which outcomes favor one dose and which favor its competitor.
Each term \(w_k\{m_k(d_a)-m_k(d_b)\}\) is the contribution of component \(k\) to the utility difference. Positive terms favor \(d_a\); negative terms favor \(d_b\). The terms sum exactly to the total utility difference, so the decomposition is an accounting identity rather than a separate statistical model.
Worked Example: Finding What Drives the Recommended Dose
Consider three hypothetical doses, with expected scores from 0 to 100, where higher is better. The components are benefit, symptom control, and safety. Suppose the preference weights are 0.50, 0.20, and 0.30, respectively.
| Dose | Benefit score | Symptom-control score | Safety score |
|---|---|---|---|
| 10 mg | 40 | 80 | 95 |
| 20 mg | 70 | 75 | 85 |
| 30 mg | 82 | 60 | 55 |
For 10 mg, the contributions are \(0.50(40)=20\), \(0.20(80)=16\), and \(0.30(95)=28.5\). Thus \(U(10)=20+16+28.5=64.5\) utility points. For 20 mg, the contributions are \(35\), \(15\), and \(25.5\), giving \(U(20)=75.5\). For 30 mg, they are \(41\), \(12\), and \(16.5\), giving \(U(30)=69.5\). Under these scores and weights, 20 mg has the highest utility.
The 20 mg versus 30 mg comparison explains why. The utility difference is \(0.50(70-82)+0.20(75-60)+0.30(85-55)=-6+3+9=6\) points. The higher benefit score at 30 mg favors 30 mg by 6 weighted points, but symptom control favors 20 mg by 3 points and safety favors 20 mg by 9 points. The net difference is 6 points in favor of 20 mg. A component plot would show this pattern visually; the table and subtraction establish its exact size.
Counterfactual Weight Changes
A counterfactual weight analysis asks, “If the outcome preferences were different, would the recommended dose change?” It holds the component scores fixed and recalculates utility under alternative weights. This isolates sensitivity to preferences; it does not account for uncertainty in the estimated component scores.
For two doses, the switch occurs when their utility difference reaches zero. In the example above, let \(b\), \(s\), and \(t\) denote the weights on benefit, symptom control, and safety, with \(b+s+t=1\). The utility difference between 30 mg and 20 mg is \(12b-15s-30t\), because 30 mg scores 12 points higher on benefit, 15 lower on symptom control, and 30 lower on safety. Substituting \(t=1-b-s\) gives \(42b+15s-30\). Therefore, 30 mg is preferred to 20 mg exactly when \(42b+15s-30>0\), provided this pairwise comparison is the relevant one among all candidate doses.
Worked Example: How Much Must Benefit Preference Change?
Use the same hypothetical component scores. Consider a counterfactual preference set with benefit weight \(b=0.67\), symptom-control weight \(s=0.13\), and safety weight \(t=0.20\). These weights are nonnegative and sum to 1.
At 20 mg, utility is \(0.67(70)+0.13(75)+0.20(85)=46.9+9.75+17=73.65\). At 30 mg, it is \(0.67(82)+0.13(60)+0.20(55)=54.94+7.8+11=73.74\). Thus 30 mg exceeds 20 mg by \(0.09\) utility points. At 10 mg, utility is \(0.67(40)+0.13(80)+0.20(95)=26.8+10.4+19=56.2\), so 30 mg is highest among the three doses under this counterfactual.
The pairwise threshold gives the same result: \(42(0.67)+15(0.13)-30=28.14+1.95-30=0.09\). Holding \(s=0.13\) fixed and assigning the remaining weight to safety, the switch threshold is \(b=(30-15(0.13))/42=28.05/42\approx0.6679\). The chosen \(b=0.67\) is only slightly above that threshold, so the recommendation changes, but the comparison is close. The calculation diagnoses sensitivity; it does not establish that these counterfactual preferences are appropriate.
For more than two doses, checking only the leading pair may be insufficient: another dose could become optimal under the new weights. Recalculate utility for every candidate dose under each counterfactual weight set. Useful scenarios include stakeholder-supported alternatives, a small perturbation around the primary weights, and plausible ranges for weights. A counterfactual should preserve the weight constraints; for example, increasing one weight while keeping the others unchanged would generally make the weights no longer sum to 1. One must specify which other weight or weights decrease to compensate.
Worked Example: Checking Robustness Over a Range of Weights
Return to the three-dose example. Suppose the plausible preference region is \(0.45\leq b\leq0.55\), \(0.15\leq s\leq0.25\), and \(t=1-b-s\). These limits describe a hypothetical sensitivity region, not a universal standard.
To compare 20 mg with 10 mg, the utility difference is \(30b-5s-10t\). Substituting \(t=1-b-s\) gives \(40b+5s-10\). Its smallest value within the stated ranges occurs at \(b=0.45\) and \(s=0.15\): \(40(0.45)+5(0.15)-10=18+0.75-10=8.75\). Thus 20 mg exceeds 10 mg throughout the region.
To compare 20 mg with 30 mg, the difference is \(-12b+15s+30t\), or \(30-42b-15s\) after substitution. Its smallest value occurs at the largest \(b\) and \(s\): \(30-42(0.55)-15(0.25)=30-23.1-3.75=3.15\). Therefore 20 mg also exceeds 30 mg throughout this region. Since it beats both alternatives at every allowed weight pair, 20 mg remains the recommended dose across the specified region. This is a bounded sensitivity result; it does not imply robustness to preferences outside that region or to uncertainty in the component scores.
Diagnostics, Alternatives, and Limitations
A good diagnostic combines a table, a visual decomposition, and explicit sensitivity calculations. The table makes arithmetic auditable; the component plot reveals the pattern of trade-offs at a glance; counterfactual calculations show whether that pattern is decisive or preference-sensitive. A single total-utility ranking answers which dose wins under one weight set, but not why it wins or how stable that answer is.
Several warning signs merit follow-up:
- A large total-score gap comes from one component. Check that its scale and weight express the intended clinical importance, and show the component-level difference plainly.
- The leading doses are nearly tied. Report the utility difference and examine plausible weight changes rather than describing the ranking as decisive.
- A small weight change reverses the choice. Report the switch or threshold and clarify which preferences were changed. Do not present one selected weight set as if it were uniquely determined by the outcome data.
- The plot and table seem to disagree. Recheck the component directions, weights, arithmetic, and plotted axes. Higher scores should consistently mean better outcomes in this tutorial’s formulation.
- A dose has the highest utility but violates a separately imposed constraint. Utility ranking does not override a hard eligibility or safety rule; apply such constraints as specified before making the recommendation.
A component plot is often more interpretable than a ranking alone, but it can be misleading if bars use inconsistent axes or if raw component scores are displayed alongside weighted contributions without clear labels. A dose-wise table is more precise but can be harder to scan. Counterfactual weight analysis is more informative than a single table for preference sensitivity, but it depends on the alternatives being clinically plausible and on keeping the score model fixed. None of these displays resolves uncertainty in the component estimates. If those estimates are uncertain, that uncertainty requires a separate analysis.
Also distinguish weight sensitivity from scale changes. The numerical contributions depend on the component scales and weights together. A change in score scale may require a corresponding change in weights to preserve the same trade-offs; varying weights while silently changing scales does not isolate preference sensitivity. The diagnostics are most useful when the scales, outcome directions, and preference assumptions are stated alongside the results.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions below.
- Two doses have component-score differences \(m_1(d_A)-m_1(d_B)=8\) and \(m_2(d_A)-m_2(d_B)=-10\). The weights are 0.4 and 0.6. What is the utility difference, and which dose is preferred?
- Why is it not generally valid to increase one weight in a counterfactual analysis while leaving all other weights unchanged?
- Suppose a dose has the highest utility, but its advantage over the second-ranked dose is only 0.02 points under the chosen weights. What should be examined before calling the recommendation robust?
- Under a stated range of weights, dose A has a positive utility difference over every other candidate dose. What conclusion can be drawn, and what important uncertainty does this not address?
Solutions. For question 1, the difference is \(0.4(8)+0.6(-10)=3.2-6=-2.8\) utility points, so \(d_B\) is preferred by 2.8 points. For question 2, the weights must satisfy the stated constraint that they sum to 1; changing one weight alone generally violates it. Specify how the other weights change to compensate. For question 3, inspect the component-level contributions and recalculate all candidate-dose utilities under plausible alternative weights. A 0.02-point lead is a close ranking, not evidence of a stable preference. For question 4, dose A is the utility maximizer throughout that specified weight region. This does not address uncertainty in the component-score estimates or preferences outside the region.