Why Eliciting Utilities Matters
When several treatments differ in benefit, adverse outcomes, and resource use, the evidence alone may not determine which arm should be preferred. One treatment may have a higher response rate but more serious adverse events; another may be less costly but offer less benefit. A decision requires both the posterior evidence about each arm and a statement of how important the different consequences are to the decision-maker.
Utility weights make those value judgements explicit. They do not change the observed data or the posterior distribution. They translate consequences onto a common scale so that a decision rule, such as choosing the arm with the greatest posterior expected utility, can compare them. The introductory material on posterior expected utility for arm selection motivates that rule; here the focus is how to construct and examine the utility inputs.
A utility scale is not automatically a measure of clinical importance just because it contains numbers. Its meaning comes from the preferences and trade-offs used to construct it. A utility of 10 is not necessarily twice as desirable as a utility of 5, unless the scale and elicitation method support that interpretation. For ranking arms, differences in utility are usually the relevant quantities; adding the same constant to every arm does not change the ranking.
Specify the Consequences Before the Weights
Elicitation begins by defining outcomes clearly. A benefit might be response by a prespecified time, symptom improvement, or survival; harm might be a serious adverse event within a fixed period; cost might be incremental treatment cost per patient. Definitions should specify the population, time horizon, severity, and measurement scale. Otherwise, respondents may be assigning weights to different consequences without realizing it.
For a simple additive score, let \(\Delta B_k\) denote arm \(k\)'s benefit relative to control, \(\Delta H_k\) its increase in harm, and \(C_k\) its incremental cost. One possible linear utility model is
Here \(w_B\), \(w_H\), and \(w_C\) are nonnegative weights. The signs express that more benefit is preferred, while additional harm and cost are undesirable. This formulation is a modelling choice, not a universal law: it assumes that the components can be combined additively and that the weights apply over the ranges being considered. The outcome units and weight units must match. For example, if \(\Delta B_k\) is a probability difference, \(w_B\) is measured in utility points per unit probability.
An arm can be assessed using posterior expected utility. If \(\theta_k\) denotes the uncertain outcome quantities for arm \(k\), and \(U_k(\theta_k)\) is its utility at a particular value of those quantities, then
The posterior expectation incorporates uncertainty in the outcomes. It does not remove uncertainty about whether the chosen arm is best, and it does not settle which weights are appropriate. Those are separate questions.
Eliciting Trade-Offs on a Common Scale
Weights are often easier to explain through comparisons than by asking someone to name an abstract number. First choose a reference change in benefit, harm, or cost. Then ask what change in another outcome would be regarded as equally important, or what amount of one outcome would compensate for the other. These questions reveal ratios between weights.
For instance, if a 10-percentage-point increase in response is judged equivalent in value to a 5-percentage-point increase in serious adverse events, the two changes should make equal and opposite contributions to an additive utility score. If the benefit weight is fixed at 100 points per unit increase in response probability, this trade-off implies a harm weight of 200 points per unit increase in adverse-event probability. Fixing one weight sets the scale; the trade-offs determine the relative weights.
Elicitation can use structured interviews, ranking or choice tasks, rating scales, or established preference-based instruments. A direct question about trade-offs should describe the outcomes in understandable, clinically concrete terms and allow respondents to express uncertainty or disagreement. The perspective also matters: patients, clinicians, payers, and trial teams may reasonably place different values on the same consequences. If the decision is intended to represent a particular group, the elicitation should represent that group rather than silently substituting another perspective.
Worked Example: Turning a Benefit–Harm Trade-Off Into a Weight
Suppose a hypothetical decision panel evaluates treatment effects per patient over one year. Benefit is the probability of response by one year, harm is the probability of a serious adverse event by one year, and cost is incremental cost in thousands of dollars. The panel defines utility in points and assigns \(w_B=100\) points per unit increase in response probability. It judges a 10-percentage-point response increase to be equally valuable as avoiding a 5-percentage-point increase in serious adverse events.
A 10-percentage-point response increase is a probability change of \(0.10\), so its utility contribution is
The 5-percentage-point adverse-event increase is a probability change of \(0.05\). To make its disutility equal to 10 points, solve \(w_H(0.05)=10\):
If the panel also judges an additional \$1,000 of cost per patient equivalent to a loss of 3 points, then \(w_C=3\) points per thousand dollars. The resulting score is \(U=100\Delta B-200\Delta H-3C\). For example, a 0.10 response increase, a 0.05 harm increase, and \$1,000 of added cost contribute \(10-10-3=-3\) points relative to the reference. The weights encode the stated panel preferences; the arithmetic does not establish that those preferences are appropriate for every patient or setting.
From Elicited Weights to a Multi-Arm Decision
For a linear utility function, posterior expected utility can be calculated from posterior means of the component outcomes. This follows from linearity of expectation: the expected value of a weighted sum is the same weighted sum of expectations. The result holds even when the outcomes are statistically dependent, provided their expectations exist. Dependence still matters for nonlinear utilities, probabilities of joint events, or uncertainty summaries beyond the expected utility.
Thus, posterior means are sufficient for this particular expected-utility calculation. They are not sufficient for every decision problem: if utility includes a threshold, interaction, or other nonlinear feature, the full posterior distribution of the relevant outcomes may be needed.
Worked Example: Comparing Three Arms Under One Set of Weights
Consider a hypothetical trial with control and three experimental arms. The table gives posterior mean incremental outcomes per patient relative to control. Benefit is the response-probability difference, harm is the serious-adverse-event probability difference, and cost is in thousands of dollars per patient. Use the weights from the preceding example: 100 points per unit of benefit probability, 200 points per unit of harm probability, and 3 points per thousand dollars.
| Arm | Mean benefit difference | Mean harm difference | Mean added cost |
|---|---|---|---|
| A | 0.12 | 0.03 | 0.8 |
| B | 0.08 | 0.01 | 1.4 |
| C | 0.10 | 0.06 | 0.3 |
For arm A, the posterior expected utility relative to control is
For arm B, it is
For arm C, it is
Under these posterior means and these weights, arm A has the largest expected utility, at 3.6 points relative to control. This is a decision conditional on the specified perspective, time horizon, outcomes, and weights. It is not a claim that arm A has the highest benefit alone, nor does it show that arm A is certainly the best arm.
How Alternative Weights Change the Choice
A utility analysis should examine whether a recommendation depends strongly on a debatable weight. The same posterior outcome estimates can yield a different ranking when the decision-maker places greater value on avoiding harm or conserving resources. Such sensitivity is substantive: it shows which trade-offs drive the decision, rather than indicating that the posterior evidence has changed.
Worked Example: A Harm-Sensitive Decision-Maker
Use the same hypothetical outcome summaries, but suppose a second decision-making group assigns 500 points per unit increase in serious-adverse-event probability and 1 point per thousand dollars of cost, retaining the benefit weight of 100. For arm A, the score becomes
For arm B, the score is \(100(0.08)-500(0.01)-1(1.4)=8-5-1.4=1.6\) points. For arm C, it is \(100(0.10)-500(0.06)-1(0.3)=10-30-0.3=-20.3\) points. The preferred experimental arm is now B, not A. Arm B's lower mean benefit is outweighed by its lower mean harm under this group's preferences.
The sensitivity can also be shown algebraically. Keep the benefit weight at 100 and the cost weight at 3, but let the harm weight be \(w_H\). The difference between arms A and B is
The two arms have equal expected utility when \(5.8-0.02w_H=0\), giving \(w_H=290\) points per unit increase in harm probability. Below 290, A has the higher expected utility; above 290, B does. This threshold is conditional on the given posterior means and the fixed benefit and cost weights. It is not a clinical threshold for harm in general.
Assumptions, Diagnostics, and Alternatives
An additive weighted score is useful when the consequences can be represented on defensible scales and decision-makers are willing to treat their contributions as separable. Before relying on it, check the following:
- Consistent outcome definitions: Benefit, harm, and cost use a common population and a stated time horizon. For example, response at six months should not be combined casually with adverse events observed over a different period.
- Understandable units: Each weight has units that match its outcome measure. A weight for probability per unit cannot be applied directly to a percentage without converting consistently.
- Reasonable additivity: The score assumes the value of a benefit does not change with the harm level or cost level. If a serious adverse event is especially unacceptable when benefit is small, an interaction or a different utility structure may be needed.
- Preference stability over the relevant range: A constant weight assumes that equal-sized changes have equal utility impact over the range being compared. This may be implausible near a clinically important boundary.
- Weight sensitivity: Recalculate rankings over credible weights and report any changes. A recommendation that switches under small, plausible preference changes should be presented as preference-sensitive.
- No double counting: Avoid penalizing the same consequence twice. For instance, if a benefit measure already incorporates treatment-related symptoms, separately subtracting those same symptoms may count them twice.
A common mistake is to choose weights because they make the preferred arm look attractive, then describe the result as if it followed from the data alone. The correction is to specify the decision perspective and elicit or justify weights independently of the desired ranking, then show how conclusions respond to defensible alternatives.
Expected utility is not the only decision approach. A probability-threshold rule can select arms that exceed a prespecified probability of clinically meaningful benefit, without assigning a single numerical score to every consequence. A constrained approach can require acceptable safety or a budget limit first, then compare benefit among the eligible arms. A preference-based health utility instrument may provide a standardized scale for some outcomes and populations. These approaches answer different questions: a threshold emphasizes a particular event, a constraint treats a criterion as a requirement, and expected utility trades multiple outcomes according to weights. None removes the need to state the decision context and its value judgements.
Utilities also have limitations. A single average score can hide important differences between patient groups, rare severe outcomes, or uncertainty about the weights themselves. If stakeholders disagree, it can be more informative to report their separate preferred arms or a range of utility results than to average their preferences without justification. Posterior uncertainty in the clinical outcomes and uncertainty or variation in preferences should be distinguished in reporting.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- If a 0.08 increase in response probability contributes 8 utility points under a chosen scale, what benefit weight is being used?
- Using \(U=100\Delta B-200\Delta H-3C\), calculate utility for an arm with \(\Delta B=0.05\), \(\Delta H=0.01\), and \(C=0.5\), where cost is in thousands of dollars.
- Why can posterior means of benefit, harm, and cost be used to calculate posterior expected utility for a linear additive score?
- Suppose an arm's ranking changes when the harm weight increases from 200 to 220. What should be reported or examined before making a firm recommendation?
- Give one reason a constrained decision rule could be preferred to a weighted utility score.
Solutions. (1) The weight is \(8/0.08=100\) utility points per unit increase in response probability. (2) The score is \(100(0.05)-200(0.01)-3(0.5)=5-2-1.5=1.5\) utility points relative to the reference. (3) Linearity of conditional expectation gives the weighted sum of the posterior means; this conclusion is for the specified linear score. (4) Report that the choice is sensitive to the harm weight, show the range of plausible weights and the resulting rankings, and explain how the weights were elicited. (5) A constraint may be preferable when a safety limit or resource ceiling is a firm requirement rather than a trade-off; eligible arms can then be compared on utility only after meeting that requirement.