Why Calibrate Weights with Swings?
An additive benefit-risk utility model combines component utilities for outcomes such as efficacy and toxicity. Its weights determine how much each component contributes to the overall score. But asking stakeholders to rate an outcome’s “importance” in isolation leaves a crucial question unanswered: importance over what range of change? A change in response rate from 30% to 70% is not the same prospect as a change from 60% to 70%. Similarly, reducing severe toxicity from 25% to 5% is a different swing from reducing it from 10% to 5%.
Swing weighting makes the comparison explicit. Stakeholders compare hypothetical improvements from the worst to the best value of each outcome, using clinically specified ranges. Their ratings are then normalized to form the weights of the additive utility model. These weights therefore refer to the value of the defined full-range changes, not to an outcome’s abstract importance independent of its range.
The outcome ranges and the component utility functions must be specified before interpreting the weights. A component utility \(u_j\) describes how values within an outcome’s range are scored; a swing rating compares the value of moving across that entire range with the value of the other full-range changes. Swing weighting calibrates the relative weights. It does not estimate clinical outcomes, choose dose, or determine the shape of each component utility function.
Procedure and Justification
For each outcome, define a clinically meaningful worst value and best value, and orient the scale so that the best value corresponds to \(u_j=1\) and the worst to \(u_j=0\). For a harm outcome, this usually means that the “best” value is a lower harm rate or less severe harm. Ask stakeholders to consider a profile in which every outcome is at its worst value. Then imagine improving exactly one outcome at a time from worst to best while keeping the others at their worst. Rate the value of each resulting swing on a common scale.
A practical elicitation method assigns 100 to the most valuable swing and rates the others relative to it. The scores are not percentages of patients helped or probabilities; they express relative value judgments. Normalize the ratings by their sum, then use the weights with component utilities calculated on the same defined outcome ranges.
Specify each outcome’s worst and best clinically relevant values. Record units, direction, and any population or treatment context needed to interpret the range.
Present one-at-a-time swings from the common worst-outcome profile. Rate each swing’s value relative to the most valuable swing on the agreed scale.
Divide each rating by the sum of all ratings. Check that weights sum to one, and combine them with component utilities defined over the elicited ranges.
The normalization follows directly from the desired weight scale. If raw ratings \(r_j\) are nonnegative and their sum is positive, set \(w_j=r_j/\sum_k r_k\). Then
Thus, the weights are relative shares of the total rated swing value. Multiplying every rating by the same positive constant leaves each normalized weight unchanged. The numerical scale is a device for elicitation; its meaning comes from the comparisons made by the stakeholder, not from the number 100 itself.
Worked Applications
Worked Example: Calibrating Efficacy and Severe-Toxicity Weights
Hypothetical elicitation. A stakeholder considers two outcomes. The efficacy range is a response rate from 30% (worst) to 70% (best). The severe-toxicity range is from 25% (worst) to 5% (best). The stakeholder rates the efficacy swing 80 and the toxicity-reduction swing 100, the highest rating on the scale. Find the weights and compare two hypothetical treatment profiles with component utilities already calculated on these ranges: Profile A has efficacy utility 0.70 and toxicity utility 0.80; Profile B has efficacy utility 0.85 and toxicity utility 0.60.
The total rating is \(80+100=180\). Therefore,
The exact weights are \(4/9\) and \(5/9\). Profile A’s overall utility is
Profile B’s overall utility is
Under this hypothetical additive model, Profile A is higher by \(6.8/9-6.4/9=0.4/9=0.04444\) utility units. This is a comparison of model scores on the specified utility scale; it is not, by itself, evidence that either profile causes better clinical outcomes.
Worked Example: Normalizing Three Outcome Swings
Hypothetical elicitation. A stakeholder rates full-range improvement in efficacy 75, improvement in symptoms 60, and reduction in severe toxicity 100. Suppose Profile C has component utilities \((0.80, 0.50, 0.30)\) and Profile D has \((0.60, 0.80, 0.50)\), in that outcome order. Calculate the weights and overall utilities.
The rating total is \(75+60+100=235\). The normalized weights are
They sum to \((15+12+20)/47=1\). For Profile C,
For Profile D,
Profile D has the higher weighted utility by \(4.6/47=0.09787\) utility units. The calculation uses the unrounded weights, avoiding small inconsistencies that can arise when rounded weights are used in intermediate products.
Worked Example: Why the Defined Range Changes the Weighting Question
Hypothetical elicitation. In one decision context, the efficacy swing is from a 40% to a 60% response rate, while severe toxicity ranges from 20% to 5%. A stakeholder rates these swings 70 and 100, respectively. In another context, the efficacy range is wider, from 20% to 80%, and the stakeholder rates that efficacy swing 100 and the same toxicity swing 80. Calculate the normalized weights in each context and explain why carrying weights over without review could mislead.
For the first context, the total rating is \(70+100=170\), so the efficacy and toxicity weights are \(70/170=0.4118\) and \(100/170=0.5882\). For the second context, the total is \(100+80=180\), giving weights \(100/180=0.5556\) and \(80/180=0.4444\).
The ratings differ because the stakeholder is comparing different efficacy swings in the two contexts. The weights are conditional on the full ranges presented. A weight from one range should not automatically be applied to a different range as though it expressed an invariant preference for the outcome label “efficacy.” The outcome utilities and weights must be coherent with the ranges to which the elicitation refers.
Diagnostics, Alternatives, and Limitations
Several checks help identify an elicitation that may be going wrong. First, confirm that every swing begins at the same hypothetical worst-outcome profile and changes only one outcome. If several outcomes change at once, a rating cannot be attributed to a single swing. Second, check that the best and worst values are clinically interpretable and that outcome directions are consistent; for toxicity, a lower rate may be the preferred endpoint. Third, ask stakeholders to explain why the most highly rated swing is more valuable than another. Unclear or unstable explanations may signal that the ranges are poorly specified or that the ratings are not comparable.
Repeat ratings, discuss substantial disagreements, and examine whether the resulting ranking of swings is stable across reasonable re-elicitation. A zero rating produces a zero weight under the formula; this is a strong statement that the full swing contributes no value in the stated model, so it should be checked rather than accepted as a routine response. If ratings are given as ranks only, they do not indicate the size of the differences between swings. Converting ranks to arbitrary numbers and normalizing them creates numerical weights, but those weights need not reflect stakeholders’ relative valuations.
Swing weighting is a quantitative follow-up to eliciting preferences and ranking priorities: it adds information about the relative size of full-range changes. It is not a substitute for directly eliciting component utilities within each range, and it does not remove the need to assess whether an additive model is appropriate. The weights also depend on whose preferences are elicited and how the outcome ranges are defined. A group may report a consensus rating, but the consensus process and variation in individual judgments should be documented.
A common pitfall is to say, “Efficacy is more important than toxicity, so efficacy must receive the larger weight.” That reasoning ignores the ranges. Swing weighting instead asks which specified full-range change is more valuable. A smaller swing in one outcome could be rated more highly than a larger swing in another, or the ordering could reverse when the ranges change. Another mistake is to treat a rating of 80 as an 80% probability or as a direct clinical effect. It is a relative elicitation score used to calculate a weight, not an observed outcome or probability.
Check Your Understanding
Answer each question, then compare with the solutions below.
- A stakeholder rates the efficacy swing 40 and the harm-reduction swing 60. What are the normalized weights?
- For the weights in Question 1, a profile has component utilities 0.90 for efficacy and 0.50 for harm reduction. Calculate its additive utility.
- Why are weights elicited for an efficacy swing from 30% to 70% not automatically the right weights for a swing from 60% to 70%?
- What is wrong with assigning the numbers 1, 2, and 3 to three outcomes solely because stakeholders ranked their swings first, second, and third, then treating the normalized values as calibrated weights?
Solutions. 1. The rating total is \(40+60=100\), so the weights are \(0.40\) and \(0.60\). 2. The utility is \(0.40(0.90)+0.60(0.50)=0.36+0.30=0.6600\) utility units. 3. The weights refer to the value of the full-range change presented. The two efficacy swings cover different changes, so their relative value compared with the harm swing can differ. 4. The ranks give only an ordering, not the relative sizes of the differences between swings. The numbers 1, 2, and 3 impose arbitrary magnitudes, so normalizing them does not establish calibrated relative swing values.