Why Utility Errors Matter
A dose-finding utility is a formal expression of how desirable the possible consequences of a dose are. It can combine benefit and harm into a score used to compare candidate doses. But a precisely computed score is not necessarily a clinically sensible one: a reversed sign, an inappropriate scale, duplicated consequences, or an unexplained weight can change the ranking while leaving the arithmetic flawless. The purpose of this tutorial is to make those specification errors visible before a utility is used to select a dose.
The earlier material on expected utility and utility-weight calibration provides the decision framework: choose the dose with the greatest expected utility under the specified model and preferences. Here, the focus is auditing what has been specified. The checks apply whether component outcomes are model predictions, elicited probabilities, or summaries of clinical consequences; they do not replace checking the adequacy of the underlying outcome models.
A weighted sum is one possible specification. For example, if \(u_B\) scores benefit and \(u_{H_j}\) scores each harm, a proposed utility might be \(U=w_Bu_B+\sum_j w_j u_{H_j}\). This form requires care: each component must have a defined direction and scale, the outcomes represented by the components must not inadvertently duplicate consequences, and the weights must encode interpretable preferences. An additive form also assumes that the stated component values and weights adequately represent preferences across combinations of outcomes. If preferences depend on interactions between outcomes, a simple additive sum may be inadequate.
Audit the Direction and Scale
First check the meaning of a high value in every component. If high benefit scores are desirable but high toxicity scores are undesirable, adding both with positive weights gives toxicity the wrong direction. A sign error is not repaired by the fact that the final scores look plausible. State the direction in words, inspect the formula, and test it on an outcome that should clearly be preferred.
Next check what a unit means. A response probability may range from 0 to 1, while a symptom score may range from 0 to 100. Assigning both a weight of 1 does not generally give them equal influence: a 10-point symptom change may outweigh a 0.05 probability change simply because the numerical ranges differ. Scaling can make component ranges explicit, but it does not determine preferences by itself. The weights must still express how decision makers value the changes represented by those ranges.
A useful normalization maps a clinically defined worst-to-best swing in a component to a common interval such as 0 to 1. The endpoints must be stated in clinical terms, not selected merely because they are convenient. A weight then describes the value assigned to that component's defined swing relative to the other components' swings. If a component is transformed, its weight must be reconsidered; retaining a weight unchanged after changing the scale can silently change the intended trade-off.
Worked Example: Correcting a Reversed Toxicity Score
Suppose two hypothetical doses have benefit scores of 0.70 and 0.80, and toxicity-burden scores of 0.20 and 0.40, respectively. The benefit score and toxicity burden are both on a 0-to-1 scale, but larger toxicity burden is worse. A proposed utility is \(U=0.6B+0.4H\), where \(H\) is the toxicity burden.
Check the direction. Substituting the scores gives \(U_A=0.6(0.70)+0.4(0.20)=0.42+0.08=0.50\) and \(U_B=0.6(0.80)+0.4(0.40)=0.48+0.16=0.64\). The formula rewards the higher toxicity burden at dose B. That contradicts the stated meaning of \(H\), so the resulting ranking is not a valid application of the intended preferences.
Repair the specification. If the toxicity burden is defined so that 0 is worst and 1 is best, use \(u_H=1-H\). The corrected formula is \(U=0.6B+0.4(1-H)\). Then \(U_A=0.6(0.70)+0.4(0.80)=0.42+0.32=0.74\), while \(U_B=0.6(0.80)+0.4(0.60)=0.48+0.24=0.72\). Under this utility, dose A ranks higher. The reversal is not a claim that dose A is objectively superior; it follows from the stated scores and weights after aligning the harm direction.
This correction assumes the two component scales and weights were intended to be comparable as written. If the weights were originally elicited for a different definition of the toxicity component, they must be reviewed rather than carried over automatically.
Find Hidden Scale Effects
A quick diagnostic is to calculate, for each component, its contribution to the total utility over the clinically plausible range. If a component can vary from 0 to 100 and another only from 0 to 1, equal numerical weights may give the first component up to one hundred times the numerical swing. The relevant question is not whether the component values have different units; it is whether their weighted changes match the intended clinical trade-offs.
Worked Example: Detecting an Unscaled Component
Consider two hypothetical doses with response probabilities \(B_A=0.20\), \(B_B=0.26\), and a symptom-improvement score \(S_A=12\), \(S_B=8\), measured on a 0-to-100 scale where higher is better. The proposed utility is \(U=B+S\), with equal coefficients of 1.
The proposed scores are \(U_A=0.20+12=12.20\) and \(U_B=0.26+8=8.26\), so dose A wins by 3.94 utility units. This ranking is driven overwhelmingly by the numerical scale of \(S\), not by evidence that the symptom improvement should receive that preference. Indeed, the response contribution favors B by only \(0.26-0.20=0.06\), whereas the symptom contribution favors A by \(12-8=4\).
Suppose, for illustration, stakeholders define the relevant symptom swing as 0 to 20 points and prefer the full response-probability swing from 0 to 1 and that symptom swing equally. Define \(u_S=S/20\), and use \(U=B+u_S\). Then \(U_A=0.20+12/20=0.80\), and \(U_B=0.26+8/20=0.66\). Dose A still ranks higher, but the comparison now uses explicitly defined component swings. If stakeholders instead value the full symptom swing at half the full response swing, the utility would be \(U=B+0.5u_S\): \(U_A=0.20+0.5(0.60)=0.50\), and \(U_B=0.26+0.5(0.40)=0.46\).
The numerical weights in this illustration are assumptions, not universal defaults. The example shows why the scale must be stated and why a change of scale entails a deliberate choice of weights.
The scale issue can be seen algebraically. Suppose one component \(u_1\) is rescaled to \(u_1'=a u_1+b\), with \(a>0\), but its coefficient \(w_1\) is left unchanged while other components are untouched. Its contribution becomes \(w_1a u_1+w_1b\). The constant term does not affect dose rankings if it is the same at every dose, but the factor \(a\) changes the relative influence of that component. Thus, transforming one component is not generally a harmless transformation of the whole utility. By contrast, multiplying the entire utility by the same positive constant preserves its dose ranking.
Check for Duplicated Harms
A harm can enter a utility more than once when components overlap. For example, a broad “any adverse event” indicator includes severe adverse events if severe events are a subset of all events. Penalizing both the broad indicator and the severe-event indicator may assign an additional penalty to severe events. That may be intentional, but it should not happen accidentally.
Overlap is not the same as correlation. Two distinct harms can occur together and both deserve separate penalties if that reflects the preferences being represented. The problem arises when the same consequence is represented repeatedly without a deliberate reason, or when a composite already includes a component that is added again. One diagnostic is to list mutually exclusive outcome states and write the utility assigned to each. If a severe state receives an unexplained extra penalty because it appears in two terms, the specification needs clarification.
Worked Example: Removing an Accidental Double Penalty
Suppose hypothetical patients can have no adverse event, a mild event, or a severe event, with probabilities 0.60, 0.30, and 0.10 at a candidate dose. These mutually exclusive probabilities sum to 1. A response probability of 0.60 is also predicted. The team intends to assign a benefit of 100 points per unit response probability, a penalty of 10 points for a mild event, and 30 points for a severe event.
A proposed formula subtracts 10 times the probability of any event and 30 times the probability of a severe event. Since severe events are included in “any event,” the expected utility is \(100(0.60)-10(0.30+0.10)-30(0.10)=60-4-3=53\). In this formula, a severe event incurs both the 10-point broad-event penalty and the 30-point severe-event penalty. That is not the stated intention if 30 points is the total penalty for a severe event.
Use the mutually exclusive event categories instead. The expected harm penalty is \(10(0.30)+30(0.10)=3+3=6\), so expected utility is \(100(0.60)-6=54\). Equivalently, define the event penalty as 0 for no event, 10 for mild, and 30 for severe, then average those values using the category probabilities. If stakeholders truly intend a 10-point penalty for any event plus an additional 30 points for a severe event, the original formula is coherent, but the severe-event total penalty is 40 points and should be made explicit.
Make the Weights Clinically Legible
A weight is clinically useful only when its meaning can be explained. In a weighted utility, ask what change in one outcome would compensate for a specified change in another, holding other components fixed. If that trade-off cannot be stated in ordinary clinical terms, the weight may be a convenient tuning constant rather than an elicited preference. Earlier material on swing weighting and anchoring utility scales gives ways to elicit and communicate such trade-offs; the audit here is to check whether the final formula preserves what was elicited.
For instance, if normalized benefit and harm components each range from 0 to 1, a utility \(U=4u_B+u_H\), with both higher-is-better, says that a full 0-to-1 swing in benefit contributes four times as much utility as a full swing in harm. That statement is precise, but it may still be unacceptable clinically. Presenting it clearly allows stakeholders to challenge it. Reporting only “the benefit weight is 4” does not.
A weight should not be interpreted without the component definition, units or normalization, and relevant range. Nor does an elicited preference guarantee that the decision is robust: uncertainty in estimated outcomes can still affect the recommended dose. Utility specification and model uncertainty are distinct issues, and a clear utility does not remove uncertainty in predicted benefits or harms.
A Practical Audit and Its Limits
State what outcome each component represents, its unit or scale, and the clinically relevant range.
For every component, verify that a larger score means a more preferred consequence, or include an explicit sign or transformation.
Calculate each component's weighted swing across its relevant range and translate the weights into clinical trade-offs.
List possible outcome states, identify nested or repeated consequences, and confirm that the utility assigned to each state matches the intended preference.
Recalculate expected utilities after repairs and test whether plausible alternative weights or definitions change the selected dose.
Diagnostics include checking whether an obviously worse outcome receives a higher score, whether one component dominates solely because of its numerical range, and whether a severe event receives an unintended extra penalty. Compare dose rankings under the original and corrected specifications; a ranking reversal signals that the error is decision-relevant. Sensitivity analysis over plausible utility definitions can show whether a recommendation depends on a narrow preference assumption, but it cannot establish which preferences are clinically appropriate.
The main alternative to an additive utility is not a better choice of arbitrary weights. If a safety threshold must never be traded against benefit, represent it as a hard constraint rather than burying it in a compensatory score. If trade-offs are unacceptable or inadequately elicited, report multiple outcomes or use Pareto comparisons instead of claiming that one weighted total resolves the decision. These alternatives clarify different decision objectives; none eliminates the need to define outcomes and preferences carefully.
Check Your Understanding
For each question, identify the utility-design issue and state the appropriate correction.
- A utility adds a response score, where higher is better, to a toxicity score, where higher means more severe toxicity. What is wrong, and how could the direction be repaired?
- A benefit component varies from 0 to 1 and a symptom component from 0 to 100. Both have coefficient 1. Why does this not establish equal preference?
- A utility penalizes “any adverse event” and “severe adverse event,” where every severe event is also an adverse event. What must be checked before using both terms?
- With two components normalized from 0 to 1, \(U=3u_B+u_H\), and both components higher-is-better. State the implied relative value of their full swings.
- Why might a hard safety threshold be preferable to adding a large harm penalty to an otherwise compensatory utility?
Solutions. (1) The toxicity direction is inconsistent: the formula rewards more toxicity. Reverse the score so higher means safer or less toxic, or subtract the burden with an appropriately defined coefficient, then verify that the weights still match their elicited meaning. (2) Equal coefficients apply to the numerical units, not necessarily to clinically comparable changes. Define clinically relevant component ranges, normalize them if appropriate, and elicit weights for those swings. (3) Check whether the severe-event penalty is intended to be additional to the general-event penalty. Enumerate mutually exclusive outcome states and confirm the total penalty for each; use exclusive categories if severe events should receive only their specified total penalty. (4) A full 0-to-1 swing in benefit contributes three times the utility change of a full 0-to-1 swing in the other component. Whether that is acceptable requires clinical interpretation. (5) A finite penalty permits sufficiently large benefit to compensate for harm. A hard constraint excludes doses that violate the threshold regardless of their benefit score, which is appropriate when such compensation is not allowed.