From Clinical Objective to Reward
A bandit policy needs a numerical reward for each patient assigned to a treatment. That number is not automatically the clinical objective. A trial may record survival, symptom scores, adverse events, or several outcomes together; the reward definition specifies which features of those observations count as benefit and how they are valued. A poorly chosen reward can make a policy optimize a quantity that is easy to calculate but does not represent the intended clinical goal.
The definition should start with the decision objective: what outcome, for which patients, over what follow-up period, and in what direction is better? The reward then maps the relevant patient outcome to a numerical value. For treatment arm \(a\), let \(Y(a)\) denote the outcome that a patient would have under that treatment, and let \(g\) be the prespecified reward function. The potential reward is \(R(a)=g(Y(a))\). The arm’s expected reward in a specified target population is \(\mu_a=E[R(a)]\). A larger \(\mu_a\) is preferred when the objective is to maximize reward.
For this interpretation to be meaningful, specify the population and outcome period, use the same reward rule for each arm, and measure the required outcomes in a comparable way. The observed reward for a patient assigned to arm \(a\) is the reward calculated from that patient’s outcome under \(a\); outcomes under unassigned treatments are not observed for that patient. The definition does not itself establish that the arms are comparable or that observed differences are caused by treatment. Those questions depend on the design and analysis.
Direction and Scale
A reward convention should be explicit: larger values mean more of the objective. If the clinical measure already has that direction, it may be used directly. If lower values are better, the reward can reverse the sign or express improvement relative to a reference. For example, if \(Y\) is a pain score and a lower follow-up score is better, \(R=-Y\) gives larger rewards to lower scores. Alternatively, \(R=Y_0-Y_1\), where \(Y_0\) is baseline pain and \(Y_1\) is follow-up pain, measures improvement in pain-score points.
Units matter. A difference of \(0.5\) on a reward scale measured in pain-score points is not the same quantity as a difference of \(0.5\) in a probability or a constructed utility score. State the scale and units every time a reward difference is interpreted. In particular, a numerical gap does not carry a universal clinical meaning unless the reward scale has a defensible clinical interpretation.
Worked Example: Reversing a Lower-Is-Better Outcome
In a hypothetical trial, pain is recorded on a 0–10 scale at baseline and four weeks. A lower four-week score is better. In a target population, suppose the mean four-week scores are 4.2 under arm A and 4.8 under arm B. Define reward as the negative follow-up score, \(R=-Y_1\).
The expected rewards are \(\mu_A=-4.2\) pain-score points and \(\mu_B=-4.8\) pain-score points. Thus,
Arm A has an expected reward higher by 0.6 reward points. Because the reward is the negative of the pain score, this is equivalent to a mean follow-up pain score lower by 0.6 points under A. It does not mean that an individual patient’s pain will fall by 0.6 points, or that 0.6 is necessarily a clinically important difference.
Worked Reward Definitions
Worked Example: A Binary Clinical Objective
Suppose the stated objective is to maximize the chance of being both clinically improved by day 30 and free of a serious adverse event by day 30. For each patient, define reward as 1 if both conditions hold and 0 otherwise. The endpoint and follow-up period are the same in both hypothetical arms.
Among 20 patients assigned to arm A, 12 meet both conditions; among 20 assigned to arm B, 9 do. The sample mean reward is the number meeting the definition divided by the number observed:
On this binary reward scale, the observed average-reward difference is 0.15, or 15 percentage points in the proportion meeting the joint definition. This conclusion concerns the composite reward as defined; it does not separately describe improvement and serious adverse events. The counts are hypothetical descriptive data, not evidence by themselves of a treatment effect.
Worked Example: Rewarding Improvement in a Continuous Outcome
A hypothetical study records baseline and four-week symptom scores on the same 0–10 scale, with lower scores better. Define each patient’s reward as baseline score minus four-week score, so positive values indicate improvement. Among 10 patients in arm A, the sum of the individual improvements is 31 points; among 10 in arm B, it is 24 points.
The average rewards and their difference are
The observed mean improvement is 0.7 symptom-score points greater in A than B. The unit is symptom-score points, not a percentage or a probability. This reward makes improvement the target. If the objective were instead to minimize the final score, that would be a different definition. With different baseline score distributions, average change and average final score need not rank arms the same way, so the choice should follow the clinical objective rather than convenience.
Worked Example: Combining Benefit and Harm With a Utility Score
Suppose a hypothetical clinical team states that its objective is to value symptom improvement while also assigning a cost to a particular adverse event. It proposes \(R=I-4E\), where \(I\) is improvement in symptom-score points and \(E\) equals 1 if the event occurs and 0 otherwise. The coefficient 4 means that, on this chosen scale, the event is assigned a penalty of four symptom-improvement points. This is a value judgment that must be justified, not a fact estimated from the data.
Suppose the average improvement is 3.0 points and the event rate is 0.10 in arm A; in arm B, the corresponding values are 2.5 points and 0.04. By linearity of averages, the expected reward is mean improvement minus four times the event rate:
The observed difference is \(2.60-2.34=0.26\) utility points in favor of A under this specified trade-off. It is not a 0.26-point difference in symptom improvement: the reward incorporates both outcomes. If stakeholders chose a different event penalty, the resulting reward difference—and potentially the preferred arm—could change.
What Transformations Do to Reward Gaps
Changing the reward scale can change the numerical size and interpretation of gaps. If \(R'=c+dR\), where \(c\) is a constant and \(d>0\), then
The constant shift does not change arm rankings; a positive rescaling preserves the ranking but multiplies every gap by \(d\). A negative rescaling reverses the ranking, which is why outcome direction must be checked. A nonlinear transformation is more consequential: it can change the ranking of expected rewards, even if it preserves the ordering of individual outcome values.
For a short justification, consider two hypothetical arms with a beneficial outcome measured in points. Arm A gives 0 or 10 points with equal probability, while arm B always gives 4.5 points. On the raw point scale, the means are 5 and 4.5, so A has the larger expected outcome. If the objective instead defines reward as the square root of the outcome, then the expected rewards are
The ranking reverses under that nonlinear reward. The transformation values outcome levels differently and therefore changes the objective; it is not merely a new unit of measurement. This is why one should define the clinical meaning of the reward before considering how its numerical scale affects learning or allocation.
Choosing and Checking a Definition
Three common approaches are direct endpoint rewards, threshold-based rewards, and utility-based rewards. A direct reward uses a measurement such as follow-up symptom score (with its direction handled correctly) or change from baseline. It preserves detail and has familiar units, but can emphasize small numerical changes even when clinical meaning is uncertain. A threshold reward records whether a prespecified event or response criterion is met. It is easy to interpret as a proportion, but discards distinctions among patients on the same side of the threshold and can be sensitive to the cutoff. A utility reward combines outcomes using explicit weights, which can reflect a stated trade-off but may be difficult to justify and explain.
| Approach | Reward interpretation | Key check |
|---|---|---|
| Direct endpoint | Mean outcome or change in its measurement units | Does the outcome direction match “larger is better”? |
| Threshold-based | Proportion meeting a defined criterion | Is the cutoff clinically justified and applied consistently? |
| Utility-based | Mean score under specified outcome weights | Are the weights and trade-offs explicit and defensible? |
Before using a reward, check that its definition answers the same question for every arm, that its direction is unambiguous, and that its units and range are known. If a reward combines outcomes, verify that every term is defined for each patient and that the weights reflect the intended priorities. Check also whether outcomes are missing, measured at different times, or recorded differently across arms: a reward that cannot be computed comparably does not provide a sound basis for comparing expected rewards. If the reward is intended to be bounded, verify that every possible value lies within the stated bounds; do not assume this just because observed values happen to be bounded.
A reward is an objective, not a complete account of clinical judgment. A single average can conceal variation in who benefits, and a composite score can conceal which component drives its value. Also, a large numerical reward does not guarantee that every clinically important concern has been handled. The reward definition should be reported in enough detail that another reader can calculate it from patient outcomes and understand what a reward difference represents.
Common Mistake
A frequent error is to feed a lower-is-better endpoint directly into a policy that maximizes reward and then interpret the arm with the larger average as better. For example, if reward is the raw pain score, a mean of 5.0 would be ranked above a mean of 4.0 even though lower pain is preferred. The correction is to align the reward direction with the objective—such as using the negative score or a defined improvement measure—and then state the resulting units. Another error is to treat a weighted composite as clinically self-evident. Its ranking follows from the weights; those weights and the implied trade-offs must be made explicit.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions below.
- A lower laboratory value is clinically preferred. If the reward is the negative of that value, what does a reward difference of 0.3 mean, and in what units?
- A binary reward is 1 for meeting a defined response criterion and 0 otherwise. In hypothetical data, 18 of 30 patients in A and 15 of 30 in B meet it. Calculate and interpret the observed average-reward difference.
- Why does multiplying all rewards by a positive constant preserve the arm ranking, while a nonlinear transformation may change it?
- A composite reward subtracts three points for an adverse event from a symptom-improvement score. What must be explained before interpreting its mean difference?
Solutions. (1) The arm with the higher expected reward has a laboratory value lower by 0.3 units on average, because reward is the negative value. (2) The averages are \(18/30=0.60\) and \(15/30=0.50\); the difference is \(0.60-0.50=0.10\), or 10 percentage points in the proportion meeting the criterion. (3) A positive multiplier scales each expected reward and each gap by the same positive amount, so it cannot reverse their order. A nonlinear transformation changes the relative contributions of different outcome values to the expectation and can therefore change arm rankings. (4) State the event definition and follow-up period, the symptom-improvement scale and direction, and the reason for the three-point penalty. The resulting difference is in composite reward points, not symptom points alone.