Treatment Arms as Actions and Outcomes as Rewards

From a Treatment Choice to a Reward

A bandit formulation starts by representing each available treatment choice as an action and each participant’s observed response as a numerical reward. This translation is not merely a change of labels. It determines what the allocation problem treats as desirable. If a larger reward is meant to represent a more favorable patient outcome, the outcome definition and its numerical coding must make that interpretation true for every arm.

For a clinical trial, an action is usually a treatment arm, but it should be defined precisely enough to identify the intervention: for example, a specified drug and dose with a stated schedule, rather than an ambiguous label such as “active treatment.” The reward is computed from an outcome measured for a participant assigned to that action. Choosing a reward does not, by itself, specify an allocation rule or a statistical model for outcomes. It specifies what numerical information an allocation rule will receive.

The mapping \(g\) may use the outcome alone or may depend on the assigned action—for example, if the arms use different dose-specific rules for calculating a score. Whatever the mapping, it should be defined in a way that gives rewards a consistent interpretation across actions. The outcome itself should also be stated precisely: what is measured, in what units, and at what assessment time. “Response” is not a sufficient specification if it could mean either symptom improvement at four weeks or survival at one year.

Conditions for a Meaningful Mapping

Before rewards are used, several practical and conceptual conditions should be checked. These are conditions for interpreting the resulting numbers, not a claim that one particular reward is right for every clinical question.

• Actions are well specified. Each arm identifies a treatment strategy that can be assigned and distinguished from the other arms. If a treatment label hides meaningful variation in dose, schedule, or accompanying care, then the action is not yet defined clearly enough.
• The outcome and observation window are fixed. Each participant’s reward must refer to the same prespecified outcome concept and assessment period. Otherwise, differences in reward may reflect different measurement rules rather than different clinical results.
• The direction is consistent. Larger reward values must always mean more favorable outcomes. If the recorded variable increases with harm, its coding must be reversed or otherwise transformed before it is used as a reward.
• The numerical scale expresses the intended trade-off. When several outcomes enter one reward, their weights and any thresholds should reflect a stated decision objective, rather than being chosen after seeing which arm looks best.
• Rewards can be calculated reliably. The data collection plan must make the inputs and the reward calculation available for each assigned participant. A rule for an outcome that is unmeasured or ambiguously recorded cannot produce a well-defined reward without additional decisions.

A convenient summary for an action is its mean reward in a specified patient population and assessment setting. Write \(\mu_a=E(R_t\mid A_t=a)\) when this expectation is well defined. This notation describes a population average on the chosen reward scale; it does not make the scale clinically meaningful by itself. Comparing \(\mu_a\) across actions answers a useful question only if the same reward definition and target context apply to all arms.

Rewards for Continuous and Multiple Outcomes

When an outcome is continuous, the reward can often be the outcome itself or a clearly defined transformation. The direction needs particular attention. For example, a higher symptom-severity score may be worse, even though a bandit reward is conventionally oriented so higher is better. A transformation such as baseline severity minus severity at a fixed follow-up time yields a positive value for improvement, provided that larger severity scores mean worse symptoms.

Some clinical decisions involve more than one outcome. A scalar reward can combine them, but doing so requires an explicit rule for trading one outcome against another. A weighted sum is one transparent option, not a neutral default. Its weights determine how much one unit of an outcome counts relative to a unit of another outcome. If the scales or weights change, the clinical objective may change too.

Why Scale Choices Matter

A useful property of a common positive affine transformation is that it preserves the ordering of arm mean rewards. Suppose a reward \(R_t\) is replaced for every action by \(R'_t=c+dR_t\), where \(c\) is a constant and \(d>0\). Then

Because \(d\) is positive, if \(\mu_a>\mu_b\), then \(c+d\mu_a>c+d\mu_b\). Thus, changing all rewards to a common scale by adding the same constant and multiplying by a positive constant does not reverse the ranking of mean rewards. This result concerns the ordering of means; it does not mean all other features of a reward-based decision procedure are unaffected by rescaling.

If \(d<0\), the ordering reverses: the transformation turns a larger original value into a smaller transformed value. If the transformation is nonlinear, even the ordering of mean rewards can change. For instance, an average of transformed values need not equal the transformation of the average. The derivation is simply

in general for a nonlinear function \(h\). Therefore, a reward transformation is not just cosmetic unless its effect on the objective has been checked.

This is also why composite rewards should be documented in their own units. A weighted score may be reported in utility units rather than clinical units, and the weights must be interpretable as part of the decision rule. If the intended purpose is instead to keep several outcomes distinct, reducing them to one number may hide important trade-offs. The appropriate choice depends on the clinical decision being represented.

Diagnostics, Alternatives, and Limitations

Several checks can expose a reward definition that is going wrong before it is used:

• List plausible outcome values and calculate their rewards by hand. Confirm that better clinical outcomes receive larger rewards and that boundary cases have sensible values.
• Check the reward calculation with the units written explicitly. Adding an unscaled laboratory value to a binary event indicator, for example, has no clear meaning without a stated conversion or weighting rule.
• Verify that the outcome window, thresholds, and scoring rule are the same for every arm. Arm-specific definitions can make the numerical comparison misleading.
• Review missing or delayed outcome cases. If the mapping cannot be applied, do not silently treat an unobserved outcome as zero; zero is a substantive reward value, not a generic code for missingness.
• Ask whether reasonable alternative weights would change the preferred arm. A change in ranking signals that the decision depends materially on the chosen trade-off.

There are three broad choices to distinguish. Using a single raw endpoint keeps interpretation close to its clinical units but may omit other important benefits or harms. Using a prespecified utility score makes a trade-off explicit and yields one reward, but the result depends on the score’s weights and assumptions. Keeping outcomes separate avoids concealing their individual patterns, but does not provide a single scalar reward without an additional decision rule. None is universally best; the choice should match the clinical question.

A reward definition also has limits. It cannot repair poorly measured outcomes, remove bias in the trial, establish that an observed arm difference is caused by treatment, or guarantee that a scalar score represents every patient’s preferences. Nor does a reward definition specify how participants should be allocated. Its role is narrower and essential: it makes explicit which numerical consequence of an assigned treatment is being treated as favorable.

Common Mistake: Treating the Recorded Outcome as the Reward

A common error is to copy a recorded value directly into the reward field without checking its direction. For example, if an adverse-event count is coded as \(0,1,2,\ldots\), a larger raw count means more harm. Calling that count a reward would favor arms with more adverse events, the opposite of the intended clinical objective. The correction is to define and verify a directionally consistent mapping, such as a negative event count, or a prespecified utility rule that includes the event appropriately. The mapping should be checked with concrete cases before it is used.