Cumulative Reward as a Patient-Benefit Criterion

Why Sum Reward Across Enrolled Patients?

An adaptive trial assigns treatments sequentially, learning from outcomes as enrollment proceeds. One way to describe the trial’s patient-facing objective is to ask how much modeled reward accrues across all enrolled patients. This is a total over the trial horizon, not a statement about what happened to any one patient and not a guarantee that an adaptive policy benefits every participant.

The meaning of “benefit” depends on how the outcome is coded. If reward is a favorable binary outcome, such as being alive and free of a specified event at 30 days, the total expected reward is interpretable as the expected number of favorable outcomes. If reward is a clinical utility score, the total is in units of that score. Reward must be defined before calculating the objective, with higher values consistently representing more desirable outcomes.

Definition: Consider a trial enrolling $T$ patients under allocation policy $\pi$. Let $A_t$ be the treatment assigned to patient $t$, and let $Y_t$ be that patient’s observed reward under the assigned treatment. The cumulative reward is $\sum_{t=1}^{T}Y_t$. The expected cumulative reward under policy $\pi$ is $$ \mathcal{R}_T(\pi)=E_{\pi}\left[\sum_{t=1}^{T}Y_t\right]. $$ The expectation averages over the trial’s treatment assignments and outcomes under the specified policy and reward model. It is a total expected reward across the $T$ enrolled patients.

The subscript $T$ makes the horizon explicit: changing the planned number of patients can change the total even if the allocation policy and outcome model stay the same. The policy matters too. Two policies applied to the same reward model may yield different assignment probabilities over time, and hence different expected totals.

For this tutorial, write $K$ for the number of treatment arms, $A_t\in\{1,\ldots,K\}$ for the assignment at enrollment $t$, and $H_{t-1}$ for the information available just before that assignment. The policy specifies $P(A_t=k\mid H_{t-1})$. Because only the outcome under the assigned arm is observed, $Y_t$ is the observed reward, not a record of all possible outcomes for patient $t$.

Definition, Assumptions, and a Useful Formula

The definition of expected cumulative reward requires the expectations to exist; for example, it is enough that every $Y_t$ has a finite expected value. To interpret the total as patient benefit, additional care is needed. The reward should represent a clinically meaningful outcome or utility, the direction of preference should be clear, and its units should be stated. If rewards are meant to represent the consequences of assigning treatments, consistency and an appropriate definition of each patient’s potential outcome are also needed. In particular, the potential outcome under the assigned treatment must equal the observed outcome, and one patient’s assignment should not change another patient’s outcome if outcomes are being interpreted as individual treatment effects.

A convenient model assumes that, for each arm $k$, the conditional expected reward is a fixed arm-specific mean $\mu_k$: given the assignment to arm $k$ and the available history, $E(Y_t\mid A_t=k,H_{t-1})=\mu_k$. This assumption allows outcomes to be random; it says their conditional means do not change with enrollment time or the observed history. Define the marginal assignment probability $q_{t,k}=P_{\pi}(A_t=k)$, averaging over possible histories before patient $t$.

Result: Under finite expected rewards and the fixed conditional-mean model above, $$ \mathcal{R}_T(\pi)=\sum_{t=1}^{T}\sum_{k=1}^{K}q_{t,k}\mu_k. $$ Here $q_{t,k}$ is the probability that patient $t$ is assigned to arm $k$, and $\mu_k$ is the expected reward for a patient assigned to arm $k$.

This equation does not require the assignments to be independent across patients. An adaptive policy generally makes later assignments depend on earlier outcomes. The equation uses the marginal assignment probabilities for each enrollment time, after averaging over the histories that the policy could encounter.

Why the Formula Holds

The justification is linearity of expectation and conditioning on the assigned arm. It does not require observing the outcomes that patients would have had under unassigned treatments. For each patient, the possible assignments partition the event space, so the expected reward is the sum of the arm-specific conditional means weighted by the probabilities of those assignments.

\begin{aligned} E_{\pi}\left[\sum_{t=1}^{T}Y_t\right] &=\sum_{t=1}^{T}E_{\pi}[Y_t]\\ &=\sum_{t=1}^{T}\sum_{k=1}^{K} P_{\pi}(A_t=k)\,E_{\pi}[Y_t\mid A_t=k]\\ &=\sum_{t=1}^{T}\sum_{k=1}^{K}q_{t,k}\mu_k. \end{aligned}

The last equality uses the fixed conditional-mean assumption. If expected reward depends on time or history, replace $\mu_k$ with the appropriate conditional mean and average over possible histories. It would then generally be incorrect to substitute one fixed $\mu_k$ for every patient assigned to arm $k$.

Worked Example: Fixed Allocation and Expected Favorable Outcomes

Suppose a hypothetical 100-patient trial assigns 60 patients to arm 1 and 40 to arm 2 according to a fixed schedule. The binary reward is 1 for a favorable 30-day outcome and 0 otherwise. Assume the modeled favorable-outcome probabilities are $\mu_1=0.70$ and $\mu_2=0.50$, and that these conditional means apply to every assigned patient.

The total reward is the number of favorable outcomes, so its expectation is $$ \mathcal{R}_{100}=60(0.70)+40(0.50)=42+20=62. $$

The expected cumulative reward is 62 favorable outcomes across the 100 patients under this allocation and reward model. It need not be an integer in other examples, and it does not assert that exactly 62 favorable outcomes will occur. The model also does not imply that every patient assigned to arm 1 will have a favorable outcome.

Adaptive Assignment Changes the Expected Total

Under adaptation, the assignment probability for a patient can depend on earlier observed outcomes. The formula still applies when the fixed conditional-mean assumption is appropriate, but the marginal probabilities $q_{t,k}$ must reflect the policy’s behavior over all possible histories. Knowing the policy’s rule for a particular history is not, by itself, the same as knowing the marginal probability of assignment at that enrollment.

Worked Example: A Two-Patient Adaptive Rule

Consider a hypothetical two-patient trial with arms A and B and binary reward. The first patient is assigned to A. The modeled success probabilities are 0.60 for A and 0.80 for B, constant across patients. The second patient receives B if the first patient succeeds and A if the first patient does not succeed. Assume the first patient’s outcome follows the stated probability under A.

The first patient’s expected reward is $0.60$. The second patient receives B with probability $0.60$, and receives A with probability $0.40$. Therefore the second patient’s expected reward is $$ (0.60)(0.80)+(0.40)(0.60)=0.48+0.24=0.72. $$

Adding the two expectations gives $$ \mathcal{R}_2=0.60+0.72=1.32. $$

Equivalently, the four outcome paths give the same result: first-patient success followed by a success on B has probability $(0.60)(0.80)=0.48$ and yields total reward 2; first-patient success followed by failure on B has probability $(0.60)(0.20)=0.12$ and yields total 1; first-patient failure followed by success on A has probability $(0.40)(0.60)=0.24$ and yields total 1; and two failures have probability $(0.40)(0.40)=0.16$ and yield total 0. Thus the expectation is $(0.48)(2)+(0.12)(1)+(0.24)(1)+(0.16)(0)=1.32$. This is a policy-specific expected total of 1.32 successes, not a promise about the realized total.

Choosing the Reward Scale

A total only has a useful clinical interpretation if its reward scale is meaningful. A binary reward can be easy to explain, but it may omit important differences among outcomes. A utility score can incorporate multiple outcome dimensions, provided its definition and units are justified. A reward can also be defined as an incremental outcome relative to a reference, in which case the total is an expected sum of increments—not the expected number of favorable outcomes.

If every patient’s reward is multiplied by a positive constant, the expected cumulative reward is multiplied by the same constant. If a constant $c$ is added to every patient’s reward, the total increases by $Tc$. Such transformations can change the numerical scale without changing which policy has the greater expected total, but adding a constant changes the reported total and its interpretation. Consequently, reports should state the reward definition, direction, and units rather than presenting a bare total.

Worked Example: Total Reward in Utility Units

Suppose a hypothetical 50-patient trial uses a prespecified utility reward measured in utility points per patient, with higher values representing better outcomes. The modeled mean rewards are 8 points per patient for arm 1 and 5 points per patient for arm 2. During an initial group of 20 patients, the policy assigns each patient to arm 1 with probability 0.50. During the remaining 30 patients, its marginal probability of assigning to arm 1 is 0.70. Assume these conditional means apply throughout the trial.

The initial group contributes an expected $$ 20\{(0.50)(8)+(0.50)(5)\} =20(4+2.5)=130 $$ utility points. The remaining patients contribute an expected $$ 30\{(0.70)(8)+(0.30)(5)\} =30(5.6+1.5)=213 $$ utility points. Thus $$ \mathcal{R}_{50}=130+213=343 $$ utility points across the enrolled patients.

The value 343 is a total in the chosen utility scale; it is not 343 utility points for each patient. Dividing by the horizon gives an average expected reward of $343/50=6.86$ utility points per patient. That average can aid comparisons across trials with different horizons, but it is a different summary from the total.

Interpreting and Checking the Criterion

Expected cumulative reward summarizes modeled patient outcomes under a particular policy, horizon, and reward definition. It is useful when the question concerns the aggregate expected experience of patients enrolled in the trial. It differs from a goal focused on learning which treatment is best: a policy could allocate efficiently to arms with high modeled reward while providing less information about other arms. Conversely, an allocation designed to maintain comparison or learn about uncertain treatments may not maximize the expected reward accrued during the trial itself. These are distinct objectives, and a single criterion does not automatically optimize all of them.

Before using the total, check the following:

Outcome direction and scale: verify that larger reward really means a more desirable clinical outcome and that all arms use the same scale.
Horizon and enrolled population: count the patients included in $T$. A total for 100 patients should not be compared directly with a total for 200 without accounting for the difference in horizon.
Assignment probabilities: for each $t$, confirm that the marginal probabilities across arms sum to 1. For example, $\sum_{k=1}^{K}q_{t,k}=1$.
Outcome model: assess whether arm-specific expected rewards are plausibly stable across time and relevant patient histories. Changes in case mix, treatment delivery, or outcome ascertainment can undermine a constant-$\mu_k$ calculation.
Missing outcomes: specify how missing rewards are handled. Dropping patients with unavailable outcomes can make the reported total represent a selected subset rather than the intended enrolled population.

A common mistake is to multiply each arm’s mean reward by its expected number of assignments and then describe the result as the realized benefit. The calculation gives an expectation under a model, not the trial’s observed outcome. Another mistake is to use a favorable-outcome probability as a reward when the actual reward is a utility score with different units. In the first case, the conclusion overstates what an expectation says; in the second, the arithmetic may be valid but the interpretation is wrong.

A further limitation is that an aggregate can conceal how benefits and harms are distributed. The same total might arise from different patterns across patients or treatment arms. Expected cumulative reward alone does not report uncertainty, subgroup outcomes, safety constraints, or the quality of treatment-effect estimates. Those considerations require additional summaries or design criteria.

Key takeaway: Expected cumulative reward is the expected sum of the rewards actually accrued by enrolled patients under a specified allocation policy. It is interpretable as total modeled patient benefit only when the reward, horizon, outcome model, and units are clearly defined.

Check Your Understanding

Answer each question using the stated reward model. Solutions follow the questions.

A hypothetical 40-patient fixed-allocation trial assigns 25 patients to an arm with mean binary reward 0.60 and 15 to an arm with mean reward 0.40. What is the expected cumulative reward, and what does it mean?
Why can an adaptive policy’s expected cumulative reward still be calculated by summing $q_{t,k}\mu_k$ when assignments depend on previous outcomes?
A researcher reports an expected total of 18 utility points from a 30-patient trial. What is the average expected reward per patient, and what information is needed to interpret the total clinically?
Suppose the mean reward on an arm changes with patient enrollment time, but the analyst uses one constant arm mean for all patients. What assumption is at risk, and what should the calculation use instead?
Explain why expected cumulative reward should not be described as the exact number of favorable outcomes observed in the trial.

Solutions. 1. The calculation is $25(0.60)+15(0.40)=15+6=21$. The expected total is 21 favorable outcomes across the 40 enrolled patients under the stated model; the realized number may differ. 2. Linearity of expectation still holds, and conditioning on the assigned arm gives the arm-specific expected reward. The probabilities $q_{t,k}$ are marginal probabilities averaged over the possible histories, so they account for adaptation without requiring independent assignments. 3. The average is $18/30=0.600$ utility points per patient. Clinical interpretation also requires the definition and direction of the utility scale, its units, the trial horizon and patient population, and the model under which the expected total was calculated. 4. The fixed conditional-mean assumption is at risk. Use a time- or history-specific conditional mean and average it over the relevant assignments and histories instead of applying one constant arm mean. 5. An expectation is an average over the outcome and assignment randomness described by the model. The observed total is random and can be above or below that expectation.