Discounted and Average-Reward Objectives

Why the Reward Objective Matters

A sequential treatment-allocation policy assigns each arriving patient using information available at that time. The policy affects not only which treatment each patient receives, but also the sequence of rewards generated across patients. A policy can give more reward early and less later, or sacrifice early reward to improve later outcomes. To compare such policies, we must specify how rewards across time are combined.

Three common summaries answer different questions: the total reward over a fixed number of patients, a discounted total that gives earlier rewards greater weight, and the long-run average reward per patient. These are not interchangeable descriptions of a policy. The appropriate objective depends on the clinical and operational question, including whether the trial has a fixed endpoint and how outcomes at different times should count.

Allocation and Reward Notation

Suppose there are \(K\) available treatments. At patient \(t\), let \(A_t\) denote the assigned treatment and \(R_t\) the resulting reward, defined so that larger values represent more desirable outcomes. Let \(H_{t-1}\) denote the information available before assigning patient \(t\). A randomized policy may choose treatment \(a\) with probability \(P(A_t=a\mid H_{t-1})\). The policy is adaptive when these probabilities can depend on the observed history.

For the basic comparisons here, assume rewards have finite means and are bounded in absolute value by a constant \(M\): \(|R_t|\leq M\). This condition guarantees that the discounted infinite sum below is finite when the discount factor is less than one. Examples use specified expected rewards; their arithmetic concerns those expectations, not claims that every patient's observed outcome equals its mean.

A finite-horizon objective explicitly depends on the chosen endpoint \(T\). A discounted objective assigns weight \(1\) to the first reward, \(\gamma\) to the second, and \(\gamma^{t-1}\) to the reward at time \(t\). When \(\gamma=0\), only the first reward contributes. As \(\gamma\) approaches one, later rewards receive more weight. The average-reward objective instead asks about the asymptotic expected reward rate; it does not give a lasting advantage to a fixed number of unusually good early outcomes.

Discounted Reward: Meaning and Convergence

The discount factor is a modeling choice, not an estimated treatment effect. It can represent a decision-maker's preference for earlier benefit, limited relevance of distant outcomes, or an approximation to a finite planning horizon. In a clinical setting, discounting patient rewards requires a substantive justification: it should not be introduced merely because a mathematical objective is convenient.

Under the bound \(|R_t|\leq M\), the absolute value of the discounted tail after period \(L\) is at most \(M\sum_{t=L+1}^{\infty}\gamma^{t-1}\). For \(0\leq\gamma<1\), this geometric series yields the following useful justification.

Thus, the influence of sufficiently distant rewards can be bounded, and the infinite discounted sum converges. The bound also helps assess whether a finite calculation is an adequate approximation: it gives a conservative maximum omitted tail, not the actual tail value. If rewards are not bounded, convergence instead requires suitable conditions on their magnitudes; the bounded-reward assumption avoids that complication here.

Sometimes the discounted total is reported in normalized form, \((1-\gamma)J_\gamma(\pi)\). This rescales the objective and can make its magnitude comparable to a reward per period, especially when rewards are stable over time. For a fixed \(\gamma\), multiplying every policy's objective by the same positive constant does not change their ranking. The normalized quantity is still a discounted summary, not automatically the same as the long-run average reward.

Long-Run Average Reward

The average-reward objective is appropriate when the main question concerns performance per patient over a continuing or very long allocation process. Its definition uses a limit inferior so that it remains defined even when the sequence of expected averages does not converge. If the ordinary limit exists, the limit inferior equals that limit. For a finite trial, however, the long-run objective is a limiting criterion, not a substitute for calculating the actual expected total over the trial's planned enrollment.

A simple special case clarifies the interpretation. If a policy has the same expected reward \(c\) for every patient, then \(E_\pi[S_T^\pi]=Tc\), and therefore \(g(\pi)=\lim_{T\to\infty}Tc/T=c\). More generally, the average depends on the whole expected reward sequence, but a finite early advantage is divided by an increasingly large \(T\) and vanishes in the long-run rate.

Comparing the Objectives

The following table summarizes the interpretation of each objective. The word “best” is always relative to the chosen reward definition, horizon, and policy set.

A comparison should use the same reward scale and a clearly stated time horizon or discount factor for all candidate policies. If one policy's expected reward sequence is at least as large as another's at every time point, all three objectives rank it at least as highly, provided the objectives are well-defined. When the sequences cross, rankings can differ, as in the preceding example.

Assumptions, Diagnostics, and Common Mistakes

The definitions themselves require a specified reward and time index. The convergence argument for an infinite discounted total additionally used bounded rewards and \(0\leq\gamma<1\). Interpreting the average as a stable long-run rate is clearest when the process has a continuing meaning and its expected averages settle; the limit-inferior definition does not require that they settle, but a single rate may then conceal important variation.

• Check the reward definition. Confirm that larger values genuinely mean better outcomes and that units are consistent across patients and treatments. If rewards combine several outcomes, the combination determines what is being optimized.
• Check the time weighting. State the finite horizon or \(\gamma\), and explain why it represents the decision. A large change in the ranking under plausible choices of \(\gamma\) is a warning that conclusions depend on the weighting convention.
• Check the tail. For an infinite discounted calculation, use a bound such as \(M\gamma^L/(1-\gamma)\) when rewards are bounded. Do not silently treat a truncated sum as the full discounted objective.
• Check the time scale. An average over an indefinitely long process may be a poor description of a trial expected to stop after a fixed number of patients. Conversely, a short fixed horizon may fail to capture sustained performance.

A common mistake is to say, “The policy with the highest average reward must be best, so it should also be preferred for this trial.” That reasoning confuses a limiting per-patient rate with a finite-horizon total. In the transient-advantage example, P has the higher three-patient total, but Q has the higher long-run average. The correction is to identify the decision question first, compute the matching objective, and report the horizon or discounting used.

These objectives evaluate reward, not every feature of a clinical trial. A policy with a high expected reward under the chosen definition need not be acceptable if it violates safety or allocation constraints, and a reward objective alone does not describe the quality of treatment-effect estimation. Those considerations require explicit criteria alongside, or instead of, reward maximization.