Stochastic Reward Models for Trial Outcomes

Why Specify a Reward Distribution?

In a bandit formulation, a treatment arm is an action and the outcome observed after assigning that action is its reward. To describe the stochastic part of this setup, we need more than a list of possible outcomes: we need a probability model for the reward associated with each arm. That model states what variation is expected among participants assigned to the same arm and which features of the outcome are being represented.

Let \(t=1,\ldots,T\) index participants in the order they are assigned, let \(A_t\) denote the arm assigned to participant \(t\), and let \(Y_t\) be the numerical reward recorded for that participant. Suppose there are \(K\) arms, indexed by \(k=1,\ldots,K\). A stationary reward model assigns a fixed distribution \(F_k\) to the reward whenever arm \(k\) is assigned. In the simplest version, conditional on the chosen arm, each new reward is generated from that arm’s distribution independently of the previous rewards.

Definition: A stationary arm-specific reward model specifies a time-invariant probability distribution \(F_k\) for the reward from each arm \(k\). Under the basic model, \(Y_t\mid A_t=k\sim F_k\), and the reward at time \(t\), given the assigned arm, is independent of the earlier observed rewards. Thus the distribution for arm \(k\) does not change with participant index \(t\) or the observed history.

“Stationary” means the outcome distribution associated with an arm stays the same over the modeled trial period. It does not mean all arms have the same distribution: \(F_1\) and \(F_2\), for example, can have different means, variances, or shapes. Nor does stationarity mean that the sequence of observed rewards must look constant. Random samples from one fixed distribution will vary.

Assumptions and Scope

The model is useful only when its assumptions are plausible for the outcome and trial population. First, the reward must be defined and coded consistently. If larger values are meant to represent greater benefit, the coding must actually have that direction for every participant and arm. Second, the arm-specific distribution is assumed stable over time. Changes in eligibility, clinical practice, outcome measurement, or participant mix can make that assumption doubtful. Third, the basic model treats outcomes from different participants as conditionally independent once their assigned arms are specified; shared clusters, repeated measurements, or other dependence require a model that represents that dependence.

The basic model also represents each arm by one distribution rather than by participant-specific distributions. This can be an inadequate simplification if outcomes vary systematically with baseline risk or other participant characteristics. In such cases, the distribution of participants enrolled at a particular time may matter: a changing mix of participants can change the observed reward distribution even if treatment response within every subgroup is stable. These assumptions concern the outcome model; adaptive assignment by itself does not make a reward model stationary or nonstationary.

A reward may be a directly recorded clinical outcome or a numerical utility derived from one. For example, a binary indicator of improvement can be coded as 1 for improvement and 0 otherwise. A continuous symptom change can be coded so that positive values represent improvement. The coding defines what “reward” means; it does not by itself establish that the chosen probability distribution is appropriate.

Binary Rewards: Bernoulli Models

For a binary outcome, code the event of interest as 1 and its absence as 0. For arm \(k\), a Bernoulli model has one parameter \(p_k\), the probability of reward 1. This parameter is fixed but usually unknown in an application. Its interpretation depends on the event and coding chosen.

Definition: A binary stationary reward model specifies \(Y_t\mid A_t=k\sim\operatorname{Bernoulli}(p_k)\), where \(0\leq p_k\leq1\). Thus \(P(Y_t=1\mid A_t=k)=p_k\), and \(P(Y_t=0\mid A_t=k)=1-p_k\).

For a Bernoulli variable, its mean and variance follow directly from its two possible values. Because \(Y_t^2=Y_t\) when \(Y_t\) is either 0 or 1, the variance is \(E(Y_t^2)-[E(Y_t)]^2=p_k-p_k^2\).

E(Y_t\mid A_t=k)=0(1-p_k)+1p_k=p_k,\qquad \operatorname{Var}(Y_t\mid A_t=k)=p_k-p_k^2=p_k(1-p_k).

If \(n_k\) participants receive arm \(k\) and \(S_k\) have reward 1, the likelihood for those binary observations is proportional to \(p_k^{S_k}(1-p_k)^{n_k-S_k}\). The observed proportion \(S_k/n_k\) is a sample summary and, under this model, the maximum-likelihood estimate of \(p_k\). It is not automatically the model parameter itself.

Worked Example: Binary Improvement in One Arm

Suppose, hypothetically, that 10 participants assigned to arm A have improvement indicators \(1,1,0,1,0,1,1,0,1,1\). Assume the basic stationary Bernoulli model for this example and treat these observations as independent conditional on assignment to A. There are seven 1s, so \(n_A=10\) and \(S_A=7\).

The likelihood as a function of the arm’s success probability is \(\mathcal{L}(p_A)=p_A^7(1-p_A)^3\). The observed success proportion is \(\hat{p}_A=7/10=0.700\), which is the maximum-likelihood estimate under this model. The observed data do not establish that the underlying model parameter equals 0.700.

For an illustration of the model’s implications, suppose we additionally specify \(p_A=0.700\). Then the model says a new participant assigned to A has probability 0.700 of an improvement coded as 1, and the reward variance is \(p_A(1-p_A)=0.700(0.300)=0.210\). The parameter value in this last calculation is an explicit illustrative assumption; using the observed proportion instead would produce a plug-in estimate of the variance, not a known model variance.

Continuous Rewards: Normal and General Models

A continuous reward can be represented by an arm-specific distribution with a mean and, when relevant, a variance. A common working model is normal: \(Y_t\mid A_t=k\sim N(\mu_k,\sigma_k^2)\), where \(\mu_k\) is the arm-specific mean reward and \(\sigma_k^2>0\) is its variance. The normal model allows a convenient description of location and spread, but it assumes a symmetric, unbounded distribution. It may be a poor description for strongly skewed outcomes, values with a strict bound, or data with substantial outliers.

Definition: Under an arm-specific normal reward model, \(Y_t\mid A_t=k\sim N(\mu_k,\sigma_k^2)\), with fixed parameters \(\mu_k\) and \(\sigma_k^2\) for arm \(k\). The mean \(\mu_k\) is the expected reward in the outcome’s units, and \(\sigma_k\) is the standard deviation in those same units.

A continuous outcome need not be normally distributed to have a stationary reward model. More generally, \(F_k\) may be any fixed distribution appropriate to the outcome, such as a bounded, skewed, or heavy-tailed distribution. Choosing a normal model is a substantive assumption, not a consequence of the outcome being measured on a numerical scale. The empirical mean \(\bar{x}_k\) and sample standard deviation \(s_k\) summarize observed data; they are not known values of \(\mu_k\) and \(\sigma_k\).

Worked Example: A Normal Model With an Explicit Mean Assumption

Suppose, hypothetically, that five observed scores in arm B are \(4.8, 5.2, 5.0, 4.6,\) and \(5.4\) points. Their sum is \(25.0\), so the sample mean is \(\bar{x}_B=25.0/5=5.00\) points. The sum of squared deviations from 5.00 is \(0.04+0.04+0+0.16+0.16=0.400\), giving sample variance \(s_B^2=0.400/(5-1)=0.100\) square points and sample standard deviation \(s_B=0.316\) points.

Now consider a separate, explicitly specified model assumption for illustration: \(Y_t\mid A_t=B\sim N(5.00,0.60^2)\). Under that assumed model, the probability that a new participant assigned to B has a score above 5.5 points is

P(Y_t>5.5\mid A_t=B) =1-\Phi\!\left(\frac{5.5-5.00}{0.60}\right) =1-\Phi(0.8333) \approx 0.2023.

This is a model-based probability because both the mean of 5.00 points and the standard deviation of 0.60 points were specified as model parameters for the calculation. The observed \(\bar{x}_B=5.00\) happens to equal the assumed mean here, but the sample mean alone does not establish \(\mu_B=5.00\). The assumed standard deviation also differs from the sample standard deviation of 0.316 points; neither value should be substituted for the other without stating a model or estimation choice.

Choosing the Reward Representation

The measurement and scientific question should guide the reward model. A binary model represents the probability of a specified event; it discards distinctions among outcomes within each category. A continuous model can retain the magnitude of an outcome, but its distributional assumptions must suit the measurement scale. The same clinical measurements can yield different rewards if they are coded differently, and those choices change the question being represented.

Worked Example: Continuous Change or Binary Threshold

In a hypothetical arm, five participants have symptom-improvement scores of \(2,4,4,6,\) and \(9\) points, with larger scores meaning greater improvement. If the reward is the continuous score, the observed mean is \(\bar{x}=(2+4+4+6+9)/5=25/5=5.00\) points. The squared deviations from 5 are \(9,1,1,1,\) and \(16\), so \(s^2=28/(5-1)=7.00\) square points and \(s=\sqrt{7}=2.646\) points.

If the question instead defines success as an improvement of at least 5 points, the coded rewards are \(0,0,0,1,1\). The observed success proportion is \(\hat{p}=2/5=0.400\). The continuous and binary representations answer different questions: the first summarizes average improvement in points, while the second summarizes the proportion meeting the specified threshold. A Bernoulli model for this threshold assumes a fixed success probability for the arm; a normal model for the original score would separately require a plausible normal distribution for the continuous scores. Neither model choice follows merely from the five observations.

Reward representation	What its parameters describe	Key modeling concern
Bernoulli	Probability of a specifically coded event	Event definition and consistent 0/1 coding
Normal continuous	Mean and variance of a continuous reward	Symmetry, tails, and plausibility of unbounded values
Other fixed distribution	Features of the arm-specific distribution suited to the outcome	Whether its shape and assumptions match the measurement

Diagnostics, Failure Modes, and Limits

Observed data cannot prove stationarity, but they can reveal problems. Plot outcomes against enrollment order and compare early and late summaries; large systematic shifts may indicate drift or a changing participant mix. Check that binary coding is valid and that the event is defined identically throughout the trial. For continuous rewards, inspect the scale, range, skewness, and unusually influential observations before relying on a normal model. Repeated or clustered measurements also call the basic conditional-independence assumption into question.

These checks are diagnostic rather than definitive. Small samples can hide meaningful drift, and apparent differences over time can arise by chance. A single arm-specific distribution can also obscure important subgroup patterns. If diagnostics indicate time trends, participant heterogeneity, or dependence, the remedy is not to claim stationarity more strongly; it is to consider a model that represents the relevant structure. The simple stationary model remains a useful baseline, but its limitations matter when interpreting arm summaries or making predictions.

Misreading a sample summary as a known parameter. The reasoning “seven of ten improved, so \(p_A=0.700\)” confuses the observed proportion with the underlying probability. The correction is to report \(\hat{p}_A=0.700\) as an estimate, unless a value for \(p_A\) has been explicitly assumed.
Using a normal probability without specifying its parameters. A sample mean is not automatically the model mean, and a sample standard deviation is not automatically the assumed population standard deviation. State whether a calculation uses specified parameters or plug-in estimates.
Calling an outcome stationary because it is random. Random variation is compatible with a fixed distribution, but a changing distribution is not stationary. Check for changes in measurement, enrollment, and outcome patterns over time.
Treating two reward codings as interchangeable. A thresholded binary outcome and a continuous score represent different targets. Choose and describe the reward before interpreting its model parameters.

Define the reward.
Specify the recorded outcome, its units or categories, and which direction represents greater reward.

Choose a distributional form.
Use Bernoulli for a coded binary event, or a continuous distribution suited to the measured scale and observed behavior.

State the assumptions.
Make clear whether arm-specific distributions are fixed over time and whether outcomes are treated as conditionally independent and homogeneous.

Separate parameters from summaries.
Label \(\mu_k\), \(\sigma_k\), and \(p_k\) as model parameters, and label \(\bar{x}_k\), \(s_k\), and \(\hat{p}_k\) as sample summaries or estimates.

Key takeaway: A stationary reward model assigns each arm a fixed distribution for participant outcomes. Bernoulli and normal models are useful specific choices, but their parameters, coding, and assumptions must be stated explicitly and checked against the clinical outcome.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions.

An arm has binary outcomes, with 12 successes among 20 participants. What is the observed success proportion, and does that establish the arm’s model parameter?
For \(Y_t\mid A_t=C\sim N(8,2^2)\), state the mean and standard deviation with units if the outcome is measured in points.
Why might coding a continuous symptom change as success or failure at a threshold change the reward question?
Name two patterns or features of observed trial data that could raise concern about the basic stationary reward model.
If enrollment order is associated with changing outcome distributions, which basic assumption is questionable?

Solutions. (1) The observed proportion is \(\hat{p}=12/20=0.600\). It estimates the Bernoulli parameter \(p\) under the model; it does not prove \(p=0.600\). (2) The model mean is 8 points and the standard deviation is 2 points; the variance is \(4\) square points. (3) Thresholding replaces the magnitude of each change with a 0/1 indicator, so it targets the probability of meeting the threshold rather than average change. (4) Examples include a systematic shift in outcomes over enrollment order, inconsistent binary coding, strong skew or extreme values under a normal model, and dependence among clustered observations. (5) The assumption that the arm-specific reward distribution is fixed over time is questionable.