Thompson Sampling With Normal Outcomes

Why Normal Outcomes Need a Posterior Model

Many clinical outcomes are continuous: a change in a symptom score, a laboratory measurement, or a blood-pressure reduction. When larger values represent better outcomes, Thompson sampling can allocate the next participant by sampling each treatment’s unknown mean response from its posterior distribution and assigning the participant to the treatment with the largest draw. The calculation therefore starts with an outcome model and a prior for each treatment mean.

This tutorial uses a normal likelihood with a known outcome variance and a normal prior. The known-variance assumption makes the update particularly direct. The resulting posterior distribution describes uncertainty about a treatment’s mean response; it is not the distribution of an individual participant’s outcome.

Model and assumptions: For treatment arm $j$, let $Y_{ij}$ be the continuous response for participant $i$ assigned to that arm, and let $\mu_j$ be its unknown mean response. Conditional on $\mu_j$, responses are independent and $Y_{ij}\sim N(\mu_j,\sigma_j^2)$, where the variance $\sigma_j^2$ is treated as known. Before observing the current trial responses, assign $\mu_j\sim N(m_{0j},\tau_{0j}^2)$, with $\tau_{0j}^2>0$. For the basic arm-by-arm update here, assume the arm means have independent priors and the likelihood factors by arm. Larger $\mu_j$ is taken to be better; if smaller outcomes are preferable, define the response on a direction-reversed scale or select the smallest posterior draw instead.

Updating Each Arm’s Posterior

Suppose arm $j$ has $n_j$ observed responses with sample mean $\bar{x}_j$. Under the model, the sample mean has distribution $\bar{x}_j\mid\mu_j\sim N(\mu_j,\sigma_j^2/n_j)$. The posterior mean combines the prior mean and the observed sample mean, weighting each by its precision (the reciprocal of its variance).

Posterior update: Under the stated normal likelihood and normal prior, $$ \mu_j\mid\text{data}\sim N(m_{nj},v_{nj}), \qquad v_{nj}=\left(\frac{1}{\tau_{0j}^2}+\frac{n_j}{\sigma_j^2}\right)^{-1}, \qquad m_{nj}=v_{nj}\left(\frac{m_{0j}}{\tau_{0j}^2}+\frac{n_j\bar{x}_j}{\sigma_j^2}\right). $$ Here $v_{nj}$ is the posterior variance, not the outcome variance. The data contribute $n_j/\sigma_j^2$ units of precision: more observations or less variable outcomes make the posterior more concentrated.

Equivalently, the observed sample mean has variance $\sigma_j^2/n_j$, so the posterior mean is a precision-weighted average of $m_{0j}$ and $\bar{x}_j$. The weights are the prior precision $1/\tau_{0j}^2$ and the data precision $n_j/\sigma_j^2$, divided by their sum. This gives a useful check: the posterior mean lies between the prior mean and the observed mean, and the posterior variance is smaller than either the prior variance or the sample-mean variance.

Worked Example: Updating One Treatment Mean

Hypothetical data. A trial measures improvement on a symptom scale, in which higher scores indicate greater improvement. Before the trial data, the treatment mean has prior distribution $N(4,4)$: prior mean 4 improvement points and prior variance 4 points squared. Assume the known response variance is $\sigma^2=36$ points squared. The nine observed responses are 5, 6, 6, 7, 7, 7, 8, 8, and 9 points, which sum to 63 and have $\bar{x}=63/9=7$ points.

The posterior variance is $$ v_n=\left(\frac{1}{4}+\frac{9}{36}\right)^{-1} =(0.25+0.25)^{-1}=2\ \text{points}^2. $$

The posterior mean is $$ m_n=2\left(\frac{4}{4}+\frac{9(7)}{36}\right) =2(1+1.75)=5.5\ \text{points}. $$

Thus $\mu\mid\text{data}\sim N(5.5,2)$. The estimated mean improvement is pulled from the observed 7 points toward the prior mean of 4 points, while uncertainty about the mean is reduced from the prior variance of 4 to 2 points squared. The 36-point-squared outcome variance has not become 2; it remains the assumed variability of individual responses.

From Posterior Distributions to Allocation

For $K$ treatments, Thompson sampling draws one candidate mean from each arm’s current posterior distribution, then allocates to the arm with the best draw. With larger outcomes preferred, the posterior probability that arm $j$ is best is

p_j=P\left(\mu_j=\max_{1\leq k\leq K}\mu_k\mid\text{data}\right).

For continuous, independent normal posteriors, exact ties have probability zero, so these probabilities sum to 1. Repeating the posterior-draw-and-select step would assign arm $j$ with probability $p_j$. A single draw produces one assignment, not an estimate of $p_j$ by itself. In a two-arm comparison, the best-arm probability can be calculated directly from the posterior distribution of the difference.

Worked Example: Comparing Two Posterior Means

Hypothetical data. Consider two treatments, A and B, measured by blood-pressure reduction in mmHg, with larger reductions preferred. Each arm has four responses, and the known outcome variance in each arm is $36\ \text{mmHg}^2$. The responses are 11, 12, 14, and 15 mmHg for A, giving $\bar{x}_A=13$, and 10, 11, 13, and 14 mmHg for B, giving $\bar{x}_B=12$. Use independent priors $\mu_A\sim N(10,25)$ and $\mu_B\sim N(9,16)$.

For A, the posterior variance and mean are $$ v_A=\left(\frac{1}{25}+\frac{4}{36}\right)^{-1}=6.618, \qquad m_A=6.618\left(\frac{10}{25}+\frac{4(13)}{36}\right)=12.206. $$ Therefore $\mu_A\mid\text{data}\sim N(12.206,6.618)$.

For B, $$ v_B=\left(\frac{1}{16}+\frac{4}{36}\right)^{-1}=5.760, \qquad m_B=5.760\left(\frac{9}{16}+\frac{4(12)}{36}\right)=10.920. $$ Therefore $\mu_B\mid\text{data}\sim N(10.920,5.760)$.

Because the posterior means are independent and normal, their difference is normal: $$ \mu_A-\mu_B\mid\text{data}\sim N(12.206-10.920,\ 6.618+5.760) =N(1.286,12.378). $$ Its standard deviation is $\sqrt{12.378}=3.518\ \text{mmHg}$. Hence $$ P(\mu_A>\mu_B\mid\text{data}) =P\left(Z>\frac{0-1.286}{3.518}\right) =\Phi(0.366)\approx 0.643. $$ Under this model and these data, the posterior probability that A has the larger mean reduction is about 0.643. In a two-arm Thompson-sampling step, A is selected with this probability and B with probability about 0.357. This is a posterior probability conditional on the model and observed data, not a frequentist confidence level.

A Multi-Arm Allocation Draw

When there are more than two treatments, the arm posteriors are still updated separately under the independent-arm model. One can then generate independent draws from those posteriors and compare the draws. The following example shows how the posterior distributions enter one allocation decision; it does not claim that one draw estimates the overall allocation probabilities.

Worked Example: Sampling Among Three Treatments

Hypothetical data. Three treatments produce continuous improvement scores, with larger values preferred. Each arm has five observed responses; the known outcome variance is $25$ score-points squared in every arm. The data are A: 20, 21, 22, 23, 24, with $\bar{x}_A=22$; B: 20, 21, 22, 22, 22.5, with $\bar{x}_B=21.5$; and C: 21, 22, 23, 24, 25, with $\bar{x}_C=23$. Use independent priors A: $N(20,16)$, B: $N(21,9)$, and C: $N(19,25)$.

For A, $$ v_A=\left(\frac{1}{16}+\frac{5}{25}\right)^{-1}=3.810, \qquad m_A=3.810\left(\frac{20}{16}+\frac{5(22)}{25}\right)=21.524. $$ For B, $$ v_B=\left(\frac{1}{9}+\frac{5}{25}\right)^{-1}=3.214, \qquad m_B=3.214\left(\frac{21}{9}+\frac{5(21.5)}{25}\right)=21.321. $$ For C, $$ v_C=\left(\frac{1}{25}+\frac{5}{25}\right)^{-1}=4.167, \qquad m_C=4.167\left(\frac{19}{25}+\frac{5(23)}{25}\right)=22.333. $$

The current posteriors are therefore approximately $N(21.524,3.810)$ for A, $N(21.321,3.214)$ for B, and $N(22.333,4.167)$ for C. For one illustrative posterior draw, take independent standard-normal values $z_A=0.80$, $z_B=-0.50$, and $z_C=0.20$. Transforming each value gives $$ \tilde{\mu}_A=21.524+0.80\sqrt{3.810}=23.085,\quad \tilde{\mu}_B=21.321-0.50\sqrt{3.214}=20.425,\quad \tilde{\mu}_C=22.333+0.20\sqrt{4.167}=22.742. $$

Arm A has the largest draw, so this draw assigns the next participant to A. A different draw could select B or C. The fact that C has the highest posterior mean does not make Thompson sampling a rule that always selects C: the sampled values reflect both estimated response and posterior uncertainty. The probability of selecting each arm over repeated independent draws is that arm’s posterior probability of being best.

Assumptions, Diagnostics, and Alternatives

The update and allocation interpretation depend on the model being a reasonable description of the trial. The response scale should be meaningfully continuous, and the normal likelihood should be plausible for responses within each arm. The variance $\sigma_j^2$ must be specified as known for this model; replacing it with an estimate changes the model and its posterior uncertainty. The prior mean and variance must use the response’s units and be defensible for the treatment mean. Arm-specific posterior updates also require the stated independence structure; shared or correlated treatment effects need a joint posterior model instead.

Useful diagnostics include plotting outcomes by arm, checking for strong skewness or outliers, and looking for floor or ceiling effects. Such features can make a normal likelihood misleading, especially with small arm samples. Verify that the assumed variance is on the correct scale (variance units are squared response units), that each arm’s sample mean agrees with its recorded data, and that posterior variances decrease as data precision increases. A posterior that is unexpectedly concentrated often signals a variance or unit error.

The normal-normal model is attractive because its posterior is easy to calculate and sample from. It is not automatically appropriate merely because the outcome is numeric. Equal randomization does not use posterior evidence to favor arms, while a plug-in greedy rule that always chooses the largest posterior mean ignores uncertainty. Thompson sampling differs from both: posterior uncertainty affects each arm’s chance of receiving the next participant. The model’s limitations remain important, however; a convenient posterior cannot repair a poor likelihood, a poorly calibrated prior, informative missing responses, or a clinically inappropriate definition of which outcome direction is better.

Approach	How the next assignment is chosen	Main distinction
Equal randomization	Assign among arms at fixed equal probabilities.	Does not adapt probabilities to observed responses.
Greedy plug-in allocation	Choose the arm with the largest posterior mean.	Does not sample from posterior uncertainty.
Thompson sampling	Draw each arm’s mean from its posterior and choose the best draw.	Allocation probabilities correspond to posterior best-arm probabilities.

Common pitfall: “Arm C has the largest posterior mean, so Thompson sampling must assign the next participant to C.” This is the greedy rule, not Thompson sampling. Thompson sampling compares random draws from the arm posteriors; a lower-mean arm can be selected on a particular draw because its posterior uncertainty also matters.

Check Your Understanding

Try each question, then compare your work with the solutions below.

A treatment has prior $\mu\sim N(10,4)$, known outcome variance $\sigma^2=16$, and $n=4$ responses with $\bar{x}=12$. Find its posterior mean and variance.
Why is the posterior variance in Question 1 not the variance of an individual response?
Two independent arm posteriors are $N(5,1)$ and $N(4,1)$. Find the posterior probability that the first arm’s mean is larger.
In one three-arm posterior draw, arm B has the largest sampled mean, even though arm A has the largest posterior mean. Which arm receives the assignment under Thompson sampling, and why?

Solutions. (1) The posterior variance is $(1/4+4/16)^{-1}=2$. The posterior mean is $2(10/4+4(12)/16)=11$, so the posterior is $N(11,2)$. (2) The individual-response variance is the assumed $16$ outcome-units squared. The posterior variance $2$ quantifies uncertainty about the unknown arm mean after combining the prior and four responses. (3) The difference has distribution $N(1,2)$, so the probability is $\Phi(1/\sqrt{2})\approx 0.760$. (4) Arm B receives the assignment for that draw because Thompson sampling selects the arm with the largest sampled mean; selecting A every time based only on its posterior mean would be greedy allocation.