Tempered Probability Matching

Why Temper Probability Matching?

Posterior probability matching allocates in proportion to the posterior probabilities that each treatment arm is best. This links assignment to current evidence while retaining randomization. But the degree of adaptation may be too strong or too weak for a particular design. A temperature parameter modifies the allocation probabilities: it can flatten them to preserve more exploration, or sharpen them to concentrate assignments on arms with higher posterior probability of being best.

The temperature is a design choice, not a quantity learned automatically from the response data. Its value should be specified and examined as part of the design, alongside the outcome model and allocation-update schedule. This tutorial focuses on the transformation itself and its consequences; it does not impose allocation floors or caps.

Definition and Assumptions

Definition: For $K$ treatment arms, let $q_k$ be the posterior probability that arm $k$ is selected when ties for the largest response probability are broken uniformly at random, where $k=1,\ldots,K$. For a temperature $T>0$, the tempered probability-matching allocation probability for arm $k$ is $$ \pi_k(T)=\frac{q_k^{1/T}}{\sum_{j=1}^{K}q_j^{1/T}}. $$ The $q_k$ are posterior probabilities, so they are nonnegative and sum to one. The $\pi_k(T)$ are the probabilities used to randomize the next eligible participant.

At $T=1$, $\pi_k(1)=q_k$, recovering ordinary posterior probability matching. Values $T>1$ flatten the allocation probabilities; values $0<T<1$ make them more concentrated. This convention is important: some methods use a parameter in the exponent directly, whereas here temperature appears as its reciprocal.

The rule presumes that the posterior probabilities $q_k$ are defined for the trial's target outcome and current information. For a binary response, one possible model has $Y_{ki}\mid p_k\sim\operatorname{Bernoulli}(p_k)$ within arm $k$, with a specified prior for each $p_k$. To compute posterior best-arm probabilities, the analysis must use the model and prior consistently and account for the observed responses. The allocation formula itself does not require a particular prior or likelihood, but its inputs do.

The formula gives nonnegative probabilities summing to one when at least one $q_k>0$. If every arm has positive posterior probability of being best, every arm retains positive allocation probability at any finite $T>0$. If an arm has $q_k=0$, this transformation keeps its allocation probability at zero. In computation, an estimated zero can reflect too few posterior simulation draws rather than genuine posterior certainty.

What Temperature Changes

The transformation changes relative allocation odds in a simple way. For two arms with positive posterior probabilities, $$ \frac{\pi_i(T)}{\pi_j(T)}=\left(\frac{q_i}{q_j}\right)^{1/T}. $$ Thus, if arm $i$ has greater posterior probability of being best than arm $j$, it retains greater allocation probability at every positive temperature. Temperature changes the magnitude of that advantage, not the ranking.

For example, if $q_i/q_j>1$, raising this ratio to a power below one moves it closer to one. As $T$ increases, $1/T$ decreases, so the allocation ratio moves toward one. If all $q_k>0$, then as $T$ tends to infinity, every $q_k^{1/T}$ tends to one and the allocation approaches $1/K$ for each arm. As $T$ tends to zero, allocation concentrates on the arm or arms with the largest $q_k$. With a unique maximum, that arm's allocation probability tends to one; with a tie among $m$ maxima, the limit is equal allocation among those $m$ arms.

The ratio identity also gives a short justification for the intended direction of the adjustment: the denominator is common to all arms, so it cancels when taking an allocation ratio. The power $1/T$ compresses ratios above one when $T>1$, and enlarges them when $T<1$. Normalization then ensures that the adjusted weights form probabilities.

Worked Example: Tempering Posterior Evidence in a Binary-Response Trial

Consider a hypothetical three-arm trial with binary responses. Each arm has an independent $\operatorname{Beta}(1,1)$ prior for its response probability $p_k$, and responses within an arm are modeled as independent Bernoulli outcomes conditional on $p_k$. Suppose the observed data are 8 responses among 20 participants on arm A, 12 among 20 on B, and 10 among 20 on C. The conjugate posteriors are respectively $\operatorname{Beta}(9,13)$, $\operatorname{Beta}(13,9)$, and $\operatorname{Beta}(11,11)$. These distributions follow from adding each arm's successes and failures to the prior parameters.

Suppose a hypothetical posterior simulation using these posteriors produces 200,000 draws of the three response probabilities, with the largest drawn probability belonging to A in 16,000 draws, B in 136,000 draws, and C in 48,000 draws. The resulting Monte Carlo estimates of the posterior best-arm probabilities are

(q_A,q_B,q_C)=\left(\frac{16{,}000}{200{,}000},\frac{136{,}000}{200{,}000},\frac{48{,}000}{200{,}000}\right)=(0.08,0.68,0.24).

At $T=2$, take square roots and normalize:

\begin{aligned} (w_A,w_B,w_C)&=(\sqrt{0.08},\sqrt{0.68},\sqrt{0.24})\\ &\approx(0.2828,0.8246,0.4899),\\ \sum_k w_k&\approx1.5974,\\ (\pi_A(2),\pi_B(2),\pi_C(2)) &\approx(0.1771,0.5162,0.3067). \end{aligned}

The next participant would therefore have approximately a 17.71% chance of assignment to A, 51.62% to B, and 30.67% to C under this rule. B remains most likely to receive the assignment, but its probability is lower than its untempered value of 68%; probability has shifted toward A and C. The response model and posterior simulation assumptions are stated here to make clear what the $q_k$ represent; the temperature calculation is applied only after those probabilities have been obtained.

Calculating the Transformation

For a given temperature, calculate each arm's weight $w_k=q_k^{1/T}$, add the weights, and divide every weight by their sum. The unnormalized weights are not allocation probabilities until this final normalization is done. Because rounding can make displayed probabilities sum slightly above or below one, carry adequate precision through the calculation and round only the final results.

Worked Example: Comparing Temperatures for Three Arms

Suppose, hypothetically, that the current posterior probabilities of being best are $q=(0.20,0.55,0.25)$ for arms A, B, and C. First consider $T=2$. The unnormalized weights are

(\sqrt{0.20},\sqrt{0.55},\sqrt{0.25}) \approx(0.4472136,0.7416198,0.5000000),

whose sum is approximately $1.6888334$. Therefore,

(\pi_A(2),\pi_B(2),\pi_C(2)) =\left(\frac{\sqrt{0.20}}{1.6888334}, \frac{\sqrt{0.55}}{1.6888334}, \frac{\sqrt{0.25}}{1.6888334}\right) \approx(0.2648,0.4391,0.2961).

These displayed probabilities sum to 1.0000. At $T=1$, they would simply be $(0.2000,0.5500,0.2500)$, so $T=2$ has made the allocation more even without changing the ranking.

Now use $T=0.5$, for which the exponent is $1/T=2$. The weights are $(0.20^2,0.55^2,0.25^2)=(0.0400,0.3025,0.0625)$, with sum $0.4050$. Thus,

(\pi_A(0.5),\pi_B(0.5),\pi_C(0.5)) =\left(\frac{0.0400}{0.4050}, \frac{0.3025}{0.4050}, \frac{0.0625}{0.4050}\right) \approx(0.0988,0.7469,0.1543).

The smaller temperature increases the assignment probability of B, the arm with the largest $q_k$, and decreases the probabilities for the other arms. These are randomization probabilities, not guarantees that any particular arm will receive a specified fraction of all participants.

Worked Example: Interpreting an Allocation Ratio

Suppose two arms have posterior best-arm probabilities $q_A=0.60$ and $q_B=0.15$, with other arms also present. At $T=2$, their allocation ratio is

\frac{\pi_A(2)}{\pi_B(2)} =\left(\frac{0.60}{0.15}\right)^{1/2} =\sqrt{4}=2.

Arm A is therefore assigned with twice the probability of arm B, regardless of the other arms' weights, because normalization cancels in the ratio. Under ordinary matching ($T=1$), the ratio would be $0.60/0.15=4$. The calculation shows that temperature 2 moderates the relative preference; it does not make the arms equally likely unless their $q_k$ values are equal.

Diagnostics, Alternatives, and Limitations

A numerical transformation can be calculated correctly while still producing a poor design. Check the posterior model and the $q_k$ estimates first: a temperature cannot repair a misspecified response model, biased outcome ascertainment, or an unsuitable prior. When $q_k$ values come from posterior simulation, repeat or enlarge the simulation if estimated probabilities are unstable. The allocation probabilities can be sensitive when two arms have similar $q_k$ values, and Monte Carlo error can change their apparent ordering when the values are close.

Inspect the resulting probabilities for every planned update, not just one illustrative state. A very small $T$ can leave one arm with almost all assignments, reducing information on alternatives and making later comparisons less precise. A large $T$ can preserve exploration but weaken preferential allocation. In finite simulation, an estimated $q_k=0$ produces zero allocation for that arm at every finite temperature; increase the number of posterior draws and investigate whether the zero is numerical before interpreting it as certainty. Do not silently replace zeros or alter probabilities without specifying that as a separate design rule.

Tempered probability matching differs from the softmax allocation covered in the earlier material on softmax allocation across multiple arms. Softmax transforms arm utilities through exponentials; this rule instead raises posterior probabilities of being best to the power $1/T$. The two procedures can produce different allocations from the same data because their inputs and transformations differ. Nor does choosing a temperature impose a minimum or maximum allocation probability. Any additional constraints would be separate elements of a design.

Finally, randomization probabilities alone do not establish that an adaptive trial has acceptable operating characteristics. They do not guarantee a desired treatment effect estimate, control type I error, or ensure balanced enrollment. Those properties depend on the full design and analysis and should be examined through simulation under plausible scenarios, including scenarios in which arms have equal response probabilities or the model is inaccurate. Because assignment depends on accumulating outcomes, the rule is adaptive rather than a fixed randomization ratio; its interpretation and evaluation should reflect that feature.

Common Mistakes

Putting temperature in the exponent instead of its reciprocal. Under the convention here, $T>1$ flattens allocations because the exponent is $1/T$. Using $q_k^T$ would reverse that behavior.
Using unnormalized weights as probabilities. The values $q_k^{1/T}$ must be divided by their sum before they can be used to randomize.
Thinking tempering changes which arm is best. It changes allocation probabilities, not the posterior probabilities $q_k$ or the ordering of positive $q_k$ values.
Interpreting an allocation probability as a final sample proportion. A 0.50 assignment probability at one update does not guarantee that half of the full trial will be assigned to that arm.
Treating a simulated zero as certainty. A finite posterior simulation may miss rare events. Check simulation precision before concluding that an arm truly has zero posterior probability of being best.

Key takeaway: Tempered probability matching uses $\pi_k(T)=q_k^{1/T}/\sum_j q_j^{1/T}$. With this convention, $T=1$ gives ordinary probability matching, $T>1$ encourages more even allocation, and $0<T<1$ concentrates allocation more strongly on arms with higher posterior probability of being best.

Check Your Understanding

Answer each question, then compare with the solutions.

If $q=(0.50,0.30,0.20)$, what allocation probabilities result at $T=1$?
For positive $q_A=0.40$ and $q_B=0.10$, calculate $\pi_A(2)/\pi_B(2)$.
For $q=(0.25,0.75)$, calculate the allocation probabilities at $T=0.5$.
What happens to the allocation probabilities as $T$ tends to infinity when all arms have positive $q_k$?
Why might an arm's estimated posterior best-arm probability of zero require further investigation?

Solutions. (1) At $T=1$, $\pi=q$, so the allocation probabilities are $(0.50,0.30,0.20)$. (2) The ratio is $(0.40/0.10)^{1/2}=\sqrt{4}=2$. (3) Since $1/T=2$, the weights are $0.25^2=0.0625$ and $0.75^2=0.5625$, summing to $0.6250$; the probabilities are $(0.0625/0.6250,0.5625/0.6250)=(0.1000,0.9000)$. (4) Each weight tends to one, so the allocation approaches equal probabilities $1/K$. (5) A finite number of posterior draws may fail to include a rare event in which that arm is best; increase or assess the simulation before treating the estimate as a true zero.

What You'll Learn