Softmax Allocation Across Multiple Arms

From Arm Utilities to Randomization Probabilities

An adaptive trial may have several eligible arms and a current posterior expected utility for each. Allocating everyone to the arm with the largest utility is deterministic: it gives no chance of assignment to the other arms. The softmax rule instead maps the utilities to probabilities, so the most attractive arms can receive greater allocation while other eligible arms retain positive probability.

The rule is a randomization mechanism conditional on the current information state. It does not itself specify how the posterior or utilities are obtained. The utilities must be defined on a common, meaningful scale—for example, expected net benefit after accounting for efficacy and harm—and the temperature must use that same scale.

Definition: Suppose there are $K$ eligible arms, indexed by $k=1,\ldots,K$, with finite arm-specific posterior expected utilities $u_k$. For a temperature $\tau>0$, the softmax allocation probability for arm $k$ is $$ \pi_k(\tau)=\frac{\exp(u_k/\tau)}{\sum_{\ell=1}^{K}\exp(u_\ell/\tau)}. $$ At the current information state, a participant is assigned to arm $k$ with probability $\pi_k(\tau)$. These probabilities are nonnegative and sum to one.

Here, $u_k$ is a utility summary for arm $k$, not the probability that the arm is best. The temperature $\tau$ controls how sharply differences in utility translate into differences in allocation. It is a design parameter, not a posterior standard deviation, a measure of uncertainty, or a treatment effect.

Calculating the Probabilities

Worked Example: Three Arms at a Fixed Temperature

Suppose a hypothetical trial has three eligible arms with posterior expected utilities of $0$, $1$, and $2$ utility points. Set $\tau=1$ utility point. The softmax weights are the exponentials of the utilities divided by the temperature:

w_1=e^{0/1}=1,\qquad w_2=e^{1/1}\approx2.7183,\qquad w_3=e^{2/1}\approx7.3891.

Their sum is $1+2.7183+7.3891=11.1074$. Dividing each weight by that total gives

\pi_1\approx\frac{1}{11.1074}=0.0900,\qquad \pi_2\approx\frac{2.7183}{11.1074}=0.2447,\qquad \pi_3\approx\frac{7.3891}{11.1074}=0.6652.

The probabilities sum to approximately $1.000$, allowing for rounding. At this information state, arm 3 has the largest allocation probability, but arms 1 and 2 still have positive probabilities. These are assignment probabilities for a participant under this rule; they do not guarantee that the realized trial will allocate those exact percentages.

How Temperature Changes Concentration

For any two arms $i$ and $j$, their allocation odds have a particularly useful form:

\frac{\pi_i(\tau)}{\pi_j(\tau)} =\exp\!\left(\frac{u_i-u_j}{\tau}\right).

Thus, the odds depend on the utility difference measured in units of temperature. A larger temperature makes the same utility gap less influential; a smaller temperature makes it more influential. This relationship also shows why temperature must be expressed in the same units as utility.

Worked Example: Comparing Three Temperatures

Use the hypothetical utilities $u=(0,1,2)$ from the previous calculation. Compare temperatures of $0.5$, $1$, and $2$ utility points. For $\tau=0.5$, the weights are $e^0=1$, $e^2\approx7.3891$, and $e^4\approx54.5982$. Their sum is $62.9873$, so the probabilities are approximately

\pi(0.5)=(0.0159,\ 0.1173,\ 0.8668).

At $\tau=1$, the calculation above gives $\pi(1)\approx(0.0900,\ 0.2447,\ 0.6652)$. At $\tau=2$, the weights are $1$, $e^{0.5}\approx1.6487$, and $e^1\approx2.7183$, with total $5.3670$. Therefore,

\pi(2)\approx(0.1863,\ 0.3072,\ 0.5065).

As temperature rises, the allocation becomes less concentrated on the highest-utility arm. It does not follow that every arm’s probability must move monotonically in one direction: an arm’s change depends on its utility relative to the probability-weighted average, as the derivative below shows.

Derivative and Limiting Behavior

The derivative gives a precise account of how each probability responds to temperature. Write the probability-weighted average utility as $\bar u_{\pi}=\sum_{\ell=1}^{K}\pi_\ell u_\ell$. Differentiating the numerator and denominator of the softmax expression yields

\frac{d\pi_j}{d\tau} = \pi_j(\tau)\frac{\bar u_{\pi}-u_j}{\tau^2}.

To see the result, let $Z=\sum_{\ell=1}^{K}\exp(u_\ell/\tau)$, so that $\pi_j=\exp(u_j/\tau)/Z$. The derivative of $\log Z$ is

\frac{d\log Z}{d\tau} = -\frac{1}{\tau^2} \frac{\sum_{\ell=1}^{K}u_\ell\exp(u_\ell/\tau)}{Z} = -\frac{\bar u_{\pi}}{\tau^2}.

Also, $d\log(\exp(u_j/\tau))/d\tau=-u_j/\tau^2$. Since $d\pi_j/d\tau=\pi_j\,d\log(\pi_j)/d\tau$, subtracting the derivative of $\log Z$ gives the stated expression. Consequently, a probability increases when its utility is below $\bar u_{\pi}$, decreases when it is above $\bar u_{\pi}$, and has zero derivative when it equals that average.

For $K\ge2$, if there is a unique maximum-utility arm, its utility is strictly greater than $\bar u_{\pi}$, because every other arm has positive probability and strictly lower utility. Its allocation probability therefore decreases as $\tau$ increases. If $K=1$, the sole arm is also the unique maximum, but its probability is identically $1$; there are no competing arms to receive allocation.

At high temperature, all arms become nearly equally likely. In the limit $\tau\to\infty$, each exponential approaches $1$, so $\pi_k\to1/K$. At low temperature, probability concentrates on the arm or arms with maximum utility. If there is a unique maximum, its probability tends to $1$; if $m$ arms tie for the maximum, each tied arm tends to $1/m$, and each lower-utility arm tends to $0$. The formula requires $\tau>0$; $\tau=0$ is not a defined setting.

Worked Example: Tied Best Arms and One Eligible Arm

Suppose three eligible arms have hypothetical utilities $4$, $4$, and $1$ utility points, with $\tau=1$. Subtracting $4$ from every utility does not change the probabilities, so calculate with utilities $0$, $0$, and $-3$. The weights are $1$, $1$, and $e^{-3}\approx0.04979$, giving a total of $2.04979$. Thus,

\pi_1=\pi_2\approx\frac{1}{2.04979}=0.4879,\qquad \pi_3\approx\frac{0.04979}{2.04979}=0.02429.

The tied arms receive equal probabilities because their utilities are equal, while the lower-utility arm receives a smaller but positive probability. If only one arm were eligible, the denominator would contain only its weight, and its probability would be $\exp(u_1/\tau)/\exp(u_1/\tau)=1$ for every $\tau>0$.

Invariances, Implementation, and Comparisons

Adding the same constant $c$ to every utility leaves the probabilities unchanged: the factor $\exp(c/\tau)$ appears in every weight and cancels from the numerator and denominator. Multiplying all utilities by a positive constant $a$ and the temperature by $a$ always preserves the probabilities, since $(a u_k)/(a\tau)=u_k/\tau$. Conversely, if the utility vector is nonconstant, preserving the probabilities after multiplying all utilities by $a$ requires multiplying the temperature by $a$; if all utilities are equal (including when there is only one arm), the probabilities are unchanged for any positive temperature. A numerical temperature has no scale-independent meaning apart from the utility units.

These properties provide practical checks. If a common shift changes computed probabilities, there is an implementation error. If utility values are converted to a different unit, the temperature must be converted consistently. For numerical stability, especially when $u_k/\tau$ is large, subtract the maximum utility $u_{\max}$ before exponentiating:

\pi_k(\tau)= \frac{\exp((u_k-u_{\max})/\tau)} {\sum_{\ell=1}^{K}\exp((u_\ell-u_{\max})/\tau)}.

This gives the same probabilities, while keeping every exponent nonpositive and at least one weight equal to $1$. A useful diagnostic is to verify that probabilities are finite, nonnegative, and sum to $1$ within rounding tolerance; that a higher-utility arm has a higher probability than a lower-utility arm; and that tied utilities receive tied probabilities.

Allocation approach	How it assigns across arms	Key distinction
Deterministic maximum expected utility	Assigns only to an arm with the largest utility (with a separate rule needed for ties).	Softmax preserves positive probability for every eligible arm when $K\ge2$ and $\tau>0$.
Equal randomization	Assigns probability $1/K$ to each eligible arm.	Softmax approaches equal randomization as temperature grows, but is generally unequal at finite temperature.
Posterior probability matching	Uses posterior probabilities of arm-level events, such as being best, as allocation probabilities.	Softmax uses exponentiated utility summaries, not posterior probabilities of those events.

The softmax rule is useful when the design calls for a smooth, tunable response to utility differences. Its limitations matter in clinical trials. A softmax probability is not a guarantee of patient benefit, a probability that an arm is truly best, or evidence that the utility model is correct. The rule uses the supplied utility summaries; poor utility definitions, incompatible scales, or a misspecified posterior can therefore produce inappropriate allocations. The choice of temperature should be justified on the clinical utility scale and assessed for sensitivity to plausible settings. The basic softmax formula also does not by itself enforce additional design constraints.

Common Mistakes

Confusing utility with a probability. An expected utility may be negative or larger than one; it is the exponentiated, normalized value that becomes an allocation probability.
Treating temperature as uncertainty. Temperature controls the mapping from utilities to allocation probabilities. It is not a posterior variance or a measure of uncertainty about treatment effects.
Ignoring the utility scale. A temperature chosen for utilities measured in points cannot be reused unchanged if the utilities are rescaled.
Claiming the highest-utility arm always loses probability as temperature rises. This is strictly true for a unique maximum when $K\ge2$. With one eligible arm, its probability remains $1$.
Assuming the probabilities are realized allocation fractions. They govern randomized assignments at a given information state; the observed fractions vary with random outcomes and subsequent adaptation.

Key takeaway: Softmax allocation converts comparable arm utilities into normalized assignment probabilities. Temperature determines how strongly utility differences affect those probabilities: larger values flatten allocation, while smaller positive values concentrate it on the best-utility arm or arms.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions.

Two eligible arms have utilities $2$ and $3$, and $\tau=1$. Which arm has the larger probability, and what are the approximate probabilities?
What is the limiting allocation when $\tau\to\infty$ for four eligible arms?
For utilities $5$ and $5$, what are the allocation probabilities at any positive temperature?
Why does multiplying every utility by $10$ and leaving temperature unchanged generally alter allocation probabilities?

Solutions. (1) The weights are $e^2\approx7.3891$ and $e^3\approx20.0855$, with total $27.4746$. The probabilities are approximately $0.2689$ and $0.7311$, so the second arm has the larger probability. (2) Each probability tends to $1/4=0.25$, since all exponential weights tend to $1$. (3) Equal utilities give equal weights, so each arm has probability $1/2$, independent of the positive temperature. (4) The exponent changes from $u_k/\tau$ to $10u_k/\tau$, magnifying utility differences. To preserve the original probabilities, temperature must also be multiplied by $10$.