Worked Design: A Three-Arm Bernoulli Trial

From a Clinical Question to a Bandit Design

A three-arm Bernoulli trial is a useful design problem when each participant receives one of three treatments and the primary reward is binary: for example, whether a prespecified response occurs within a fixed assessment window. The design must say more than which treatment has the largest success probability. It must specify how outcomes are generated, what information is available when each assignment is made, how the assignment probabilities are chosen, and what performance objective is being optimized.

We will use a fully hypothetical setting. The three arms are treatments 1, 2, and 3, with success probabilities \(p_1=0.30\), \(p_2=0.42\), and \(p_3=0.36\). These values describe a scenario for illustrating and evaluating the design; a real trial generally does not know them when assigning participants. The proposed design has a fixed horizon of \(T=300\) participants and seeks high expected total successes while retaining some random allocation to every arm.

Definition: In a three-arm Bernoulli bandit trial, \(A_t\in\{1,2,3\}\) is the arm assigned to participant \(t\), and \(Y_t\in\{0,1\}\) is that participant’s reward, with 1 denoting success. Arm \(i\) has success probability \(p_i\). The history available before assignment \(t\) is \(H_{t-1}=(A_1,Y_1,\ldots,A_{t-1},Y_{t-1})\). An allocation rule specifies \(q_{t,i}=P(A_t=i\mid H_{t-1})\), where \(\sum_{i=1}^3 q_{t,i}=1\).

Sequential Model and Design Assumptions

The sequential formulation matters because an adaptive rule uses past outcomes to choose later assignments. We assume that, for every time \(t\) and arm \(i\),

P(Y_t=1\mid H_{t-1},A_t=i)=p_i.

Equivalently, conditional on the observed history and the current assignment, the current outcome is generated as a Bernoulli random variable with parameter \(p_i\), independently of the past given that information. The assignment itself is drawn using the stated rule conditional on \(H_{t-1}\). This is a sequential model; it does not assert that outcomes are independent after conditioning on the entire realized assignment sequence. Such conditioning can reveal information about earlier outcomes when later assignments depend on them.

For this worked design, assume the response definition and assessment window are fixed, each participant receives one arm, outcomes are observed before the next assignment, and one participant’s treatment does not alter another participant’s response. We also assume the arm-specific probabilities remain constant over enrollment. These assumptions give a simple stationary Bernoulli model. If responses are delayed, the rule must instead use the information available at the time of assignment; if response probabilities change over calendar time or participant characteristics, a constant \(p_i\) may not be an adequate model.

Objective, Horizon, and Allocation Rule

For a fixed horizon \(T\), define the total reward as \(S_T=\sum_{t=1}^{T}Y_t\). We use the finite-horizon cumulative-reward objective: choose assignments to maximize \(E(S_T)\), subject to the design’s exploration requirement. This objective values observed successes during the trial. It is distinct from an objective that focuses only on identifying the best arm for future use.

The allocation rule below is an implementable softmax rule with an initial allocation to each arm and a uniform exploration component. Let \(N_i(t-1)\) be the number assigned to arm \(i\) before time \(t\), and let \(R_i(t-1)\) be the number of successes observed on that arm. After initialization, set \(\hat p_i(t-1)=R_i(t-1)/N_i(t-1)\). With temperature \(\tau=0.10\) and exploration mixture \(\epsilon=0.10\), use

q_{t,i}=(1-\epsilon)\frac{\exp\{\hat p_i(t-1)/\tau\}} {\sum_{j=1}^{3}\exp\{\hat p_j(t-1)/\tau\}}+\frac{\epsilon}{3}.

For the first three participants, assign one to each arm in a prespecified order. Thereafter, draw \(A_t\) from the probabilities \(q_{t,1},q_{t,2},q_{t,3}\). The initial assignments make each empirical mean defined; the uniform mixture ensures that every arm has probability at least \(\epsilon/3\) on later assignments. The rule favors arms with larger observed success rates, but does not eliminate randomization.

Worked Example: Specifying the Three-Arm Trial

Consider the hypothetical probabilities \(p_1=0.30\), \(p_2=0.42\), and \(p_3=0.36\), with \(T=300\). The reward is success by the fixed assessment time, coded \(Y_t=1\); failure is coded \(Y_t=0\). The participant is the assignment unit, the arms are the three treatments, and the outcome model is the sequential Bernoulli model stated above.

The objective is to maximize expected total successes over the 300 assignments, using the softmax rule after one initial assignment per arm. Since the rule’s probabilities depend on accumulated outcomes, the actual allocation sequence is random. The specified \(p_i\) values are not inputs to the rule: the rule uses observed empirical means. They are used here to describe the hypothetical scenario and to calculate expected rewards under that scenario.

The assumptions should be checked against the trial’s operation. In particular, the endpoint must be available before the next assignment for the stated rule to use the complete \(H_{t-1}\). If, for example, the response takes several weeks to ascertain, the rule cannot use that response immediately; the history would need to include only outcomes observed by that time, or the design would need a different update schedule.

Calculating an Adaptive Assignment

A rule becomes operational only when its probabilities can be calculated from a specified history. The next example works through one update. Its observed counts are hypothetical; they illustrate how the allocation depends on the data rather than on the unknown true probabilities.

Worked Example: Softmax Probabilities at a Given History

Suppose that after initialization and further assignments, the available history gives arm 1: 12 successes among 40 participants; arm 2: 18 among 40; and arm 3: 15 among 40. The empirical means are

\hat p_1=12/40=0.300,\qquad \hat p_2=18/40=0.450,\qquad \hat p_3=15/40=0.375.

With \(\tau=0.10\), the softmax weights are proportional to \(\exp(3.00)\), \(\exp(4.50)\), and \(\exp(3.75)\). Dividing each by \(\exp(4.50)\) gives the equivalent weights \(\exp(-1.50)=0.2231\), \(1\), and \(\exp(-0.75)=0.4724\). Their sum is \(1.6955\), so the softmax probabilities before the exploration mixture are approximately \(0.1316\), \(0.5898\), and \(0.2786\).

The final probabilities are

\begin{aligned} q_{t,1}&=0.90(0.1316)+0.10/3=0.1518,\\ q_{t,2}&=0.90(0.5898)+0.10/3=0.5642,\\ q_{t,3}&=0.90(0.2786)+0.10/3=0.2841. \end{aligned}

Rounding accounts for the displayed probabilities summing to approximately 1. Arm 2 is most likely to be assigned at this history, but all three arms retain positive probability. Under the hypothetical true probabilities, the conditional expected reward for the next participant is

E(Y_t\mid H_{t-1})=\sum_{i=1}^3q_{t,i}p_i =(0.1518)(0.30)+(0.5642)(0.42)+(0.2841)(0.36) \approx 0.3848.

Thus the next assignment has an expected success probability of about 0.3848 under this particular history and scenario. This is not a claim that the realized next outcome will be 0.3848; the outcome is binary.

Why the Expected-Reward Calculation Works

At a given history, the assignment probabilities and the arm-specific success probabilities determine the conditional expected reward. First condition on which arm is assigned, then average over the randomized assignment:

\begin{aligned} E(Y_t\mid H_{t-1}) &=\sum_{i=1}^3 E(Y_t\mid H_{t-1},A_t=i) P(A_t=i\mid H_{t-1})\\ &=\sum_{i=1}^3 p_iq_{t,i}. \end{aligned}

The second equality uses the sequential outcome model. Summing over time and applying linearity of expectation gives

E(S_T)=\sum_{t=1}^{T}E(Y_t) =\sum_{t=1}^{T}E\left\{\sum_{i=1}^{3}q_{t,i}p_i\right\}.

For an adaptive rule, \(q_{t,i}\) is random across possible histories, so the expectation on the right must average over those histories. It would generally be incorrect to calculate the reward at one observed history and multiply it by the full horizon. The displayed identity is a justification for evaluating the design, not a shortcut that removes the need to account for how future histories evolve.

Worked Example: Equal Allocation and an Oracle Benchmark

For a transparent benchmark, compare equal allocation with an oracle that knows the true probabilities and assigns every participant to the best arm. Under equal allocation over \(T=300\), the expected number of successes is

300\left(\frac{0.30+0.42+0.36}{3}\right) =300(0.36)=108.

The largest hypothetical success probability is \(p_2=0.42\), so the oracle benchmark has expected total successes \(300(0.42)=126\). The difference is \(126-108=18\) expected successes. This is an oracle comparison, not an attainable guarantee: the trial does not know which arm is best at enrollment and must learn from outcomes.

For another perspective, a fixed allocation of 100 participants per arm also has expected total \(100(0.30)+100(0.42)+100(0.36)=108\). Equal allocation and the oracle benchmark are easy to calculate because their allocation counts are fixed. The expected total for the adaptive softmax design cannot be obtained from the one-step example alone; it depends on the distribution of histories generated by the rule.

Implementation Checks, Alternatives, and Limitations

Before using the rule, verify that the response is truly defined as binary and that its assessment window is identical across arms. Check that assignment probabilities are calculated from data available at that moment, sum to one, and are logged along with the history used to obtain them. A useful implementation test is to feed a known history into the allocation code and reproduce the hand-calculated probabilities above. Also verify that the lower bound \(\epsilon/3\) is maintained after rounding and software updates.

The main simple alternative is fixed equal randomization. It is easy to implement and gives predetermined expected allocation counts, but it does not shift assignments toward arms with better observed rewards. The proposed adaptive rule can allocate more often to promising arms, while the exploration mixture continues to allocate to all arms. A purely greedy rule, which always assigns to the arm with the largest current estimate, is another contrast: it can stop exploring an arm after early unfavorable outcomes. The softmax mixture avoids zero probabilities, but its performance still depends on the temperature, the exploration fraction, and the reliability of the empirical means.

The design has important limitations. A high observed mean from a small number of participants can be unstable; a constant temperature does not automatically adapt to uncertainty or to the remaining horizon. The uniform mixture improves continued sampling but does not ensure balanced sample sizes, nor does it by itself guarantee adequate precision for every treatment comparison. A constant-probability Bernoulli model can be misleading if participant mix, endpoint ascertainment, or treatment delivery changes over time. Finally, maximizing expected successes alone does not encode safety requirements, inferential precision, or the need for a confirmatory comparison; those goals may require additional constraints or a different objective.

Common pitfall: It is incorrect to say that outcomes are independent conditional on the full realized assignment sequence in an adaptive trial. Later assignments can depend on earlier outcomes, so knowing the later assignments can convey information about those earlier outcomes. State the sequential condition \(P(Y_t=1\mid H_{t-1},A_t=i)=p_i\) instead, and specify how assignments are randomized given the history.

Check Your Understanding

Answer each question, then compare with the solutions below.

With \(p_1=0.25\), \(p_2=0.40\), and \(p_3=0.35\), what is the expected reward for one participant under equal randomization?
Why does the uniform mixture in the worked allocation rule give every arm a minimum assignment probability of \(\epsilon/3\)?
At a particular history, suppose \(q=(0.20,0.50,0.30)\) and \(p=(0.25,0.40,0.35)\). Calculate the conditional expected reward.
What needs to change if outcomes are not observed before the next participant is assigned?
Why is the oracle’s expected total not a realistic performance guarantee for an adaptive trial?

Solutions. (1) The equal-randomization expectation is \((0.25+0.40+0.35)/3=1.00/3\approx0.3333\) successes per participant. (2) Each softmax probability is nonnegative, and the rule adds \(\epsilon/3\) to it, so \(q_{t,i}\geq\epsilon/3\) for every arm. (3) The expectation is \((0.20)(0.25)+(0.50)(0.40)+(0.30)(0.35)=0.050+0.200+0.105=0.355\) successes. (4) The allocation rule must use only outcomes available at assignment time; its history and update schedule must reflect the delay. (5) The oracle knows the true best arm in advance, whereas the adaptive trial does not know the \(p_i\) values and must allocate while learning them.