Why Unequal Reinforcement Matters
In a response-adaptive urn design, the current urn composition determines the probabilities of assigning the next patient to each treatment. When treatment-specific reinforcement amounts differ, the same response can therefore produce different changes in the allocation probabilities, depending on which treatment was assigned. This tutorial isolates that arithmetic: how to update the urn, and how to calculate the allocation probabilities induced by the updated state.
The key distinction is between the reinforcement rule and the allocation rule. The reinforcement rule specifies how the urn changes after an assignment and response. The allocation rule converts the resulting composition into probabilities for the next assignment. Calculating both steps explicitly prevents a reinforcement amount from being mistaken for an allocation probability.
A Treatment-Specific Reinforcement Rule
Consider a binary response, success or failure. Under the rule used first below, if patient \(t\) is assigned to treatment \(i\) and has a success, the urn receives \(c_i\) additional balls of treatment \(i\). A failure adds no balls. The constants \(c_1,\ldots,c_K\) are fixed, nonnegative reinforcement amounts and need not be equal. Write \(e_i\) for the vector with a 1 in coordinate \(i\) and 0 elsewhere. The update is
Thus, a success on treatment \(i\) increases the total urn size by \(c_i\), while a failure leaves it unchanged. After any update, the next assignment probability for treatment \(j\) is its updated count divided by the updated total. This is a particular generalized-urn rule; more general rules can use a different replacement vector for each treatment-response combination. The same calculation principle applies: update every coordinate, recalculate the total, and then divide.
For response-dependent calculations, assume the response probability on treatment \(i\) is \(\theta_i\), fixed over the patients considered, and that each patient's response, conditional on the assigned treatment, follows that treatment's response distribution. For the binary examples, \(\theta_i\) is the success probability. These assumptions let us calculate expected changes; an observed-path update itself only requires the specified assignment, response, and reinforcement rule.
How One Reinforcement Changes the Probabilities
Let the current count for the reinforced treatment \(i\) be \(y_i\), and let the current total be \(S\). A success on treatment \(i\) changes its probability from \(y_i/S\) to \((y_i+c_i)/(S+c_i)\). For another treatment \(j\), the numerator stays fixed while the total increases. Subtracting the probabilities before the update gives the exact effects:
If \(c_i>0\) and at least one other treatment has balls, a success increases the reinforced treatment's next allocation probability and decreases the probabilities of the other treatments. A larger \(c_i\) produces a larger increase for the reinforced treatment, holding the current state fixed. The changes across all treatment probabilities sum to zero, since the probabilities before and after an update each sum to 1.
Worked Example: One Success With Unequal Amounts
Suppose the urn starts at \(Y_t=(4,4,4)\) for three treatments, and the reinforcement amounts are \(c_1=3\), \(c_2=1\), and \(c_3=2\). Patient \(t\) is assigned to treatment 1 and has a success. Calculate the next allocation probabilities.
Initially, \(S_t=4+4+4=12\), so each treatment has probability \(4/12=1/3\). Because treatment 1 produced a success, add its treatment-specific amount of 3:
The next probabilities are \(7/15=0.4667\) for treatment 1, \(4/15=0.2667\) for treatment 2, and \(4/15=0.2667\) for treatment 3. They sum to 1. The reinforcement amounts for treatments 2 and 3 are not applied: the observed assignment was to treatment 1.
Following a Sequence of Assignments and Responses
An urn path is updated one patient at a time. At each step, use the composition that exists immediately before that assignment, and apply only the reinforcement triggered by that patient's assigned treatment and response. The allocation probability for a later patient is not calculated from the initial urn or from the response history alone; it is calculated from the current, updated composition.
Worked Example: Tracking Four Patients
Suppose there are three treatments, the initial urn is \(Y_1=(5,3,2)\), and the success reinforcements are \(c_1=4\), \(c_2=1\), and \(c_3=2\). Consider this hypothetical sequence: patient 1 is assigned to treatment 1 and succeeds; patient 2 is assigned to treatment 2 and fails; patient 3 is assigned to treatment 1 and succeeds; patient 4 is assigned to treatment 3 and succeeds. Find each relevant next-allocation probability and the final composition.
Initially the total is 10, so the first patient's treatment probabilities are \((5/10,3/10,2/10)=(0.5000,0.3000,0.2000)\). After the treatment 1 success, add 4 to its count. The state becomes \((9,3,2)\), with total 14, so the second patient's probabilities are \((9/14,3/14,2/14)=(0.6429,0.2143,0.1429)\).
Patient 2 is assigned to treatment 2 but fails, so there is no update: the state remains \((9,3,2)\). Patient 3 is therefore assigned using those same probabilities. After the specified treatment 1 success, add 4 to treatment 1. The state becomes \((13,3,2)\), total 18, and the fourth patient's probabilities are \((13/18,3/18,2/18)=(0.7222,0.1667,0.1111)\).
Patient 4 is assigned to treatment 3 and succeeds. Add \(c_3=2\), giving the final state \((13,3,4)\) and total 20. If another patient were to be assigned, the probabilities would be \((13/20,3/20,4/20)=(0.6500,0.1500,0.2000)\). These calculations specify the probabilities at each step; the stipulated sequence is an illustrative hypothetical path, not a claim that every listed assignment must occur.
Expected Composition Is Not Expected Allocation Probability
If treatment \(i\) is assigned with probability \(Y_{t,i}/S_t\), and its success probability is \(\theta_i\), then the probability of a treatment \(i\) success on the next patient is \((Y_{t,i}/S_t)\theta_i\). Under the success-only rule, the expected increment to count \(i\) is consequently \((Y_{t,i}/S_t)\theta_i c_i\). This gives a simple expression for the expected next composition. It does not, in general, give the expected next allocation probability by dividing expected counts by the expected total: in general, the expectation of a ratio is not the ratio of expectations.
Worked Example: Calculating an Expected Next Allocation Probability
Suppose \(Y_t=(6,3,1)\), so \(S_t=10\). The success probabilities are \((\theta_1,\theta_2,\theta_3)=(0.4,0.6,0.8)\), and the reinforcement amounts are \((c_1,c_2,c_3)=(2,1,4)\). Calculate the expected next composition and the expected next allocation probability for treatment 1.
The current assignment probabilities are \((0.6,0.3,0.1)\). The expected count increments are \(0.6(0.4)(2)=0.48\), \(0.3(0.6)(1)=0.18\), and \(0.1(0.8)(4)=0.32\). Thus the expected next composition is \((6.48,3.18,1.32)\), with expected total \(10.98\). The ratio \(6.48/10.98\) is about \(0.5902\), but it is not the expected next allocation probability.
To calculate that probability, enumerate the possible assignment-response cases for this one update. If treatment 1 is assigned, its probability is 0.6: a success occurs with probability 0.4 and makes treatment 1's next probability \(8/12\); a failure occurs with probability 0.6 and leaves it at \(6/10\). The conditional expected next probability in this case is \(0.4(8/12)+0.6(6/10)=0.6267\).
If treatment 2 is assigned, the corresponding expected probability is \(0.6(6/11)+0.4(6/10)=0.5673\). If treatment 3 is assigned, it is \(0.8(6/14)+0.2(6/10)=0.4629\). Weighting these conditional expectations by their assignment probabilities gives
The expected next allocation probability for treatment 1 is therefore about 0.5925 under the stated model, not the ratio \(6.48/10.98\). The calculation accounts for the different possible next states and their different totals.
Assumptions, Checks, and Alternatives
The proportional-allocation rule requires a positive total urn size. For the success-only rule, counts cannot become negative, and a positive initial count for each treatment ensures each treatment has positive allocation probability at every finite step. If a treatment starts with zero balls and no rule can add balls to it, it can never be assigned; this is a structural exclusion, not merely a small probability. In a more general replacement design that removes balls, nonnegative updated counts and a positive total must be checked after every possible update.
A reliable calculation has three checks. First, only the replacement specified by the assigned treatment and observed response is applied. Second, the updated total equals the sum of the updated coordinates. Third, the next allocation probabilities are nonnegative and sum to 1. For example, after a success that adds \(c_i\), the new total must be \(S+c_i\), not \(S+c_j\) for an unassigned treatment. These checks catch many errors before a longer sequence is calculated.
| Approach | What determines the update or assignment | Distinction from unequal reinforcement |
|---|---|---|
| Equal reinforcement | A success on any treatment adds the same specified number of balls to its own treatment. | The reinforced count changes by the same amount regardless of the assigned treatment. |
| Unequal reinforcement | A success on treatment \(i\) adds \(c_i\) balls to that treatment. | The size of the change depends on which treatment was assigned. |
| Fixed allocation | Assignment probabilities are set in advance and do not depend on urn updates. | Observed responses do not alter later assignment probabilities through this urn rule. |
Unequal reinforcement can make allocation more responsive to success on some treatments than on others, but that feature is a property of the chosen design. It is not, by itself, evidence that one treatment is more effective. The calculations also rely on the specified response probabilities when projecting expected changes; if response probabilities vary over time or depend on patient characteristics, the simple fixed-\(\theta_i\) calculation may not describe the design. Finally, a one-step increase in a treatment's allocation probability does not imply that its probability will keep rising on every subsequent patient: later assignments and responses can change the urn in different directions.
Common Mistakes
- Applying every treatment's reinforcement at once. A success on treatment 1 triggers \(c_1\), not \(c_1+c_2+c_3\). The response and assigned treatment jointly determine the update.
- Changing the numerator but not the denominator. Adding balls changes both the reinforced treatment's count and the total. Every next allocation probability uses the updated total.
- Using the equal-reinforcement amount for all arms. With unequal reinforcement, a success on treatment \(i\) adds \(c_i\), even when another treatment has a different reinforcement amount.
- Taking a ratio of expected values as an expected probability. The ratio of expected count to expected total need not equal the expected ratio. To find an expected next allocation probability, average the probabilities over possible next states.
Check Your Understanding
Answer each question, then compare your reasoning with the brief solutions.
- An urn has counts \((2,5,3)\). A success on treatment 2 adds 2 balls to treatment 2. What is the updated state and the next allocation probability for treatment 2?
- For a state with total \(S\), treatment \(i\) has count \(y_i\) and receives \(c_i>0\) balls after a success. State the change in its allocation probability and explain its sign when another treatment has at least one ball.
- An urn has state \((4,2)\), and the rule adds 3 balls to the assigned treatment only after success. What is the next state after assigning treatment 1 and observing failure?
- Why is the ratio of expected next count to expected next total generally not enough to find the expected next allocation probability?
Solutions. (1) The updated state is \((2,7,3)\), total 12, so treatment 2's next probability is \(7/12=0.5833\). (2) The change is \(\frac{c_i(S-y_i)}{S(S+c_i)}\), which is positive when \(c_i>0\) and \(S-y_i>0\). (3) A failure causes no update, so the next state remains \((4,2)\). (4) The next probability is a ratio that varies across possible next states; in general, averaging those ratios does not equal the ratio of the averages.