Changing the Play-the-Winner Reinforcement Size

Why Reinforcement Size Matters

In a play-the-winner urn, the current mix of balls determines the chance that the next patient is assigned to each treatment. A response changes that mix, so it can change subsequent allocation probabilities. The reinforcement size specifies how many balls the rule adds after each response. Adding several balls instead of one can therefore make a single response affect later assignments more strongly.

This tutorial uses the two-arm rule in which a successful response reinforces the treatment just assigned, while a failure reinforces the other treatment. The patient’s drawn ball is returned to the urn before the reinforcement is added. That return matters: the update adds balls to the current counts rather than replacing or removing the drawn ball.

Definition: Let \(a\) and \(b\) be the positive integer numbers of balls for treatments A and B in the urn immediately before an assignment. The state is \((a,b)\). One ball is drawn at random, its treatment is assigned, and the ball is returned. For a reinforcement size \(m\), a successful response adds \(m\) balls of the assigned treatment’s colour; a failure adds \(m\) balls of the other colour. Here \(m\) is a positive integer.

The rule describes the randomization mechanism, not the chance of a patient responding successfully. No assumption that the treatments have equal success probabilities—or that either treatment is more effective—is needed to calculate an allocation probability from a specified urn state. In an actual trial, the observed response determines which update applies.

The Update and the Next Allocation Probability

If the urn is in state \((a,b)\), the probability of assigning the next patient to A is the fraction of balls that are for A. The corresponding probability of assigning the patient to B is the fraction for B.

P(\text{next assignment is A}\mid (a,b))=\frac{a}{a+b}, \qquad P(\text{next assignment is B}\mid (a,b))=\frac{b}{a+b}.

To update the state, first identify the treatment assigned by the draw, then apply the response rule. For example, if the draw assigns treatment A, a success adds \(m\) A balls and a failure adds \(m\) B balls. Returning the drawn ball does not add another ball to the count: it was already included in the state before the draw.

Update rule: With reinforcement size \(m\), the four possible updates are

\begin{aligned} \text{A assigned, success:} &\quad (a,b)\longrightarrow(a+m,b),\\ \text{A assigned, failure:} &\quad (a,b)\longrightarrow(a,b+m),\\ \text{B assigned, success:} &\quad (a,b)\longrightarrow(a,b+m),\\ \text{B assigned, failure:} &\quad (a,b)\longrightarrow(a+m,b). \end{aligned}

The next assignment probability is calculated from the new state, not the state that preceded the completed patient’s assignment.

The rule can also be summarized by whether the response succeeds: success adds \(m\) balls of the assigned treatment’s colour; failure adds \(m\) balls of the opposite colour. This summary is useful, but keeping the four cases visible helps prevent reversing the update when treatment B is drawn.

Worked Example: Three A Balls Added After Success

Suppose a hypothetical trial has an urn state of \((4,3)\), with four A balls and three B balls, and uses reinforcement size \(m=3\). A patient draws an A ball and has a successful response. Find the updated state and the probability of assigning the next patient to A.

Before the draw, the chance of assignment to A is \(4/(4+3)=4/7\). The A ball is returned. Because the response is successful on A, three A balls are added:

(4,3)\longrightarrow(4+3,3)=(7,3).

The updated urn contains \(7+3=10\) balls. Therefore the next patient’s conditional probability of assignment to A is

\frac{7}{7+3}=\frac{7}{10}=0.700.

The probability has risen from \(4/7\approx0.571\) before this response to \(0.700\) afterward. The increase follows from adding three A balls; the returned drawn ball does not create an additional change in the counts.

Worked Example: Opposite-Colour Reinforcement After Failure

Use the same hypothetical starting state, \((4,3)\), and reinforcement size \(m=3\). This time the patient is assigned to A and has a failure. Find the updated state and the next probability of assignment to A.

A failure after assignment to A adds three balls of the opposite colour, B. Thus

(4,3)\longrightarrow(4,3+3)=(4,6).

The new total is \(4+6=10\), so the next allocation probability for A is

\frac{4}{4+6}=\frac{4}{10}=0.400.

The next probability of A is lower than its starting value, \(4/7\approx0.571\), because the failure added B balls while leaving the A count unchanged. In particular, this rule does not remove A balls after failure.

Why the Total Increases by the Reinforcement Size

The urn update has a simple arithmetic check. Before an assignment, there are \(a+b\) balls. The drawn ball is returned, so this operation leaves the total unchanged. The response then adds exactly \(m\) balls of one colour. Consequently every completed assignment and response increases the total by \(m\), regardless of which arm was assigned or which response occurred.

(a+b)_{\text{after update}}=(a+b)+m.

This also gives a check after several completed patients. If the initial total is \(T_0\) and \(t\) assignments have each been followed by an update, then the total is \(T_0+tm\). This statement assumes that the drawn ball is returned and that no other balls are removed or added. It applies to the total, not to either individual arm count: which colour receives the \(m\) balls depends on the assignment and response.

Justification: In each of the four cases in the update rule, one arm count increases by \(m\) and the other count is unchanged. Therefore the new total is either \((a+m)+b\) or \(a+(b+m)\), both equal to \(a+b+m\). Repeating this argument for \(t\) updates gives \(T_0+tm\).

Worked Example: Comparing Reinforcement Sizes

Suppose a hypothetical urn begins in state \((2,5)\), and the next patient is assigned to A and responds successfully. Compare the next patient’s probability of assignment to A under reinforcement sizes \(m=1\) and \(m=4\).

With \(m=1\), one A ball is added. The updated state is \((2+1,5)=(3,5)\), so

P(\text{next assignment is A})=\frac{3}{3+5}=\frac{3}{8}=0.375.

With \(m=4\), four A balls are added. The updated state is \((2+4,5)=(6,5)\), so

P(\text{next assignment is A})=\frac{6}{6+5}=\frac{6}{11}\approx0.5455.

Before the response, the chance of A was \(2/(2+5)=2/7\approx0.2857\) under either rule. A larger reinforcement produces a larger increase here because the successful response adds more A balls. The allocation probability is still calculated from the resulting composition; it is not equal to the reinforcement size or to the number of balls added.

Calculating an Update Reliably

For any one completed patient, the calculation can be organized into three steps. This procedure keeps the response rule separate from the probability calculation and works for either treatment.

Record the state and assignment.
Write the pre-update counts \((a,b)\), then identify which treatment the draw assigned.

Apply the response rule.
Add exactly \(m\) balls of the assigned treatment’s colour after success, or of the opposite colour after failure. Keep the other count unchanged.

Recalculate from the updated state.
Divide the updated A count by the updated total for the next probability of A; divide the updated B count by that total for B.

The two next-allocation probabilities must sum to one. If the updated state is \((a',b')\), then \(a'/(a'+b')+b'/(a'+b')=1\). This is a useful check on the denominator and on which count is used for each arm.

What Reinforcement Size Changes—and What It Does Not

Feature	Effect of using a larger \(m\)
Number of balls added after a response	Increases from \(m\) balls to the specified larger count.
Total balls after each update	Increases by the chosen \(m\) at each completed update.
Arm counts after an update	Only the colour selected by the response rule increases, by exactly \(m\).
Next allocation probability	Must be recalculated from the updated counts; it depends on the whole urn composition.
Probability of a patient response	Not specified by the urn update rule; it is not calculated from the ball counts.

The one-ball rule is the special case \(m=1\). A larger \(m\) makes each response produce a larger count change, but it does not generally multiply the next allocation probability by \(m\). For example, the probability is a ratio with the updated total in its denominator. Nor does a larger reinforcement guarantee that treatment A becomes more likely: after a failure on A, it is B that receives the additional balls.

Compared with a fixed allocation probability that does not respond to outcomes, a play-the-winner urn makes later allocation probabilities depend on the current composition, which in turn reflects earlier assignments and responses. Compared with one-ball reinforcement, a larger \(m\) changes the size of each individual update while preserving the same success/failure direction. These are different properties: adaptivity comes from using response-dependent updates, while reinforcement size controls the magnitude of each update.

Diagnostics, Pitfalls, and Limitations

A calculation is suspect if the new total does not equal the old total plus \(m\), if both arm counts changed in a single update, or if the two next-allocation probabilities do not sum to one. Check the assigned arm and response before doing arithmetic. In particular, success on B reinforces B, whereas failure on B reinforces A.

Wrong reasoning: “The patient drew an A ball, so the urn total increases by one when the ball is returned, and then increases by \(m\).” Correction: The drawn ball was already counted and is returned, so the draw-and-return step leaves the total unchanged. Only the \(m\) added balls increase it.
Wrong reasoning: “Failure on A means subtract \(m\) from the A count.” Correction: Under this rule, failure adds \(m\) balls of the opposite colour. It does not remove balls from the assigned arm.
Wrong reasoning: “After reinforcement, the probability of A is \(m/(a+b+m)\).” Correction: The numerator is the updated A count, not the number added. Use \(a'/(a'+b')\).
Wrong reasoning: “A larger reinforcement means the next patient must receive the treatment that just succeeded.” Correction: The probability can increase without becoming one. The next draw is random, and the other colour’s balls remain in the urn.

This calculation is conditional on a specified state and a specified response. It does not, by itself, determine the response probabilities, establish that either treatment is superior, or quantify long-run trial performance. Also, a larger reinforcement increases the urn total more quickly, so the same number of completed patients under different reinforcement sizes does not generally produce the same state. Conclusions about allocation probabilities must use the actual counts, not only the number of patients treated.

Key takeaway: Return the drawn ball, add \(m\) balls of the assigned colour after success or the opposite colour after failure, and compute the next allocation probability from the updated counts. The total rises by exactly \(m\) per completed update.

Check Your Understanding

Try each calculation, then compare your work with the solutions below.

An urn is in state \((5,2)\). With \(m=2\), a patient is assigned to B and succeeds. What is the updated state and the next probability of assignment to A?
An urn is in state \((3,4)\). With \(m=3\), a patient is assigned to B and fails. What is the updated state and the next probability of assignment to B?
An urn starts with 4 A balls and 6 B balls. Each of five patients completes an update using \(m=2\). How many balls are in the urn afterward, regardless of the sequence of assignments and responses?
After a successful assignment to A, the state changes from \((2,5)\) to \((5,5)\). What reinforcement size was used, and what is the next probability of assignment to A?

Solutions. (1) Success on B adds two B balls, so \((5,2)\to(5,4)\). The next probability of A is \(5/(5+4)=5/9\approx0.5556\). (2) Failure on B adds three A balls, so \((3,4)\to(6,4)\). The next probability of B is \(4/(6+4)=0.400\). (3) The initial total is \(4+6=10\), and five updates each add two balls, giving \(10+5(2)=20\) balls. (4) The A count rose from 2 to 5, so \(m=3\). The next probability of A is \(5/(5+5)=0.500\).