Deriving Play-the-Winner Transition Probabilities

Why Condition on Both Assignment and Response?

In a play-the-winner urn, the next assignment depends on the urn’s current composition, and the urn update depends on the response to the treatment just assigned. To find the probability of a next urn state, we must account for both parts of that process. The probability of drawing a treatment alone is not the probability of a particular next state: the patient’s response determines which color receives the added ball.

This tutorial derives the one-patient transition probabilities for the two-arm rule with one-ball reinforcement. The treatment-specific response probabilities are treated as given model quantities. The calculations do not estimate those probabilities or evaluate treatment efficacy.

Model and Assumptions

Call the treatments 1 and 2. Immediately before assigning patient \(t\), let the urn contain \(a\) balls for treatment 1 and \(b\) balls for treatment 2, where \(a+b>0\). Write the state as \(S_t=(a,b)\). The ball drawn determines the treatment assigned to that patient. The drawn ball is returned to the urn after the assignment, and exactly one additional ball is added after the response is observed.

Definition: Under the two-arm, one-ball play-the-winner rule, a success on treatment 1 adds one treatment-1 ball, and a failure on treatment 1 adds one treatment-2 ball. A success on treatment 2 adds one treatment-2 ball, and a failure on treatment 2 adds one treatment-1 ball. Thus the drawn ball is always returned, and the outcome determines the color of the one additional ball.

Let \(Y_t=1\) denote a success and \(Y_t=0\) a failure. The treatment-specific response probabilities are \(p_1\) and \(p_2\), where \(0\leq p_i\leq 1\). The response model assumes that, conditional on the treatment assigned and the history before the response, the success probability is \(p_i\) for treatment \(i\). In particular, these probabilities do not change with the current urn composition or earlier responses.

The derivation also assumes that the draw is made uniformly from the balls currently in the urn; each ball therefore has the same chance of being selected. The response is observed before the next patient is assigned, there is no missing response that alters the update, and the stated rule is applied after every patient. These assumptions make the urn composition sufficient for calculating the next-state probabilities. If responses depend on additional history, those variables may also be needed to describe the process.

Deriving the Transition Probabilities

Given state \((a,b)\), the probability of assigning treatment 1 is \(a/(a+b)\), and the probability of assigning treatment 2 is \(b/(a+b)\). There are four possible treatment-response paths. Each has a probability obtained by multiplying the treatment-assignment probability by the relevant conditional response probability.

Treatment drawn	Response	Probability of the path given state \((a,b)\)	Next state
1	Success	\(\frac{a}{a+b}p_1\)	\((a+1,b)\)
1	Failure	\(\frac{a}{a+b}(1-p_1)\)	\((a,b+1)\)
2	Success	\(\frac{b}{a+b}p_2\)	\((a,b+1)\)
2	Failure	\(\frac{b}{a+b}(1-p_2)\)	\((a+1,b)\)

The first and fourth paths both lead to \((a+1,b)\), so their probabilities must be added. The second and third paths both lead to \((a,b+1)\), and their probabilities must also be added. These are disjoint paths: for example, a patient cannot simultaneously receive treatment 1 and treatment 2.

\begin{aligned} P\{S_{t+1}=(a+1,b)\mid S_t=(a,b)\} &=\frac{a}{a+b}p_1+\frac{b}{a+b}(1-p_2)\\ &=\frac{a p_1+b(1-p_2)}{a+b},\\[6pt] P\{S_{t+1}=(a,b+1)\mid S_t=(a,b)\} &=\frac{a}{a+b}(1-p_1)+\frac{b}{a+b}p_2\\ &=\frac{a(1-p_1)+b p_2}{a+b}. \end{aligned}

There are no other possible next states under this rule. The returned ball leaves the original composition in place, and the single added ball increases exactly one of the two counts by one.

Key result: Given current state \((a,b)\), the one-step transition probability to \((a+1,b)\) is \(\{a p_1+b(1-p_2)\}/(a+b)\), and the probability to \((a,b+1)\) is \(\{a(1-p_1)+b p_2\}/(a+b)\). These conditional probabilities depend on both the urn composition and the treatment-specific response probabilities.

Justification Checks

A transition calculation should satisfy two basic checks. First, the possible next-state probabilities must sum to one. Adding the two expressions gives

\frac{a p_1+b(1-p_2)+a(1-p_1)+b p_2}{a+b} =\frac{a+b}{a+b}=1.

Second, every listed transition must obey the update rule. Each next state has total \(a+b+1\), exactly one more ball than the current state. A transition that leaves the total unchanged or adds two balls is inconsistent with the specified return-and-reinforce rule.

Because the assignment probabilities and response probabilities are determined by the current state and the stated model, these transitions also show that the state sequence is Markov under the assumptions above: conditional on the current state, the distribution of the next state does not require the earlier state history. This conclusion would not follow if, for example, response probability depended on an unrecorded feature of earlier assignments.

Worked Example: Combining Two Paths to the Same State

Suppose the urn contains 3 treatment-1 balls and 2 treatment-2 balls. Let \(p_1=0.70\) and \(p_2=0.40\). Find the probabilities of the two possible next states.

The treatment-assignment probabilities are \(3/5=0.60\) for treatment 1 and \(2/5=0.40\) for treatment 2. The four path probabilities are:

Treatment 1 success: \((3/5)(0.70)=0.42\), leading to \((4,2)\). Treatment 1 failure: \((3/5)(0.30)=0.18\), leading to \((3,3)\). Treatment 2 success: \((2/5)(0.40)=0.16\), leading to \((3,3)\). Treatment 2 failure: \((2/5)(0.60)=0.24\), leading to \((4,2)\).

Thus the two paths to \((4,2)\) combine to give \(0.42+0.24=0.66\). Equivalently,

P\{S_{t+1}=(4,2)\mid S_t=(3,2)\} =\frac{3(0.70)+2(1-0.40)}{5} =\frac{2.10+1.20}{5}=0.660.

The two paths to \((3,3)\) combine to give \(0.18+0.16=0.34\), or

P\{S_{t+1}=(3,3)\mid S_t=(3,2)\} =\frac{3(1-0.70)+2(0.40)}{5} =\frac{0.90+0.80}{5}=0.340.

The results sum to \(0.660+0.340=1.000\), and both next states contain six balls, one more than the initial five.

Worked Example: A Treatment with Few Balls Can Still Gain a Ball

Suppose the current state is \((1,4)\), with \(p_1=0.80\) and \(p_2=0.50\). Calculate the chance that the treatment-1 count increases.

Treatment 1 is drawn with probability \(1/5\); a success on it adds a treatment-1 ball. Treatment 2 is drawn with probability \(4/5\); a failure on it also adds a treatment-1 ball. Therefore,

\begin{aligned} P\{S_{t+1}=(2,4)\mid S_t=(1,4)\} &=\frac{1}{5}(0.80)+\frac{4}{5}(1-0.50)\\ &=0.160+0.400=0.560. \end{aligned}

The other next state occurs after treatment 1 fails or treatment 2 succeeds:

P\{S_{t+1}=(1,5)\mid S_t=(1,4)\} =\frac{1}{5}(0.20)+\frac{4}{5}(0.50) =0.040+0.400=0.440.

Although treatment 1 begins with only one ball, its count increases with probability \(0.560\) in this example. That probability is not simply the chance of drawing treatment 1: treatment-2 failures also reinforce treatment 1.

Worked Example: Equal Response Probabilities

Suppose the state is \((2,3)\) and both treatments have success probability \(0.60\), so \(p_1=p_2=0.60\). Find both next-state probabilities and explain why they need not equal the treatment-allocation probabilities.

Treatment 1 receives an added ball after a treatment-1 success or a treatment-2 failure. Hence

P\{S_{t+1}=(3,3)\mid S_t=(2,3)\} =\frac{2(0.60)+3(0.40)}{5} =\frac{2.40}{5}=0.480.

Treatment 2 receives an added ball after a treatment-1 failure or a treatment-2 success:

P\{S_{t+1}=(2,4)\mid S_t=(2,3)\} =\frac{2(0.40)+3(0.60)}{5} =\frac{2.60}{5}=0.520.

The assignment probabilities are \(0.40\) for treatment 1 and \(0.60\) for treatment 2, but these are not the transition probabilities. A state increment can result from either treatment, depending on its response. Equal success probabilities do not make the next urn update equal to the treatment draw.

Common Mistakes, Diagnostics, and Comparisons

A frequent error is to calculate the probability of adding a treatment-1 ball using only treatment-1 assignments. That omits failures on treatment 2, which also add a treatment-1 ball. The opposite-direction error is to treat a success on treatment 2 as reinforcement of treatment 1; under this rule it adds a treatment-2 ball. The safest approach is to list all four treatment-response paths before combining paths with the same destination.

Other useful diagnostics are straightforward:

Check that \(p_1\) and \(p_2\) are between 0 and 1 and that the treatment-draw probabilities use the current urn counts.
Check that the two next-state probabilities add to 1.
Check that each next state increases exactly one count by one.
Check that the response probability in each branch belongs to the treatment actually assigned.

These transitions describe a response-adaptive rule: the treatment assigned is sampled from the current urn composition, while that composition changes in response to outcomes. By contrast, complete randomization or a permuted-block scheme bases assignments on its randomization design rather than on the preceding patient’s response. Those designs have different allocation mechanisms; their assignment probabilities should not be substituted into the urn transition formula.

The derivation has important limits. It represents a binary response with fixed treatment-specific probabilities and the particular one-ball update rule stated above. If the response is delayed until after another patient is assigned, if outcomes are missing, or if the response probability changes over time, the four-path calculation may not describe the actual procedure without additional modeling. Also, a transition probability describes the urn’s next state under the assumed model; it is not evidence by itself that either treatment is more effective.

Check Your Understanding

For each question, identify the relevant paths before calculating. Solutions are included below for checking your work.

At state \((2,2)\), let \(p_1=0.75\) and \(p_2=0.25\). Find the probability of moving to \((3,2)\) and to \((2,3)\).
At state \((3,1)\), suppose \(p_1=0.40\) and \(p_2=0.80\). Which two treatment-response paths lead to \((3,2)\), and what is their combined probability?
Explain why a treatment-1 failure contributes to the transition probability for \((a,b+1)\), rather than \((a+1,b)\).
Suppose a calculation gives next-state probabilities \(0.62\) for \((a+1,b)\) and \(0.31\) for \((a,b+1)\). Name one immediate diagnostic failure, regardless of the response probabilities used.

Solutions. (1) Treatment 1 success and treatment 2 failure lead to \((3,2)\): \(\{2(0.75)+2(1-0.25)\}/4=(1.50+1.50)/4=0.750\). Treatment 1 failure and treatment 2 success lead to \((2,3)\): \(\{2(0.25)+2(0.25)\}/4=0.250\). The probabilities sum to 1. (2) Treatment 1 failure and treatment 2 success both lead to \((3,2)\). Their combined probability is \(\{3(1-0.40)+1(0.80)\}/4=(1.80+0.80)/4=0.650\). (3) A failure on treatment 1 adds a treatment-2 ball under the rule, so the count changes from \((a,b)\) to \((a,b+1)\). (4) The probabilities sum to \(0.62+0.31=0.93\), not 1, so at least one possible path or arithmetic contribution has been omitted or calculated incorrectly.