Coding Success and Failure Correctly

Why the Success–Failure Distinction Matters

In a play-the-winner urn, an observed outcome changes the urn that will determine later treatment assignments. A success reinforces the treatment that produced it; a failure reinforces the other treatment. Reversing either outcome coding or treatment labels therefore does more than misdescribe the past: it changes the probabilities used to assign future participants.

This tutorial focuses on a two-arm urn and on checking each update against the assigned treatment and the observed response. The update rule must be fixed before coding it. In particular, “success” must mean the prespecified favorable outcome, not whichever outcome happens to be represented by a convenient value in a data file.

The state records the urn composition, not the number of participants who have received each treatment and not the number of successes observed. For example, a state of \((5,3)\) means that five balls represent \(A\) and three represent \(B\). Given this state, the next assignment is \(A\) with probability \(5/8\) and \(B\) with probability \(3/8\). Those probabilities follow from drawing one ball uniformly from the urn.

The update rule assumes that the assigned treatment is recorded correctly, the binary response can be classified as success or failure under a prespecified definition, and the reinforcement size and update convention are applied consistently. It also assumes the drawn ball is returned before reinforcement. If a response is missing or cannot be classified, it is not automatically a failure: the protocol must specify how such cases are handled.

The Four Updates to Keep Separate

The safest audit is to ask two questions for every participant: Which treatment was assigned? and Was the observed outcome success or failure? Only then should the urn be changed. The resulting cases are:

A useful verbal check is that a success adds balls matching the assigned treatment, whereas a failure adds balls matching the other treatment. This is not the same as “success always adds \(A\)” or “failure always adds \(B\).” The treatment assigned determines which color corresponds to “same” and which to “other.”

A Compact Derivation for Auditing Updates

The four cases can be written as one rule, which is helpful when checking code that processes many records. Let \(Y_t=1\) denote success and \(Y_t=0\) denote failure, and let \(Z_t\) denote the treatment assigned to participant \(t\). Define \(\mathbf{1}\{\cdot\}\) to equal 1 when its condition is true and 0 otherwise. The state after the response is:

To justify the first equation, \(A\) balls are added in exactly two cases: a success on \(A\), because the assigned treatment is reinforced, or a failure on \(B\), because the other treatment is reinforced. The second equation follows by exchanging \(A\) and \(B\). In every case exactly one indicator is 1, so the total number of balls increases by \(r\):

This total-size check can reveal an implementation problem, such as adding reinforcement twice or failing to add any balls. It cannot by itself detect a reversed outcome code: a reversed update can still add exactly \(r\) balls. The arm-specific state must also be checked against the assignment and outcome.

Worked Path: Keeping Assignment and Outcome Labels Aligned

A common source of error is not the update formula itself but a mismatch among the treatment assigned, the response record, and the arm names used by the program. The next example follows a short hypothetical path, with \(r=1\), and makes every state explicit.

If a data file stores “success” as 0 and “failure” as 1, that is not automatically wrong. It is wrong only if the update code assumes the opposite mapping. A separate danger is swapping the arm labels—for example, treating a database code of 1 as \(A\) in one step and as \(B\) in another. In the path above, if participant 2’s assignment were accidentally read as \(A\), the success update would add an \(A\) ball, producing \((3,2)\) rather than \((2,3)\). The total would still increase by one, so the assignment label must be audited directly.

Diagnostics, Alternatives, and Limits

A practical audit should preserve enough information to reconstruct each update: the state before the draw, the drawn ball or assigned treatment, the response as observed, the success/failure mapping, the reinforcement size, and the state after reinforcement. Compare the recorded post-update state with the applicable row of the four-case table. Investigate discrepancies rather than silently forcing the state to match.

• Reversed response coding: verify what each stored response value means, then compare that mapping with the code’s success and failure branches.
• Swapped arm labels: compare treatment names and codes across the assignment record, outcome record, and urn-state variables.
• Wrong reinforcement size: check that exactly \(r\) balls are added, not \(r+1\), \(2r\), or a value from a different setting.
• Missing or repeated update: verify that each eligible participant contributes one update and that an outcome is not processed twice.
• Unexpected total: check whether the drawn ball was returned and whether reinforcement was added once. A changing total may be intentional under a different rule, but it is not the rule defined here.

The table distinguishes a coding mistake from a different allocation design. A fixed allocation rule is not an incorrectly coded urn; it simply does not use response-based reinforcement. Likewise, reversing which outcome is called success may be coherent only if the scientific endpoint and the entire update rule are deliberately redefined in advance. Silently changing the coding convention midway through a trial does not implement a consistent play-the-winner rule.

The urn update also does not establish that an outcome is clinically meaningful, that responses are measured without error, or that the adaptive design has desirable operating characteristics. Those are separate design and analysis questions. This tutorial’s checks verify only that the specified two-arm update is translated correctly into states. In particular, checking arithmetic cannot validate the prespecified definition of success or repair a treatment assignment that was recorded incorrectly.