From a Protocol to an Executable Mechanism
A response-adaptive protocol describes how assignment probabilities change as responses accrue. To generate reference assignments, that description must be executable: at each decision time, it must specify the information available, the probability assigned to each treatment, how the random assignment is made, and which newly available outcomes update the allocation rule. A vague instruction such as “favor the better-performing treatment” is not enough. Different implementations of that instruction can generate different assignment distributions.
The goal here is to reconstruct the assignment mechanism, not to recover the random numbers used in the original trial. The reconstruction needs the protocol’s decision rules and the relevant trial history. For a reference run, it also needs the outcomes that the rule should use in that run. Under a sharp null hypothesis, each participant’s outcome is fixed regardless of treatment, so the observed outcome for that participant can be supplied to the rule after the participant’s simulated assignment.
Write \(A_i\) for participant \(i\)’s assignment, \(H_i\) for the information available immediately before that assignment, and \(q_i(a\mid H_i)=P(A_i=a\mid H_i)\) for the protocol’s conditional assignment probability. The earlier material on sequential assignment probabilities establishes the importance of conditioning on the history actually available at each decision. Reconstructing the algorithm makes that conditioning operational.
Specify the State, Timing, and Random Draw
Before writing code or calculating probabilities, translate the protocol into an ordered set of decisions. The state \(H_i\) may include treatment-specific response counts, covariates, prior assignments, enrollment time, pending outcomes, and any deterministic variables used by the rule. Include a quantity only if the protocol uses it, and do not silently omit one that it does use.
For a two-treatment rule, a common implementation draws \(U_i\sim\mathrm{Uniform}(0,1)\), independently at each assignment, and assigns treatment A when \(U_i<q_i(A\mid H_i)\); otherwise it assigns treatment B. This convention makes the random operation explicit. The protocol must also settle boundary cases, such as what happens if \(U_i\) equals the threshold. With a continuous uniform draw, equality has probability zero, but implementations should still use a consistent rule.
The state update is a separate operation from the assignment draw. If a response is not yet available, it cannot update the rule at that decision time. If outcomes arrive in batches, the update must occur at the specified batch time, rather than being applied immediately in a replay merely because the outcome is known to the analyst.
Initialize all counts, response summaries, covariate information, and pending-outcome records exactly as specified by the protocol.
Use only information available before the current assignment to compute \(q_i(a\mid H_i)\) for each treatment.
Apply the stated random operation, using a fresh random draw where the protocol requires one, and record the assignment and its probability.
When a response becomes available, update the state using the protocol’s exact rule, then proceed to the next decision.
This order can be summarized as a probability factorization along the realized sequence: the chance of a complete assignment sequence, under a specified sequence of histories, is the product of its successive conditional assignment probabilities. The history itself depends on earlier assignments and on outcomes available under the rule.
The factorization follows by repeated application of the conditional-probability chain rule. It does not license substituting probabilities calculated from a different history: each factor must use the state that would actually precede that assignment.
Worked Example: Replaying a Response-Rate Rule
Worked Example: Replaying a Response-Rate Rule
Consider a hypothetical two-treatment trial with binary responses. Let \(s_j\) be the number of observed successes and \(n_j\) the number of observed responses in treatment \(j\). The protocol calculates a smoothed response rate \(r_j=(s_j+1)/(n_j+2)\), then assigns the next participant to A with probability \(q(A)=r_A/(r_A+r_B)\). Otherwise, the participant receives B. Responses are available before the next participant is assigned. For a short illustration, suppose the outcomes supplied to the replay, in participant order, are 1, 0, and 1, where 1 denotes success. Use fresh draws \(U_1=0.32\), \(U_2=0.80\), and \(U_3=0.41\).
Initially \(s_A=s_B=n_A=n_B=0\), so \(r_A=r_B=1/2\), and \(q_1(A)=(1/2)/(1/2+1/2)=0.5000\). Since \(U_1=0.32<0.5000\), assign participant 1 to A. The supplied response is a success, so update to \(s_A=1,n_A=1\); B still has \(s_B=0,n_B=0\).
Before participant 2, \(r_A=(1+1)/(1+2)=2/3\), while \(r_B=(0+1)/(0+2)=1/2\). Thus \(q_2(A)=(2/3)/(2/3+1/2)=(2/3)/(7/6)=4/7\approx0.5714\). Since \(U_2=0.80\) is not less than 0.5714, assign participant 2 to B. The supplied response is a failure, so update \(n_B=1,s_B=0\).
Before participant 3, \(r_A=2/3\) and \(r_B=(0+1)/(1+2)=1/3\). Therefore \(q_3(A)=(2/3)/(2/3+1/3)=2/3\approx0.6667\). Since \(U_3=0.41<0.6667\), assign participant 3 to A. The reconstructed assignments are A, B, A. Each probability was calculated from the responses available before that participant’s assignment, not from a final summary of all three outcomes.
In a sharp-null reference run, the response sequence in this example can be interpreted as the outcomes that would be observed for these participants under either treatment. The same fixed outcome for each participant is supplied to the allocation rule after that participant’s simulated assignment. The assignment sequence can therefore differ across runs, and the treatment-specific response summaries can differ too, even though the participant-level outcomes supplied under the sharp null are held fixed.
Worked Example: Respecting Delayed Outcomes
Worked Example: Respecting Delayed Outcomes
Suppose a hypothetical protocol uses the same smoothed-rate rule as above, but responses are reviewed in pairs: outcomes for participants 1 and 2 are not used until both have been recorded. At the start, both treatments have no responses. Participant 1 is assigned using \(q_1(A)=0.5\), and receives a success. Participant 2 is assigned before that success is released to the allocation rule.
Because no response has yet been released, the probability for participant 2 remains \(q_2(A)=0.5\), even though the analyst knows participant 1’s response. If a fresh draw is \(U_2=0.30\), participant 2 is assigned to A. Suppose participant 2’s response is a failure. Only now does the batch update occur: treatment A has one success and one failure, so \(s_A=1,n_A=2\); treatment B still has \(s_B=0,n_B=0\).
For participant 3, \(r_A=(1+1)/(2+2)=1/2\) and \(r_B=(0+1)/(0+2)=1/2\), giving \(q_3(A)=0.5\). Updating immediately after participant 1 instead would have used \(r_A=2/3\), \(r_B=1/2\), and \(q_2(A)=4/7\). That is a different mechanism. In this protocol, the delayed release is part of the assignment rule, not an administrative detail that can be ignored.
This example also illustrates why the state should record which outcomes are pending. A reconstruction that stores only the outcomes eventually observed, but not their availability times, cannot reliably reproduce the protocol.
Worked Example: Checking a Probability Log
Worked Example: Checking a Probability Log
Consider a hypothetical two-treatment protocol with a rule that assigns A with probability \(q(A)=0.5+0.1(s_A-s_B)\), restricted to the interval from 0.2 to 0.8 if necessary. The protocol starts with both success counts at zero. Participant 1 receives a success in A. Before participant 2, the correct probability is \(0.5+0.1(1-0)=0.60\).
Suppose the assignment record says participant 2 had probability 0.50, was assigned to A, and used a logged draw \(U_2=0.42\). The assignment is compatible with the logged probability because \(0.42<0.50\); however, the logged probability is not compatible with the stated protocol and history. The rule implies 0.60, not 0.50. The discrepancy points to a possible update, logging, or implementation error. Simply checking whether the realized assignment could occur under the logged probability would miss it.
If instead the record showed probability 0.60 and draw 0.42, the recorded assignment to A would be consistent with the draw rule. That consistency does not by itself prove the whole implementation correct: the starting state, outcome timing, and probability calculation must also be checked at every decision.
What Must Be Fixed in a Reference Run?
For a reference assignment sequence, preserve the elements the design holds fixed and regenerate the random operations the design makes random. Under a sharp null, observed participant-level outcomes are held fixed as assignment-independent outcomes; the adaptive allocation rule is then rerun sequentially using those outcomes when they become available. Baseline covariates and enrollment order are likewise held fixed when the protocol conditions on them. The allocation probabilities are recalculated as the replayed history changes.
Do not hold fixed the original treatment labels while also claiming to generate new assignments. Nor should the observed final allocation totals generally be imposed as a constraint unless the protocol itself enforces those totals. A sequential adaptive rule may produce different totals in different runs. The reference procedure must reproduce that rule, rather than replace it with a simpler procedure that happens to preserve the observed number assigned to each arm.
| Approach | What it does | Key distinction |
|---|---|---|
| Replay the adaptive algorithm | Recomputes each probability from the replay’s current history and draws each assignment in sequence. | Matches a response-adaptive mechanism when its state and timing are specified correctly. |
| Shuffle treatment labels | Permutes the observed labels without rerunning the probability updates. | Generally does not reproduce a rule whose assignment probabilities depend on past responses. |
| Use fixed allocation probabilities | Draws each assignment from a constant probability, or conditions on fixed group totals. | Changes the mechanism if the actual protocol adapted over time or did not fix totals. |
These are not interchangeable ways to generate assignments. A replay is only as accurate as its specification. If the protocol is incompletely documented, there may be several plausible reconstructions; choosing one silently does not resolve that uncertainty.
Diagnostics and Common Failure Modes
- Wrong update time. Applying an outcome before it was available changes later probabilities. Check each decision against the protocol’s outcome-release schedule.
- Incomplete state. Omitting a count, covariate, enrollment-time variable, or pending response used by the rule can produce plausible but incorrect probabilities. Write down the full state before implementation.
- Probability and draw mismatch. Recalculate whether the recorded assignment follows the stated threshold convention. For example, under the rule “assign A if \(U_i<q_i(A\mid H_i)\),” a draw of 0.42 and probability 0.60 imply A.
- Unspecified boundaries or truncation. If probabilities are capped, rounded, or treated specially at a boundary, code that operation explicitly. Rounding a probability before the draw can alter assignments near the threshold.
- Using a final outcome summary too early. The allocation rule uses the history available at the time, not information from the end of the trial.
- Confusing regeneration with recovery. New random draws generate valid replayed assignments under the specified mechanism; they do not reveal the unobserved original random numbers from treatment labels alone.
Check Your Understanding
Answer each question, then compare your reasoning with the brief solutions below.
- A protocol assigns A with probability 0.65 and uses the rule “assign A if \(U_i<0.65\).” If \(U_i=0.72\), what is the assignment?
- A response is known to the analyst but the protocol releases it only after the next assignment. Should it affect the next assignment probability? Why?
- Under a sharp null, what participant-level outcome information is held fixed in a reference replay, and what is regenerated?
- A log reports probability 0.55 and draw 0.40, followed by assignment to A. What can this check establish, and what can it not establish by itself?
Solutions. (1) Assign B, because \(0.72\) is not less than \(0.65\). (2) No. The allocation rule can use only information available under its specified release schedule, so the response remains pending for that decision. (3) The participant-level outcomes are treated as fixed and assignment-independent under the sharp null; assignments and their sequential probabilities are regenerated by replaying the algorithm. (4) The draw is consistent with A under the logged probability and threshold rule, since \(0.40<0.55\). This check alone does not verify that 0.55 was calculated from the correct state, that the state was updated at the correct time, or that the implementation followed the full protocol.