Why replay the allocation rule?
In an adaptive trial, the probability of assigning the next participant can depend on outcomes observed so far. Under a sharp null of no treatment effect for any participant, the outcomes can be held fixed while the treatment assignments are varied. But each varied assignment path can produce a different history of arm-specific outcomes, and therefore different probabilities for later assignments. A valid randomization test must reproduce that sequence: assign a participant, update the trial’s estimates, and calculate the next probability using the updated history.
This tutorial works with a two-arm doubly adaptive biased coin (DBCD) design. It focuses on a sharp-null test of the difference in observed arm means, using the actual sequential allocation rule as the reference distribution. The key practical principle is simple: replay the rule on every candidate path; do not hold the observed path’s later probabilities fixed.
Specify the design and the null hypothesis
Let \(A_i=1\) denote assignment to treatment and \(A_i=0\) assignment to control for participant \(i\). Let \(Y_i(1)\) and \(Y_i(0)\) be that participant’s potential outcomes under treatment and control. The sharp null is
Under this null, the observed outcome \(Y_i\) is the same whichever arm participant \(i\) is assigned to. Thus the vector of outcomes is fixed across candidate assignment paths. This is stronger than a null of zero average treatment effect: it specifies no individual treatment effect.
For clarity, suppose the response is binary, with 1 denoting success and 0 denoting failure. After \(i-1\) assignments, let \(N_{T,i-1}\) and \(N_{C,i-1}\) be the numbers assigned to treatment and control, and let \(S_{T,i-1}\) and \(S_{C,i-1}\) be their success counts. Use the smoothed response estimates
The added 1 success and 1 failure give finite estimates even when an arm has no successes or no failures. Define the estimated target treatment proportion as
Write \(\pi_{T,i-1}=N_{T,i-1}/(i-1)\) and \(\pi_{C,i-1}=1-\pi_{T,i-1}\) for the current allocation proportions. In this tutorial, the DBCD probability of treatment assignment is
Here \(\mathcal H_{i-1}\) is the assignment and outcome history available before participant \(i\), and \(\gamma\geq 0\) controls the strength of correction toward the estimated target. At \(\gamma=0\), \(q_i=\rho_{T,i-1}\); larger values make departures of the current allocation proportions from the target more influential. A design must specify its initial assignments and any rule for updating estimates. We will assume the first two participants are assigned one to each arm, after which the displayed rule is used with \(\gamma=1\). This seed ensures both current allocation proportions are positive.
What must be replayed?
For each generated path, the first two assignments follow the specified seed. At each subsequent step, calculate \(q_i\) from the assignments and outcomes already on that path, generate \(A_i\), and then update the arm counts and success counts using the fixed outcome \(Y_i\). Do not update the estimates using the observed assignment history unless the generated path happens to match it.
A useful statistic for binary outcomes is the difference in sample success proportions,
For a two-sided test, compare \(|T|\) with the observed absolute statistic \(|T_{\mathrm{obs}}|\). The randomization p-value is the probability, under the specified sequential assignment mechanism and the fixed outcomes, of obtaining a statistic at least as extreme:
The conditioning notation emphasizes that the outcome vector and design are fixed. The probability is over assignment paths, including the randomness at every adaptive step.
Worked Example: Calculating the Next DBCD Probability
Suppose the two seeded participants give one treatment success and one control failure. Thus \(N_T=N_C=1\), \(S_T=1\), and \(S_C=0\). Calculate the treatment probability for participant 3 under the specified rule with \(\gamma=1\).
The smoothed response estimates are
Therefore \(\rho_T=2/3\), \(\rho_C=1/3\), and the current allocation proportions are \(\pi_T=\pi_C=1/2\). Substitution into the DBCD formula gives
The treatment probability is 0.8000, not the target proportion \(2/3\): with \(\gamma=1\), the rule also responds to the current allocation proportions. This calculation uses only the history available before participant 3 is assigned.
Exact randomization distribution in a small trial
The path probability is obtained by multiplying the sequential conditional assignment probabilities along that path. This follows from the probability chain rule: each factor is calculated using the history generated by the preceding assignments and the fixed outcomes. In general, those factors differ across paths because the histories differ.
The seed assignments in this example have probability one. The following example enumerates all remaining paths, so its randomization p-value is exact for the stated design and null.
Worked Example: Enumerating All Paths and Testing the Sharp Null
Consider four participants with fixed binary outcomes, in order, \(\mathbf Y=(1,0,1,0)\). The first participant is seeded to treatment and the second to control. The observed assignments for participants 3 and 4 are treatment and control, respectively. Use \(\gamma=1\) and the statistic \(T=\hat p_T-\hat p_C\).
For the observed path, the treatment outcomes are \(1,1\) and the control outcomes are \(0,0\), so
At participant 3, the seed history is the one-success treatment arm and the one-failure control arm. As calculated above, \(P(A_3=1)=8/9\) and \(P(A_3=0)=1/9\).
If participant 3 is assigned to treatment, the fixed outcome \(Y_3=1\) is added to that arm. The resulting estimates are \(\hat p_T=(2+1)/(2+2)=3/4\) and \(\hat p_C=1/3\), so \(\rho_T=9/13\), while \(\pi_T=2/3\). The next treatment probability is
If instead participant 3 is assigned to control, that arm then has one success and one failure. The estimates are \(\hat p_T=2/3\) and \(\hat p_C=1/2\), giving \(\rho_T=4/7\) and \(\pi_T=1/2\). Consequently,
The four paths, their probabilities, and their statistics are:
| Assignments for participants 3 and 4 | Path probability | \(T\) |
|---|---|---|
| Treatment, treatment | \((8/9)(81/113)=648/1017\) | \(2/3\) |
| Treatment, control | \((8/9)(32/113)=256/1017\) | \(1\) |
| Control, treatment | \((1/9)(16/25)=16/225\) | \(0\) |
| Control, control | \((1/9)(9/25)=9/225\) | \(2/3\) |
The probabilities sum to \(1\). For example, the first two sum to \(904/1017\), and the last two sum to \(1/9\), giving \(1\) overall. Only the observed treatment-control path has \(|T|\geq |T_{\mathrm{obs}}|=1\). Therefore the exact two-sided p-value is
At significance level \(\alpha=0.05\), this test does not reject the sharp null. The p-value is a tail probability under the sharp null and the specified design; it is not the probability that the null is true.
Monte Carlo replay for larger trials
The number of possible paths can grow exponentially with the number of randomized participants. When exact enumeration is impractical, generate \(B\) independent paths using the same sequential replay algorithm. For each path, retain the fixed outcomes in participant order, draw each assignment from the probability computed from that path’s current history, and calculate the terminal statistic. If \(b\) generated statistics are at least as extreme as the observed statistic, a common Monte Carlo p-value estimate is
The plus-one form includes the observed assignment as an additional reference draw and avoids reporting a zero p-value. Monte Carlo error is distinct from the trial’s randomization uncertainty: it is numerical uncertainty from using a finite number of simulated paths. A rough standard error is \(\sqrt{\hat p_{\mathrm{MC}}(1-\hat p_{\mathrm{MC}})/B}\), useful for judging whether more paths are needed near a decision threshold.
Worked Example: Reporting a Monte Carlo p-Value
Suppose a larger hypothetical DBCD trial has observed \(|T_{\mathrm{obs}}|=0.240\). Replay the design independently \(B=9{,}999\) times, recomputing probabilities at every step. Assume \(b=2{,}468\) paths have \(|T|\geq0.240\). Then
The approximate Monte Carlo standard error is
Thus the simulated tail probability is about 0.2469, with Monte Carlo standard error about 0.00431. This describes simulation precision, not a confidence interval for a treatment effect. The reported p-value remains far above 0.05 in this hypothetical calculation, although close decisions would merit enough simulation that Monte Carlo error cannot plausibly change the conclusion.
Justification, diagnostics, and alternatives
Under the sharp null, every participant’s outcome is known regardless of the assigned arm: it is the observed outcome. The design then specifies the conditional probability of every next assignment from the history so far. Multiplying these conditional probabilities gives the probability of each complete assignment path. Consequently, enumerating paths with those probabilities, or sampling paths from them, reproduces the assignment distribution under the null. Comparing the observed statistic with this distribution is therefore a randomization-based test for that sharp null.
Under the sharp null, retain each participant’s observed response for every candidate assignment path.
Apply the seed assignments, then calculate each new DBCD probability from that candidate path’s accumulated assignments and outcomes.
At the end of each path, calculate the same statistic used on the observed trial.
Sum exact path probabilities, or estimate their total by Monte Carlo, for statistics at least as extreme as observed.
Before trusting a result, check that the replay reproduces the protocol: initialization, estimator smoothing, target formula, \(\gamma\), any covariate dependence, and any stopping rule must match the trial. Check that each conditional probability is between 0 and 1, that complementary arm probabilities sum to 1, and that enumerated path probabilities sum to 1. For simulations, inspect code on a tiny case that can be enumerated exactly, and verify that the observed path itself has positive probability under the stated rule.
The central failure mode is to calculate later probabilities once from the observed history and reuse them for every simulated path. In the four-participant example, after participant 3 is assigned to treatment the next treatment probability is \(81/113\); after assignment to control it is \(16/25\). Replacing both with one fixed probability changes the path probabilities and therefore the reference distribution. Similarly, permuting assignments as though they came from a fixed-probability design generally ignores the adaptive mechanism. Conditioning on allocation counts is another possible randomization approach, but it defines a different reference distribution and must be justified for the design and test being used.
This procedure’s exact interpretation depends on the specified assignment mechanism, the sharp null, and the recorded outcome and assignment history being correctly represented. It does not test every weaker statement about an average effect. If outcomes are missing, treatment is not received as assigned, or the protocol deviates from the stated algorithm, the analyst must define a suitable null and assignment mechanism rather than silently replaying an inaccurate design. Also, extreme probabilities can make some paths very rare; this can slow Monte Carlo convergence even when the replay code is correct.
Compared with a large-sample test based on a reference distribution such as a normal approximation, replay uses the actual sequential randomization probabilities and can be especially valuable in small trials. Its cost is that a valid test requires a faithful, potentially computationally demanding replay. The randomization test is also specific to the sharp null: an average-effect question requires a different inferential target and is not answered merely by relabeling this p-value.
Check Your Understanding
Answer each question, then compare with the solutions below.
- Under a sharp null, what happens to a participant’s outcome when that participant’s assignment is changed in a replayed path?
- In the four-participant example, why are the probabilities for participant 4 different after the two possible assignments for participant 3?
- If \(B=4{,}999\) paths are generated and \(b=499\) are at least as extreme as observed, calculate the plus-one Monte Carlo p-value.
- State one diagnostic that can detect a replay implementation error.
- Does a sharp-null randomization p-value directly provide a test of a zero average treatment effect? Explain briefly.
Solutions. (1) It remains fixed at the observed outcome, because the sharp null states that the participant’s outcome is identical under either assignment. (2) The assignment changes the arm counts and response estimates, which changes the estimated target and the current allocation proportions; hence it changes \(q_4\). (3) \(\hat p_{\mathrm{MC}}=(499+1)/(4{,}999+1)=500/5{,}000=0.1000\). (4) For example, in a small case enumerate all paths and verify that their probabilities sum to 1 and agree with the replay algorithm; also verify that each pair of arm probabilities sums to 1. (5) No. The sharp null asserts no effect for every participant, whereas a zero average effect is a weaker statement that can allow individual effects to differ.