Adaptive Randomization With Data-Dependent Stopping

Why the Stopping Rule Belongs in the Test

In an adaptive trial, the analysis time can itself depend on accumulating assignments and responses. A trial might stop when an interim statistic crosses a boundary, when a response pattern meets a rule, or when a maximum sample size is reached. Once stopping depends on observed data, the observed endpoint is not simply a statistic measured at a fixed sample size. It is a statistic evaluated at a random time selected by the data.

For a randomization test, the reference distribution must represent the random process that generated the observed data. Under data-dependent stopping, this means applying the stopping rule again to every assignment path in the randomization distribution. Using the statistic from the observed stopping time while treating that time as if it had been fixed in advance can give the wrong tail probability.

Definition: Let \(H_t\) denote the trial history through time \(t\), including assignments and observed outcomes. A stopping time \(\tau\) is a rule for which the decision whether \(\tau \le t\) can be made using \(H_t\) alone. For a maximum of \(N\) stages, a typical rule is \(\tau=\min\{t\le N:g_t(H_t)=1\}\), with stopping at \(N\) if no earlier rule is met.

The focus here is the sharp null hypothesis that each unit’s outcome would be the same under every treatment assignment. Under this null, outcomes are fixed with respect to the randomized assignments; the assignment process is the source of randomness in the test. The test also requires a fully specified assignment mechanism, including how each assignment probability depends on the history. For a response-adaptive design, the probability used at the next assignment must be recalculated from the history of each simulated or enumerated path.

A further condition is easy to overlook: the outcome information needed to construct alternative paths must be available under the randomization framework being used. This is straightforward for a fixed cohort whose outcomes are observed regardless of the stopping decision, or for a design that explicitly specifies the outcome sequence under the sharp null. If stopping accrual means later participants are never enrolled and their outcomes are unknown, the observed data alone generally do not supply those unobserved outcomes. An exact conditional randomization test then requires additional design-based information or assumptions; it cannot be obtained simply by pretending those outcomes were observed.

Constructing the Stopped-Path Distribution

Let \(A_i\) indicate the assignment at stage \(i\), and let \(p_i(H_{i-1})=P(A_i=1\mid H_{i-1})\) be the specified assignment probability when the trial reaches that stage. Under the sharp null, replay a candidate assignment path, reveal the corresponding fixed outcomes, update the history, and apply the stopping rule. The statistic is evaluated at the stopping time generated by that path, not at the stopping time of the observed path.

For a deterministic stopping rule, the probability of a path that stops at \(\tau\) is the product of its assignment probabilities through \(\tau\). If stopping or continuation also involves a random decision, its conditional probability must be included in that product. These path probabilities form the randomization distribution.

P(A_1=a_1,\ldots,A_\tau=a_\tau) = \prod_{i=1}^{\tau} p_i(H_{i-1})^{a_i} \{1-p_i(H_{i-1})\}^{1-a_i}

For a prespecified statistic \(T_\tau\), an upper-tail randomization p-value is the probability, under this stopped-path distribution, of obtaining a value at least as large as the observed one. A two-sided p-value can be defined using a specified measure of extremeness, such as \(|T_\tau|\), with that same rule applied to every path. With discrete assignment distributions, including ties in the tail gives a valid, potentially conservative, test.

p_{\mathrm{rand}} = \sum_{\omega:\,T_{\tau(\omega)}(\omega)\,\ge\,T_{\mathrm{obs}}} P(\omega)

Here \(\omega\) ranges over complete assignment paths allowed by the design, and \(\tau(\omega)\) is the stopping time produced by that path. For a two-sided test based on absolute values, replace the tail condition by \(|T_{\tau(\omega)}(\omega)|\ge |T_{\mathrm{obs}}|\). The statistic and tail definition should be chosen before examining which path was observed.

Worked Examples

Worked Example: Enumerating a Stopped-Path Test

Consider a hypothetical fixed cohort of three units, each with outcome 1 under either treatment, as specified by the sharp null. Assignments are independent with probability \(1/2\) of treatment. Define \(S_t=\sum_{i=1}^t(A_i-1/2)Y_i\). Stop after stage 2 if \(|S_2|\ge 1\); otherwise continue to stage 3. Use \(|S_\tau|\) as the two-sided statistic. Suppose the observed assignments are \(A_1A_2=11\), so the trial stops at stage 2 and \(S_\tau=1\).

Each of the eight three-assignment sequences has probability \(1/8\), but paths stop at different times. Sequences 11 and 00 stop after stage 2, with statistics 1 and \(-1\). The other four sequences continue. For 01 and 10, the stage-2 score is 0; the third assignment makes the terminal score either \(1/2\) or \(-1/2\). Thus the stopped-path distribution has probability \(1/4\) at each of 1 and \(-1\), and probability \(1/4\) at each of \(1/2\) and \(-1/2\).

The two-sided p-value is therefore \[ P(|S_\tau|\ge 1)=P(S_\tau=1)+P(S_\tau=-1)=\frac14+\frac14=0.50. \] This is the test for the statistic evaluated at the stopping time generated by each path.

For contrast, if the statistic were evaluated at the fixed horizon of three stages regardless of the stopping rule, \(S_3\) would be \(3/2\) or \(-3/2\) with probability \(1/8\) each, and \(1/2\) or \(-1/2\) with probability \(3/8\) each. Its two-sided tail probability at the observed threshold 1 would be \(2/8=0.25\). That is a different reference distribution and answers a different question; it is not the stopped-path p-value.

Worked Example: Replaying Response-Adaptive Assignment Probabilities

In a second hypothetical trial, the sharp-null outcome sequence for a two-stage fixed cohort is \(Y_1=0\), \(Y_2=1\). At stage 1, treatment is assigned with probability \(1/2\). If the first assignment is treatment, the rule stops immediately. Otherwise, stage 2 is reached and treatment is assigned with probability \(3/4\). Define \(T=\sum_{i=1}^{\tau}(A_i-p_i)Y_i\), where \(p_i\) is the treatment probability used at that stage. Suppose the observed path is \(A_1=0,A_2=1\).

The observed statistic is \[ T=(0-\tfrac12)(0)+(1-\tfrac34)(1)=0.25. \] Its path probability is \[ P(A_1=0,A_2=1)=\frac12\cdot\frac34=0.375. \] To obtain the upper-tail p-value, enumerate all paths that the rule permits. If \(A_1=1\), the trial stops and \(T=(1-\tfrac12)(0)=0\), with probability \(1/2\). If \(A_1=0\), the trial continues: \(A_2=1\) gives \(T=0.25\) with probability \(0.5(0.75)=0.375\), and \(A_2=0\) gives \(T=-0.75\) with probability \(0.5(0.25)=0.125\).

Consequently, \[ P(T\ge 0.25)=0.375. \] The adaptive probability \(3/4\) is used only on paths that reach stage 2. Assigning that probability to paths that stopped after stage 1, or assigning equal probability to all terminal paths, would not reproduce the design.

Worked Example: Monte Carlo Approximation to an Exact Tail

Return to the three-unit design in the first example, where the exact two-sided stopped-path p-value for the observed 11 path is 0.50. Suppose full enumeration is inconvenient in a larger version of the design, so 10,000 paths are generated by repeatedly applying the assignment probabilities and stopping rule. In a hypothetical run, 4,978 paths have a stopped-path statistic at least as extreme as the observed one.

A common simulation estimate that avoids reporting a zero p-value is \[ \widehat p_{\mathrm{MC}}=\frac{b+1}{M+1} =\frac{4978+1}{10000+1} =0.4979, \] where \(b=4{,}978\) is the number of simulated tail events and \(M=10{,}000\) is the number of simulated paths. The estimate is close to the exact value 0.50. Its approximate Monte Carlo standard error, using the tail-event proportion, is \[ \sqrt{\frac{0.4978(1-0.4978)}{10000}}\approx 0.00500. \] This uncertainty reflects simulation error, not uncertainty about the treatment effect. Every simulated path must still use the correct assignment probabilities and reproduce the stopping rule.

Why the Full Path Matters

The basic justification is that, under the sharp null and the specified design, the observed assignment path is generated from the same randomization distribution used to calculate the test. Applying the stopping rule to every candidate path makes the reference statistic \(T_{\tau(\omega)}(\omega)\) follow the distribution of the statistic actually reported by the design. The probability of a tail at least as extreme as the observed tail rank is then at most its nominal level; discreteness and ties can make the test conservative.

Fix the null outcomes.
Under the sharp null, hold outcomes fixed with respect to assignments, using only outcome information justified by the design and conditioning scheme.

Generate an assignment.
At each reached stage, draw from the probability prescribed by that path’s current history.

Update and stop.
Reveal the outcome, update the history, and apply the stopping rule. Do not continue a path that stops or truncate a path that continues.

Evaluate the statistic.
Compute the prespecified statistic at that path’s own stopping time, then calculate its exact or simulated tail probability.

A fixed-look test instead evaluates a statistic at a prespecified time, such as stage \(k\), and uses the randomization distribution for that fixed-look statistic. It is appropriate when the analysis is genuinely specified at that look and the data used at the look are observed according to the design. It does not automatically become a valid test for a terminal statistic selected by an outcome-dependent stopping rule. Earlier material on interim looks and anytime-valid randomization p-values addresses procedures designed to support repeated monitoring; those procedures should not be replaced by an ordinary fixed-sample p-value after the stopping decision.

Conditioning on the observed stopping time is another possible analysis, but it is not the same as replaying the full stopped path. A conditional test must use the randomization distribution conditional on that event and must have a non-degenerate, correctly specified conditional distribution. In the first example, conditioning on stopping at stage 2 leaves only paths 11 and 00. Both have \(|S_\tau|=1\), so the conditional two-sided p-value for the observed 11 path is \(1\), rather than the unconditional stopped-path value \(0.50\). Conditioning can change the question and may discard useful variation.

Diagnostics, Alternatives, and Limitations

Before trusting a stopped-path test, verify that every simulated path updates the adaptive allocation probability from its own history, that stopping is evaluated at the same point in the sequence as in the trial, and that path probabilities sum to 1 (or simulated paths follow the normalized design). For small designs, enumerate paths and check the sum explicitly. For simulation, compare estimates across independent batches and report Monte Carlo precision when it is material to the conclusion.

Fixed-look analysis: targets a prespecified look and uses its fixed-look randomization distribution. It is not a substitute for a terminal stopped-path analysis when stopping depends on the data.
Full stopped-path randomization test: targets the statistic and stopping rule actually used, under the specified sharp null and design. It can be computationally demanding and depends on having the outcome information needed for the paths.
Sequentially valid procedures: can support monitoring without selecting a conventional fixed-sample reference distribution after the fact. Their validity relies on their own stated conditions and they answer a related, but not identical, inferential question.

A randomization test of a sharp null does not, by itself, provide a test of every weaker average-effect null. Nor can correct path enumeration repair a misspecified assignment mechanism, an undocumented stopping rule, or unavailable outcomes for units who would enter only on alternative paths. These are design and estimand limitations, not computational details.

Common Mistake

Common mistake: “The trial stopped at stage \(\tau\), so I can hold \(\tau\) fixed and permute assignments as if the trial had been planned to stop there.”

Correction: If \(\tau\) was selected from accumulating trial data, the terminal statistic and its reference distribution must reflect that selection. Replay the stopping rule on each path, or use a separately justified fixed-look or sequentially valid procedure. A test conditional on the stopping event is a distinct analysis and must be calibrated conditionally.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions.

What information must be included in a randomization path when assignment probabilities adapt to observed outcomes?
A trial stops when an interim statistic crosses a boundary. Why can the fixed-horizon distribution of a terminal statistic give the wrong p-value?
In the second worked example, what is the probability of the observed path \(A_1=0,A_2=1\), and why?
When might a conditional-on-stopping-time test be appropriate, and what must be checked?
Why may an exact conditional randomization test fail to be specified when stopping accrual leaves future participants unobserved?

Solutions. (1) Include each assignment probability conditional on that path’s history, reveal outcomes under the null as justified by the design, and apply the stopping rule after each relevant observation. (2) A fixed-horizon reference distribution does not generally account for the data-dependent choice of the terminal time; the stopped statistic is generated at a random time. (3) Its probability is \(0.5\times0.75=0.375\): the trial first continues with probability \(0.5\), then assigns treatment with probability \(0.75\). (4) It can be appropriate when the conditional randomization law is correctly specified and non-degenerate; conditioning changes the target distribution and can remove variation. (5) Alternative paths may include participants whose outcomes were never observed, so those outcomes cannot be supplied by conditioning on the observed data alone without additional design information or assumptions.