Why Randomized Adaptation Needs a Conditional Error Check
An adaptation decision need not be deterministic. At an interim analysis, a trial might use a random draw to select one of several continuation designs, perhaps to distribute participants across operationally feasible options. Randomization does not itself invalidate a test, but it changes the question that must be answered: is type I error controlled only after averaging over the random draw, or is it controlled for each possible selected design?
For a guarantee that holds after the selected outcome is known, check the conditional rejection probability separately for every outcome the randomizer can produce. A bound only on the average across outcomes can be sufficient for an overall type I error guarantee, but it can allow one selected continuation to exceed the original conditional error.
The notation \(\phi_r\) allows the continuation test, and its future data, to differ across outcomes. Each \(\gamma_r(H)\) must therefore be calculated under the conditional null distribution appropriate to that selected continuation. Assume throughout that the original design has type I error at most \(\alpha\), that the randomization probabilities are specified in advance, and that the random draw is genuinely random and independent of future outcomes conditional on \(H\). If a choice changes future assignments or data distributions, those changes belong in the calculation of \(\gamma_r(H)\).
Two Ways to Condition
There are two useful conditioning choices, and they answer different questions. Conditioning only on \(H\) asks about error before the randomizer's result is known. Conditioning on both \(H\) and \(R\) asks about error after that result is known. If the random draw is independent of the original design's future data conditional on \(H\), it does not change the original conditional error:
The equality is important: the reference conditional error remains \(C(H)\) for every possible draw. To preserve error after each outcome, require
A less stringent condition bounds only the average over the randomizer. If \(q_r(H)=P(R=r\mid H)\), the rejection probability before the draw is revealed is
Thus \(\sum_r q_r(H)\gamma_r(H)\le C(H)\) protects the conditional error given \(H\), but it does not imply that each \(\gamma_r(H)\le C(H)\). This difference is not a technicality: once \(R\) is observed, an outcome-specific excess is no longer averaged away.
Worked Example: An Average Bound That Fails After One Outcome
Suppose that at a particular interim history the original conditional error is \(C(H)=0.12\). A randomizer selects continuation A with probability 0.75 and continuation B with probability 0.25. Under the null, their conditional rejection probabilities are 0.08 and 0.20, respectively.
Before the draw is revealed, the new design's conditional rejection probability is $$ 0.75(0.08)+0.25(0.20)=0.06+0.05=0.11. $$ Since \(0.11\le 0.12\), the average conditional error is bounded by the original one.
After outcome B, however, the conditional rejection probability is \(0.20\), which exceeds \(0.12\). Therefore this randomization preserves the bound on average over \(R\), but it does not preserve conditional error for every outcome. If the requirement is an outcome-by-outcome guarantee, this design fails.
Why Outcome-Specific Bounds Preserve Type I Error
The justification follows by conditioning first on the interim information and then on the random draw. For a finite set of outcomes, the conditional expectation of the new rejection indicator given \(H\) is the probability-weighted sum of its outcome-specific conditional rejection probabilities. If each is bounded by \(C(H)\), then their average is also bounded by \(C(H)\).
Taking expectations over the interim information gives the overall guarantee. By the law of total expectation, \(E_0[C(H)]=P_0(\phi_0=1)\), so
This argument also explains the stronger, outcome-specific version. If \(\gamma_r(H)\le C(H)\) for every \(r\), then the new test's conditional rejection probability given \(H,R=r\) is bounded by the original conditional error for each draw. Averaging over \(R\) and then over \(H\) yields the same overall type I error guarantee. If only the probability-weighted average is bounded, the overall guarantee still follows, but the bound after an individual outcome need not hold.
For a continuous random draw, replace the sum by an integral over its conditional distribution given \(H\). The same reasoning applies: a bound for almost every draw gives an average bound, while a guarantee for every possible outcome requires the corresponding pointwise bound wherever the randomizer can select an outcome.
Worked Example: Preserving Error for Both Outcomes
At an interim history, let \(C(H)=0.12\). The trial selects continuation A with probability 0.4 and continuation B with probability 0.6. Suppose their conditional rejection probabilities under \(H_0\) are 0.08 and 0.12.
For outcome A, \(0.08\le0.12\). For outcome B, \(0.12\le0.12\). Thus each possible continuation meets the outcome-specific bound.
The conditional rejection probability before the draw is revealed is $$ 0.4(0.08)+0.6(0.12)=0.032+0.072=0.104. $$ This is also bounded by \(0.12\). After either outcome is observed, the conditional rejection probability is no greater than the original conditional error. If the original trial's overall type I error is at most \(\alpha\), the conditional-expectation argument shows that the randomized continuation also has type I error at most \(\alpha\).
Calculating the Conditional Error for Selected Tests
A practical calculation starts with the original design's conditional error at the observed interim history. Then, for each randomization outcome, calculate the new continuation's null rejection probability conditional on that same history and outcome. Compare each value with the reference \(C(H)\). For example, when a continuation rejects for a conditional null \(p\)-value no larger than a threshold \(c_r(H)\), and that \(p\)-value is conditionally uniform under \(H_0\), its conditional rejection probability is \(c_r(H)\). If it is only conditionally super-uniform, the rejection probability is at most \(c_r(H)\), which still gives a valid upper bound.
Worked Example: Checking Choices at Two Interim Histories
Consider two possible interim histories, \(H=h_1\) and \(H=h_2\). Under the null, suppose their probabilities are 0.70 and 0.30. The original conditional errors are \(C(h_1)=0.06\) and \(C(h_2)=0.18\). At each history, a randomizer selects A or B. The continuation tests use conditionally uniform null \(p\)-values, with the following rejection thresholds:
| Interim history | Original conditional error | Randomizer probabilities | Threshold for A | Threshold for B |
|---|---|---|---|---|
| \(h_1\) | 0.06 | A: 0.50; B: 0.50 | 0.04 | 0.06 |
| \(h_2\) | 0.18 | A: 0.25; B: 0.75 | 0.12 | 0.18 |
Conditional uniformity means each threshold is also the corresponding conditional rejection probability. At \(h_1\), both \(0.04\) and \(0.06\) are no larger than \(C(h_1)=0.06\). At \(h_2\), both \(0.12\) and \(0.18\) are no larger than \(C(h_2)=0.18\). Hence every history-outcome pair meets the outcome-specific bound.
The rejection probability conditional on \(h_1\), before the draw, is $$ 0.50(0.04)+0.50(0.06)=0.05. $$ At \(h_2\), it is $$ 0.25(0.12)+0.75(0.18)=0.03+0.135=0.165. $$ The new design's overall null rejection probability is therefore $$ 0.70(0.05)+0.30(0.165)=0.035+0.0495=0.0845. $$ The original overall type I error is \(0.70(0.06)+0.30(0.18)=0.096\). The new value, \(0.0845\), is no greater than \(0.096\). The example checks both the outcome-specific condition and the resulting overall error bound.
Assumptions, Diagnostics, and Alternatives
The calculation depends on a correctly specified reference conditional error and valid conditional null rejection probabilities for each continuation. In particular, conditional uniformity in the preceding example is an assumption about each continuation's \(p\)-value given the interim history and selected outcome; ordinary marginal uniformity alone does not establish it. If the randomization affects future assignments, information, or the distribution of a test statistic, use that outcome-specific conditional distribution rather than reusing an unchanged distribution by assumption.
Useful checks include confirming that the randomization probabilities are nonnegative and sum to one at each interim history; that every selectable outcome has been checked; and that the conditional rejection probabilities use the correct null distribution and conditioning information. A numerical result above the original conditional error for even one outcome signals failure of the outcome-specific guarantee. Simulations under the null can help identify implementation mistakes, but they do not replace the mathematical bound, especially when the relevant conditional histories are rare.
The main alternative is to require only the probability-weighted average bound. That can be appropriate when the design goal is control conditional on \(H\) before revealing the draw, and its validity follows from the same averaging argument. It is weaker than outcome-specific preservation. A deterministic adaptation has no randomization outcomes to average over, so its selected continuation must itself meet the conditional-error bound. In either case, conditional error preservation controls type I error; it does not establish power, clinical usefulness, or operational feasibility.
Check Your Understanding
Answer each question, then compare with the brief solutions.
- If \(C(H)=0.10\), a randomizer selects A and B with probabilities 0.8 and 0.2, and their conditional rejection probabilities are 0.08 and 0.16, what is the conditional rejection probability before the draw is revealed? Is the bound preserved after each outcome?
- State the assumption that allows the original conditional error given \(H,R=r\) to equal \(C(H)\).
- Suppose each selectable continuation satisfies \(\gamma_r(H)\le C(H)\). Explain why the overall type I error is no greater than the original design's type I error.
- A continuation uses a conditional null \(p\)-value that is super-uniform, and rejects when the \(p\)-value is at most \(c\). What upper bound can be used for its conditional rejection probability?
Solutions. (1) The average is \(0.8(0.08)+0.2(0.16)=0.064+0.032=0.096\), which is below 0.10. Outcome B has conditional rejection probability 0.16, so the bound is not preserved after each outcome. (2) The draw is independent of the original design's future data and rejection decision conditional on \(H\). (3) Averaging the outcome-specific bounds gives \(P_0(\phi_R=1\mid H)\le C(H)\); taking expectations gives \(P_0(\phi_R=1)\le E_0\{C(H)\}=P_0(\phi_0=1)\le\alpha\). (4) The conditional rejection probability is at most \(c\), by conditional super-uniformity.