Randomized Continuation Tests

Why Randomize a Continuation Test?

After an interim analysis, a trial’s remaining opportunities for rejection may be constrained by a conditional error bound: at each interim history, the new procedure must not have a greater chance of rejecting the null than the original design had at that history. A deterministic continuation rule can be too coarse to use that allowance fully. For example, a discrete future statistic may offer only a 3% or a 13% null rejection probability, while the available conditional error is 8%. The first rule is conservative; the second exceeds the bound.

Randomization fills the gap. At a specified boundary outcome, an independent random draw determines whether to reject or continue. Its probability is chosen so that the conditional rejection probability equals, or stays below, the available error. The test remains controlled because the randomization is included in the calculation, not because it changes the data or makes a hypothesis more likely.

Definition: Let $X$ denote the interim data and let $\phi_0(X,Y)$ be the original test’s rejection indicator, where $Y$ represents the remaining data. Its conditional error at interim history $X=x$ is $$ A(x)=E_0\{\phi_0(X,Y)\mid X=x\}. $$ Here $E_0$ means expectation under the null model. A randomized adapted test is represented by a function $\phi_1(X,Z)$ between 0 and 1, where $Z$ is the post-interim data: conditional on the observed data, $\phi_1$ is the probability of rejection. It can be implemented using an independent $U\sim\mathrm{Uniform}(0,1)$, by rejecting when $U\leq\phi_1(X,Z)$.

The examples and result below concern a specified null distribution, so the conditional probabilities can be calculated under that distribution. The interim history must be defined before the continuation rule is applied, and the conditional model for the post-interim data must be the one used to calculate the bound. If a test’s conditional error varies with interim data, the new rule must respect that bound for each relevant history, not merely on average.

The Conditional Error Argument

The key requirement is that the adapted procedure’s conditional rejection probability does not exceed the original design’s conditional error. More precisely, under the null, suppose $$ E_0\{\phi_1(X,Z)\mid X\}\leq A(X) $$ with probability one. The conditional expectation averages over the post-interim data and any randomization in the new test, holding the interim history fixed.

Theorem: If the original test has null rejection probability at most $\alpha$, and the adapted test satisfies $E_0\{\phi_1\mid X\}\leq A(X)$ for almost every interim history, then the adapted test also has null rejection probability at most $\alpha$. If equality holds in the conditional bound at every history, the adapted test has the same overall null rejection probability as the original test.

To see why, apply the law of total expectation to the adapted test. The first inequality uses the assumed conditional bound; the last equality follows from the definition of the original conditional error.

\begin{aligned} E_0\{\phi_1(X,Z)\} &=E_0\!\left[E_0\{\phi_1(X,Z)\mid X\}\right]\\ &\leq E_0\{A(X)\}\\ &=E_0\{\phi_0(X,Y)\}\\ &\leq \alpha. \end{aligned}

This argument explains precisely what randomization does and does not do. It can make a conditional rejection probability take an intermediate value between two probabilities available from deterministic rules. It cannot justify exceeding the conditional error at any history. A bound that holds only after averaging across interim histories is not a substitute for the pointwise conditional requirement.

Randomizing at a Discrete Boundary

Suppose a future statistic $S$ has a discrete null distribution. Let $c$ be a boundary value. A test might always reject when $S>c$, never reject when $S<c$, and reject with probability $\gamma$ when $S=c$. Its conditional rejection probability, given the interim history, is $$ P_0(S>c\mid X)+\gamma P_0(S=c\mid X). $$ If the available conditional error is $A(X)$, choosing $$ \gamma=\frac{A(X)-P_0(S>c\mid X)}{P_0(S=c\mid X)} $$ uses the bound exactly when $0\leq\gamma\leq1$. If the ratio is below 0 or above 1, this boundary cannot attain the bound in the proposed way: the boundary or the rejection rule must be changed, or a conservative rule used.

Worked Example: Filling the Gap at a Discrete Boundary

At a particular interim history, suppose the conditional error is 0.08. Under the null, the post-interim statistic $S$ has probabilities $P_0(S=3\mid X)=0.03$, $P_0(S=2\mid X)=0.10$, and $P_0(S=1\mid X)=0.87$. Consider a rule that rejects for $S=3$, randomizes at $S=2$, and does not reject for $S=1$.

Rejecting only for $S=3$ gives conditional rejection probability 0.03, below the available 0.08. Always rejecting for $S\geq2$ gives $0.03+0.10=0.13$, above the bound. Let $\gamma$ be the probability of rejection when $S=2$. Set the rule’s conditional rejection probability equal to 0.08:

0.03+\gamma(0.10)=0.08,\qquad \gamma=\frac{0.08-0.03}{0.10}=0.50.

Thus, the test always rejects at $S=3$, rejects at $S=2$ with probability 0.50, and does not reject at $S=1$. Its conditional null rejection probability is $0.03+(0.50)(0.10)=0.08$. The discrete probabilities and the conditional error are hypothetical; the calculation assumes they are the correct null probabilities for this interim history.

Operationally, after observing $S=2$, generate $U$ independently from a uniform distribution on $[0,1]$, and reject if $U\leq0.50$. At $S=3$, reject without randomization; at $S=1$, do not reject. Independence matters: if the randomizer were related to the trial data, the conditional probability calculation could fail.

Using Different Budgets at Different Histories

The conditional error can vary with interim data. In that case, the randomization probability may also vary, but it must be determined by a prespecified rule based on the observed interim history and the corresponding null conditional distribution. The following example shows how to allocate the bound separately in two interim strata.

Worked Example: Matching Two Conditional Error Budgets

Suppose the interim history falls in stratum $A$ with null probability 0.40 and stratum $B$ with null probability 0.60. The original design has conditional errors 0.10 in $A$ and 0.04 in $B$. In stratum $A$, the new continuation statistic has a strict-tail probability of 0.02 and a boundary probability of 0.10. In stratum $B$, those probabilities are 0.01 and 0.06. All probabilities are hypothetical and refer to the appropriate null distribution conditional on the stratum.

In stratum $A$, choose $\gamma_A$ to use the 0.10 bound:

0.02+\gamma_A(0.10)=0.10,\qquad \gamma_A=\frac{0.10-0.02}{0.10}=0.80.

In stratum $B$, choose $\gamma_B$ to use the 0.04 bound:

0.01+\gamma_B(0.06)=0.04,\qquad \gamma_B=\frac{0.04-0.01}{0.06}=0.50.

The new conditional rejection probabilities are therefore 0.10 in $A$ and 0.04 in $B$, each equal to its original conditional error. The overall null rejection probability is the probability-weighted average:

$$ (0.40)(0.10)+(0.60)(0.04)=0.040+0.024=0.064. $$

The original design also has overall null rejection probability 0.064, so the adapted test preserves it in this example. Checking only the overall value would not be enough to establish conditional error control: each stratum’s new conditional rejection probability must also be compared with its own bound.

Randomizing Whether to Continue

Randomization can also be applied to the continuation decision itself. At an interim history, suppose continuing leads to a test whose conditional null rejection probability is $r$. If the trial continues with probability $q$, and stopping produces no rejection, then the overall conditional rejection probability at that history is $qr$. This calculation assumes the continuation randomizer is independent of the future data under the null and that the conditional rejection probability $r$ is the one appropriate to that history.

Worked Example: Randomizing the Continuation Decision

At one interim history, the available conditional error is 0.05. A specified continuation test would have conditional null rejection probability 0.08 if it were always run. A deterministic choice to continue would therefore exceed the bound; a deterministic choice to stop would use none of it. Let $q$ be the probability of continuing, and assume stopping cannot reject.

The conditional rejection probability is $q(0.08)$. To use the full bound, solve

q(0.08)=0.05,\qquad q=\frac{0.05}{0.08}=0.625.

Thus, continue with probability 0.625 and otherwise stop without rejecting. The conditional null rejection probability is $(0.625)(0.08)=0.05$, exactly the available error. The calculation does not mean that 62.5% of patients are assigned to treatment; it is a probability governing the design’s continuation decision at this interim history.

The choice between randomizing at a test boundary and randomizing whether to continue depends on the design and what actions are scientifically and operationally meaningful. Either way, the full conditional rejection probability—including the probability of entering the continuation test—must be calculated. Randomization is not a way to disregard the test’s conditional error; it is a way to calibrate the action so the error remains within it.

Assumptions, Diagnostics, and Limitations

A randomized continuation test is justified when its probabilities can be specified and implemented reliably. Before using one, verify the following:

Correct conditional error: The original design’s conditional error is calculated for the relevant interim history under the null model.
Correct conditional distribution: The probabilities for future statistics are conditional on the same interim information used to define the error bound.
Valid randomization: The random draw is independent of the trial data, and its probability is exactly the prespecified value.
Pointwise control: At every relevant history, the adapted conditional rejection probability is no greater than that history’s conditional error.
Feasible boundary probability: For boundary randomization, the boundary has positive probability and the calculated $\gamma$ lies between 0 and 1.

Useful checks include enumerating the possible statistic values under a discrete null model, verifying that their probabilities sum to 1, and recomputing the conditional rejection probability after including the randomization. If the boundary has probability zero, the displayed formula for $\gamma$ is undefined and that boundary cannot be used to fill the gap. If the calculation gives $\gamma<0$ or $\gamma>1$, it does not specify a valid randomization probability.

The main deterministic alternative is to choose the largest nonrandomized rejection region that does not exceed the conditional error. This is valid when properly calibrated, but can be conservative for discrete data. Randomization can attain the bound, though some trial teams or participants may find a chance-based rejection decision difficult to explain or implement. A different prespecified test may also have a suitably fine range of conditional rejection probabilities, but its validity still depends on satisfying the conditional bound. No approach is protected from a misspecified null distribution, an incorrectly computed conditional error, or a rule chosen after seeing the results.

Common mistake: “The deterministic boundary rule has a 3% conditional rejection probability and the budget is 8%, so it is acceptable to report that the full 8% was used.” The rule is valid but uses only 3%; it does not have an 8% conditional rejection probability. To use the remaining allowance, randomize at an outcome with positive null probability and calculate the randomization probability so that the total conditional rejection probability is 8%. Conversely, a rule with 13% rejection probability cannot be justified by saying that the overall average across interim histories is below the nominal level.

Randomization is most useful when the conditional distribution is discrete or otherwise coarse, the conditional error is known, and the resulting decision can be prespecified and documented. It is not required for validity: a conservative deterministic test remains an option. Its special role is to make a conditional rejection probability attainable at a value between the available deterministic choices.

Check Your Understanding

Work through each question, then compare your reasoning with the solutions.

A conditional error is 0.06. Under the null, a strict-tail event has probability 0.02 and the boundary event has probability 0.08. What boundary rejection probability uses the full bound?
Why does the conditional error argument require control at each interim history rather than only an overall average?
A continuation test has conditional null rejection probability 0.10, while the available conditional error is 0.04. If stopping cannot reject, what continuation probability uses the bound exactly?
Name two checks that can reveal an invalid randomized continuation calculation.

Solutions. (1) Solve $0.02+\gamma(0.08)=0.06$, giving $\gamma=(0.06-0.02)/0.08=0.50$. (2) The law of total expectation gives overall control from the conditional bound at each history; an average bound could conceal histories where the new test exceeds the original conditional error. (3) Solve $q(0.10)=0.04$, giving $q=0.40$. (4) Examples include checking that the randomizer is independent of the trial data, that the conditional probabilities use the correct null distribution given interim history, that the boundary probability is positive, and that the calculated randomization probability lies in $[0,1]$.

What You'll Learn