Selection Rules Across Multiple Interim Looks

Why Selection Must Be Defined Across Looks

In a multi-stage trial, an arm that continues after one interim analysis may be assessed again at a later look. The sequence matters: a rule that is sensible when applied once may be applied incorrectly if the set of arms eligible for evaluation is not carried forward consistently. In particular, a treatment arm that has been dropped should not quietly reappear in a later analysis under an irreversible-dropping design.

The central object is therefore not just the rule used at one look, but the sequence of active-arm sets. Each set records which experimental arms remain under evaluation after the corresponding interim decision. Scores and thresholds may change from look to look, and may differ by arm, but the update must preserve the rule that past dropping decisions are final.

Definition: Let the experimental arms be indexed by $i=1,\ldots,J$, and let interim looks be indexed by $k=1,\ldots,K$. The active-arm set $A_k$ is the set of experimental arms still under evaluation after the decision at look $k$. Initially, $A_0=\{1,\ldots,J\}$. An irreversible-dropping rule requires $A_k\subseteq A_{k-1}$ for every $k=1,\ldots,K$. An arm removed from an active set cannot re-enter a later active set.

The definition concerns membership in the active set, not whether an arm's outcomes remain in the trial database. Historical data may still be retained for reporting or other prespecified analyses. But for the selection process, only arms in the current active set are eligible for a new continuation decision.

The Set Update at Each Look

Write $Q_{ik}$ for the interim selection score for arm $i$ at look $k$, with larger values defined to be more favorable. A rule may compare that score with a threshold $c_{ik}$, rank scores, or combine several criteria. The particular score and selection criterion are chosen in advance; the set update specifies how that criterion is applied across looks.

Definition: At look $k$, let $C_k$ be the set of arms that meet the prespecified selection rule when evaluated using the information available at that look. The active set is updated by $$ A_k=A_{k-1}\cap C_k. $$ Equivalently, the rule is applied only to arms in $A_{k-1}$, and the resulting retained arms form $A_k$. The arms dropped at look $k$ are $A_{k-1}\setminus A_k$.

The intersection makes the eligibility restriction explicit. Even if an arm not in $A_{k-1}$ would satisfy the current score threshold, it is not in $A_k$. For a simple arm-specific threshold rule, the update can be written as

A_k=\{i\in A_{k-1}: Q_{ik}\geq c_{ik}\}.

Here $c_{ik}$ may vary both by look and by arm. Other rules can replace the threshold comparison without changing the set logic. For example, a rule may retain the highest-scoring eligible arm or retain arms satisfying a joint screening condition; in each case, the rule acts on the current active set and its output is the next active set.

This construction assumes that the looks and the selection rules are specified in advance, the score direction is consistent, and the information used at look $k$ is available for the eligible arms being assessed. The set update itself does not require independent arm scores. In a shared-control trial, scores may be statistically dependent because they use common control data. That dependence matters when evaluating the operating characteristics of the full design, but it does not alter the definition of $A_k$.

Why the Active Sets Are Nested

Irreversibility follows directly from the update, rather than from any special property of the scores. Because an intersection cannot contain an element outside either set,

A_k=A_{k-1}\cap C_k\subseteq A_{k-1}.

Repeated application gives $A_k\subseteq A_{k-1}\subseteq\cdots\subseteq A_0$. Thus an arm absent from $A_k$ cannot belong to any later active set. The cumulative dropped set through look $k$ is $D_k=A_0\setminus A_k$, and the new drops at look $k$ are $A_{k-1}\setminus A_k$. Since $A_k\subseteq A_{k-1}$, the cumulative set updates as

D_k=D_{k-1}\cup(A_{k-1}\setminus A_k), \qquad D_{k-1}\cap(A_{k-1}\setminus A_k)=\varnothing.

The disjointness says that a newly dropped arm was active immediately before the current decision, whereas an arm dropped earlier is already in $D_{k-1}$. This provides a useful audit: the active and dropped sets together should account for every experimental arm, with no arm counted in both.

Applying the Rule Over Successive Looks

Worked Example: Arm-Specific Thresholds at Three Looks

Suppose a hypothetical trial evaluates three experimental arms, A, B, and C, against a shared control. At each look, $Q_{ik}$ is a standardized interim score, with larger values more favorable. The prespecified rule retains an active arm when $Q_{ik}\geq c_{ik}$. The thresholds are 1.5 for A, 1.6 for B, and 1.4 for C at every look. The scores below are illustrative, not data from a real trial.

At look 1, the scores are 1.8 for A, 1.2 for B, and 1.5 for C. Starting from $A_0=\{\mathrm{A},\mathrm{B},\mathrm{C}\}$, apply the threshold separately:

A_1=\{\mathrm{A}:1.8\geq1.5,\ \mathrm{B}:1.2\geq1.6,\ \mathrm{C}:1.5\geq1.4\} =\{\mathrm{A},\mathrm{C}\}.

Thus B is dropped at look 1. At look 2, the scores are 1.1 for A, 2.0 for B, and 1.7 for C. B's score exceeds its threshold, but B is no longer eligible. Apply the rule only to $A_1$:

A_2=\{\mathrm{A}:1.1\geq1.5,\ \mathrm{C}:1.7\geq1.4\} =\{\mathrm{C}\}.

A is dropped at look 2. At look 3, suppose C has score 1.6, which exceeds its threshold of 1.4. Then $A_3=\{\mathrm{C}\}$. The active sets are nested: $\{\mathrm{A},\mathrm{B},\mathrm{C}\}\supseteq\{\mathrm{A},\mathrm{C}\}\supseteq\{\mathrm{C}\}\supseteq\{\mathrm{C}\}$. B's favorable second-look score does not reverse its first-look decision.

Worked Example: Calculating and Applying Interim Scores

Consider a hypothetical study of symptom improvement, measured in points, where a larger estimated improvement relative to control is favorable. For this example, define $Q_{ik}=\hat{\Delta}_{ik}/SE_{ik}$, where $\hat{\Delta}_{ik}$ is the estimated improvement for arm $i$ relative to control at look $k$. At look 1, two active arms have estimates and standard errors: A, 4.2 points with $SE=2.0$ points; B, 3.0 points with $SE=2.0$ points. The prespecified rule retains an arm if $Q_{i1}\geq1.6$.

For A, $Q_{\mathrm{A}1}=4.2/2.0=2.10$. For B, $Q_{\mathrm{B}1}=3.0/2.0=1.50$. Therefore, with $A_0=\{\mathrm{A},\mathrm{B}\}$,

A_1=\{\mathrm{A}:2.10\geq1.6,\ \mathrm{B}:1.50\geq1.6\} =\{\mathrm{A}\}.

At look 2, suppose the estimates are 3.6 points for A and 5.0 points for B, each with $SE=2.0$ points. The scores would be $Q_{\mathrm{A}2}=3.6/2.0=1.80$ and $Q_{\mathrm{B}2}=5.0/2.0=2.50$. Only A is still active, so the update is $A_2=\{\mathrm{A}:1.80\geq1.6\}=\{\mathrm{A}\}$. B's computed score is shown to make the eligibility restriction clear; it does not make B eligible again. The score is a selection quantity here, not by itself a declaration of statistical significance or proof of clinical benefit.

Worked Example: A Stage-Specific Rule

Suppose two hypothetical arms, D and E, begin with $A_0=\{\mathrm{D},\mathrm{E}\}$. The prespecified score threshold is 1.4 at look 1 and 1.8 at look 2, with larger scores favorable. At look 1, $Q_{\mathrm{D}1}=1.7$ and $Q_{\mathrm{E}1}=1.5$, so

A_1=\{\mathrm{D}:1.7\geq1.4,\ \mathrm{E}:1.5\geq1.4\} =\{\mathrm{D},\mathrm{E}\}.

Both arms continue. At look 2, their scores are $Q_{\mathrm{D}2}=1.9$ and $Q_{\mathrm{E}2}=1.6$. Using the look-2 threshold gives

A_2=\{\mathrm{D}:1.9\geq1.8,\ \mathrm{E}:1.6\geq1.8\} =\{\mathrm{D}\}.

The threshold change is part of the planned rule, not a reason to re-evaluate the earlier decision using the new cutoff. Both arms passed the look-1 rule; at look 2, only D meets the look-2 rule.

Implementation Checks and Common Failure Modes

A multi-look selection rule is easiest to implement when each decision is recorded as a transition from the previous active set. A compact audit table can list the look, the eligible arms before the decision, the score and threshold for each eligible arm, and the resulting active set. It should also record which arms are newly dropped. These records make it possible to check both the calculation and the sequence of eligibility.

Re-entry through a fresh full-arm analysis. Recomputing a threshold rule for every original arm at each look and taking all current passers can bring back an arm dropped earlier. Instead, intersect the current rule's result with $A_{k-1}$.
Using the wrong look's rule. If thresholds differ across looks, apply $c_{ik}$ at look $k$. Reusing a previous threshold changes the specified selection rule.
Mixing eligibility with score availability. A score may be calculable for an arm that is no longer active. Availability of a score does not restore eligibility.
Changing rules after seeing results. Choosing a new threshold or a different selection criterion in response to observed interim scores means the implemented procedure is no longer the prespecified one. Any statistical evaluation must match the rule actually used.
Reading a selection score as a final inferential conclusion. Passing an interim selection rule means the arm is retained under that rule. It does not alone establish efficacy, clinical relevance, or a particular error rate.

A useful diagnostic is to verify $A_k\subseteq A_{k-1}$ at every transition and $A_k\subseteq A_0$ throughout. If either check fails, there is an eligibility or record-keeping error under an irreversible rule. Separately, the thresholds, score direction, and timing should match the design specification. These checks do not establish that the chosen rule has desirable operating characteristics; those require evaluating the complete multi-look design, including the joint behavior of interim statistics.

How This Formulation Differs From Other Approaches

A one-look selection rule makes a single mapping from the original set of arms to a retained subset. It does not by itself describe how a retained set is updated when another interim decision occurs. The recursive formulation adds that missing history: the next rule acts on the current active set, not automatically on the original set.

A re-entry approach would permit an arm that failed an earlier screen to become eligible again after a later score. That is a different design, not an alternative interpretation of irreversible dropping. A fixed-set approach, in which the same arms remain under evaluation at every look, also differs: it makes no interim removal decisions. When the design specifies irreversible dropping, the nested-set update is the appropriate representation. Its simplicity is also a limitation: once an arm is removed, later evidence cannot restore it, so the initial rules and their statistical consequences need careful planning.

Key takeaway: At each look, evaluate only the arms in the current active set, and define the next active set as the subset that passes that look's prespecified rule. The resulting sets are nested, so dropping is irreversible by construction.

Check Your Understanding

Answer each question using the active-set update. Solutions follow the questions.

If $A_1=\{\mathrm{A},\mathrm{C}\}$ and, at look 2, only C passes the rule among active arms, what is $A_2$?
At look 1, $A_0=\{\mathrm{A},\mathrm{B}\}$ and a threshold rule retains scores at least 1.5. Scores are 1.7 for A and 1.4 for B. At look 2, B's score is 2.2. Can B enter $A_2$?
For an active arm, the estimated improvement is 2.4 points and its standard error is 1.5 points. Calculate the score $Q=\hat{\Delta}/SE$. Does it pass a threshold of 1.6?
State the set relationship that must hold between successive active sets under irreversible dropping, and explain what a violation indicates.

Solutions. 1. $A_2=\{\mathrm{C}\}$, because only C passes among the arms in $A_1$. 2. No. At look 1, $A_1=\{\mathrm{A}\}$, since $1.7\geq1.5$ but $1.4<1.5$. B is ineligible at look 2 even though its score is 2.2, so it cannot enter $A_2$. 3. $Q=2.4/1.5=1.60$. It meets the threshold exactly and passes when the rule uses $Q\geq1.6$. 4. The required relationship is $A_k\subseteq A_{k-1}$. A violation indicates that an arm has been reintroduced or that the active-set update or record keeping is inconsistent with irreversible dropping.