Stopping When No Experimental Arms Qualify

From an Empty Selection Set to a Trial-Level Action

At an interim analysis, each experimental arm is assessed against a continuation rule. When at least one arm qualifies, the selection rule identifies which arms may continue. When none qualifies, the selection set is empty—but that fact alone does not fully specify what happens next. The protocol must translate the empty set into an operational action: for example, stopping the trial’s planned recruitment, or stopping recruitment to experimental arms while continuing specified follow-up.

That distinction matters because an arm-level result and a trial-level action answer different questions. The result says that no experimental arm met its prespecified criterion at this look. The action says what the study will do as a consequence. An empty set is not, by itself, proof that all experimental treatments are ineffective, and it does not determine whether existing participants will continue to be followed.

Definition: At interim analysis $k$, let $E_k$ be the set of active experimental arms. For arm $i\in E_k$, let $S_{ik}$ be its interim selection score and $c_{ik}$ its continuation threshold. Under the rule that an arm qualifies when $S_{ik}\ge c_{ik}$, the qualifying set is $$ Q_k=\{i\in E_k:S_{ik}\ge c_{ik}\}. $$ The no-qualifier event is $Q_k=\varnothing$, equivalently $S_{ik}<c_{ik}$ for every $i\in E_k$. A trial-level stopping policy is a prespecified mapping from the interim decision state—including whether $Q_k$ is empty—to an operational action.

The inequality convention is part of the rule. Here, equality with the threshold qualifies; failure is strictly below it. If a design instead specifies a strict continuation rule, equality must be handled according to that rule. This detail can matter when scores are rounded or have discrete distributions.

Why the Trial-Level Rule Must Be Explicit

A protocol should state what “stop” means in operational terms. It may mean no further recruitment under the planned trial, or it may mean that no experimental arm will recruit additional participants while the study completes specified follow-up or safety procedures. The protocol should also state how data already collected will be handled and what participant follow-up remains required. Those choices should not be inferred from the fact that the qualifying set is empty.

The trial-level rule also needs to agree with the rest of the design. Earlier material on minimum numbers of arms to continue distinguishes a threshold-based qualifying set from any minimum-retention requirement. If a prespecified minimum rule modifies what happens when few or no arms qualify, the protocol must specify its precedence. It should not silently relabel a nonqualifying arm as having met its continuation threshold.

The action concerning experimental arms does not, on its own, specify every other arm’s disposition. The protocol must state the complete study action, including recruitment, treatment, and follow-up arrangements, without treating those operational details as consequences that follow automatically from $Q_k=\varnothing$.

Probability of No Experimental Arm Qualifying

The probability of an empty qualifying set is useful when planning a design or assessing how often its stopping policy may be triggered. The general expression depends on the joint distribution of the arm scores. If the scores are independent, the probability factors into arm-specific failure probabilities. Independence is a substantive assumption, not a default feature of a multi-arm trial: shared participants, common controls, or shared sources of variation can make scores dependent.

Result: If the arm scores $S_{ik}$, for $i\in E_k$, are mutually independent, then $$ P(Q_k=\varnothing) = P(S_{ik}<c_{ik}\text{ for every }i\in E_k) = \prod_{i\in E_k}P(S_{ik}<c_{ik}) = \prod_{i\in E_k}F_{ik}(c_{ik}^{-}), $$ where $F_{ik}(x)=P(S_{ik}\le x)$ is the cumulative distribution function of $S_{ik}$, and $F_{ik}(c^-)=P(S_{ik}<c)$. Without independence, the probability is the joint probability $P(S_{ik}<c_{ik}\text{ for every }i\in E_k)$, which generally cannot be obtained by multiplying marginal probabilities.

The left limit $F_{ik}(c_{ik}^{-})$ is important. Since equality qualifies under the stated rule, the failure probability is $P(S_{ik}<c_{ik})$, not necessarily $P(S_{ik}\le c_{ik})$. For a continuous score distribution, the probability of equality at a fixed threshold is zero, so $F_{ik}(c_{ik}^{-})=F_{ik}(c_{ik})$. For a discrete or rounded score, that equality need not hold.

Worked Example: Applying a Prespecified Stop Policy

Suppose a hypothetical interim analysis has three active experimental arms, A, B, and C. The protocol says that an arm qualifies if its interim score is at least its threshold. The scores and thresholds are A: $0.42$ versus $0.50$, B: $1.10$ versus $1.00$, and C: $0.67$ versus $0.70$. The protocol further states that if no experimental arm qualifies, recruitment under the planned trial stops, but required follow-up for already enrolled participants continues.

For A, $0.42<0.50$, so A does not qualify. For B, $1.10\ge1.00$, so B qualifies. For C, $0.67<0.70$, so C does not qualify. Thus $Q_k=\{\text{B}\}$, not the empty set. The no-qualifier trial-level action is not triggered; the protocol’s continuation rules for a nonempty qualifying set determine the next action.

Now consider a second hypothetical set of scores: A $0.42$, B $0.96$, and C $0.67$, with the same thresholds. Each score is strictly below its threshold, so $Q_k=\varnothing$. The prespecified action is to stop planned recruitment while continuing required follow-up. It would be inaccurate to say that all follow-up stops: that is not what this protocol states.

Justifying the Probability Calculation

The factorization follows by writing the no-qualifier event as an intersection of arm-specific failures. Mutual independence allows the probability of that intersection to be written as the product of the individual probabilities. This is a probability statement about the selection event; it does not by itself give the probability of a particular operational action unless the action is defined by that event.

\begin{aligned} \{Q_k=\varnothing\} &=\bigcap_{i\in E_k}\{S_{ik}<c_{ik}\},\\ P(Q_k=\varnothing) &=\prod_{i\in E_k}P(S_{ik}<c_{ik}) &&\text{if the scores are mutually independent},\\ &=\prod_{i\in E_k}F_{ik}(c_{ik}^{-}). \end{aligned}

Worked Example: Independent Continuous Scores

Suppose, for illustration, two hypothetical arm scores are independent and normally distributed with variance 1. For arm 1, the score has mean $0$ and the continuation threshold is $0.5$. For arm 2, the score has mean $0$ and the threshold is $0.8$. Each arm qualifies when its score is at least its threshold.

For a normal score with mean $0$ and standard deviation $1$, the probability of falling below a threshold $c$ is $\Phi(c)$, where $\Phi$ is the standard normal cumulative distribution function. The scores are continuous, so the probability of equality to either threshold is zero. Therefore, $$ P(S_{1k}<0.5)=\Phi(0.5)\approx0.6915,\qquad P(S_{2k}<0.8)=\Phi(0.8)\approx0.7881. $$

Using independence, $$ P(Q_k=\varnothing) = 0.6915\times0.7881 \approx0.5450. $$

Thus the probability that neither experimental arm qualifies is approximately $0.5450$, or 54.50%, under these hypothetical score distributions and the independence assumption. This is not a probability that the null hypothesis is true; it is a probability of a specified interim selection event under the stated model.

Worked Example: A Discrete Score at the Threshold

Suppose, hypothetically, arm A has a binary score $S_{Ak}$ that is 1 with probability $0.50$ and 0 with probability $0.50$. Its continuation threshold is 1, and equality qualifies. Arm B has a binary score that is 1 with probability $0.40$ and 0 with probability $0.60$, also with threshold 1. Assume the two scores are independent.

Arm A fails only when its score is 0, so $P(S_{Ak}<1)=0.50$. Arm B fails only when its score is 0, so $P(S_{Bk}<1)=0.60$. Hence, $$ P(Q_k=\varnothing)=0.50\times0.60=0.30. $$

For arm A, $F_{Ak}(1)=P(S_{Ak}\le1)=1$, which is not the failure probability. The correct quantity is $F_{Ak}(1^-)=P(S_{Ak}<1)=0.50$; similarly, $F_{Bk}(1^-)=0.60$. Using the CDF at the threshold instead of its left limit would incorrectly give $1\times1=1$. This example shows why the strict failure event must be matched to the threshold convention.

Dependence, Diagnostics, and Alternatives

When scores are dependent, multiplying marginal failure probabilities can substantially misstate the chance that no arm qualifies. For example, shared-control comparisons can induce dependence between experimental-arm scores. The correct quantity remains the joint probability of all strict failures; it can be obtained from an appropriate joint model or design simulation. Any such calculation should use the same score definitions, thresholds, and interim data structure as the proposed design.

Worked Example: Why Dependence Changes the Answer

Consider a hypothetical pair of arm decisions with the following joint probabilities: both arms fail, $0.30$; only arm 1 fails, $0.20$; only arm 2 fails, $0.10$; and neither fails, $0.40$. These four mutually exclusive outcomes sum to $0.30+0.20+0.10+0.40=1.00$.

The marginal failure probabilities are $P(F_1)=0.30+0.20=0.50$ and $P(F_2)=0.30+0.10=0.40$. Their product is $0.50\times0.40=0.20$, but this is not the probability that both fail because the decisions are not independent. The joint table gives $P(Q_k=\varnothing)=P(F_1\cap F_2)=0.30$. A trial policy that stops planned recruitment exactly when neither arm qualifies would therefore be triggered with probability $0.30$, not $0.20$, under this hypothetical joint distribution.

Several checks help reveal whether the no-qualifier rule is being implemented or evaluated incorrectly:

Check the boundary convention. Verify whether equality qualifies and use the corresponding strict or non-strict event in code and probability calculations.
Check the active-arm set. The event concerns arms in $E_k$, not arms already dropped at an earlier look.
Check score dependence. Do not multiply arm-specific probabilities unless independence is justified. If scores share data or a control group, examine the joint model or simulate the full interim process.
Check that the action matches the protocol. A flag for an empty qualifying set should trigger the stated operational response, including any specified follow-up, rather than an assumed universal meaning of “stop.”
Check numerical implementation. Reproduce threshold comparisons on boundary values and confirm that rounded scores are not being compared when the rule specifies unrounded scores.

A global futility rule is one alternative way to describe a trial-level decision: it specifies a condition under which the trial stops for lack of sufficient interim evidence. An arm-by-arm continuation rule instead first determines which arms qualify; an empty set may then trigger a trial-level policy. These approaches can be aligned, but they are not automatically identical. Their event definitions and boundary conventions must match. Another operational choice is to cease experimental recruitment while retaining specified follow-up. That is distinct from ending all planned trial activity, and the protocol should say which meaning applies.

Limitations and Common Pitfalls

The no-qualifier event is a decision event under a specified rule, not a definitive scientific conclusion. An arm may fail to qualify because its interim estimate is imprecise, because the threshold is demanding, or because the observed interim data are unfavourable. The event alone does not establish that every treatment is ineffective or harmful. Interpretation should remain tied to the design’s score, threshold, timing, and target population.

A common mistake is to reason, “No experimental arm qualified, so the whole study and all participant follow-up must stop.” The correction is to separate the statistical trigger from its operational consequence. The trigger is $Q_k=\varnothing$; the protocol-defined policy determines what recruitment, treatment, data collection, and follow-up do next. A second common mistake is to calculate the empty-set probability by multiplying marginal failure probabilities without checking independence. Use the joint probability unless independence is supported by the design and model.

Key takeaway: An empty qualifying set is a precise interim selection result. A trial-level stop is a separate, prespecified policy decision. Define the boundary rule, the active arms, the probability model, and the practical meaning of stopping before interpreting or implementing the result.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions below.

Three active arms have scores $0.6, 0.8, 0.7$ and thresholds $0.6, 0.9, 0.7$, respectively. Under a rule that equality qualifies, is the qualifying set empty?
Two independent continuous scores have failure probabilities $0.25$ and $0.40$. What is the probability that neither arm qualifies?
For a discrete score with positive probability at its threshold, why might $F(c)$ differ from the failure probability when qualification requires $S\ge c$?
State one distinction between stopping planned recruitment and stopping follow-up.

Solutions. (1) Arms 1 and 3 qualify because their scores equal their thresholds; arm 2 fails. Thus $Q_k$ is not empty. (2) Independence gives $P(Q_k=\varnothing)=0.25\times0.40=0.10$. (3) The failure event is $S<c$, whereas $F(c)=P(S\le c)$ includes probability mass at equality; use $F(c^-)=P(S<c)$. (4) Stopping planned recruitment means no new participants are enrolled under the specified plan; follow-up of participants already enrolled may still continue if the protocol requires it.