Top-K Selection Under a Fixed Continuation Budget

Why a Fixed Continuation Budget Needs a Rule

At an interim analysis, several experimental arms may show promising results, while the next stage can accommodate only a limited number. A rule that simply says “continue the best arms” is incomplete: it must specify how arms qualify, how they are ranked, and what happens if fewer arms qualify than there are available places. These choices affect which treatments receive further study, so they should be stated before interim results are examined.

This tutorial develops a rule for continuing up to a fixed number \(K\) of arms. It first applies a minimum continuation criterion, then ranks the arms that pass that criterion. The distinction matters: a capacity of \(K\) is a ceiling, not a requirement to continue \(K\) arms regardless of their interim results.

The scores \(G_i\) and \(T_i\) may be the same statistic, but they need not be. For example, a protocol might screen arms using one interim criterion and rank those passing it using a different prespecified statistic. The rule is fully specified only when the criterion, direction of favorable scoring, ranking score, cutoff, capacity, and tie handling are clear. If higher values indicate less favorable results for a particular statistic, it must be reoriented or the rule must be written for the appropriate direction.

The ranking score should support the intended comparison across eligible arms. A standardized score and an effect estimate answer different questions: the former reflects an effect relative to its standard error, while the latter is in the outcome’s measurement units. Which one to use is a design choice; the top-\(K\) rule does not make those scores interchangeable. In either case, the choice should be prespecified rather than selected after looking at the interim results.

Definition and Justification of the Rule

Write \(q=|A|\), the number of eligible arms, and \(m=\min(K,q)\), the number that continue. Order the eligible arms by ranking score from largest to smallest, denoting the ordered scores by \(T_{(1)}\geq T_{(2)}\geq\cdots\geq T_{(q)}\). When \(m=0\), no arm continues under this rule. When \(m>0\), the selected arms are those occupying the first \(m\) positions in that ordering, subject to the protocol’s tie convention.

The ranking step has a simple justification. Among eligible sets of a specified size \(m\), choosing the \(m\) largest ranking scores maximizes the sum of those scores. If a proposed set includes an eligible arm with score \(T_a\) but excludes an eligible arm with score \(T_b>T_a\), replacing \(a\) with \(b\) increases the sum by \(T_b-T_a>0\). A set that omits a higher-ranked eligible arm in favor of a lower-ranked one therefore cannot maximize that sum. Repeating this replacement leads to the top \(m\) arms.

This justification concerns ranking among eligible arms; it does not show that the selected arms have beneficial effects, nor does it establish a treatment’s efficacy. It also assumes the protocol intends to use all available places when at least \(K\) arms qualify, and to continue every qualifying arm when fewer than \(K\) qualify. If the design instead permits fewer than \(m\) arms to continue for another reason, that additional stopping rule must be specified separately.

Assumptions, Implementation Checks, and Failure Modes

The top-\(K\) rule is an operational selection procedure, not a claim that interim scores are independent or that the best observed score identifies the truly best treatment. To apply it coherently, the design should satisfy several practical conditions:

• Prespecified eligibility and ranking. The score definitions, favorable direction, cutoff, and capacity are fixed before the interim data are used to choose arms.
• Scores available on a comparable basis. Each arm has a score defined under the intended interim analysis. If scores have different meanings or scales, the planned ranking must still be scientifically interpretable.
• Complete decision rules. The protocol specifies how to handle ties, unavailable scores, and any data-quality problem that prevents a score from being calculated. These cases should not be resolved informally after results are seen.
• Correct application of the eligibility screen. Only members of \(A\) are ranked for continuation. The capacity is applied to that set, not to all \(J\) arms.

Shared-control contrasts can be correlated. That dependence does not prevent sorting the observed scores, but it can affect the joint behavior of the selection procedure—for example, how often a particular arm ranks first or how often several arms pass a criterion together. Treating arm scores as independent when evaluating the design can therefore give misleading operating characteristics. The ranking algorithm itself does not remove that dependence.

A common implementation error is to sort all arms first and then take the first \(K\), without checking eligibility. Another is to apply the criterion but fill every available place from the nonqualifying arms when fewer than \(K\) pass. Both change the stated rule. A direct check is to record the eligible set first, count it, and then select only from within it.

Ties at the selection boundary also require attention. If tied eligible arms compete for the last available place, “take the top \(K\)” alone does not identify a unique set. The protocol needs a prespecified convention that respects the capacity, such as a deterministic secondary rule or a prespecified random procedure. Similarly, if an arm’s score cannot be calculated, the design must say how that affects eligibility and ranking; silently treating a missing score as favorable or unfavorable is not a defined method.

How This Rule Differs From Other Continuation Approaches

The fixed-capacity rule is useful when continuation resources are limited but the design still requires a minimum screen for further study. Its limitations follow from those same choices: selected arms depend on the interim scores, arms close to the cutoff may be treated differently, and unused capacity is possible. Selection does not by itself provide confirmatory evidence or correct for the fact that arms were chosen using interim data. Those are separate design and analysis questions.