From an Interim Estimate to a Decision Rule
An interim treatment-effect estimate is useful for adaptation only when the design specifies what action it triggers. A rule such as “continue the promising treatments” is not reproducible: it leaves “promising” undefined, does not say how many arms may continue, and offers no answer for ties or for the case in which no arm looks promising. A prespecified selection rule turns the interim information into an operational decision before those data are examined.
Suppose a trial has \(K\) experimental arms and a control arm. For experimental arm \(i\), let \(\Delta_i\) denote the treatment effect for the target contrast defined for that arm, with larger values representing greater benefit, and let \(\hat{\Delta}_i\) be its interim estimate. The sign convention matters: for an endpoint where lower values are better, the effect or its reported estimate must be oriented so that “larger is better” if the rule is written in that direction. A rule may instead use another prespecified orientation, but its comparisons must agree with that orientation.
The estimate is an input to the rule, not the rule itself. A complete specification identifies the decision time, which arms are eligible for selection, whether the control arm is retained, and whether the rule returns a fixed number of experimental arms or any number that meets a condition. It also says what happens if an estimate is unavailable or if no arm meets a stated condition. These details make the same interim results lead to the same action when the rule is applied by different people.
Two Different Questions: Ranking and Thresholding
A ranking rule asks which arms have the largest interim estimates relative to the other eligible arms. For example, a top-two rule retains the two arms with the largest \(\hat{\Delta}_i\), regardless of how large or small those estimates are in absolute terms. A threshold rule asks whether each estimate clears a specified value \(\tau\): it retains arm \(i\) when \(\hat{\Delta}_i \geq \tau\). The threshold has the same units as the treatment effect and needs a contextual justification; it is not a significance level.
These rules answer different design questions. Ranking allocates a limited number of places to the best-ranked arms, even if all interim estimates are modest. Thresholding retains arms that meet a fixed estimate-based criterion, possibly leaving none or several. A combined rule can first apply a threshold and then rank the eligible arms if there is a limit on how many can continue. In that case, both stages and the order in which they are applied must be stated.
| Rule type | Decision question | Typical output | Specification that must be explicit |
|---|---|---|---|
| Ranking | Which estimates are largest among eligible arms? | A fixed number of top-ranked arms | Number to retain and tie handling |
| Threshold | Which estimates meet a fixed criterion? | All qualifying arms, possibly none | Cutoff, comparison operator, and any fallback |
| Combined | Which arms qualify, and which qualify first? | Qualifying arms up to a limit | Threshold, ranking order, limit, ties, and fallback |
Comparisons are meaningful only when the estimates have the intended common interpretation. For instance, comparing estimates with different units, different directions, or different target contrasts can produce a numerical ordering that does not answer the design question. The target contrasts should therefore be fixed in the design, and estimates should be expressed on the agreed scale before the rule is applied.
Worked Example: Retaining the Two Highest-Ranked Arms
In a hypothetical trial, three experimental arms target the same treatment-effect scale, with larger effects indicating greater benefit. At the prespecified interim analysis, the estimates are \(2.4\), \(1.8\), and \(1.1\) points for arms A, B, and C. The rule retains exactly two experimental arms with the largest unrounded estimates and uses a prespecified arm code to break an exact tie.
The ranking is \(2.4 > 1.8 > 1.1\). Thus, arm A ranks first, arm B ranks second, and arm C ranks third. The rule retains A and B and drops C from the set of continuing experimental arms. This result follows from the relative ordering; the rule does not require either retained estimate to exceed a minimum effect.
The example assumes the estimates are available at the specified decision time, refer to the same oriented target effect, and are calculated using the prespecified interim analysis method. If one estimate were unavailable, the design would need a rule for that case rather than an improvised substitution.
Worked Example: Applying a Fixed Effect Threshold
In a second hypothetical trial, the rule retains every arm whose interim estimate is at least \(1.5\) points on the prespecified benefit scale. Estimates are \(0.7\) points for arm A, \(1.6\) points for arm B, \(2.1\) points for arm C, and \(1.4\) points for arm D.
For each arm, compare its estimate with \(\tau=1.5\): \(0.7 < 1.5\), \(1.6 \geq 1.5\), \(2.1 \geq 1.5\), and \(1.4 < 1.5\). The rule therefore retains B and C and does not retain A or D. The comparison is made using the unrounded estimates, so display rounding cannot change a decision near the cutoff.
This is an estimate-based selection decision, not a test of whether an arm’s true treatment effect exceeds \(1.5\) points. In particular, crossing the cutoff does not by itself establish efficacy or quantify uncertainty about the effect.
Writing the Rule as an Algorithm
A useful way to make a selection rule auditable is to write its steps in the order they will be executed. The following is a general construction. The exact choices depend on the selection objective, but none should be left to the discretion of the person reviewing the interim results.
State the decision time, eligible arms, target contrast for each estimate, benefit direction, and interim estimator. Specify how missing or unavailable estimates are handled.
For a ranking rule, state how many arms to retain. For a threshold rule, state the cutoff, units, and whether equality qualifies. If the rule combines both, state which operation comes first.
Give a deterministic tie procedure and state what happens if no arm qualifies, too few qualify, or a maximum number would otherwise be exceeded.
State which arms continue, which stop, and whether control continues. Apply the rule to the prespecified inputs without changing its criteria after seeing the results.
For example, a hybrid rule could retain arms with estimates at least \(\tau\), then retain no more than \(m\) of those arms by ranking their estimates, with an explicit tie rule. If no arm meets \(\tau\), the protocol must say whether no experimental arm continues or whether a fallback applies. Merely adding “use clinical judgment” would not define a reproducible statistical rule.
Worked Example: A Threshold Rule With a Fallback
Consider a hypothetical rule that retains arms with an interim estimate of at least \(2.0\) points. If no arm reaches \(2.0\), it retains the arm with the largest estimate. Exact ties for the fallback are resolved by a prespecified arm code. The estimates are \(1.3\) points for A, \(1.8\) points for B, and \(1.6\) points for C.
First apply the threshold: \(1.3 < 2.0\), \(1.8 < 2.0\), and \(1.6 < 2.0\). No arm qualifies. The fallback therefore applies. Comparing the estimates gives \(1.8 > 1.6 > 1.3\), so arm B has the largest estimate and is retained. This outcome is not the same as silently lowering the threshold: the fallback is a separate, prespecified branch of the rule.
The assumption here is that all three estimates use the same target scale and are available at the decision time. The example also shows why the fallback must be described before the results are known: without it, the rule would not determine what to do when the qualifying set is empty.
Why Ranking and Thresholding Behave Differently
The distinction can be shown directly. Suppose the same constant \(c\) is added to every interim estimate, so each becomes \(\hat{\Delta}_i+c\). For any pair of arms \(i\) and \(j\),
Therefore, adding a common constant does not change the ranking. A top-\(m\) rule is invariant to this shift, apart from ties that remain ties. A fixed-threshold rule is not generally invariant, because the condition becomes \(\hat{\Delta}_i+c \geq \tau\), which can change whether an arm qualifies. This algebra explains why a ranking rule is inherently relative to the other estimates, whereas a threshold rule depends on an absolute benchmark on the chosen scale.
This property is not a claim that rankings are statistically stable. It concerns only how a deterministic rule responds to a common shift in its inputs. If estimates are noisy, their order may change across interim samples; evaluating that uncertainty and its consequences requires separate operating-characteristic work. Nor does a threshold decision certify that the underlying treatment effect exceeds the cutoff: it compares an estimate with a cutoff.
Assumptions, Diagnostics, and Failure Modes
A selection rule is well-defined only if its inputs and operational details match the design. Before applying it, check that the interim data support the specified estimates, the estimation method and analysis time are fixed, and the benefit direction and units are consistent across arms. Also verify that the rule preserves any required control arm and that the number of arms it returns is allowed by the trial design.
Useful checks include applying the algorithm to boundary cases before the trial begins: an exact tie, an estimate exactly equal to the threshold, no arm above the threshold, and an unavailable estimate. A simple implementation can be tested against hand-calculated examples. If software rounds estimates before comparing them, values near the cutoff can be assigned incorrectly; comparisons should use the specified precision and unrounded values where available.
- Changing the criterion after viewing interim results. Choosing a cutoff or retention count that makes the observed result look attractive is not prespecification. Set the rule and its branches before examining the decision data.
- Mixing effect directions or units. A larger number may represent less benefit for some endpoints. Orient the target effects consistently or define the rule in the appropriate direction.
- Leaving ties or an empty qualifying set unresolved. A verbal rule may then permit different actions by different reviewers. Give a deterministic tie procedure and an explicit fallback or no-selection outcome.
- Treating selection as proof. Retaining an arm says that the rule selected it, not that its true effect is beneficial, clinically important, or statistically established.
Ranking, thresholding, and combined rules are alternatives for different decision objectives, not interchangeable versions of the same method. Ranking is natural when the design has a fixed capacity and must choose the strongest estimates relative to one another. Thresholding is suitable when the design wants to retain arms meeting a prespecified estimate criterion, even if the number retained varies. A combined rule may be appropriate when both a minimum estimate and a capacity limit matter, but its extra branches also create more ways for an underspecified rule to fail.
All three approaches have limitations. A rule based on interim estimates may be affected by estimation variability, and selected arms can have estimates that are more favorable than their underlying effects. A deterministic rule does not remove that uncertainty. Selection also changes which treatments proceed, so later analyses must account for the design and the selection process when appropriate. The selection criterion and the inferential claims made after selection are distinct design questions.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- A top-one rule is applied to estimates of \(0.2\), \(0.1\), and \(-0.4\) points. Which arm is selected, and what does this selection establish?
- A rule retains arms with estimates strictly greater than \(1.0\). If an arm’s estimate is exactly \(1.0\), does it qualify? What detail would change if the rule instead used “at least \(1.0\)”?
- Two arm estimates are \(1.2\) and \(0.8\). A common constant of \(0.5\) is added to both. Explain what happens to their ranking and to eligibility under a threshold of \(1.0\).
- Name two boundary cases that should be addressed when specifying a threshold-based rule.
Solutions. (1) The arm with estimate \(0.2\) is selected because it has the largest estimate. This establishes only that it ranked first under this rule; it does not establish that its true effect is beneficial. (2) It does not qualify, because “strictly greater than” excludes equality. Under “at least \(1.0\),” equality qualifies. (3) The estimates become \(1.7\) and \(1.3\), so their ranking remains unchanged. Before the shift, only \(1.2\) meets the threshold; after it, both shifted estimates meet the threshold, showing that threshold eligibility can change under a common shift. (4) Examples include no arm meeting the threshold, an estimate exactly equal to the cutoff, a tie for the largest estimate, and an unavailable estimate. A complete rule specifies how the relevant cases are handled.