Stage-Specific Selection Thresholds

Why Use Different Cutoffs at Different Stages?

An interim continuation rule determines which experimental arms remain eligible for further study. A single cutoff is simple, but the same numerical requirement at every stage need not represent the same policy as information accumulates. A design may instead set one cutoff for an early look and another for a later look, reflecting its prespecified priorities at each point in the trial.

The key is to define the rule before applying it: identify the score, its favorable direction, the stage at which it is evaluated, and the cutoff assigned to that stage. These are selection decisions. A continuation cutoff, by itself, is not a confirmatory boundary and does not establish that an arm is effective.

Definition: Let $S_{jk}$ be the prespecified interim selection score for experimental arm $j$ at interim stage $k$, with larger values representing more favorable evidence. Let $c_k$ be the continuation cutoff assigned to stage $k$. Under a threshold-only rule, arm $j$ qualifies to continue at that stage exactly when $S_{jk} \geq c_k$. The set of qualifying arms is $\{j:S_{jk}\geq c_k\}$, among arms eligible for assessment at stage $k$.

A stage-specific policy is the sequence of cutoffs $c_1,c_2,\ldots,c_K$ for the prespecified interim stages. The cutoffs may differ, but they must be expressed on the scale of the score being compared. For instance, a cutoff of 1.2 on a standardized score is not interchangeable with a cutoff of 1.2 units on an estimated treatment effect.

Assumptions and Specification

The rule is meaningful when its components are settled in advance and interpreted consistently. In particular, the following conditions should hold:

The interim stages and the cutoff for each stage are specified before the outcomes used for selection are examined.
The score has a defined favorable direction at every stage. If larger values are favorable, the rule uses $S_{jk}\geq c_k$ at every stage.
The score measures the same target and uses a comparable scale across stages, or the stage-specific cutoffs explicitly account for any scale change.
The threshold is applied to the correct stage and only to arms eligible for assessment under the design.
The cutoff is understood as a selection rule. Any claim about type I error, power, or confirmatory evidence requires separate design and analysis considerations.

These conditions do not require the scores for different arms to be independent. With a shared control, scores can be dependent; that dependence affects joint selection behavior, even though the rule can still be applied arm by arm. Nor does the definition require thresholds to increase or decrease with stage. The sequence reflects the chosen policy and the score scale.

Worked Example: Applying Two Standardized-Score Cutoffs

Suppose a hypothetical trial uses a standardized selection score for which larger values are more favorable. The prespecified cutoff is $c_1=1.0$ at the first interim stage and $c_2=1.8$ at the second. At stage 1, three active arms have scores 0.7, 1.2, and 2.1. At stage 2, the two arms that continued have scores 1.6 and 2.0.

At stage 1, compare each score with $c_1$:

0.7<1.0,\qquad 1.2\geq1.0,\qquad 2.1\geq1.0.

Thus, the first arm does not qualify at stage 1, while the second and third do. At stage 2, use $c_2$, not $c_1$:

1.6<1.8,\qquad 2.0\geq1.8.

Of the two arms assessed at stage 2, only the arm with score 2.0 qualifies to continue. These decisions follow the specified selection rule; they are not conclusions that an arm is or is not efficacious.

How Accumulating Information Changes a Standardized Score

A threshold's practical meaning depends on the score scale. Consider a simple model for a treatment-effect estimate at stage $k$. Let $\hat{\theta}_{jk}$ estimate the effect for arm $j$, with favorable effects positive. Suppose its standard error is $1/\sqrt{I_k}$, where $I_k$ is the information available at stage $k$. Under a normal approximation,

\hat{\theta}_{jk}\sim N\left(\theta_j,\frac{1}{I_k}\right), \qquad Z_{jk}=\frac{\hat{\theta}_{jk}}{1/\sqrt{I_k}} =\hat{\theta}_{jk}\sqrt{I_k}.

Here $\theta_j$ is the true effect, and $Z_{jk}$ is the standardized score. The stated model is an illustration, not a requirement for defining a threshold. It assumes the estimate is approximately normal, its standard error is correctly represented by $1/\sqrt{I_k}$, and the direction of benefit is defined in advance.

Under this model, the score has mean $\theta_j\sqrt{I_k}$ and variance 1. Thus, if information increases, a fixed positive effect tends to produce a larger standardized score. A fixed cutoff on $Z_{jk}$ therefore does not correspond to a fixed cutoff on the estimated effect: the equivalent effect threshold is $c_k/\sqrt{I_k}$. Conversely, if the policy requires an estimated effect to exceed a fixed value $d$, the equivalent standardized-score cutoff at stage $k$ is $d\sqrt{I_k}$.

Derivation: Under the normal model, continuation occurs when $Z_{jk}\geq c_k$. Since $Z_{jk}=\hat{\theta}_{jk}\sqrt{I_k}$ and $I_k>0$, this is equivalent to $\hat{\theta}_{jk}\geq c_k/\sqrt{I_k}$. Also, $Z_{jk}\sim N(\theta_j\sqrt{I_k},1)$, so the probability of qualifying at that stage, if assessed, is $$ P(Z_{jk}\geq c_k)=\Phi(\theta_j\sqrt{I_k}-c_k), $$ where $\Phi$ is the standard normal cumulative distribution function. The probability depends on both the cutoff and the information available.

This calculation explains why a later cutoff may appropriately differ from an earlier one, but it does not prescribe a universal sequence. A design that intends to maintain a particular effect-scale hurdle can choose stage cutoffs that reflect the information at each look. A design with another selection objective may choose a different sequence. In either case, the policy should be explicit about what the score and cutoff mean.

Worked Example: Matching a Cutoff to the Score Scale

In a hypothetical trial, the stage 1 information is $I_1=25$, and the stage 1 cutoff on the standardized score is $c_1=1.0$. At stage 2, the information is $I_2=100$, and the cutoff is $c_2=2.0$. The estimated effects at the two stages are $\hat{\theta}_{j1}=0.24$ and $\hat{\theta}_{j2}=0.21$, in hypothetical effect units.

At stage 1, the standard error is $1/\sqrt{25}=0.20$, so

Z_{j1}=\frac{0.24}{0.20}=1.20.

Because $1.20\geq1.0$, the arm qualifies at stage 1. The equivalent effect threshold is $c_1/\sqrt{I_1}=1.0/5=0.20$, and $0.24\geq0.20$ gives the same decision.

At stage 2, the standard error is $1/\sqrt{100}=0.10$, giving

Z_{j2}=\frac{0.21}{0.10}=2.10.

Because $2.10\geq2.0$, the arm qualifies at stage 2 as well. The equivalent effect threshold is $c_2/\sqrt{I_2}=2.0/10=0.20$. Although the estimated effect fell from 0.24 to 0.21 effect units, the arm still meets the later standardized-score cutoff. The calculation shows why checking the estimate alone, without its stage-specific standard error and cutoff, can give the wrong selection decision.

Worked Example: How Information Affects Qualification Probability

Suppose a hypothetical arm has true effect $\theta_j=0.30$ effect units. Under the normal model above, let $I_1=25$, $I_2=100$, and use standardized-score cutoffs $c_1=1.0$ and $c_2=2.0$. Assume for this calculation that the arm is assessed at each stage; the values below are stage-specific marginal probabilities, not the probability of reaching and passing both stages.

At stage 1, the standardized score has mean $\theta_j\sqrt{I_1}=0.30(5)=1.5$ and variance 1. Therefore,

P(Z_{j1}\geq1.0)=\Phi(1.5-1.0)=\Phi(0.5)\approx0.6915.

At stage 2, its mean is $\theta_j\sqrt{I_2}=0.30(10)=3.0$, so

P(Z_{j2}\geq2.0)=\Phi(3.0-2.0)=\Phi(1.0)\approx0.8413.

Under these assumptions, the arm's probability of meeting the stage-specific cutoff is about 69.2% at stage 1 and 84.1% at stage 2 when assessed at each look. The higher later probability reflects both increased information and the chosen cutoff. It is not a confirmatory power calculation, and it does not account for selection at the earlier stage.

Choosing, Comparing, and Checking a Policy

A constant cutoff is the simplest alternative: it applies $c$ at every stage. It is easy to communicate, but on a standardized-score scale it can represent a less demanding effect estimate as information grows, because the equivalent effect threshold is $c/\sqrt{I_k}$. Stage-specific cutoffs allow the rule to account for that change or to express a deliberately different policy at each stage.

Confidence-bound selection rules, introduced in the earlier material on confidence-bound selection, compare an uncertainty bound with a criterion rather than comparing only a score with a cutoff. Such a rule may be represented as a score threshold after algebraic rearrangement in particular settings, but its interpretation depends on the bound, its confidence level, and the scale. A calibrated confirmatory boundary has a different purpose: it is designed in relation to error-rate control. Do not treat a selection cutoff as if that calibration had been performed.

Before applying stage-specific cutoffs, check the following:

Confirm that each cutoff is paired with the intended stage and that the score direction is consistent.
Check that the score and cutoff use the same scale and units. A numerical threshold without its scale is incomplete.
Verify that the information or standard error used to form a standardized score corresponds to the data available at that interim analysis.
Review the written decision rule against borderline values, including whether equality qualifies. Under the rule defined here, $S_{jk}=c_k$ qualifies.
Keep the distinction between an arm's marginal probability of meeting one stage's cutoff and its probability of being selected through a sequence of stages. The latter depends on the joint behavior of interim scores and earlier selection.

A common failure is to apply the same cutoff at every stage simply because it is convenient, then describe it as preserving the same effect requirement. That claim is generally wrong for a standardized score when information changes. The correction is to translate each standardized cutoff to its implied effect threshold, $c_k/\sqrt{I_k}$, and check whether those thresholds express the intended policy. Another error is to call a qualifying score “statistically significant” or to infer a controlled error rate from the selection rule alone. The correction is to report the selection decision accurately and reserve confirmatory claims for an analysis designed and calibrated for that purpose.

Stage-specific thresholds make the policy transparent, but they do not eliminate the need to evaluate its operating behavior under plausible effects, variances, and dependencies. Results can be sensitive to the chosen cutoffs, misspecified information, or a score whose direction or scale changes between stages. Prespecification and clear reporting help make the rule reproducible; they do not make it universally optimal.

Key takeaway: Assign a cutoff to each interim stage, apply it to a score with a consistent favorable direction and scale, and interpret the resulting decision as selection. A changing cutoff can respond to accumulating information, but it does not automatically provide confirmatory error-rate control.

Check Your Understanding

Answer each question, then compare your reasoning with the brief solutions.

An arm has score 1.4 at stage 1, where $c_1=1.5$, and score 1.7 at stage 2, where $c_2=1.6$. At which stage does it qualify?
For a standardized score with $I_k=64$ and $c_k=1.6$, what estimated-effect threshold is equivalent to the score rule?
Under $Z_{jk}\sim N(\theta_j\sqrt{I_k},1)$, write the probability that an arm meets cutoff $c_k$.
Why does meeting a continuation cutoff not, by itself, establish confirmatory statistical significance?

Solutions. (1) It does not qualify at stage 1 because $1.4<1.5$; it qualifies at stage 2 because $1.7\geq1.6$. (2) The equivalent threshold is $c_k/\sqrt{I_k}=1.6/8=0.20$ effect units. (3) The probability is $\Phi(\theta_j\sqrt{I_k}-c_k)$. (4) The cutoff defines a selection policy; confirmatory significance requires an inferential procedure with the relevant calibration and error-rate control.

Stage-Specific Selection Thresholds

What You'll Learn