Continuation Thresholds for Strong Interim Evidence

Why Set a Continuation Threshold?

At an interim analysis, an MAMS trial must decide which experimental arms will receive further study. A continuation threshold makes one part of that decision explicit: an arm must show at least a prespecified amount of interim evidence in the favorable direction to remain under consideration. The threshold is an operational selection criterion. It is not, by itself, a declaration that the treatment is effective or a substitute for the trial’s final confirmatory analysis.

The threshold should be chosen before the interim data are examined. Its practical meaning depends on the statistic, its direction, the timing of the interim analysis, and the outcome scale. A numerical cutoff cannot be interpreted independently of those choices.

Definition: Let \(\hat{\theta}_j\) be the interim estimated treatment effect for experimental arm \(j\) relative to control, oriented so that larger values favor the experimental arm, and let \(SE_j\) be its standard error. The interim standardized statistic is \(Z_j=\hat{\theta}_j/SE_j\). A one-sided continuation threshold \(c_{\mathrm{cont}}\) retains arm \(j\) when \(Z_j\geq c_{\mathrm{cont}}\). Equivalently, if the one-sided interim p-value is calculated for \(H_0:\theta_j\leq 0\) against \(H_a:\theta_j>0\), the arm meets a p-value threshold \(\gamma\) when \(p_j\leq\gamma\), where \(c_{\mathrm{cont}}=\Phi^{-1}(1-\gamma)\) under the standard normal reference distribution.

This definition assumes that the direction of benefit has been specified in advance. For a time-to-event outcome where a smaller hazard is favorable, for example, the effect or statistic must be oriented so that greater values still represent greater benefit. If the statistic is not oriented consistently, an upper-threshold rule can retain the wrong arms.

Conditions for Interpreting the Rule

The rule is meaningful only if the interim estimates and standard errors are suitable for the planned analysis. In the common large-sample setting, the working approximation is \(Z_j\sim N(\delta_j,1)\), where \(\delta_j\) is the standardized interim effect under the assumed true treatment effect and interim information. This approximation is not automatic: the estimator, variance calculation, endpoint, and interim data cut must all be appropriate.

The comparison and direction of benefit are defined, including which control observations contribute to each contrast.
The interim information is sufficiently mature and the estimator and \(SE_j\) use a prespecified, defensible analysis method.
The threshold and its reference distribution are fixed before reviewing the interim results; if a different reference distribution is used, the cutoff must be derived for it.
Missing data, deviations from the analysis plan, or an incorrect standard error have not made the interim statistic materially misleading.

For a binary or time-to-event endpoint, a normal approximation may be reasonable with adequate information, but sparse events can make it poor. For an outcome where standard errors are estimated, the relevant standardized statistic may have a different reference distribution in small samples. A threshold calibrated using a standard normal distribution should not be applied mechanically if the actual planned statistic does not follow that approximation.

Calibrating the Threshold

A continuation threshold trades off two operating characteristics. A higher threshold makes continuation less likely under a null or weak effect, but also makes it less likely that an arm with a promising effect passes at the interim. One transparent way to calibrate the threshold is to choose a target continuation probability \(q\) at a specified standardized effect \(\delta^*\). This is a design input, not a universal standard.

Z_j\sim N(\delta^*,1) \quad\Longrightarrow\quad P(Z_j\geq c_{\mathrm{cont}}\mid \delta^*) =1-\Phi(c_{\mathrm{cont}}-\delta^*).

Setting this probability to \(q\) and solving gives the threshold. The same calculation also shows the continuation probability under a null effect, by setting \(\delta=0\).

q=1-\Phi(c_{\mathrm{cont}}-\delta^*) \quad\Longrightarrow\quad c_{\mathrm{cont}}=\delta^*+\Phi^{-1}(1-q), \qquad P(Z_j\geq c_{\mathrm{cont}}\mid \delta=0)=1-\Phi(c_{\mathrm{cont}}).

This derivation uses a normal approximation with unit variance for the standardized statistic and assumes the specified standardized effect is appropriate for the interim information. In an actual design, those assumptions should be checked against the planned analysis. The calculation describes one arm’s probability of crossing the threshold; it does not, by itself, describe the probability that at least one of several arms crosses.

Worked Example: Translating a Prespecified Evidence Level

Suppose an interim protocol requires one-sided interim evidence at level \(\gamma=0.10\), using a standard normal reference distribution. The treatment effect has been oriented so larger values favor treatment. Find the equivalent standardized-score threshold.

By definition, the threshold is the standard normal quantile with upper-tail probability 0.10:

c_{\mathrm{cont}}=\Phi^{-1}(1-0.10)=\Phi^{-1}(0.90)\approx 1.282.

An arm with \(Z_j\geq1.282\) meets this continuation criterion; an arm with \(Z_j<1.282\) does not. This is a rule for continuing an arm in this hypothetical design. The value 0.10 is not the final significance level, and crossing the threshold does not establish a confirmatory treatment effect.

Worked Example: Calibrating for a Promising Effect

A design team considers a continuation threshold that gives an 80% chance of retaining an arm when its standardized interim effect is \(\delta^*=2.00\). Assume \(Z_j\sim N(\delta,1)\) for this calculation. Find the threshold, then calculate the null continuation probability.

Using the calibration formula,

c_{\mathrm{cont}}=2.00+\Phi^{-1}(1-0.80) =2.00+\Phi^{-1}(0.20) \approx 2.00-0.8416=1.158.

Under the specified promising effect, the continuation probability is \(P(Z_j\geq1.158\mid\delta=2.00)=1-\Phi(1.158-2.00)=\Phi(0.842)\approx0.800\). Under the null effect, it is \(P(Z_j\geq1.158\mid\delta=0)=1-\Phi(1.158)\approx0.123\). Thus, under these assumptions, the chosen rule retains about 80% of arms at the stated promising effect and about 12% under the null. These are operating characteristics of the threshold, not probabilities that an individual treatment is effective.

Applying the Rule to Interim Results

The calculation for an observed arm is direct: compute the prespecified interim statistic, compare it with the prespecified threshold, and record whether the arm satisfies the rule. The decision should not be altered after seeing results by changing the threshold, the contrast direction, or the standard error method. In a shared-control trial, different treatment-versus-control statistics can be correlated because they use common observations. That dependence does not change the arm-by-arm comparison with a fixed cutoff, but it matters when evaluating joint operating characteristics across arms.

Worked Example: Screening Three Arms Against One Control

In a hypothetical trial, larger outcome values are beneficial. At an interim analysis, the estimated treatment-minus-control effects and standard errors are: arm A, 3.4 points with \(SE_A=2.0\) points; arm B, 2.8 points with \(SE_B=2.0\) points; and arm C, 4.0 points with \(SE_C=2.0\) points. Suppose the prespecified continuation threshold is \(c_{\mathrm{cont}}=1.50\).

For arm A, \(Z_A=3.4/2.0=1.70\), so \(1.70\geq1.50\) and arm A meets the criterion. For arm B, \(Z_B=2.8/2.0=1.40\), so \(1.40<1.50\) and arm B does not meet it. For arm C, \(Z_C=4.0/2.0=2.00\), so \(2.00\geq1.50\) and arm C meets it. Under this rule, arms A and C continue and arm B does not.

These are decisions from the defined operational rule, not final conclusions about efficacy. Also, the three statistics need not be independent because the arms share a control group. The individual comparisons above remain correct, but multiplying separate crossing probabilities as if the statistics were independent would generally give an incorrect probability for joint retention.

Continuation Is Not Confirmation

A continuation threshold answers a design question: which arms should proceed to later stages under the selection plan? Final confirmatory testing answers a different question: whether the planned evidence supports a treatment claim while controlling the error rates required by the trial. A threshold can be deliberately chosen to retain promising arms without being a final rejection boundary. Interim data may also contribute to the final analysis, so the full design must account for the interim selection and testing plan rather than treating the final test as an unrelated analysis.

For example, suppose a hypothetical protocol sets \(c_{\mathrm{cont}}=1.50\) but requires a final standardized statistic of at least 2.40 for a particular confirmatory decision. An arm with interim \(Z_j=1.70\) passes the continuation rule. That result alone neither meets the stated final criterion nor proves the effect is positive. The two cutoffs serve different purposes and are judged at different stages.

A single-threshold futility rule, by contrast, specifies evidence below which an arm is stopped for lack of promise. A continuation threshold specifies evidence at or above which an arm qualifies to continue. They are related operational ideas, but the thresholds need not be complements or use the same scale. A rule that retains only arms exceeding a high evidence bar may stop arms that would have survived a less demanding futility screen.

Diagnostics, Failure Modes, and Limitations

Before relying on observed threshold crossings, compare the analysis actually performed with the design specification. Check the endpoint direction, the interim data cut, the calculation of the treatment-control contrast and its standard error, and the reference distribution used to convert evidence to a threshold. If any of these differ, the reported crossing may not correspond to the planned rule.

Direction reversal: If a lower outcome is favorable but the contrast is left unreversed, a large favorable effect can produce a negative statistic and fail an upper-threshold rule.
Unstable approximation: With sparse events or very small samples, a standard normal tail probability may be inaccurate. Use the reference distribution and analysis specified for the design.
Post hoc threshold choice: Selecting a cutoff after inspecting interim statistics makes the operating characteristics calculated for a prespecified cutoff inapplicable.
Ignoring dependence: Shared-control statistics may be correlated. Per-arm decisions can still be made, but joint retention rates require accounting for dependence.
Overinterpretation: Crossing the threshold is not proof of treatment benefit, and failing it is not proof that the treatment has no effect. The decision is conditional on a design rule and interim information.

No single continuation cutoff is optimal for every design. Its consequences depend on the timing and information at interim, plausible treatment effects, the cost of carrying an arm forward, and the trial’s confirmatory objectives. A normal-theory calculation gives a clear initial calibration, but important design-level probabilities may require evaluating the joint distribution and the full planned analysis. The threshold alone also does not specify what happens to the control group or how many arms must continue.

Key takeaway: A continuation threshold is a prespecified interim selection boundary. Define the statistic, favorable direction, reference distribution, and cutoff; assess its operating characteristics under relevant effects; and keep the continuation decision distinct from final confirmatory testing.

Common Mistake: Calling a Passed Threshold “Statistically Significant”

Wrong reasoning: “This arm crossed the continuation threshold, so the trial has shown that the treatment works.” This confuses a rule for allocating further study with a final confirmatory conclusion.

Correction: State only that the arm met the prespecified interim continuation criterion. Whether the final trial supports a treatment claim depends on the confirmatory hypotheses, analysis, and error-rate control specified for that trial. Likewise, an arm that does not meet the continuation rule has not thereby been shown to have exactly zero effect.

Check Your Understanding

Answer each question using the threshold definitions and assumptions above.

With a one-sided standard normal evidence threshold of \(\gamma=0.05\), what is the corresponding continuation cutoff?
An arm has an interim effect estimate of 1.8 units and \(SE=1.2\) units. If \(c_{\mathrm{cont}}=1.50\), does it meet the criterion?
Under the model \(Z\sim N(1.5,1)\), what is the continuation probability for \(c_{\mathrm{cont}}=1.0\)?
Why does an arm’s passing a continuation cutoff not establish a confirmatory treatment effect?

Solutions. (1) \(c_{\mathrm{cont}}=\Phi^{-1}(0.95)\approx1.645\). (2) \(Z=1.8/1.2=1.50\); because the rule uses \(Z\geq c_{\mathrm{cont}}\), the arm meets it exactly. (3) \(P(Z\geq1.0)=1-\Phi(1.0-1.5)=\Phi(0.5)\approx0.6915\). (4) The cutoff determines an interim selection decision; final confirmation requires the planned confirmatory test and its error-rate control, and the threshold crossing alone is not that test.