Selecting Every Arm Above a Minimum Effect

Why Select by a Minimum Estimated Effect?

At an interim analysis, a trial may need to decide which experimental arms should continue. One transparent rule is to retain every arm whose estimated effect exceeds a prespecified minimum. Unlike a rule that keeps only the best-performing arm, this criterion can retain several arms when each clears the cutoff. It can also retain none.

The rule is simple to calculate, but its meaning depends on details that must be fixed in advance: which interim estimator is used, what direction counts as benefit, what units the effect is measured in, and whether the cutoff is strict or inclusive. The resulting set is a decision based on interim estimates. It is not, by itself, a hypothesis test or a statement that retained arms have established efficacy.

For a continuous outcome where higher values are better, one common estimator is the difference between the interim sample means: \(\hat{\theta}_{i,1}=\bar{x}_{i,1}-\bar{x}_{0,1}\), where arm 0 is control. If lower values are better, orient the effect so that positive values still indicate benefit; for example, define the effect as control mean minus treatment mean. All arms must be scored in a consistent direction. The cutoff has the same units as the effect estimate: a cutoff of 2 points is not interchangeable with a standardized cutoff of 0.2 standard deviations.

Applying the Rule to Interim Data

Consider a trial with a common control and several experimental arms. The required operational assumptions are that the interim estimates refer to the same endpoint and time point, use a consistent treatment-effect definition, and are available for every arm being considered. The rule also assumes that the cutoff was fixed before inspecting the interim estimates. These conditions make the comparison meaningful and prevent the cutoff or scoring convention from being chosen to favor observed results.

The following examples use hypothetical interim summaries. In each, the outcome is measured in points and higher values indicate benefit. Each treatment-control contrast is computed using the same interim control mean. The sample sizes are included to make clear that the summaries come from interim groups; the selection rule itself compares the estimated effects with the cutoff.

Uncertainty and the Probability of Retention

A cutoff applied to an estimate does not remove sampling variability. To describe that variability, suppose the interim estimates are approximately jointly normal: \((\hat{\theta}_{1,1},\ldots,\hat{\theta}_{K,1})^\mathsf{T}\sim N_K(\boldsymbol{\theta},\Sigma)\), where \(\theta_i\) is the true treatment effect for arm \(i\), and \(\Sigma\) is the covariance matrix of the estimates. This approximation is appropriate when the estimators have an approximately normal sampling distribution under the trial's design and analysis model. Its adequacy depends on the outcome, sample sizes, and estimator; it is not guaranteed merely because the selection rule is defined.

For an individual arm with standard error \(SE_i\), the normal model gives \(\hat{\theta}_{i,1}\sim N(\theta_i,SE_i^2)\). Standardizing the cutoff yields the probability that this arm is retained:

Here \(\Phi\) is the standard normal cumulative distribution function. The probability depends on the true effect relative to the cutoff and on the standard error. An arm with a true effect at the cutoff has probability one half of exceeding it under this symmetric model. More precise interim estimation makes retention more predictable, but does not change the rule's cutoff.

Why Shared-Control Dependence Matters

When treatment-control estimates use the same interim control group, they are generally correlated. Under the standard independent-group mean model with a common control sample, the contrasts share the random control mean. For two different arms \(i\) and \(j\), $$ \operatorname{Cov}(\bar{x}_{i,1}-\bar{x}_{0,1},\bar{x}_{j,1}-\bar{x}_{0,1}) =\operatorname{Var}(\bar{x}_{0,1}), $$ provided the experimental-arm samples are independent of one another and of the control sample. The resulting covariance and correlation are properties of the estimator and design, not of the observed effect sizes.

The marginal probability calculation above remains useful arm by arm, but probabilities of particular combinations of retained arms must use the joint distribution when dependence is present. For example, multiplying two marginal retention probabilities gives the probability that both arms qualify only if their qualification events are independent. Shared-control contrasts generally do not satisfy that condition. However, the expected number of retained arms does not require independence: writing \(I_i=1\) when arm \(i\) qualifies and \(I_i=0\) otherwise, $$ E(|\mathcal{S}_1|)=E\left(\sum_{i=1}^{K} I_i\right) =\sum_{i=1}^{K}P(i\in\mathcal{S}_1), $$ by linearity of expectation. This identity gives the expected count, not the probability of retaining a specific combination.

A practical diagnostic is to check the estimator definitions, interim sample sizes, common-control use, and covariance calculations before interpreting joint selection probabilities. If estimates have unequal precision, the same raw-effect cutoff still defines the rule, but it does not give every arm the same probability of passing when their true effects are equally distant from the cutoff. A normal approximation can also be unreliable with small samples, highly skewed outcomes, or an estimator whose sampling distribution is poorly approximated by a normal distribution.

How This Rule Differs From Other Selection Approaches

The minimum-effect rule answers a specific question: which arms have interim point estimates above this fixed effect value? It is distinct from several nearby procedures:

• Standardized selection scores: comparing effects divided by their standard errors ranks evidence relative to uncertainty, whereas this rule compares effects in their original units with an effect-scale cutoff. The two rules can select different sets.
• Significance-based screening: a test compares data with a null hypothesis using a test statistic and a significance level. Passing an effect cutoff is not equivalent to obtaining a small p-value, and the cutoff alone does not control a type I error rate.
• Ranking-based selection: a ranking rule orders arms, while this rule retains all arms meeting the threshold. A qualifying arm need not be the highest-ranked arm, and multiple arms can qualify.
• Combined interim gates: if a trial protocol requires both a minimum-effect condition and another continuation condition, the final continuation set must be defined using both conditions. Passing this minimum-effect rule alone does not imply that every other gate has been passed.

The rule is limited in what its binary decision conveys. It discards how far an estimate lies above or below the cutoff, and it does not account for uncertainty in the decision unless uncertainty is incorporated through a different rule. Because selection is based on noisy estimates, retained arms will tend to have estimates that are high relative to their true effects, especially when many arms are screened. Later estimates and confirmatory inference should therefore be interpreted under the trial's prespecified design and analysis plan rather than treating selection as proof of benefit.

Common Mistakes and a Reliable Workflow

A frequent error is to treat a minimum-effect cutoff as if it were a significance boundary. For example, an analyst might say, “The estimate exceeded 0.5 points, so the treatment effect is statistically significant.” That conclusion does not follow: the estimate's standard error, a specified hypothesis test, and the relevant error-rate procedure are not part of the cutoff comparison. The correct statement is that the observed interim estimate exceeded the prespecified cutoff and the arm was retained under this rule.

Another common failure is mixing signs or scales. If a negative contrast indicates benefit but the cutoff is positive, applying the rule without reorienting the contrast can reverse the intended decision. Likewise, an estimate in standardized units cannot be compared with a cutoff in outcome points. Before applying the rule, verify the effect direction, units, estimator, cutoff, and strictness of the inequality. Then calculate each arm's contrast separately and list all qualifying indices; do not stop after identifying the largest estimate.