Confidence-Bound Selection Rules

Why Use a Confidence Bound at an Interim Analysis?

An interim estimate can lie above a clinical benchmark simply because of sampling variation. Conversely, an estimate below the benchmark may still be compatible with a worthwhile treatment effect if its standard error is large. A confidence-bound rule makes this uncertainty visible: rather than comparing only the estimate with the benchmark, it asks whether the data support effects on one side of that benchmark with a specified degree of confidence.

Consider a multi-arm trial in which larger treatment effects are better. For experimental arm \(j\), let \(\theta_j\) be its treatment effect relative to control, measured in the outcome’s units, with positive values indicating benefit. Let \(\Delta\) be the prespecified clinical benchmark: an effect is considered clinically worthwhile when \(\theta_j>\Delta\). The benchmark is a design input, not a value estimated from the interim data.

These are distinct decisions. Requiring the lower bound to exceed \(\Delta\) is a stringent positive-evidence rule. Dropping only when the upper bound is below \(\Delta\) is a more cautious futility rule: it avoids dropping an arm when its plausible effects still include clinically worthwhile benefit. A design may use one rule or define how both rules operate together. The rule, confidence level, benchmark, and handling of equality should be specified before examining interim results.

Model and Construction of the Bounds

Suppose the interim estimator \(\hat{\theta}_j\) is approximately unbiased for \(\theta_j\), and its estimated standard error \(SE_j\) adequately describes its sampling variability. For the basic normal approximation, assume \(\left(\hat{\theta}_j-\theta_j\right)/SE_j\) is approximately standard normal. This may be justified by a normal outcome model with suitable variance estimation or by a large-sample approximation. The estimator and standard error must correspond to the same treatment contrast and interim data.

For a one-sided error probability \(\gamma\), let \(z_{1-\gamma}\) be the \(1-\gamma\) quantile of the standard normal distribution. The one-sided lower and upper confidence bounds are \(L_j=\hat{\theta}_j-z_{1-\gamma}SE_j\) and \(U_j=\hat{\theta}_j+z_{1-\gamma}SE_j\). Each is a one-sided \(1-\gamma\) bound under the stated approximation. A two-sided \(1-2\gamma\) interval has these same endpoints; a two-sided 95% interval instead uses \(z_{0.975}\) at both ends. The confidence level must therefore be attached to the actual tail convention, not inferred from the word “bound.”

For example, with a one-sided 95% bound, \(\gamma=0.05\) and \(z_{0.95}\approx1.645\). Under the normal approximation, the lower bound has coverage \(P(L_j\leq\theta_j)\approx0.95\); the upper bound has coverage \(P(\theta_j\leq U_j)\approx0.95\). These are repeated-sampling statements about the procedure. They do not mean that, after observing the data, there is a 95% probability that the fixed parameter is above or below the realized bound.

The clinical benchmark need not be zero. If \(\Delta>0\), the rule concerns whether an effect is large enough to be worthwhile, not merely whether it differs from no effect. When the outcome is measured in minutes, points, or another unit, \(\theta_j\), \(\Delta\), \(\hat{\theta}_j\), and \(SE_j\) must all use compatible units. The direction of benefit must also be consistent: if lower outcomes are better, define the effect or reverse the inequalities accordingly.

What the Bound Rule Implies

The connection between a lower bound and a one-sided test follows directly from the bound’s construction. If \(L_j>\Delta\), then \(\hat{\theta}_j-z_{1-\gamma}SE_j>\Delta\), or equivalently \((\hat{\theta}_j-\Delta)/SE_j>z_{1-\gamma}\). Under the boundary value \(\theta_j=\Delta\), that standardized statistic has an approximately standard normal distribution. Thus, for a single prespecified comparison, the probability of satisfying this strict lower-bound rule at the boundary is approximately \(\gamma\). The corresponding upper-bound rule \(U_j<\Delta\) is equivalent to \((\hat{\theta}_j-\Delta)/SE_j<-z_{1-\gamma}\).

This calculation explains the marginal evidence associated with each rule; it does not calibrate the complete multi-arm, multi-stage trial. Applying the same marginal rule to several arms does not generally control the probability of any false selection at \(\gamma\). Shared control can make arm estimates correlated, and repeated looks can also affect error rates. A confidence-bound screen is therefore a decision component, not a substitute for the design’s prespecified final testing and error-rate calibration.

A joint confidence statement across arms may require an adjustment. One simple option is Bonferroni: for \(m\) arms and a desired familywise noncoverage probability no greater than \(\eta\), construct each one-sided bound at noncoverage probability \(\eta/m\). The probability that any of the \(m\) bounds fails to cover in its relevant direction is then at most the sum of the marginal failure probabilities, hence at most \(\eta\). This bound does not require independence, though it can be conservative. Whether a joint confidence statement is needed depends on the purpose of the rule; the adjustment still does not automatically calibrate the full trial’s final error rate across stages.

Choosing and Diagnosing a Rule

The right bound depends on the decision the trial is designed to make. A lower-bound requirement is appropriate when continuation is reserved for arms with affirmative interim evidence that the effect exceeds a clinically worthwhile threshold. An upper-bound futility rule is appropriate when the design seeks to drop only arms for which worthwhile benefit is sufficiently implausible under the chosen uncertainty criterion. A combined rule can retain arms with \(L_j>\Delta\), drop arms with \(U_j<\Delta\), and leave arms with \(L_j\leq\Delta\leq U_j\) unresolved. It must specify what happens to unresolved arms, since the bounds alone do not make that operational decision.

Several checks help identify a rule that is being applied incorrectly:

• Check the benefit direction and units. A reversed effect definition reverses the appropriate inequality. A benchmark in points cannot be compared with a bound in standardized units without a justified conversion.
• Check the tail convention. A one-sided 95% bound uses \(z_{0.95}\approx1.645\); an endpoint of a two-sided 95% interval uses \(z_{0.975}\approx1.960\). Substituting the wrong quantile changes the operating rule.
• Check the interim standard error. A standard error from the wrong analysis population, contrast, or information time produces a bound that does not describe the intended uncertainty.
• Check whether the claim is marginal or joint. Separate arm-wise bounds are not automatically a simultaneous confidence statement. Dependence from shared control does not remove the need to identify the intended guarantee.
• Check whether the rule is being mistaken for final inference. Selecting or dropping arms using interim bounds does not itself establish that the final confirmatory tests have their intended type I error.

Compared with a point-estimate threshold, confidence-bound selection explicitly accounts for precision: the same estimate can qualify when \(SE_j\) is small and fail to qualify when \(SE_j\) is large. Compared with ranking arms by a standardized score, a bound rule uses an interpretable effect benchmark in the outcome’s units; a high score alone need not represent a clinically worthwhile effect. A p-value rule tests a specified null hypothesis, but a small p-value against no effect does not necessarily show that the effect exceeds a positive clinical benchmark. These approaches can be combined in a design, but their criteria and inferential roles should not be conflated.

Confidence-bound selection has important limitations. At an early interim analysis, a wide interval may leave many arms unresolved, so a cautious upper-bound futility rule can retain arms that later prove unpromising. Conversely, a stringent lower-bound rule may drop an arm whose true effect is worthwhile because its estimate is imprecise. Confidence bounds also rely on the chosen model and standard error; severe small-sample distortion, outcome changes, or inappropriate variance estimates can undermine their nominal coverage. The rules do not account automatically for the consequences of repeated selection or repeated testing. Those features must be addressed by the full design’s operating characteristics and final inferential plan.

Common Mistake