Selecting Only the Highest-Ranked Arm

Why Keep Only the Interim Leader?

At an interim analysis, a design may have resources to continue only one experimental arm. A direct rule is to retain the arm with the largest observed treatment-control estimate. This rule is simple to state, but its meaning is easy to overstate: it identifies the current leader, not necessarily an effective treatment or the truly best treatment. Its behavior is especially uncertain when several arms have similar effects.

The rule below concerns ranking on the unstandardized interim treatment-control estimates. As in the earlier material on ranking treatment arms with a shared control, the estimates are dependent because they use the same control group. For pairwise ranking, however, the common control estimate cancels. That fact is central to understanding how close arms behave.

Definition and Working Model

Suppose $m$ experimental arms are active at an interim analysis, and let $\mathcal A$ denote their prespecified index set. Let $\bar Y_j$ be the interim sample mean in experimental arm $j$, and let $\bar Y_0$ be the interim control mean. Assume outcomes have been oriented so that larger values are better.

Definition: The interim treatment-control estimate for arm $j$ is $\hat\theta_j=\bar Y_j-\bar Y_0$. The best-arm-only rule retains an arm whose estimate is largest: $$ \hat j=\arg\max_{j\in\mathcal A}\hat\theta_j. $$ If estimates are distinct, this retains exactly one experimental arm. The rule does not require that the largest estimate be positive or exceed a minimum effect.

For the calculations in this tutorial, suppose interim participants are distinct across randomized arms, outcomes are independent across participants, and each arm's sample mean estimates its arm's population mean without bias. For the shared-variance formulas, also suppose outcomes have a common variance $\sigma^2$, with sample sizes $n_j$ in arm $j$ and $n_0$ in control. Normal outcomes give exact normal sampling distributions; with adequate sample sizes, a normal approximation for the means may be used. These assumptions are for the calculations, not a requirement that every operational selection rule use normal outcomes.

Write $\theta_j=\mu_j-\mu_0$ for the true treatment-control effect. Under the common-variance model,

\operatorname{Var}(\hat\theta_j)=\sigma^2\left(\frac{1}{n_j}+\frac{1}{n_0}\right), \qquad \operatorname{Cov}(\hat\theta_j,\hat\theta_k)=\frac{\sigma^2}{n_0}\quad (j\ne k).

The covariance is positive: both estimates contain the same random control mean with a minus sign. This dependence matters when describing the joint set of estimates. But for ranking two arms,

\hat\theta_j-\hat\theta_k =(\bar Y_j-\bar Y_0)-(\bar Y_k-\bar Y_0) =\bar Y_j-\bar Y_k.

Thus a common control shift moves every treatment-control estimate together and does not change their ordering. This cancellation assumes the same control estimate is used for both contrasts and that the ranking score is the unstandardized treatment-control estimate.

Worked Example: Choosing the Observed Leader

Suppose a hypothetical interim analysis has a control mean of 12.4 symptom-free days. The means for three experimental arms are 14.1, 14.0, and 13.2 days. Use the best-arm-only rule.

The treatment-control estimates are $14.1-12.4=1.7$ days, $14.0-12.4=1.6$ days, and $13.2-12.4=0.8$ days. The largest is 1.7 days, so the rule retains the first arm.

The first and second estimates differ by only $1.7-1.6=0.1$ day. The rule still chooses one arm, but this calculation alone does not show that the first arm has a meaningfully larger true effect. The observed ordering may reflect sampling variation.

Probability of Selecting the Better of Two Arms

With two arms, selection probability can be calculated directly under a normal sampling model. Let $D=\hat\theta_1-\hat\theta_2$. Arm 1 is selected when $D>0$. Under the common-variance model, and with independent experimental-arm samples and a shared control sample,

E(D)=\theta_1-\theta_2,\qquad \operatorname{Var}(D)=\sigma^2\left(\frac{1}{n_1}+\frac{1}{n_2}\right).

The control contribution cancels from the difference. If $D$ is normal, the probability that arm 1 is selected is

P(\hat j=1) =P(D>0) =\Phi\left( \frac{\theta_1-\theta_2} {\sigma\sqrt{1/n_1+1/n_2}} \right),

where $\Phi$ is the standard normal cumulative distribution function. The ratio compares the true effect gap with the standard deviation of the observed ranking difference. A small ratio means that random variation can readily reverse the ranking. The formula assumes the stated sampling model and no exact tie; a continuous normal difference has probability zero of an exact tie.

Worked Example: A Small True Gap

Consider two hypothetical arms with equal interim sample sizes of 100 per arm, a shared control sample of 100, and a common outcome standard deviation of 10 days. Suppose their true effects are 5.0 and 4.5 days. The first arm is truly better by 0.5 day.

Each treatment-control estimate has variance $10^2(1/100+1/100)=2$ days squared. Their covariance is $10^2/100=1$ day squared, so the variance of their difference is $2+2-2(1)=2$ days squared. Equivalently, the shared control cancels and the difference variance is $10^2(1/100+1/100)=2$.

Therefore $D$ has mean 0.5 day and standard deviation $\sqrt{2}=1.414$ days. The probability of selecting the truly better first arm is

P(D>0)=\Phi\left(\frac{0.5}{1.414}\right) =\Phi(0.3536)\approx 0.638.

Under this model, the first arm is selected about 63.8% of the time, while the second is selected about 36.2% of the time. The first arm is more likely to lead, but the observed leader is far from certain to be the truly better arm. These are selection probabilities under the hypothetical model, not probabilities that either treatment is effective.

Several Similar Arms and the Winner's Estimate

With more than two arms, selecting the maximum is a joint ranking problem. Under a multivariate normal model, the probability of selecting arm $j$ is the probability that its estimate exceeds every other active estimate:

P(\hat j=j)=P\left(\hat\theta_j-\hat\theta_k>0 \text{ for every }k\in\mathcal A,\ k\ne j\right).

The pairwise differences are generally dependent, so their probabilities cannot usually be multiplied as if each comparison were independent. The covariance structure must be respected, using an appropriate multivariate calculation or simulation when numerical selection probabilities are needed.

If all arms have the same true effect and their estimates are exchangeable and continuous, each arm has the same probability of being the maximum. Since exactly one arm is selected, the probability for each is $1/m$. A selected winner is therefore guaranteed to be the observed leader, but under this equal-effect model no arm has a higher chance than another of being selected.

The selected estimate is also optimistic on average when arms are tied. For a simple derivation, suppose $\hat\theta_j=\theta+\epsilon_j$, where the errors are independent, identically distributed, continuous, symmetric about zero, and have finite means. The reported estimate of the selected arm is $\theta+\max_j\epsilon_j$. For $m>1$, the maximum is at least the average of the errors in every realization:

\max_j\epsilon_j\geq \frac{1}{m}\sum_{j=1}^m\epsilon_j.

The inequality is strict with positive probability when the errors are continuous and not all equal. Taking expectations gives $E(\max_j\epsilon_j)>0$, because each error has mean zero. Thus the estimate belonging to the winner tends to exceed the common true effect. This is selection-induced optimism, not evidence that the estimated effect is biased for a fixed arm before selection.

Worked Example: Equal Effects, One Winner

Suppose three hypothetical experimental arms truly each improve a measure by 2.0 points relative to control. At an interim analysis their observed estimates are 1.5, 2.4, and 2.1 points. The best-arm-only rule selects the second arm because 2.4 is the largest value.

The selected estimate is 0.4 point above the common true effect, but one realization cannot establish that the second arm is superior. If the three estimates are exchangeable with continuous sampling distributions, symmetry implies that each arm has probability $1/3$ of being selected over repeated trials under this equal-effect scenario.

The selection rule does not average the three estimates or preserve uncertainty about the ranking; it reports the identity of the maximum. Across repetitions, the identity varies, and the selected estimate is systematically pulled upward relative to the common effect under the independent-error assumptions in the derivation.

When the Rule Is Useful—and What It Does Not Establish

Best-arm-only selection is useful when the design genuinely requires one experimental arm to continue and the goal at this decision is relative prioritization. It differs from selecting every arm above a minimum effect: a threshold rule can retain several arms or none, whereas the best-arm-only rule retains one whenever the active set is nonempty and a maximum is defined. The choice between them is a design objective, not a statistical conclusion forced by the data.

A maximum estimate is not a test of benefit. If every estimated effect is negative, the rule still selects the least negative estimate. Likewise, selecting an arm does not show that its effect is clinically important, that it is statistically significant, or that it would remain the leader with additional data. Separate stopping rules, confirmatory inference, and control of error rates must be specified where required; ranking alone provides none of these guarantees.

Practical checks should focus on the stability and meaning of the ranking. Report the leading estimates and their gaps, alongside their uncertainty, rather than reporting only the winner. When arms are close relative to the variability of pairwise differences, describe selection as uncertain. For planning, simulate the complete interim rule under plausible effect patterns and the planned covariance structure; summarize how often each arm is selected, including under equal or nearly equal effects. This is more informative than treating a single observed ranking as decisive.

Common mistake: “Arm 1 has the largest estimate, so it is the best treatment.” The rule establishes only that arm 1 has the largest observed interim estimate among the candidates.
Correction: Sampling variation can change the order, particularly when true effects are close. Interpret the selection as a design decision based on interim evidence, not proof of true superiority.
Common mistake: Treating the shared-control treatment estimates as independent when calculating the probability of a winner.
Correction: Their covariance is positive. Use the joint covariance structure for multi-arm probabilities, while recognizing that a common control estimate cancels in each raw pairwise ranking difference.
Common mistake: Concluding the selected arm is beneficial because it was selected.
Correction: The maximum can be negative, and ranking does not itself test whether any effect exceeds zero or a clinically relevant value.

With discrete or rounded outcomes, exact ties may occur. A complete protocol needs a prespecified way to handle them; the continuous-model probability calculations above do not assign positive probability to exact ties.

Key takeaway: Selecting only the highest-ranked arm is a relative rule: it chooses the largest interim treatment-control estimate. Its reliability depends on the gaps between true effects relative to sampling variability, and the winning estimate is prone to upward selection bias.

Check Your Understanding

Answer each question, then compare with the solutions below.

Two arms have interim treatment-control estimates of 0.3 and 0.1 units. What does the best-arm-only rule select, and what does that choice establish?
Under the equal-sample common-variance model, explain why the shared control variance does not appear in the variance of the difference between two raw treatment-control estimates.
Two arms have a true effect gap of zero and exchangeable continuous estimates. What is the probability that either specified arm is selected?
Why can the mean estimate among selected winners exceed the common true effect when several arms have equal effects?

Solutions. (1) It selects the arm with estimate 0.3 units. This establishes only that it is the observed leader; it does not establish benefit or true superiority. (2) Subtracting the estimates gives $(\bar Y_1-\bar Y_0)-(\bar Y_2-\bar Y_0)=\bar Y_1-\bar Y_2$, so the common control term cancels. (3) Each arm has probability $1/2$ by exchangeability and the probability of a tie is zero. (4) The maximum of the sampling errors is at least their average and is strictly greater with positive probability; since the average error has expectation zero, the maximum has positive expectation.