The Selected Arm and the Truly Best Arm

Two Different Questions After Arm Selection

A MAMS design may use interim results to choose which treatment arms continue. At the end, the treatment with the largest observed estimate may be called the winner. That label answers a data-dependent question: which arm was selected by this design? It does not necessarily answer the scientific question which candidate treatment has the largest true effect? Keeping these questions separate is essential when interpreting treatment-effect estimates after selection.

Suppose the outcome is defined so that a larger treatment effect means greater benefit. Let \(\theta_j\) be the true effect for candidate arm \(j\), measured relative to the control, and let \(\hat{\theta}_j\) be an estimate of that effect. The candidate set is fixed for the comparison. In a MAMS design, the set of arms still eligible at a selection stage may be smaller; conclusions about the best arm must say whether “best” refers to all original candidates or only to the eligible arms.

The maximum true effect is a fixed feature of the candidate treatments, although it is usually unknown. By contrast, \(J\) depends on observed data and is therefore random across hypothetical repetitions of the design. Before a particular arm is selected, \(\theta_J\) is also a random quantity across repetitions: its value depends on which arm the rule selects. Once \(J\) is observed, the selected-arm effect is the fixed parameter for that arm. These are different targets, even when their numerical values happen to be close.

The selection rule matters to the definition. In general, let \(S_j\) be the quantity used to rank eligible arms at a selection stage, such as an interim estimate, and let \(\mathcal{A}\subseteq\mathcal{C}\) be the arms then eligible. If the rule selects the largest score, \(J=\arg\max_{j\in\mathcal{A}} S_j\), with a prespecified rule for ties. An arm excluded before that stage cannot be selected, but it may still be the truly best arm among all original candidates. A report should therefore identify the comparison set and the stage at which selection occurred.

Why the Distinction Changes the Interpretation

An estimate reported for the selected arm, \(\hat{\theta}_J\), is an estimate associated with the treatment actually selected. It is not automatically an estimate of \(\theta_*\), nor does its size prove that the selected arm is truly best. The maximum of the observed estimates, \(\max_j\hat{\theta}_j\), is a data-based quantity; it is not the maximum of the unknown true effects. Selection can make the largest observed estimate look especially compelling because the same data helped identify the winner.

There is also a repeated-design summary that should not be confused with either target. If selection always produces an arm, the average true effect of the arm selected over repetitions is \(E(\theta_J)\). This averages the true effects of whichever arms the rule selects. It is not generally equal to \(\theta_*\). If a design sometimes selects no arm, the corresponding average must specify whether it is conditional on selection occurring or how no selection is handled.

Selection Probability in a Simple Model

Selection errors can be described by a probability of correct selection. For a fixed candidate set and a rule that always selects an arm, this is \(P(J\in\mathcal{B})\). The probability depends on the true effects, the variability and dependence of the estimates, and the selection rule. The following two-arm calculation makes the distinction explicit without requiring a multi-stage correction procedure.

Assume two arms are compared, both are always eligible, and the arm with the larger estimate is selected. Suppose the estimates are unbiased and independent, with a common known standard error \(\sigma\):

where the two errors are independent. Assume \(\theta_1>\theta_2\), so arm 1 is uniquely truly best. Under these assumptions, the probability of correct selection can be derived by considering the difference between the estimates.

To justify the result, define \(D=\hat{\theta}_1-\hat{\theta}_2\). Correct selection occurs exactly when \(D>0\). Independence and normality imply that the difference has mean \(\theta_1-\theta_2\) and variance \(2\sigma^2\). Standardizing \(D\) gives:

This expression shows that a larger true-effect gap relative to the estimation noise makes correct selection more likely. It also shows why “the arm with the higher estimate” is not synonymous with “the truly best arm.” Equal true effects give no unique best arm; in that case, the relevant event is selecting any arm in the tied best set, rather than selecting one designated label.

Interim Selection and Final Estimation

In a MAMS design, the selection score and the final estimate need not use the same observations. For example, interim data may determine which arms continue, while later data contribute to the selected arm’s final estimate. This changes the information used to define \(J\), but it does not erase the distinction between \(\theta_J\) and \(\theta_*\). The selected-arm target refers to the effect of the arm that the rule actually retained; the best-arm target refers to the largest true effect in the specified candidate set.

Diagnostics, Comparisons, and Limitations

Before interpreting a selected-arm result, check the decision rule and the target it supports. In particular, establish whether selection was based on an interim statistic or a final estimate, which arms were eligible at each stage, and whether the claim concerns the original candidate set or only the surviving arms. If the rule can stop without selecting a treatment, state how that outcome is handled in summaries of selection performance.

The main alternatives answer different questions. Reporting \(\hat{\theta}_J\) describes the estimated effect for the selected treatment. Reporting all arm-specific estimates makes the observed pattern visible, although the estimates still do not reveal with certainty which true effect is largest. Reporting a probability of correct selection describes how often a specified rule would select a truly best arm under a stated model. Reporting \(E(\theta_J)\), when relevant, summarizes the true effect of the arm selected over repetitions. None of these quantities can be substituted for another without changing the question.

A common mistake is to reason, “Arm 2 had the largest estimate, so arm 2 is the truly best treatment, and its estimate is an estimate of the maximum true effect.” The correction is that the largest observed estimate identifies the winner under the stated rule; the effect parameter for that winner is \(\theta_J\), while the maximum true effect is \(\max_j\theta_j\). Establishing which arm is truly best requires more than identifying which estimate happened to be largest.

The simple probability calculation above assumes independent normal estimates, equal standard errors, two arms, and a rule that always selects the larger estimate. MAMS comparisons can violate these assumptions: estimates may be correlated because arms share a control group, precision may differ across arms, and eligibility depends on earlier stages. In those settings, the simple formula is not justified without modification. More generally, the probability of correct selection depends on the full joint distribution and the actual selection path, not just the final ranking.

Finally, the maximum true effect is an especially demanding target when several treatments have similar effects. A small change in observed data can change which arm ranks first, even if the effects are practically very similar. Conversely, a rule may reliably select one arm without showing that its advantage over other arms is clinically important. State the target, the candidate set, and the uncertainty relevant to the claim; do not treat a selected label as proof of a unique best treatment.