Defining the Estimand After Arm Selection

Why the Target Must Be Named After Selection

In a multi-arm, multi-stage (MAMS) trial, an interim rule may stop some experimental arms and retain others for further study. The final report may then emphasize one arm. “The treatment effect” is not automatically a single, well-defined target: it might mean the true effect of the arm the rule selected, the effect of the truly best arm in the original set, or an average effect across the arms randomized at the start. These quantities answer different questions and need not be equal.

The target must be specified before choosing an estimator or interpreting a result. For example, an estimate for the selected arm does not by itself estimate the average effect of every randomized arm. Nor does the fact that an arm was selected establish that it is truly best. The selection rule determines which arm is highlighted; the estimand describes the effect the analysis is intended to learn about.

Arm-Specific Effects and Three Distinct Targets

Suppose a trial has one control arm, labelled \(0\), and \(K\) experimental arms, labelled \(1,\ldots,K\). Let \(Y(j)\) denote the outcome a member of the target population would have under assignment to arm \(j\), and let \(\mu_j=E\{Y(j)\}\). Assume for now that larger outcomes are better. Define each experimental arm’s treatment effect on the mean-difference scale relative to control as \(\delta_j=\mu_j-\mu_0\), in the outcome’s units. If smaller outcomes are better, the sign convention should be reversed or stated explicitly.

The selected-arm effect and best-arm effect can coincide, but they are defined differently. The former refers to the arm chosen by a rule that may depend on trial data; the latter refers to the arm with the largest true effect, whether or not the rule chooses it. The all-arm average instead summarizes the effects of the arms in a stated set. Equal weights answer a different question from weights based, for example, on a prespecified target distribution of treatment use.

Before the trial’s data are observed, \(J\) can vary across repeated trials, so \(\delta_J\) is a random quantity when the true effects are treated as fixed and the selection rule is data-dependent. Once a particular arm has been selected, its true effect is the corresponding fixed parameter \(\delta_j\); the observed estimate is not that parameter. A repeated-trial summary of selected-arm performance is \(E(\delta_J)=\sum_{j=1}^{K}P(J=j)\delta_j\), when the rule always selects one of these arms. This average over selection outcomes is not generally the best-arm effect or the average across all randomized arms.

Assumptions and What the Selection Rule Must Specify

These definitions rely on a clearly defined treatment contrast. The outcome, measurement time, target population, and scale must be stated, and the control must be identified. For the mean-difference formulation above, the arm-specific means must exist. In a randomized trial, random assignment supports comparisons of arm means for the randomized population, subject to consistent outcome measurement and appropriate handling of missing outcomes. Interpreting these comparisons as causal effects also requires consistency (the observed outcome under the assigned arm corresponds to that arm’s potential outcome) and no interference between participants.

Selection does not change the definitions of \(\delta_j\); it changes which arm is emphasized and can affect how an estimate behaves. To define the selected-arm target, specify the candidate arms, the stages at which decisions occur, the data or criteria used, the direction of benefit, any stopping conditions, and how ties are handled. Also state what happens if no experimental arm is selected. If the rule can stop all arms, a selected-arm effect is defined only for trials in which an arm is selected; an unconditional summary may instead need to include the probability of selecting none.

A MAMS rule may retain several arms rather than choose one. In that case, “selected-arm effect” is insufficiently specific: name the particular arm, define an effect for each retained arm, or define an aggregate over the retained set. If the target is an average, specify whether it covers all arms randomized initially or only those retained. These are distinct sets, and the choice changes the estimand.

A Short Derivation: Average Performance of the Selection Rule

If the rule always selects exactly one of the \(K\) experimental arms, let \(P(J=j)\) denote the probability that arm \(j\) is selected over repeated trials under the specified true effects and design. The selected effect equals \(\delta_j\) whenever \(J=j\). Partitioning over the possible selected arms gives

Here \(\mathbf{1}(J=j)\) is 1 if arm \(j\) is selected and 0 otherwise. The second equality follows because the true effects are fixed: on the event \(J=j\), the selected effect is \(\delta_j\). Since \(\sum_jP(J=j)=1\), this is a probability-weighted average of the arm effects, with weights determined by how often the rule selects each arm. It is not ordinarily the equal-weighted average \(\sum_j\delta_j/K\), which gives every randomized arm the same weight regardless of its selection probability.

This derivation describes the target’s repeated-trial average; it does not supply an estimator or correct an estimate for selection. The selection probabilities depend on the rule and the data-generating scenario. The key point here is definitional: one must say whether the goal is the selected arm in a particular trial, the repeated-trial average performance of the rule, the truly best arm’s effect, or an average over a specified set of arms.

Which Set of Arms Does the Average Cover?

An all-arm average must name its arm set. In MAMS designs, an interim decision can remove arms from further study, but those arms were still part of the original randomization. Averaging over the original randomized experimental arms describes that original set. Averaging only over surviving arms instead describes a data-dependent subset and is a different target. Neither is intrinsically wrong; the question and the weights determine which is relevant.

Choosing and Reporting the Estimand

A useful comparison is to ask what each target is intended to summarize. The selected-arm effect focuses on the arm chosen by a specified rule. The best-arm effect is an oracle benchmark: it refers to the maximum true effect among the arms in a stated candidate set, regardless of the rule’s actual choice. An all-arm average summarizes the entire stated set using prespecified weights. The last two do not become interchangeable because one arm was selected.

These targets also differ from their estimators. A sample mean difference calculated in the selected arm is an estimate, not the estimand itself. The estimate may be affected by random variation and by the fact that selection used trial data. Defining the target correctly is necessary before assessing an estimator’s properties, but it does not by itself establish that an estimator is unbiased or that an interval has its nominal coverage.

Diagnostics, Pitfalls, and Limitations

Several checks can reveal that a target has not been adequately defined. If a report says “effect in the winning arm” without stating the selection rule, the selected-arm target is incomplete. If “best treatment” is used for the arm with the largest observed estimate, observed leadership has been confused with the arm having the largest true effect. If an average is reported without arm weights and an arm set, its meaning is ambiguous. If the rule can retain multiple arms or select none, a single selected-arm effect may not be defined for every possible trial outcome.

• Wrong reasoning: “The arm with the largest interim estimate was selected, so its true effect is the largest.”
  Correction: Selection identifies the rule’s chosen arm, not necessarily the arm with the largest true effect. Define \(\delta_J\) and \(\max_j\delta_j\) separately.
• Wrong reasoning: “The average effect is the mean over the arms that reached the final stage,” even when the question concerns all initially randomized arms.
  Correction: State whether the target set is the original randomized arms or the retained arms. Those averages can differ.
• Wrong reasoning: “The estimate in the selected arm is the effect of that arm.”
  Correction: The effect is a population parameter; the estimate is calculated from observed data and is subject to sampling variation.

The best-arm effect is useful as a benchmark but may be difficult to learn precisely when several arms have similar true effects. The all-arm average can be scientifically relevant when the aim is to summarize a portfolio of treatments, but an equal-weighted average may not match a target population’s intended use of those treatments. The selected-arm target is directly tied to the decision rule, but a single trial’s selected arm need not represent the arm the rule would choose most often across repeated trials. These limitations concern the question being answered, not merely the calculation used to answer it.