Intervals After Selecting the Best Treatment Arm

Why Selecting the Largest Estimate Changes the Interval

In a multi-arm trial, investigators may select the treatment arm with the largest estimated effect for reporting or further study. The estimate for that arm is not an ordinary, prespecified arm estimate: it was chosen because it was the largest among several noisy estimates. Even when every arm has the same true effect, the selected estimate tends to be high. An ordinary interval centered on it therefore need not have its advertised coverage for the selected arm.

This tutorial develops an exact interval in a deliberately simple setting, then identifies what changes when the arms have unequal effects. The symmetric model gives a transparent calculation, not a universal recipe. In more general designs, the selection rule, joint distribution of the estimates, and unknown effects all matter.

The common-effect assumption is crucial: it makes the selected estimate the maximum of identically distributed estimates of one shared parameter. It is not an assumption that can be silently imposed when the arms may have different true effects. The known-\(\sigma\) assumption likewise makes the normal calculations exact; substituting an estimated standard error does not automatically preserve exact coverage.

Distribution of the Selected Estimate

For any real number \(m\), the event \(M\le m\) occurs precisely when all \(K\) estimates are at most \(m\). Independence therefore gives

where \(\Phi\) is the standard normal cumulative distribution function. By symmetry, each arm is equally likely to be selected, and the distribution of \(M\) conditional on any particular arm being the winner is the same as its unconditional distribution. Thus the formula also describes \(M\) conditional on \(J=j\), for any fixed \(j\), under this common-effect model.

To obtain a central \(100(1-\alpha)\%\) interval, invert the distribution at probabilities \(\alpha/2\) and \(1-\alpha/2\). For observed maximum \(m\), the confidence set consists of values of \(\theta\) satisfying \(\alpha/2\le [\Phi((m-\theta)/\sigma)]^K\le 1-\alpha/2\). Taking the \(K\)th root and then applying the increasing function \(\Phi^{-1}\) yields the endpoints below.

For a short justification of the coverage statement, define \(U=[\Phi((M-\theta)/\sigma)]^K\). The distribution calculation shows that \(P_\theta(U\le u)=u\) for \(0<u<1\), so \(U\) is uniform on \((0,1)\). The interval includes \(\theta\) exactly when \(\alpha/2\le U\le1-\alpha/2\), an event with probability \(1-\alpha\). Symmetry makes this same result hold conditional on the identity of the winner.

How Ordinary Arm-Specific Intervals Compare

If arm \(i\) had been specified before looking at the data, its usual known-standard-error 95% interval would be \(Y_i\pm1.96\sigma\). For that fixed arm, the interval covers \(\theta\) with probability 0.95 under the normal model. But after selecting the largest estimate, the ordinary interval centered on \(M\) is generally not a 95% interval for \(\theta\).

In the common-effect model, the ordinary interval for the winner covers if and only if \(\theta-1.96\sigma\le M\le\theta+1.96\sigma\). Using the maximum’s distribution gives its exact coverage:

The subtraction matters: the event that the maximum is below the lower boundary must be removed as well as the event that it exceeds the upper boundary. For three arms, using the displayed rounded normal probabilities,

Thus the ordinary interval covers in about 92.7% of repetitions in this particular three-arm model, rather than 95%. Its coverage falls further as the number of equally uncertain arms increases. This comparison concerns the probability of covering the shared effect; it is not a claim that the ordinary interval is invalid for a genuinely prespecified arm.

Unequal Effects and the Selection Event

When arms have different true effects, write \(Y_i\sim N(\theta_i,\sigma_i^2)\), initially still assuming independent estimates and known standard errors. The maximum then has distribution \(\prod_i\Phi((m-\theta_i)/\sigma_i)\), which depends on every arm’s effect and standard error. In particular, it is not generally a distribution involving only the effect \(\theta_J\) of the selected arm. A confidence interval for that effect cannot be obtained by simply replacing \(\theta\) in the common-effect formula.

The selection event itself can be written \(Y_J\ge Y_i\) for every competing arm \(i\). Conditioning on that event changes the distribution of the selected estimate. For example, for two independent estimates with standard errors \(\sigma_1,\sigma_2\), the conditional cumulative distribution of \(Y_1\), given it wins, is

where \(\phi\) is the standard normal density. The numerator integrates the density of \(Y_1=t\) times the probability that \(Y_2<t\); the denominator is the probability that arm 1 wins. This expression depends on the competing effect \(\theta_2\) as well as \(\theta_1\). Inverting it for \(\theta_1\) therefore requires a way to handle the nuisance parameter \(\theta_2\), such as a justified conditional procedure, a specified joint model, or a method designed for the actual selection rule. The equal-effects formula does not solve that general problem.

Diagnostics, Alternatives, and Limitations

Before reporting a post-selection interval, check that the mathematical selection rule matches the trial analysis. Selecting the largest point estimate differs from selecting an arm based on a test statistic, posterior probability, interim boundary, or clinical constraint. The distribution used for interval construction must reflect the rule actually used and the dependence structure of the estimates. Shared control groups, unequal allocation, estimated variances, and adaptive changes to enrollment can all invalidate the independent, equal-variance model above.

A useful diagnostic is to compare each arm’s estimate and standard error, then ask whether the assumed symmetry and common effect are scientifically tenable. Also check whether the interval targets the common effect under a null-like symmetric model or the selected arm’s own effect when effects may differ. These are distinct inferential targets. The common-effect interval is exact only for the common parameter under the stated assumptions; it does not automatically provide coverage for \(\theta_J\) in a heterogeneous-effect setting.

The main alternative is to construct inference that explicitly accounts for the selection event under a more general model. Depending on the design and target, this may involve conditional inference, test inversion, or an unconditional method that averages over the selection process. These approaches require their own assumptions and may yield different intervals. Ordinary arm-specific intervals remain appropriate for prespecified arms under their model, but simply reporting one for the winner does not adjust for selection. Simultaneous intervals for several arms address a different target and are not a substitute for identifying the selected-arm target precisely.

Common mistake: “Each arm has a 95% interval, so the interval for whichever arm wins also has 95% coverage.” The premise describes each fixed arm’s marginal coverage, not the coverage conditional on being selected. Selection favors estimates with positive noise; in the three-arm example, the ordinary interval for the winner covers only about 92.7% of the time. The correction is to evaluate the interval procedure under the full selection rule and the target parameter, rather than carrying over the prespecified-arm coverage claim.

Finally, the exact normal calculations rely on known \(\sigma\), independent estimates, equal standard errors, and a common true effect. They are valuable as a benchmark: they quantify the effect of choosing the maximum and show how inversion repairs coverage in that benchmark. For a real multi-arm trial, the model and selection process must be checked before using any post-selection interval.