Selection-Adjusted Confidence Intervals For Winning Arms

Why selection changes interval coverage

When several experimental arms are compared, the arm with the largest observed treatment effect is often called the winner. If the same data both choose the winner and estimate its effect, an ordinary confidence interval can be too optimistic: selection favors estimates that happen to be high, including those elevated by sampling variation. A confidence interval computed as if the winning arm had been specified in advance does not automatically account for that selection.

The goal here is an interval for the effect of the selected arm that remains valid despite the selection rule. One practical route is to construct intervals that cover all candidate effects simultaneously; then the interval for whichever arm wins inherits the coverage guarantee. A more model-specific route uses the sampling distribution of the maximum itself. These approaches answer related but not identical questions, and their assumptions matter.

This probability statement is not generally a guarantee conditional on every possible selected label. In particular, \(P(\theta_J\in I_J)\geq 1-\alpha\) does not imply \(P(\theta_j\in I_j\mid J=j)\geq 1-\alpha\) for each \(j\). The distinction is important when interpreting coverage after a specific arm is observed to win.

Simultaneous intervals protect the selected arm

Suppose there are \(K\) candidate treatment effects and intervals \(I_1,\ldots,I_K\) satisfy the simultaneous coverage condition \(P(\theta_i\in I_i\text{ for every }i)\geq 1-\alpha\). The estimates may be correlated, as often occurs when treatment arms share a control group. Simultaneous coverage means the whole collection covers all candidate effects together with probability at least \(1-\alpha\); it is stronger than requiring each interval separately to have 95% coverage.

A simple construction uses Bonferroni intervals. Assume each \(\hat{\theta}_i\) is approximately normal with known or appropriately estimated standard error \(SE_i\), and that its marginal interval procedure is calibrated so that its two-sided noncoverage probability is at most \(\alpha/K\). For known standard errors, the intervals are

where \(z_q\) is the \(q\) quantile of the standard normal distribution. No independence among the \(K\) estimates is required. The method does require valid marginal standard errors and a prespecified family of \(K\) candidate comparisons; if uncertainty is estimated using a small sample, an appropriate finite-sample procedure or a justified approximation is needed.

An exact interval in a common-effect maximum model

A second construction illustrates directly how selection changes the distribution of a winner. Assume, for this model only, that the \(K\) candidate estimates are independent and identically distributed as \(X_i\sim N(\mu,\sigma^2)\), with known \(\sigma\) and the same true effect \(\mu\) for every arm. The selected estimate is the maximum \(M=\max_i X_i\). This equal-effect assumption is restrictive, but it gives a transparent exact calculation for the selected maximum.

For any value \(m\), independence gives \(P(M\leq m)=P(X_1\leq m,\ldots,X_K\leq m)=\Phi((m-\mu)/\sigma)^K\), where \(\Phi\) is the standard normal cumulative distribution function. Inverting this distribution produces an interval for the common effect \(\mu\). The interval is asymmetric around the observed maximum because the maximum is shifted upward by selection.

To verify the inversion, let \(a=\Phi^{-1}((\alpha/2)^{1/K})\) and \(b=\Phi^{-1}((1-\alpha/2)^{1/K})\). The central \(1-\alpha\) event for the maximum is \(\alpha/2\leq\Phi((M-\mu)/\sigma)^K\leq1-\alpha/2\). Taking the \(K\)th root and applying \(\Phi^{-1}\) gives \(a\leq(M-\mu)/\sigma\leq b\). Rearranging for \(\mu\) yields \(M-\sigma b\leq\mu\leq M-\sigma a\), as stated. This is exact under the model and has unconditional coverage \(1-\alpha\). Because the estimates are exchangeable under the common-effect model, its coverage is also the same conditional on any particular arm label winning (ties have probability zero).

Choosing a procedure and checking its assumptions

Bonferroni intervals are broadly useful because they require only valid marginal intervals, not a particular dependence structure. Their cost is conservatism: as the number of arms grows, each interval becomes wider. A maxT simultaneous procedure can often give narrower intervals by using the joint distribution and correlations of the treatment contrasts. For example, its critical value can be calibrated from the distribution of the largest absolute standardized contrast. That gain depends on specifying or estimating the joint covariance structure adequately. Both approaches protect the selected interval through simultaneous coverage.

An ordinary interval uses a critical value for one prespecified comparison, such as \(z_{0.975}\) for a two-sided 95% normal interval. It is usually narrower, but after choosing the largest estimate from several arms it does not generally retain nominal coverage. The exact maximum interval above instead models the distribution of the selected statistic itself. Such direct selection-model intervals can be efficient when the selection rule and sampling model are correctly specified; their validity can be fragile if the effects differ, estimates are correlated, or the actual selection rule is more complex than taking a maximum.

For any of these methods, define the family of candidate arms and the selection rule before interpreting the interval. If selection also depends on interim data, dropping arms, or several stages of analysis, a final-stage interval calibrated as though there had been only one selection may fail to account for the full process. The joint sampling distribution or a valid simultaneous procedure must cover the decisions actually made.

Common pitfalls and diagnostics

• Using the winner's ordinary interval without qualification. The wrong reasoning is that an interval has 95% coverage for each arm, so it must have 95% coverage for whichever arm wins. The arm is random after selection; per-arm coverage alone does not establish coverage for the selected arm. Use a simultaneous or appropriately selection-model-based procedure.
• Confusing marginal post-selection coverage with conditional coverage. A simultaneous interval ensures coverage for the selected arm across repeated trials, but does not automatically guarantee 95% coverage conditional on observing a particular arm as winner. State which coverage statement the method supports.
• Ignoring the selection rule or the number of candidates. If more arms or decision stages contributed to choosing the reported winner than the interval accounts for, the claimed adjustment may be inadequate. Record which estimates could have been selected and how the decision was made.
• Applying a special maximum model to unequal effects. The exact interval derived above targets a common \(\mu\), not an unrestricted \(\theta_J\). When arm effects may differ, use a procedure that covers the arm-specific effects under its stated assumptions, such as a simultaneous method.

A useful diagnostic is to compare an ordinary interval with the adjusted interval and ask which source of multiplicity the wider procedure addresses. Also check that the reported standard errors and critical values match the contrast definitions, endpoint units, covariance assumptions, and full selection process. A numerical adjustment cannot repair a misspecified sampling model or unaccounted-for adaptation.