Why Coverage Changes When an Arm Is Selected
A confidence interval can have its advertised coverage for each treatment considered separately and still perform differently after a data-dependent rule chooses which treatment to report. The key is to specify the repeated-sampling event being assessed. Is the question whether the interval for the ultimately selected treatment covers its effect on average over all possible selections, or whether it covers at the advertised rate among trials that select a particular arm? These are different properties.
This distinction matters when treatment arms are selected using estimates that are also used to construct the reported intervals. An arm can be selected partly because its estimate happened to be favorable. Selection therefore changes the distribution of the selected estimate, and may also change the coverage of its ordinary interval. A coverage claim should state both the interval procedure and whether coverage is unconditional or conditional on the selection event.
These probabilities have different denominators. Unconditional coverage averages over the possible selected arms, with each arm weighted by its probability of selection. Conditional coverage restricts attention to repetitions in which one specified arm was selected. In both definitions, the treatment effects are fixed; it is the trial data, intervals, and selected index that vary across repetitions.
What Simultaneous Intervals Guarantee
A collection of intervals has simultaneous coverage at least \(1-\alpha\) if the probability that every arm's interval covers its effect is at least \(1-\alpha\). This is stronger than requiring each interval separately to have coverage \(1-\alpha\). It is useful after selection because simultaneous coverage protects whichever arm the rule chooses, even if the selection rule depends on the same data.
The unconditional statement follows because if all intervals cover, the interval for the selected arm covers too. For the conditional statement, that implication only bounds the joint probability of a miss and selection; conditioning divides by the selection probability.
For a specific selection event \(J=j\), the miss event \(\{J=j,\theta_j\notin C_j\}\) is contained in the event that at least one interval fails. Consequently,
This can be a weak bound when an arm is rarely selected: if \(P(J=j)\leq\alpha\), it gives no positive lower bound on conditional coverage. The bound concerns a joint probability before conditioning; it must not be mistaken for a conditional error rate of at most \(\alpha\).
Worked Example: Simultaneous Coverage for Three Arms
Worked Example: Simultaneous Coverage for Three Arms
Suppose three treatment-effect estimates are normally distributed with known standard errors of 0.8 effect units, and each estimate is unbiased with its stated standard error. A rule selects the arm with the largest observed estimate. In one hypothetical trial, the estimates are 1.8, 2.4, and 1.1 effect units, so arm 2 is selected. Construct Bonferroni simultaneous 95% intervals and explain the coverage guarantee.
For each of \(K=3\) two-sided intervals, use a normal critical value with tail probability \(\alpha/(2K)=0.05/6\). The corresponding quantile is approximately 2.394. The intervals are \(\hat{\theta}_j\pm 2.394(0.8)\); the half-width is \(2.394(0.8)=1.915\) effect units.
For the selected arm 2, the interval is \(2.4\pm1.915\), or approximately \([0.485,4.315]\) effect units. The other intervals are \([ -0.115,3.715]\) and \([-0.815,3.015]\) effect units. The procedure has simultaneous coverage at least 95% by the Bonferroni inequality, which does not require independence among the estimates. Thus the selected interval has at least 95% unconditional coverage under the model. This guarantee does not say that its coverage conditional on selecting arm 2 is at least 95%.
The assumptions used here are that each estimate has a normal sampling distribution centered at its treatment effect and that its standard error is correctly specified. Bonferroni does not require a particular correlation structure, but incorrect marginal distributions or standard errors would still invalidate the claimed marginal coverage and hence the guarantee.
Worked Example: Conditional Coverage Can Be Much Lower
Worked Example: Conditional Coverage Under a Two-Arm Rule
Consider two hypothetical treatment arms with fixed effects \(\theta_1=2\) and \(\theta_2=0\) effect units. Suppose the estimates are independent and satisfy \(\hat{\theta}_i\sim N(\theta_i,1)\), so each has a known standard error of 1 effect unit. Select the arm with the larger estimate, and report its ordinary two-sided 95% interval \(\hat{\theta}_i\pm1.96\). This model gives exact marginal 95% coverage for each arm before selection.
Write \(Z_i=\hat{\theta}_i-\theta_i\), so \(Z_1,Z_2\) are independent standard normal variables. Arm 2 is selected when \(Z_2-Z_1>2\). Since \(Z_2-Z_1\sim N(0,2)\), its selection probability is \(P(J=2)=\Phi(-2/\sqrt{2})\approx0.07865\), where \(\Phi\) is the standard normal cumulative distribution function. Arm 1 is selected with probability approximately \(0.92135\).
For a selected arm 2 interval to miss, \(|Z_2|>1.96\) must hold as well as \(Z_2-Z_1>2\). Given \(D=Z_2-Z_1=d\), \(Z_2\) is normal with mean \(d/2\) and variance \(1/2\). Thus the conditional miss probability can be evaluated as
where \(\phi\) is the standard normal density. Numerical evaluation gives a joint probability of selecting arm 2 and missing its interval of about 0.01558. Dividing by the arm 2 selection probability gives a conditional miss probability of about \(0.01558/0.07865=0.198\). Therefore, coverage conditional on selecting arm 2 is about \(1-0.198=0.802\), or 80.2%.
For arm 1, the corresponding calculation conditions on \(D\leq2\); given \(D=d\), \(Z_1\) is normal with mean \(-d/2\) and variance \(1/2\). Numerical evaluation gives coverage conditional on selecting arm 1 of about 96%. The unconditional selected-arm coverage is therefore about \(0.92135(0.960)+0.07865(0.802)=0.948\), or 94.8%. These conditional values differ substantially despite the ordinary intervals having 95% marginal coverage for each arm before selection.
A Short Justification and Practical Alternatives
The Bonferroni construction illustrates why simultaneous intervals give a selection-robust unconditional guarantee. If each of \(K\) marginal two-sided intervals has miss probability at most \(\alpha/K\), then the probability of at least one miss is at most the sum of those probabilities:
Thus all intervals cover with probability at least \(1-\alpha\), and the selected interval covers whenever all of them cover. This argument is complete regardless of how the rule chooses \(J\), provided the simultaneous intervals truly have the stated coverage. Bonferroni may be conservative, especially when estimates are strongly correlated, but correlation cannot make its union-bound guarantee fail.
The main alternatives answer different inferential questions:
| Approach | Coverage target | Main qualification |
|---|---|---|
| Ordinary pointwise intervals | Coverage for each fixed arm before selection | Do not generally retain nominal coverage after data-dependent selection. |
| Simultaneous intervals | Joint coverage, and therefore unconditional selected-arm coverage | Can be wider; conditional coverage for a particular selected arm is not guaranteed at the nominal level. |
| Selection-adjusted intervals | Coverage accounting for a specified selection event | Require a correct model for the estimates and the selection rule; rare events can yield wide or unstable intervals. |
| Independent selection and confirmation | Coverage conditional on the selection data, using fresh independent data for inference | Requires valid independence and a suitable confirmation-stage estimator and interval. |
Worked Example: Independent Confirmation After Selection
Worked Example: A Fresh Estimate for the Selected Arm
Suppose a hypothetical trial uses pilot data to choose one of three arms. The pilot data are independent of a later confirmation sample. For the selected arm, the confirmation estimate of its effect versus control is 4.2 effect units with a known standard error of 2 effect units. Assume the confirmation estimate, conditional on the pilot data and selected arm, is normally distributed around that arm's fixed effect with the stated standard error.
The ordinary 95% confirmation interval is \(4.2\pm1.96(2)\). Its half-width is \(3.92\) effect units, giving \([0.28,8.12]\) effect units. Conditional on the pilot data, the selected index is fixed, and the independent confirmation estimate has its assumed normal distribution. Therefore, under the model, this interval covers the selected arm's effect with 95% probability conditional on the pilot data, and hence also conditional on the pilot selecting that arm.
This calculation is valid only if the confirmation data are genuinely independent of the selection data and the stated model and standard error apply. Reusing pilot observations in the estimate or standard error can reintroduce selection dependence; independence cannot be asserted merely because the analysis occurs at a later calendar time.
Diagnostics, Limitations, and Common Mistakes
Coverage should be evaluated for the actual adaptive procedure, not for an interval formula in isolation. In a simulation or design calculation, generate trial data under specified fixed treatment effects, apply the full selection rule, construct the reported interval, and record whether it contains the effect of the selected arm. The proportion of successful repetitions estimates unconditional selected-arm coverage. To estimate coverage conditional on \(J=j\), use only repetitions selecting arm \(j\), and report that selection frequency as well. When the event is rare, a large total number of simulations may still leave too few repetitions for a stable conditional estimate.
Check that the simulated joint distribution reflects the design: estimates may be correlated through shared control data, unequal allocation, interim looks, or adaptive changes. A normal approximation may also be poor for small samples or binary outcomes. In such cases, the relevant marginal and joint distributions must be justified for the actual procedure; plugging in a familiar critical value does not establish coverage. If the design includes interim selection and later data, the analysis must represent the information used at each stage rather than treating all estimates as independent by default.
A common mistake is: “The simultaneous failure probability is at most 5%, so the interval misses at most 5% of the time among trials selecting arm 2.” That conclusion is not justified. Simultaneous coverage bounds the joint probability of a miss and selecting arm 2 by 5%; dividing by \(P(J=2)\) can make the conditional miss bound much larger than 5%. The correction is to state the guarantee as unconditional, or to use a method designed to provide the desired conditional coverage.
Selection-adjusted procedures depend on the selection rule and the data model, and simultaneous intervals can be conservative. Sample splitting avoids reuse of selection data for inference but can reduce precision because fewer observations contribute to each stage. No one method is uniformly best: the appropriate choice depends on whether the scientific claim concerns the selected treatment on average, a treatment conditional on having been selected, or a confirmatory estimate from independent data.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- State the difference between unconditional selected-arm coverage and coverage conditional on selecting arm \(j\).
- Three intervals have simultaneous coverage of at least 95%. What guarantee follows for the interval of whichever arm a data-dependent rule selects?
- If \(P(J=2)=0.10\) and simultaneous coverage is at least 95%, what upper bound does the general argument give for \(P(\theta_2\notin C_2\mid J=2)\)?
- Why can an independent confirmation sample support conditional coverage after an arm is selected using pilot data?
Solutions. (1) Unconditional coverage averages across repetitions and their selected arms; conditional coverage restricts repetitions to those selecting the specified arm. (2) The selected interval has unconditional coverage of at least 95%; this alone does not establish 95% coverage conditional on selecting any particular arm. (3) The conditional miss bound is \(\min(1,0.05/0.10)=0.50\), so the argument guarantees only a conditional miss probability no greater than 50%, not 5%. (4) Conditional on the pilot data, the selected arm is fixed; if confirmation data are independent and the confirmation interval is valid for that fixed arm, its coverage statement remains valid conditional on the pilot selection.