Why Adjust Intervals After Selecting Effects?
A trial may compare several experimental arms with a control, then report an interval for the arm with the most favorable estimated effect. That interval is chosen because the data made its estimate stand out. A conventional 95% interval for that arm, calculated as if it had been named in advance, generally does not retain 95% coverage after this selection. The issue is not that confidence intervals cannot be reported after selection; it is that the coverage target must account for the chance of errors across the candidate effects from which the report was selected.
The approach here is to construct intervals that cover the entire prespecified family of candidate effects with high probability. If all candidate effects are covered at once, then any subset chosen from them is covered on that same event. This gives a direct way to protect intervals for selected arms without needing to know in advance which arm will be selected.
For the methods below, the candidate family and its size \(K\) are specified before examining the results. Assume each estimated effect \(\hat{\theta}_j\) is unbiased for \(\theta_j\), and that the standardized estimation errors have the stated distribution. For the normal-theory formulas, treat the standard errors \(SE_j\) as known or as sufficiently well estimated for the stated normal approximation to be valid. Correlations among estimates matter for some adjustments, but not for the basic Bonferroni guarantee. The trial’s design and analysis must also yield valid estimates and standard errors; adjusting for selection does not repair an invalid sampling model.
Bonferroni Simultaneous Intervals
The Bonferroni method allocates a total error probability of \(\alpha\) across the \(K\) two-sided intervals. It assigns each interval a marginal error probability \(\alpha/K\), with half in each tail. If each standardized error is standard normal, the resulting interval for effect \(j\) is
Here \(z_q\) is the \(q\)-quantile of the standard normal distribution. No independence assumption is required: the estimates may be correlated.
The key selection argument follows from set inclusion. If every candidate interval covers its effect, the interval for whichever candidate is selected must cover that candidate’s effect. Thus the probability that a selected interval misses is at most the probability that at least one interval in the full family misses. This conclusion holds for selecting the largest estimate, selecting several arms, or using another rule based on the trial data, provided the selected effects are members of the family covered by the simultaneous intervals.
Worked Example: Bonferroni Interval for the Highest Estimate
Hypothetical data. A trial compares four experimental arms with control on a continuous endpoint, measured in points; positive effects favor the experimental arm. The estimated effects are \(1.20,\ 0.90,\ 0.40,\ 0.10\) points, each with \(SE=0.30\) points. The arm with the highest estimated effect is selected. Construct Bonferroni 95% simultaneous intervals and report the selected arm’s interval.
The family has \(K=4\) effects and \(\alpha=0.05\). The two-sided critical value is \(z_{1-0.05/(2\times4)}=z_{0.99375}\approx2.498\). The margin of error for each interval is \(2.498(0.30)=0.7494\) points. For the arm with estimate \(1.20\), the interval is
The intervals use the same margin for all four effects, so the other intervals are \((0.1506,1.6494)\), \((-0.3494,1.1494)\), and \((-0.6494,0.8494)\) points. Under the stated normal and standard-error assumptions, the family of four intervals covers all four true effects with probability at least 0.95. Consequently, the reported interval for the highest-estimated arm has at least 95% coverage as part of this family-wise guarantee. It is not a claim that the interval has 95% coverage conditional on that arm having been selected.
Using Independence or the Joint Distribution
Bonferroni is simple and robust to dependence, but it may be conservative. If the standardized errors are mutually independent and normal, the probability that all \(K\) marginal intervals cover is the product of their coverage probabilities. Setting this product to \(1-\alpha\) gives the Šidák per-interval error probability \(\alpha_{\mathrm{S}}=1-(1-\alpha)^{1/K}\). The corresponding two-sided critical value is \(z_{1-\alpha_{\mathrm{S}}/2}\). Independence is a real assumption, not a consequence of having different treatment arms. Estimates that share a control group are typically correlated.
Worked Example: Šidák Intervals for Independent Estimates
Hypothetical data. Suppose three experimental-versus-control effect estimates, in points, have estimates \(0.55,\ -0.10,\ 0.35\) and standard errors \(0.25,\ 0.30,\ 0.20\) points. Assume their standardized errors are mutually independent normal variables. For a 95% family-wise target, the per-interval error probability is
The two-sided critical value is \(z_{1-0.01695/2}\approx z_{0.99152}\approx2.386\). The margins of error are \(2.386(0.25)=0.5965\), \(2.386(0.30)=0.7158\), and \(2.386(0.20)=0.4772\) points. The three intervals are therefore
Under the independence assumption, each interval has coverage \(1-\alpha_{\mathrm{S}}\), so the probability that all three cover is \((1-\alpha_{\mathrm{S}})^3=0.95\). If the first estimate is selected as the largest, its interval is still covered by the simultaneous event. If the estimates are correlated, however, this Šidák calculation is not justified merely because the arms are distinct; a dependence-aware method or Bonferroni adjustment is needed.
When the joint distribution is known or can be validly estimated, a max-statistic interval can use the correlation structure directly. For jointly normal standardized errors \(Z_j=(\hat{\theta}_j-\theta_j)/SE_j\), define \(M=\max_{1\leq j\leq K}|Z_j|\). Choose \(c\) so that \(P(M\leq c)=1-\alpha\) under their joint distribution. Then the intervals \(\hat{\theta}_j\pm cSE_j\) have simultaneous coverage \(1-\alpha\). If exact calibration is difficult, a conservative value satisfying \(P(M\leq c)\geq1-\alpha\) is sufficient. The covariance model used to find \(c\) must correspond to the actual estimators.
Worked Example: Correlation-Aware Max-Statistic Intervals
Hypothetical data. Two experimental-versus-control estimates are \(0.72\) and \(0.61\) points, with standard errors \(0.25\) and \(0.30\) points. Suppose the standardized errors are jointly normal with correlation \(0.50\). Numerical calibration of the two-sided maximum statistic for these two standardized errors gives \(c\approx2.23\) for 95% simultaneous coverage. The intervals are
If the first arm is selected because its estimate is larger, its interval is \((0.1625,1.2775)\) points. The family-wise interpretation comes from calibrating the maximum of both absolute standardized errors, not from treating the selected estimate as an isolated comparison. The correlation matters because the chance of at least one extreme error depends on the joint distribution; if the correlation or the distribution is misspecified, the stated max-statistic coverage may fail. Bonferroni would instead use \(z_{0.9875}\approx2.241\) for both arms, avoiding the need to model their correlation but giving a slightly wider interval here.
Choosing and Checking an Adjustment
The adjustment should match the inferential family and the information available about dependence. Bonferroni is often a sensible default when the family is finite and marginal intervals are valid, particularly when dependence is uncertain. Šidák can be more efficient under its independence assumption. A max-statistic adjustment can exploit correlation and may shorten intervals, but relies on a correctly specified joint distribution and reliable calibration. These methods target simultaneous interval coverage; they are not interchangeable with reporting unadjusted marginal intervals and describing them as family-wise.
| Method | Dependence requirement | Main trade-off |
|---|---|---|
| Bonferroni | No independence assumption; valid marginal error bounds are required | Simple and broadly applicable, but can be conservative |
| Šidák | Independent standardized errors for the stated formula | Can be less conservative than Bonferroni under independence |
| Max-statistic | Valid joint distribution or calibration for the standardized errors | Uses correlation information; sensitive to joint-model misspecification |
Before reporting adjusted intervals, check that the candidate family includes every effect from which selection could occur, that the standard errors and joint model correspond to the trial’s analysis, and that the selected arm or arms really are members of that family. If selection can involve unlisted subgroups or contrasts, protecting only the listed arms does not protect the broader search. Also examine whether the normal approximation is credible, especially for small samples or highly skewed outcomes. The separate earlier material on small-sample exact intervals and adaptation-respecting bootstrap intervals addresses alternatives when ordinary normal approximations are unsuitable.
A frequent mistake is to calculate an ordinary 95% interval only for the arm with the largest observed effect and argue, “It is one interval, so no adjustment is needed.” The reported interval is one interval, but the identity of the arm was chosen after comparing several estimates. The relevant error risk arises across the full search. Define the family first, adjust intervals for that family, and then report the selected member’s interval. Another mistake is to interpret simultaneous coverage as a 95% probability that the selected effect lies in its realized interval. The guarantee is a repeated-sampling statement: across repetitions under the assumed model, the complete set of intervals covers all family effects at least 95% of the time. It does not generally guarantee 95% coverage conditional on the observed selection outcome.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions below.
- There are five candidate effects and a family-wise error target of 0.05. What is the Bonferroni two-sided critical value?
- Why does simultaneous coverage protect the interval for the arm with the largest estimate?
- When is the Šidák formula in the tutorial justified, and why might it fail for treatment-versus-control estimates?
- For two correlated estimates, what information is needed to use a max-statistic interval?
Solutions. (1) Use \(z_{1-0.05/(2\times5)}=z_{0.995}\approx2.576\). (2) The event that all candidate intervals cover implies that the interval for whichever candidate is selected covers; therefore the probability of a miss for the selected interval cannot exceed the probability of at least one miss in the family. (3) The formula assumes independent standardized errors. Treatment-versus-control estimates often share the control observations, inducing correlation, so independence cannot be assumed from the arm labels. (4) One needs a valid joint distribution for the standardized errors, including their covariance or correlation, and a critical value calibrated to the maximum absolute standardized error under that distribution.