Why Selection Changes the Coverage Question
A confidence interval can have its advertised coverage for every contrast considered separately and still fail to cover the contrast reported after examining the data. The issue is not that the interval formula changes: it is that the reported contrast is selected partly because its estimate looks unusually large or small. This selection can make the reported estimation error atypical.
The key distinction is between a prespecified family of contrasts and a family assembled or narrowed after seeing the data. A simultaneous procedure calibrated for a fixed candidate family can protect a data-dependent choice from that family. By contrast, pointwise intervals—or intervals calibrated only for the selected contrasts while ignoring how they were selected—need not have their nominal postselection coverage.
This definition concerns coverage of the reported, random target. It does not say that the target itself is fixed before data collection. For instance, if a researcher reports whichever of two treatment contrasts has the larger absolute estimate, the reported estimand is whichever corresponding contrast was selected. The probability of covering that estimand must include the possibility that either contrast is selected.
A Coverage Guarantee That Survives Selection
Suppose the candidate family \(C\) is finite and fixed before the data are observed. For each \(c\in C\), let \(I_c\) be an interval for \(c^\mathsf{T}\theta\). The intervals have simultaneous coverage at least \(1-\alpha\) if \(P(c^\mathsf{T}\theta\in I_c\text{ for every }c\in C)\geq 1-\alpha\). This is a joint statement: it requires the entire collection of intervals to cover together with the stated probability.
Proof. Define the event \(A=\{c^\mathsf{T}\theta\in I_c\text{ for every }c\in C\}\). On \(A\), every candidate interval covers its own estimand. Since \(\hat c\in C\), the selected interval also covers: \(A\subseteq\{\hat c^\mathsf{T}\theta\in I_{\hat c}\}\). Taking probabilities gives \(P(\hat c^\mathsf{T}\theta\in I_{\hat c})\geq P(A)\geq 1-\alpha\). \(\square\)
The theorem does not require the selection rule to be independent of the estimates. It can select the largest estimate, the largest absolute estimate, or a contrast chosen by any other rule. What matters is that the procedure provides simultaneous coverage over the whole fixed candidate family and that the selected contrast is in that family. The result gives unconditional coverage; it does not promise the same coverage conditional on the particular selection that occurred.
A useful way to construct the required simultaneous intervals is Bonferroni adjustment. If \(m\) prespecified two-sided intervals each have noncoverage probability at most \(\alpha/m\), the probability that any one of them fails is at most \(\alpha\). This guarantee does not require the contrast estimates to be independent.
Proof. The event that at least one interval fails is the union of the \(m\) individual noncoverage events. By the union bound, its probability is no greater than the sum of their probabilities, which is at most \(m(\alpha/m)=\alpha\). Taking the complement gives simultaneous coverage at least \(1-\alpha\). The selection guarantee then follows from the preceding theorem. \(\square\)
Worked Examples
Worked Example: Selecting the Largest Absolute Estimate
Suppose \(X_1\) and \(X_2\) are independent with \(X_i\sim N(\theta_i,1)\), and the true values are \(\theta_1=\theta_2=0\). For each coordinate, use the usual pointwise 95% interval \(X_i\pm z_{0.975}\), where \(z_{0.975}\approx1.96\). After observing the data, select the coordinate with the largest absolute estimate and report its interval. Find the exact coverage probability of the reported interval.
Let \(J\) be the selected coordinate, so \(|X_J|=\max(|X_1|,|X_2|)\). The reported interval is \(X_J\pm z_{0.975}\), and its target is \(\theta_J=0\). It covers exactly when \(|X_J|\leq z_{0.975}\). Since \(|X_J|\) is the larger of the two absolute estimates, this event is equivalent to both \(|X_1|\leq z_{0.975}\) and \(|X_2|\leq z_{0.975}\).
For a standard normal variable, \(P(|X_i|\leq z_{0.975})=0.95\). Independence therefore gives \(P(\text{reported interval covers})=0.95\times0.95=0.9025\). Thus the interval has 95% coverage for either coordinate when considered separately, but only 90.25% coverage after selecting the larger absolute estimate. The selection rule favors an estimate with a large absolute error when both true coordinates are zero.
Worked Example: Protecting a Selection From Two Coordinates
Keep the same model, \(X_1\sim N(\theta_1,1)\) and \(X_2\sim N(\theta_2,1)\), independently. Instead of pointwise 95% intervals, construct Bonferroni intervals for the two prespecified coordinate contrasts at familywise level \(\alpha=0.05\), then select the coordinate with the largest absolute estimate. What coverage does this guarantee?
There are \(m=2\) two-sided intervals. Allocate noncoverage probability \(\alpha/m=0.025\) to each interval, or \(0.0125\) to each tail. The critical value is \(z_{1-0.0125}=z_{0.9875}\approx2.241\), giving intervals \(X_i\pm2.241\), for \(i=1,2\).
Each interval has coverage \(1-0.025=0.975\). By independence, the probability that both cover is \(0.975^2=0.950625\), which exceeds 0.95. More generally, Bonferroni guarantees at least 0.95 even without independence. Whenever both intervals cover, the selected interval covers, regardless of which coordinate was selected. The selection therefore has coverage at least 0.95, at the cost of wider intervals than the pointwise intervals in the previous example.
Worked Example: Screening Ten Contrasts but Calibrating for Two
Suppose ten prespecified candidate contrasts have independent standard normal estimation errors under the null, with known standard errors equal to 1. A screening rule selects the two contrasts with the largest absolute estimates. The analyst then applies the two-interval Bonferroni critical value \(z_{0.9875}\approx2.241\) to just those two intervals. What is the probability that both reported intervals cover?
Let the ten standardized estimates be \(Z_1,\ldots,Z_{10}\), each independently \(N(0,1)\). The selected two intervals both cover exactly when the two largest absolute estimates are at most \(2.241\). Equivalently, all ten absolute estimates must be at most \(2.241\).
For one estimate, \(P(|Z_i|\leq2.241)=0.975\), because the critical value leaves 0.0125 in each tail. By independence, \(P(\text{both selected intervals cover})=P(|Z_i|\leq2.241\text{ for all }i=1,\ldots,10)=0.975^{10}\approx0.7763\).
The two-interval Bonferroni calculation would be appropriate if those two contrasts had been specified in advance. Here, however, the selection searched all ten candidates and picked the two most extreme estimates. The chance that the maximum absolute estimate exceeds the cutoff is much larger than the chance of a failure among two prespecified intervals. To obtain a 95% guarantee by Bonferroni in this setting, the analyst could instead adjust for all ten candidates, using \(z_{1-0.05/(2\cdot10)}=z_{0.9975}\approx2.807\), and then report the intervals for the selected two.
What Must Be Fixed Before Selection?
The examples distinguish two different strategies. One is to define a broad candidate family in advance, construct a simultaneous procedure for that family, and then select and report any member of it. The other is to let the observed data determine which contrasts count as the family and calibrate only for the final list. The first has a clear coverage guarantee; the second does not inherit one merely because the final list is short.
The protective family must include every contrast the selection rule could report. If the rule can choose from ten candidates, intervals adjusted as though only two candidates existed do not satisfy the theorem’s hypothesis. Likewise, a simultaneous procedure for one fixed family does not protect a contrast created by an unanticipated data-dependent search outside that family. Defining the candidate set is therefore part of the coverage argument, not a detail that can be settled after the results are inspected.
The guarantee from simultaneous intervals is an unconditional one over repeated samples. It says that the probability of covering the selected estimand is at least \(1-\alpha\), averaging over the data and the resulting selection. It does not, by itself, say that the interval has at least \(1-\alpha\) coverage conditional on a specific selection event. Conditioning changes the probability statement and needs its own justification.
Common Mistake
Wrong reasoning: “I report two intervals, and each uses a Bonferroni adjustment for two intervals, so the probability either fails is at most 5%.” This argument assumes the two intervals were fixed independently of the data-driven search. If two contrasts were chosen from a larger candidate set because their estimates looked most extreme, the selection step was omitted from the calculation.
Correction: Identify the full set of contrasts the rule could select, then use a simultaneous procedure that covers that entire set. The fixed-family coverage event protects any later choice within it. If the procedure instead covers only the selected contrasts, a separate postselection justification is required; the nominal level for that smaller list is not enough.
Check Your Understanding
Use the distinction between pointwise and simultaneous coverage to answer these questions.
- Why can selecting the largest absolute estimate from several pointwise intervals reduce the coverage of the reported interval?
- State the event inclusion that proves simultaneous coverage over a fixed family protects any selection within that family.
- Does the fixed-family selection guarantee require the selection rule to be independent of the estimates? Explain.
- Ten contrasts are screened, and the two largest estimates are reported. Why is a two-interval Bonferroni adjustment generally insufficient?
- Does unconditional postselection coverage of at least 95% imply 95% coverage conditional on the selected contrast? Explain why or why not.