Multiplicity After Data-Driven Contrast Selection

Why Selection Changes the Inferential Question

A confidence interval or test for a contrast can have its advertised error rate when that contrast is specified independently of the data used for inference. The situation changes if an analyst examines estimated differences, test statistics, or p-values and then reports the most striking contrast. The reported contrast is no longer an arbitrary member of the family: it is the one the data made look most extreme.

This selection can make an ordinary, nominal 95% interval miss more than 5% of the time, or make a nominal 5% test reject a true null more than 5% of the time. The issue is not that each calculation is necessarily wrong for a contrast fixed in advance. The issue is that the guarantee for a fixed contrast does not automatically apply to the contrast chosen after looking.

Definition: Let \(\mathcal{C}=\{c_1,\ldots,c_m\}\) be a finite set of candidate contrasts specified before examining the inferential data. For a parameter vector \(\mu\), contrast \(j\) is \(\theta_j=c_j^\mathsf{T}\mu\), estimated by \(\hat{\theta}_j=c_j^\mathsf{T}\hat{\mu}\). A data-driven selection rule chooses an index \(\hat{j}\) using observed estimates or test results. Post-selection inference concerns the contrast \(\theta_{\hat{j}}\) that this rule selects, rather than a contrast fixed in advance.

A fixed-family guarantee is different from a guarantee for one selected result. Familywise error control at level \(\alpha\) means that the probability of at least one false rejection among the specified family of true null hypotheses is at most \(\alpha\). A procedure that controls error for a fixed finite family can protect a data-selected member of that family, provided the family actually includes every contrast the selection rule could choose.

A Direct Demonstration of the Inflation

Consider two candidate contrasts whose standardized estimates under their null hypotheses are independent standard normal variables \(Z_1\) and \(Z_2\). This is an exact normal-theory example, such as two orthogonal contrasts with known standard errors. Suppose the analyst selects the contrast with the larger absolute statistic and then applies the ordinary two-sided 5% test to that selected statistic.

Worked Example: Selecting the Larger of Two Null Statistics

Under the global null, \(Z_1,Z_2\) are independent \(N(0,1)\). Let \(z_{.975}\) be the 97.5th percentile of the standard normal distribution, approximately 1.96. The selected contrast is rejected when \(\max(|Z_1|,|Z_2|)>z_{.975}\).

For either statistic separately, the probability of staying within the ordinary two-sided acceptance region is \(P(|Z_j|\le z_{.975})=0.95\). By independence, the probability that both stay within it is

P\bigl(|Z_1|\le z_{.975},\,|Z_2|\le z_{.975}\bigr) =0.95^2=0.9025.

Therefore the probability of at least one statistic crossing the ordinary threshold—and hence of rejecting for the selected contrast—is

$$ 1-0.9025=0.0975. $$

The actual false-rejection probability is 9.75%, not 5%. Each fixed-contrast test has size 5%; choosing the larger absolute statistic changes the event that counts as a rejection. With more candidates, the same selection rule can inflate the probability further.

Independence made the calculation especially simple. Correlated contrasts need not have this same numerical error rate, but correlation alone does not make ordinary fixed-contrast inference valid after selection. The selected statistic has a distribution shaped by both the dependence among contrasts and the selection rule. Its ordinary marginal reference distribution is generally not the right reference distribution after selection.

Protect the Whole Candidate Family

One general strategy is to construct intervals or tests that protect all candidates at once. If all intervals cover their respective contrasts simultaneously, then the interval for whichever candidate is later selected also covers its contrast. The selection rule can use the data, because the guarantee already applies to every member of the family.

Theorem: Suppose \(\mathcal{C}=\{c_1,\ldots,c_m\}\) is fixed, and intervals \(I_1,\ldots,I_m\) satisfy \(P(\theta_j\in I_j\text{ for every }j)\ge 1-\alpha\). For any selection rule \(\hat{j}\) taking values in \(\{1,\ldots,m\}\), including one based on the same data, \(P(\theta_{\hat{j}}\in I_{\hat{j}})\ge 1-\alpha\).

Proof. On the event that every interval covers its contrast, the interval for the selected index covers the selected contrast. Thus the event of a selected-contrast miss is contained in the event that at least one family interval misses:

\{\theta_{\hat{j}}\notin I_{\hat{j}}\} \subseteq \bigcup_{j=1}^m\{\theta_j\notin I_j\}.

Taking probabilities, the probability of a selected-contrast miss is at most the probability of the union on the right, which is at most \(\alpha\) by the assumed simultaneous coverage. This proves the claim. \(\square\)

Bonferroni intervals provide one way to obtain the simultaneous coverage in this result: construct each of the \(m\) intervals with marginal noncoverage at most \(\alpha/m\). Bonferroni's inequality then bounds the probability of any miss by the sum of those probabilities, at most \(\alpha\), without requiring independence. Other valid choices include a max-\(t\) procedure calibrated to the prespecified finite family, or Scheffé simultaneous inference when protection is needed for all linear contrasts. Tukey intervals address the family of all pairwise mean differences under their applicable model; they do not automatically protect arbitrary non-pairwise contrasts.

Worked Example: A Bonferroni Interval After Choosing Among Four Contrasts

Suppose four candidate contrasts have known standard errors of 0.5, and their observed estimates are 1.20, 0.90, 0.35, and 0.10. The analyst selects the largest absolute standardized estimate. The four standardized values are \(2.40,1.80,0.70,\) and \(0.20\), so contrast 1 is selected.

To obtain 95% simultaneous coverage using Bonferroni, use a two-sided error probability of \(0.05/4=0.0125\) for each contrast. Thus each interval uses the standard normal critical value \(z_{1-0.05/(2\cdot4)}=z_{.99375}\), approximately 2.498. The interval for the selected contrast is

1.20\ \pm\ 2.498(0.5) =1.20\ \pm\ 1.249 =[-0.049,\ 2.449].

The interval includes zero, so the familywise procedure does not establish that the selected contrast differs from zero at the 5% familywise level. For comparison, an ordinary 95% interval for this contrast would be \(1.20\pm1.96(0.5)=[0.22,2.18]\), which excludes zero. That ordinary interval treats contrast 1 as if it had been fixed in advance; it does not account for choosing it from four candidates. The simultaneous guarantee remains valid regardless of dependence among the four estimates.

The scope of the candidate family matters. If the analyst can choose only among four contrasts, simultaneous protection for those four can suffice. If the analyst can invent a contrast after seeing the data—for example, choose any direction in a multi-parameter contrast space—a finite correction for a list that omits possible choices does not protect the actual selection. A procedure designed to cover all linear contrasts, such as Scheffé's method under its model assumptions, addresses that broader target.

Separate Selection From Confirmation

A different strategy is to use one part of the data to select a contrast and independent data to test or estimate it. Independence prevents the confirmation data from having helped determine which contrast is examined. Conditional on the selection-stage data, the chosen contrast is then fixed for the confirmation stage.

Result: Suppose selection uses data \(D_1\), and confirmation uses independent data \(D_2\). Assume that, for each candidate contrast, the confirmation-stage test has size at most \(\alpha\) under its null when that contrast is fixed. If the procedure tests only the contrast selected using \(D_1\), its probability of falsely rejecting a selected true null is at most \(\alpha\).

Justification. Condition on \(D_1\). The selected index \(\hat{j}\) is then fixed, and \(D_2\) retains its stipulated distribution because the two data sets are independent. If the selected contrast is null, the conditional probability of rejection using \(D_2\) is at most \(\alpha\) by the assumed fixed-contrast test validity. If it is not null, that selection does not constitute a false rejection. Averaging these conditional error probabilities over \(D_1\) gives an unconditional false-rejection probability at most \(\alpha\). \(\square\)

Worked Example: Selecting in a Pilot Sample and Testing Independently

A study has three prespecified contrasts. An independent pilot sample gives estimates \(-0.4,\ 1.1,\) and \(0.7\), with the same standard error for each. The analyst chooses contrast 2 because its absolute estimate is largest, but reserves a separate confirmation sample for inference.

Suppose the confirmation sample comes from a normal linear model with 18 residual degrees of freedom. For contrast 2, it gives an estimate of 0.60 and an estimated standard error of 0.30. The confirmation statistic is

t=\frac{0.60-0}{0.30}=2.00.

For a two-sided 5% test with 18 degrees of freedom, the critical value is approximately \(t_{.975,18}=2.101\). Since \(2.00<2.101\), the confirmation test does not reject. Its p-value is approximately 0.061.

The pilot choice does not invalidate this test: conditional on the pilot data, contrast 2 is fixed, and the independent confirmation data have not been used to select it. This validity would not follow if the analyst used the confirmation observations to choose contrast 2 and then tested it as though it had been prespecified. The confirmation model, contrast standard error, and degrees of freedom must also be appropriate for that independent sample.

Common Mistake and Practical Interpretation

Mistake: “Every contrast had a valid 5% test, so the one I report is valid at 5%.” Individual size applies to a contrast fixed independently of the testing data. The event that the selected contrast rejects is the union of rejection events over contrasts the procedure might select, and that union can have probability greater than 5% under the null.
Correction: match the guarantee to the selection process. Protect a prespecified family simultaneously, use a procedure whose scope includes every possible contrast, or use independent data for selection and confirmation.
Mistake: correcting only for the contrasts that look interesting after analysis. The relevant family is the set of contrasts among which the analyst could have selected, not merely the subset ultimately reported. Omitting selectable contrasts can understate the multiplicity.

Selection-aware inference is not a penalty attached mechanically to every analysis. If the contrast was genuinely prespecified, its fixed-contrast procedure may be appropriate. If the estimate or test results guided the choice, the report should make that choice process clear and use an error guarantee that covers it. Simultaneous intervals also offer a useful interpretation: after selection, report the interval for the selected contrast together with the family and simultaneous coverage level that justify it.

Check Your Understanding

Use the distinction between fixed-contrast and selection-aware guarantees to answer these questions.

Why does selecting the larger absolute value of two independent null \(Z\)-statistics make an ordinary two-sided 5% test reject with probability 9.75%?
State the event-containment argument that lets simultaneous intervals protect a contrast selected from their family.
If an analyst can select from six prespecified contrasts, what Bonferroni noncoverage probability should be assigned to each two-sided interval to obtain 95% simultaneous coverage?
Why would simultaneous protection for a finite list fail to cover a contrast invented after inspecting the data if that contrast is not in the list?
What independence condition makes a test on a pilot-selected contrast valid when a separate sample is used for confirmation?