Why Adjusted P-Values Need to Account for Selection
An unadjusted p-value answers a question about one treatment hypothesis under a specified test. But when several treatments are examined and the most promising are selected, a small p-value can arise partly because the analysis had several opportunities to find one. Confirmatory evidence must account for that multiplicity and for how the treatment hypotheses were selected.
The relevant adjustment depends on the design. If treatment selection and testing reuse the same data, simply applying a procedure to only the selected arms may not control the chance of any false positive across the original family. If arms are selected using interim data and then tested on an independent validation sample, valid p-values from that sample can be adjusted within the selected family. A third option uses the joint distribution of the treatment statistics, as in maxT procedures.
The family must be defined before interpreting adjusted values. It might be all treatment-versus-control hypotheses eligible for selection, or a smaller selected family tested on independent validation data. The family cannot be silently changed after seeing which arms have small p-values. The hypotheses should also specify the direction of testing, such as superiority of a treatment over control, before the confirmatory analysis.
Bonferroni Adjustment for Selection Using the Same Data
Suppose there are \(m\) treatment hypotheses in the original family, and each has a valid marginal p-value \(p_i\) under its null hypothesis. The Bonferroni adjusted p-value is
This adjustment is conservative in some settings, but it remains valid even if the analyst selects arms using the same data used to calculate the p-values. The reason is that the familywise probability of at least one false rejection is no greater than the sum of the individual false-rejection probabilities. Selection does not remove any of the original opportunities for a false positive, so adjustment uses the original family size \(m\), not merely the number of arms eventually reported.
Worked Example: Bonferroni Values After Same-Data Selection
In a hypothetical trial, five experimental treatments are compared with a common control using prespecified one-sided tests. The trial team uses the same data to select two promising treatments for consideration. Their raw p-values are \(0.018\) and \(0.073\). Assume each raw p-value is valid for its own null hypothesis before accounting for the five-treatment family.
Because the selection used the testing data, use the original family size, \(m=5\):
For the first selected treatment, the adjusted value is \(\min(1,5(0.018))=0.090\). For the second, it is \(\min(1,5(0.073))=0.365\). At familywise \(\alpha=0.05\), neither hypothesis is rejected. The selection of two treatments does not justify multiplying by two: all five treatment opportunities were present in the same data.
The Bonferroni guarantee follows directly. If each true \(H_i\) is rejected only when \(p_i\leq\alpha/m\), then
where \(I_0\) is the set of true hypotheses and \(|I_0|\leq m\). This union-bound argument does not require independent tests or a selection rule that is independent of the data.
Holm Adjustment for an Independently Selected Validation Family
When selection is based on interim data and confirmation uses an independent validation sample, the validation p-values can be valid conditional on the selected set, provided the validation design and analysis remain valid for each selected hypothesis. Holm’s step-down method then controls familywise error for that selected family, including when its p-values are dependent.
For \(k\) hypotheses in the confirmatory family, order their raw p-values as \(p_{(1)}\leq p_{(2)}\leq\cdots\leq p_{(k)}\). The Holm adjusted value at rank \(r\) is
The running maximum ensures that adjusted p-values do not decrease as raw p-values increase. To test at level \(\alpha\), compare the adjusted p-values with \(\alpha\), or equivalently test in order and stop at the first rank whose raw p-value exceeds \(\alpha/(k-r+1)\).
Worked Example: Holm Values for Two Independently Selected Arms
In a hypothetical adaptive trial, interim data are used to select two of five treatments. A separate validation sample is then used for the confirmatory comparisons. Assume the selection is independent of the validation outcomes, the two validation p-values are valid under their respective null hypotheses, and the confirmatory family consists of those two selected treatment hypotheses. Their raw p-values are \(0.012\) and \(0.041\). These are explicitly the two smallest raw p-values in this selected family, at ranks 1 and 2.
Here \(k=2\). At rank 1, the Holm multiplier is \(2\); at rank 2, it is \(1\). The running maximum gives
Thus the selected arm with raw p-value \(0.012\) has adjusted p-value \(0.024\), and the selected arm with raw p-value \(0.041\) has adjusted p-value \(0.041\). At \(\alpha=0.05\), both are rejected. This conclusion concerns the two-hypothesis confirmatory family and relies on the independent validation data; it would not be justified by applying Holm only to two arms selected from the same testing data.
Joint-Distribution Adjustment with maxT
A maxT procedure uses the joint null distribution of standardized treatment statistics. For one-sided superiority tests, let \(Z_i\) be the standardized statistic for treatment \(i\), with larger values favoring that treatment. A single-step maxT adjusted p-value for a treatment with observed statistic \(z_i\) is the null probability that the maximum statistic in the prespecified family is at least \(z_i\):
The probability must be calculated under an appropriate joint null distribution. When using this procedure for strong familywise control—control under every configuration of true and false hypotheses—a standard sufficient condition is subset pivotality: the joint distribution of the statistics corresponding to any subset of true null hypotheses is the same regardless of which hypotheses outside that subset are false. The joint null model and this condition must be justified for the design; a correlation estimate or a convenient reference distribution alone does not guarantee validity.
Worked Example: maxT Values for Three Independent Tests
Consider a hypothetical analysis with three prespecified one-sided treatment comparisons. Assume the three standardized statistics are independent standard normal variables under the complete null, and that the true-null statistics retain this joint distribution under partial-null configurations. This satisfies the subset-pivotal condition for this example. The observed raw p-values are \(0.010\), \(0.040\), and \(0.200\). The two treatments selected for further consideration are the ones with the two smallest raw p-values, \(0.010\) and \(0.040\).
For independent standard normal statistics, if a treatment’s raw p-value is \(p_i=1-\Phi(z_i)\), then the probability that every statistic is below \(z_i\) is \(\Phi(z_i)^3=(1-p_i)^3\). Therefore the maxT adjusted value is
For the first selected treatment, this gives \(1-(1-0.010)^3=1-0.970299=0.02970\). For the second selected treatment, it gives \(1-(1-0.040)^3=1-0.884736=0.1153\). For completeness, the third treatment’s value is \(1-(1-0.200)^3=0.4880\). At \(\alpha=0.05\), only the first selected treatment is rejected. These adjusted values refer to the full three-treatment family, not a family restricted after observing which arms ranked highest.
The maxT derivation also explains how the procedure controls familywise error under subset pivotality. Choose a cutoff \(c_\alpha\) so that the probability that the maximum of all \(m\) statistics exceeds it under the joint null is \(\alpha\). Reject a hypothesis only if its statistic is at least \(c_\alpha\). If any true hypothesis is rejected, the maximum over the true hypotheses must also be at least \(c_\alpha\). Under subset pivotality, that maximum has its calibrated null distribution, so the probability of any false rejection is at most \(\alpha\). Correlation can make maxT less conservative than Bonferroni, but it must be represented correctly.
Choosing and Interpreting the Adjustment
| Approach | When it can be used | Key requirement |
|---|---|---|
| Bonferroni over original family | Selection and testing can use the same data | Each marginal p-value is valid under its null |
| Holm within selected family | Selection uses interim data; confirmation uses independent validation data | Valid validation p-values conditional on the selected family |
| maxT over the prespecified family | Joint distribution of treatment statistics is available and selection is based on those statistics | Correct joint null calibration and conditions supporting strong control |
| Selection-conditional inference | The target inference explicitly conditions on the selection event | A valid conditional null distribution for the selection rule and relevant hypotheses |
A conditional p-value is computed from a distribution given that a particular selection event occurred. It may be useful when the scientific question is explicitly conditional on that event, but conditioning changes the reference distribution. It is not automatically a strong familywise adjusted p-value: validity must be established for the actual selection rule and all relevant null configurations. Unconditional Bonferroni or a justified maxT procedure can instead control error across the original family without conditioning on the observed selection.
For any approach, an adjusted p-value at or below \(\alpha\) means the specified procedure would reject that hypothesis at familywise level \(\alpha\). It does not mean there is that probability that the null is true. The adjustment also does not repair a biased effect estimate or an invalid raw p-value; it addresses multiplicity under its stated assumptions.
Diagnostics and Common Pitfalls
- Reducing the family after selection. Multiplying by the number of reported arms can understate error when those same data selected the arms. Keep the original family, unless a valid design feature such as independent validation justifies a selected confirmatory family.
- Ignoring ranks in Holm adjustment. Holm values depend on ordered raw p-values and their ranks. Identify the family and ranks before reporting an adjusted value.
- Using a joint distribution without checking its assumptions. Examine whether the test statistics’ dependence and null distributions match the assumed model, including any subset-pivotality requirement for strong control.
- Changing directions after seeing the results. A one-sided test must have its direction specified in advance. Choosing the favorable direction after observing the statistics invalidates the claimed raw p-values.
- Confusing adjusted and raw p-values. Report both when useful, together with the method and family. A raw value of \(0.012\) is not itself a confirmatory result if the procedure gives an adjusted value above the chosen \(\alpha\).
Check Your Understanding
Answer each question, then compare with the brief solutions below.
- Four treatment hypotheses are tested, and one is selected using the same data. Its valid raw p-value is \(0.016\). What is its Bonferroni adjusted p-value, and is it rejected at familywise \(\alpha=0.05\)?
- Three hypotheses form an independently selected validation family. Their ordered raw p-values are \(0.008\), \(0.030\), and \(0.090\). Calculate all three Holm adjusted p-values.
- For two independent one-sided tests, an arm has raw p-value \(0.05\). Calculate its maxT adjusted p-value under the complete-null model.
- Why is Holm adjustment on only the selected arms not automatically valid when the same data both select and test them?
- What does it mean to say an adjusted p-value is \(0.04\)? State what it does not mean.
Solutions. (1) \(\min(1,4(0.016))=0.064\); it is not rejected at \(0.05\). (2) At rank 1, \(3(0.008)=0.024\); at rank 2, the running maximum is \(\max(0.024,2(0.030))=0.060\); at rank 3, it is \(\max(0.024,0.060,0.090)=0.090\). (3) \(1-(1-0.05)^2=1-0.9025=0.0975\). (4) Selection from the testing data favors arms with favorable statistics, so the selected raw p-values need not behave like p-values for a prespecified family; the adjustment must account for the original selection opportunities or use a justified independent validation design. (5) It means the specified familywise procedure would reject that hypothesis at level \(0.04\) or higher; it is not a 4% probability that the null is true.