Why Adjusted P-Values Are Useful
A raw p-value summarizes evidence against one null hypothesis using its own test. When several hypotheses belong to one prespecified family, however, the rejection rule must also account for the chance of making one or more false rejections across that family. Bonferroni, Holm, and Hochberg procedures do this in different ways. Adjusted p-values put their decisions on a common scale: for each hypothesis, they report the smallest familywise significance level at which the procedure would reject it.
The adjustment does not change the observed data or turn a raw p-value into a probability that a null hypothesis is true. It gives a procedure-specific threshold for rejection. Consequently, the adjusted value depends both on the hypothesis’s raw p-value and on the other p-values in the family.
The endpoint is stated separately on purpose. If the capped adjusted value is 1, it means there is no level below 1 at which the procedure rejects that hypothesis. It does not assert that the procedure’s rejection rule at exactly \(\alpha=1\) must reject it. Standard testing decisions use levels strictly below 1; at that endpoint, do not infer a decision merely by comparing a capped adjusted value with 1.
Definitions and Assumptions
Let \(H_1,\ldots,H_m\) be the hypotheses in the family, and let \(p_i\) be the raw p-value for \(H_i\). Order the raw p-values from smallest to largest: \(p_{(1)}\leq p_{(2)}\leq\cdots\leq p_{(m)}\). The notation \(H_{(j)}\) refers to the hypothesis paired with \(p_{(j)}\). If there are ties, use any fixed ordering among tied hypotheses; the procedures’ rejection thresholds treat tied values consistently.
The family must be specified before interpreting the adjusted values. For familywise error control, each raw p-value for a true null must be valid: under that null, its probability of being at most \(u\) is no greater than \(u\), for every \(0\leq u\leq1\). Bonferroni and Holm control the familywise error rate under arbitrary dependence among the p-values. Hochberg’s step-up procedure requires additional dependence conditions, such as independence or conditions that ensure the Simes inequality for every subset of true null hypotheses. Its adjusted values can be computed without those conditions, but the claimed familywise error control cannot.
The family and the direction of each test must match the scientific question. For example, two-sided p-values belong in a family when the prespecified hypotheses are two-sided. Adjusting one-sided p-values and then interpreting them as two-sided results is not justified by the adjustment formula.
Bonferroni Adjusted P-Values
Bonferroni compares each raw p-value with \(\alpha/m\). Equivalently, multiply each raw p-value by the number of hypotheses and cap the result at 1.
This follows directly from the rejection rule. For \(0\leq\alpha<1\), Bonferroni rejects \(H_i\) when \(p_i\leq\alpha/m\), which is equivalent to \(mp_i\leq\alpha\). Since \(\alpha<1\), this is equivalent to \(\min(1,mp_i)\leq\alpha\). The cap ensures that a reported adjusted p-value remains within the probability scale.
Worked Example: Bonferroni Adjustment for Four Comparisons
Suppose a hypothetical multi-arm trial has four prespecified treatment-versus-control hypotheses. Their raw two-sided p-values are 0.012, 0.030, 0.190, and 0.700. Assume each raw p-value is valid for its own null; no independence assumption is needed for Bonferroni.
Here \(m=4\), so the adjusted values are \(\min(1,4(0.012))=0.0480\), \(\min(1,4(0.030))=0.120\), \(\min(1,4(0.190))=0.760\), and \(\min(1,4(0.700))=1\).
At familywise level \(\alpha=0.05\), the first hypothesis is rejected because its adjusted p-value is \(0.0480\leq0.05\). The other three are not rejected. The value 0.0480 is the smallest familywise level at which Bonferroni would reject the first hypothesis; it is not the probability that its null hypothesis is true.
Holm Adjusted P-Values
Holm’s procedure orders the p-values and tests them sequentially from smallest to largest. At rank \(j\), it compares \(p_{(j)}\) with \(\alpha/(m-j+1)\). It continues rejecting until the first comparison that fails; that hypothesis and all hypotheses with larger ordered p-values are not rejected.
The maximum is needed because a later hypothesis cannot be rejected after an earlier step has failed. To see the formula, the first \(i\) hypotheses pass all Holm comparisons through rank \(i\) exactly when \((m-j+1)p_{(j)}\leq\alpha\) for every \(j\leq i\). This collection of inequalities is equivalent to \(\max_{j\leq i}\{(m-j+1)p_{(j)}\}\leq\alpha\). Capping at 1 gives the reported adjusted value and preserves the rejection equivalence for \(\alpha<1\).
Worked Example: Holm Adjustment and the Stop Rule
Consider four hypothetical valid raw p-values, already ordered: 0.008, 0.021, 0.040, and 0.200. Holm’s procedure is valid under arbitrary dependence, so these calculations do not require independent comparisons.
At rank 1 the multiplier is 4, giving \(4(0.008)=0.032\). At rank 2 it is 3, giving \(3(0.021)=0.063\). At rank 3 it is 2, giving \(2(0.040)=0.080\), and at rank 4 it is 1, giving \(1(0.200)=0.200\). The running maxima are therefore 0.032, 0.063, 0.080, and 0.200. These are the Holm adjusted p-values.
At \(\alpha=0.05\), the first test passes because \(0.008\leq0.05/4=0.0125\). The second fails because \(0.021>0.05/3\approx0.0167\). Holm stops at that point, so only the first hypothesis is rejected. Its adjusted value is 0.032, which is below 0.05; the second adjusted value is 0.063, which is above 0.05.
Hochberg Adjusted P-Values
Hochberg’s procedure works in the opposite direction. It is a step-up procedure: starting at the largest p-value, it searches for a rank \(j\) satisfying \(p_{(j)}\leq\alpha/(m-j+1)\). If it finds such a rank, it rejects that hypothesis and all hypotheses with smaller ordered p-values. Its adjusted values are
For a hypothesis at rank \(i\), the step-up procedure rejects it if there is a qualifying rank \(j\geq i\). At that rank the condition is \((m-j+1)p_{(j)}\leq\alpha\). Existence of such a \(j\) is equivalent to the minimum over \(j\geq i\) being at most \(\alpha\), which gives the formula. As with the other procedures, the cap at 1 and the stated equivalence apply to levels \(\alpha<1\).
Worked Example: Hochberg Can Reject More Than Holm
Suppose three hypothetical, valid raw p-values are 0.020, 0.030, and 0.040. For the Hochberg familywise error claim, assume the p-values are independent. The values are already ordered, and the rank multipliers are 3, 2, and 1.
For rank 1, the relevant products are \(3(0.020)=0.060\), \(2(0.030)=0.060\), and \(1(0.040)=0.040\); the minimum is 0.040. For rank 2, the minimum of \(2(0.030)=0.060\) and \(1(0.040)=0.040\) is 0.040. At rank 3, the value is \(1(0.040)=0.040\). Thus all three Hochberg adjusted p-values are 0.040.
At \(\alpha=0.05\), Hochberg rejects all three because the largest p-value, 0.040, is at most 0.05. For comparison, Holm’s adjusted values are 0.060, 0.060, and 0.060: its running maxima are \(\max(0.060)=0.060\), \(\max(0.060,0.060)=0.060\), and \(\max(0.060,0.060,0.040)=0.060\). Holm therefore rejects none at 0.05. The example illustrates a possible power advantage for Hochberg under its conditions; it does not make Hochberg valid under arbitrary dependence.
How to Compare the Three Procedures
For the same family of raw p-values and a common familywise level, the procedures can yield different rejection sets and adjusted values. Bonferroni applies the same multiplier \(m\) to every p-value. Holm uses rank-specific multipliers while protecting against an early stopping point. Hochberg also uses rank-specific multipliers, but its step-up rule can use a sufficiently small p-value at a later rank to reject hypotheses at earlier ranks.
| Procedure | Adjusted-value formula by rank | Dependence condition for familywise control |
|---|---|---|
| Bonferroni | \(\min(1,mp_i)\) | Arbitrary dependence |
| Holm | \(\min(1,\max_{j\leq i}(m-j+1)p_{(j)})\) | Arbitrary dependence |
| Hochberg | \(\min(1,\min_{j\geq i}(m-j+1)p_{(j)})\) | Independence or suitable dependence conditions ensuring the needed Simes inequality |
Bonferroni is simple and transparent. Holm is at least as powerful as Bonferroni for the same family and level, while retaining familywise control under arbitrary dependence. Hochberg can be more powerful than Holm, but only when its dependence conditions are justified. The formulas do not decide which hypotheses belong in the family or whether the scientific claim is one-sided or two-sided; those choices must be prespecified and match the trial objectives.
Common Mistakes and Diagnostics
- Using the raw p-value as if it were adjusted. A raw value of 0.03 does not by itself establish rejection at familywise level 0.05. Apply the selected procedure to the entire family.
- Mixing formulas across procedures. Holm uses a running maximum over earlier ranks; Hochberg uses a minimum over later ranks. Reversing these operations changes decisions.
- Assuming Hochberg is always valid. Calculation is possible under any dependence, but the familywise error guarantee needs suitable dependence conditions. Use Bonferroni or Holm when arbitrary dependence must be covered.
- Misreading a capped value of 1 at the endpoint. For instance, with Bonferroni and \(m=2\), a raw p-value of 0.8 gives \(\min(1,2(0.8))=1\). At levels \(\alpha<1\), Bonferroni does not reject it. The capped value 1 records that no level below 1 causes rejection; it should not be used to claim rejection at exactly \(\alpha=1\).
- Changing the family after seeing results. Adding or removing hypotheses changes \(m\), the ranks, and the adjusted values. The family should reflect the prespecified set of claims, not a post-result selection.
A useful diagnostic is to calculate decisions two ways at the chosen \(\alpha<1\): apply the procedure’s original stepwise rule, then compare each adjusted value with \(\alpha\). They should agree. A mismatch often signals a rank-ordering error, an incorrect running maximum or minimum, or failure to respect the step-down stop rule.
Check Your Understanding
Work through each question, then check the solutions.
- For \(m=3\) raw p-values 0.010, 0.030, and 0.200, calculate the Bonferroni adjusted p-values.
- For ordered p-values 0.015, 0.025, and 0.060, calculate the Holm adjusted values.
- For those same ordered values, calculate the Hochberg adjusted values. What dependence qualification matters for the familywise error claim?
- Explain why an adjusted p-value of 1 does not, by itself, show that the procedure rejects at \(\alpha=1\).
Solutions. (1) Multiply each value by 3 and cap at 1: 0.030, 0.090, and 0.600. (2) The rank products are \(3(0.015)=0.045\), \(2(0.025)=0.050\), and \(1(0.060)=0.060\); their running maxima give Holm values 0.045, 0.050, and 0.060. (3) The tail minima are \(\min(0.045,0.050,0.060)=0.045\), \(\min(0.050,0.060)=0.050\), and 0.060, so Hochberg values are 0.045, 0.050, and 0.060. Hochberg’s familywise guarantee requires independence or suitable dependence conditions; unlike Holm, arbitrary dependence is not enough. (4) The adjusted value is capped at 1 and means no level strictly below 1 causes rejection. The rejection equivalence is stated for \(\alpha<1\), so the level-one decision must be evaluated from the procedure itself rather than inferred from the capped number.