Why Adjusted Results Need Careful Interpretation
A one-way ANOVA comparison may involve several questions about group means: which pairs differ, or whether particular planned contrasts are distinguishable from zero. If each question is tested separately at significance level \(\alpha\), the chance of making at least one false rejection across the whole collection can exceed \(\alpha\). Multiple-comparison procedures address this by controlling error over a defined family of comparisons.
An adjusted p-value and a simultaneous confidence interval are two ways of presenting results from such a procedure. The p-value is interpreted relative to the chosen procedure and family; the interval is constructed to provide coverage for the family together. Neither should be read as though it were an ordinary single-comparison result with no adjustment.
The phrase “for the defined family” matters. A family might be all pairwise comparisons among the groups, or it might be a smaller set of comparisons chosen for a scientific purpose. The adjustment depends on that choice and on the procedure used. A Bonferroni-adjusted p-value, a Holm-adjusted p-value, and a Tukey-adjusted p-value are not interchangeable labels for one universal calculation.
Model and the Link Between Tests and Intervals
For a one-way ANOVA with \(k\) fixed groups, use the model \(Y_{ij}=\mu_i+\varepsilon_{ij}\), where \(Y_{ij}\) is the outcome for observation \(j\) in group \(i\), and \(\mu_i\) is that group’s population mean. The usual ANOVA procedures assume independent errors with a common variance and a normal distribution: \(\varepsilon_{ij}\sim N(0,\sigma^2)\). Equivalently, within each group the outcomes are normally distributed around that group’s mean with the same variance. The residual mean square, \(MSE\), estimates \(\sigma^2\), and the residual degrees of freedom are \(df=N-k\).
For a comparison of two group means, such as \(\mu_i-\mu_j\), an ordinary two-sided test uses the estimated difference and its standard error to form a \(t\) statistic with the ANOVA residual degrees of freedom. A multiple-comparison procedure changes the rejection rule, or its equivalent p-value or interval, to account for the family. The procedure does not remove the need for the model assumptions: independence, a reasonable normal error model, and a common within-group variance still matter.
Bonferroni-Adjusted P-Values and Intervals
Suppose a family contains \(m\) two-sided comparisons, with unadjusted p-values \(p_1,\ldots,p_m\). The Bonferroni-adjusted p-value for comparison \(r\) is \(\min(1,mp_r)\). Equivalently, at family-wise level \(\alpha\), compare each raw p-value with \(\alpha/m\). A matching two-sided Bonferroni interval for each comparison uses individual confidence level \(1-\alpha/m\), or critical value \(t_{1-\alpha/(2m),df}\). Together, these intervals have at least \(1-\alpha\) simultaneous coverage.
The guarantee follows from the union bound. If each of \(m\) intervals has a miss probability no greater than \(\alpha/m\), then the probability that at least one misses is no greater than the sum of those probabilities:
This argument does not require the comparisons to be independent. Bonferroni can therefore be useful for a selected family, although it may be conservative, especially when many comparisons are made.
Worked Example: Bonferroni Results for Three Pairwise Comparisons
Consider hypothetical outcomes from three independently assigned treatment groups, with \(n_A=n_B=n_C=10\), sample means \(\bar{x}_A=50\), \(\bar{x}_B=54\), and \(\bar{x}_C=58\) units, and \(MSE=16\) square units. The total sample size is \(N=30\), so the residual degrees of freedom are \(df=N-k=30-3=27\). Assume the hypothetical study’s design supports independence and that residual checks show no serious departure from normality or common variance. We will examine all three pairwise comparisons, so \(m=3\).
For any pair, the estimated standard error is
For \(A-C\), the estimated difference is \(50-58=-8\) units, giving \(t=-8/1.789=-4.472\). Suppose the two-sided unadjusted p-value from the \(t\) distribution with \(27\) degrees of freedom is \(0.00013\). Its Bonferroni-adjusted value is
Thus this comparison is significant at family-wise \(\alpha=0.05\). For \(A-B\), the difference is \(-4\) units and \(t=-4/1.789=-2.236\); the two-sided raw p-value is about \(0.034\), giving \(p_{\text{adj}}\approx\min(1,3(0.034))=0.102\). The same calculation applies to \(B-C\), also a difference of \(-4\) units. Those two comparisons do not meet the family-wise 0.05 threshold.
For the matching Bonferroni intervals, the critical value is \(t_{1-0.05/(2\cdot3),27}\), approximately \(2.552\). The half-width is \(2.552(1.789)=4.57\) units. The interval for \(A-C\) is
It excludes zero, consistent with its adjusted p-value. Each of the other two intervals is \(-4\pm4.57=(-8.57,0.57)\) units, which includes zero. These are simultaneous intervals: the Bonferroni guarantee applies to all three comparisons together, not separately as a 95% guarantee for each interval in isolation.
Holm-Adjusted P-Values
Holm’s step-down procedure also controls the family-wise error rate. It begins by sorting the \(m\) raw p-values from smallest to largest, \(p_{(1)}\leq\cdots\leq p_{(m)}\). At level \(\alpha\), it compares \(p_{(1)}\) with \(\alpha/m\), then \(p_{(2)}\) with \(\alpha/(m-1)\), continuing until a comparison fails its threshold; it stops rejecting at that point.
Holm-adjusted p-values make the same step-down decisions by comparing each adjusted value with \(\alpha\). In sorted order, calculate \((m-r+1)p_{(r)}\) for rank \(r\), then take the running maximum so the adjusted values do not decrease as the raw p-values increase. Cap each value at 1. The running maximum is important: multiplying each raw p-value by its rank-specific factor without this step can produce values that do not represent Holm’s procedure.
Worked Example: Calculating Holm-Adjusted P-Values
A hypothetical one-way ANOVA output gives four raw p-values for a prespecified family of four contrasts: \(0.031\) for contrast \(L\), \(0.004\) for \(M\), \(0.20\) for \(N\), and \(0.018\) for \(O\). Suppose these are two-sided p-values calculated under the one-way model, with independent observations, approximately normal residuals, and a common within-group variance. Sort them before applying Holm:
| Rank \(r\) | Contrast | Raw p-value | Multiplier \(m-r+1\) | Running adjusted p-value |
|---|---|---|---|---|
| 1 | M | 0.004 | 4 | 0.016 |
| 2 | O | 0.018 | 3 | 0.054 |
| 3 | L | 0.031 | 2 | 0.062 |
| 4 | N | 0.20 | 1 | 0.20 |
The first adjusted value is \(4(0.004)=0.016\). At rank 2, the candidate is \(3(0.018)=0.054\), which exceeds \(0.016\), so the running value becomes \(0.054\). At rank 3, \(2(0.031)=0.062\), so the running value becomes \(0.062\). At rank 4, the candidate is \(1(0.20)=0.20\). No cap at 1 is needed here.
At family-wise \(\alpha=0.05\), only contrast \(M\) is rejected: its Holm-adjusted p-value \(0.016\) is below \(0.05\), while \(O\)’s is \(0.054\), just above the threshold. This matches the step-down test: the first raw p-value passes \(0.05/4=0.0125\), but the second, \(0.018\), does not pass \(0.05/3\approx0.0167\), so the procedure stops. The adjusted values depend on all four p-values and the specified family, not just on the raw p-value for a single contrast.
Holm-adjusted p-values do not by themselves specify one universal set of matching, symmetric confidence intervals. If an analysis reports Holm-adjusted p-values and intervals, check which interval procedure was used and whether it provides simultaneous coverage for the same family. Do not assume that an ordinary 95% interval for each contrast inherits Holm’s family-wise guarantee.
Tukey-Adjusted Results and Simultaneous Pairwise Intervals
Tukey’s procedure is designed for all pairwise comparisons among group means. Its adjusted p-values account for the distribution of the largest standardized pairwise difference in the family. In a balanced design, each pairwise interval has the form \((\bar{x}_i-\bar{x}_j)\pm q_{1-\alpha;k,df}\sqrt{MSE/n}\), where \(q_{1-\alpha;k,df}\) is the appropriate studentized-range critical value. Tukey-Kramer extends the method to unequal group sizes. The relevant guarantee is simultaneous coverage for the pairwise family, not a separate guarantee that each interval alone has the stated confidence level.
Worked Example: Reading a Tukey Pairwise Output
Use the same hypothetical three-group ANOVA results: group sizes of 10, means of 50, 54, and 58 units, \(MSE=16\), and residual \(df=27\). The model assumptions described above are retained. Tukey’s family here consists of all three pairwise comparisons. At \(\alpha=0.05\), take the studentized-range critical value for \(k=3\) and \(df=27\) to be approximately \(3.51\).
For each pair, the Tukey interval half-width is
The estimated difference for \(A-C\) is \(-8\) units, so its Tukey simultaneous interval is \(-8\pm4.44=(-12.44,-3.56)\) units. It excludes zero. The estimated difference for \(A-B\) is \(-4\) units, so its interval is \(-4\pm4.44=(-8.44,0.44)\) units; it includes zero. The \(B-C\) interval has the same endpoints as \(A-B\), because that estimated difference is also \(-4\) units.
A Tukey output using this same family and \(\alpha=0.05\) must agree with those interval decisions: the \(A-C\) Tukey-adjusted p-value is below \(0.05\), while the \(A-B\) and \(B-C\) adjusted p-values are above \(0.05\). For example, an output reporting \(p_{\text{adj}}<0.001\) for \(A-C\) indicates strong evidence of a difference under the Tukey procedure; it does not mean the probability that the null hypothesis is true is below \(0.001\).
How to Read an Output Without Overstating It
When reading software output, first identify the comparison family and adjustment method. Then check whether the reported p-values are raw or adjusted, and whether the intervals are ordinary or simultaneous. Finally, compare adjusted p-values with the stated family-wise \(\alpha\), or check whether the corresponding simultaneous intervals exclude the null value. For a difference of means, that null value is zero.
A simultaneous 95% family of intervals means that, under the model and procedure, repeated use of the method would produce a set containing all the family’s true differences at least 95% of the time. It does not mean there is a 95% probability that a particular fixed population difference lies in its observed interval. Nor does an adjusted p-value give the probability that its null hypothesis is true. It measures compatibility of the data with that null under the test procedure, with the adjustment accounting for the stated family.
Intervals can also show practical scale and direction. In the Bonferroni example, the \(A-C\) estimate was \(-8\) units, so the negative direction means group A’s estimated mean was 8 units lower than group C’s. The simultaneous interval quantifies uncertainty about that difference in outcome units. Statistical significance alone does not establish that the difference is important in practice.
Common Pitfall
- Wrong reasoning: “The raw p-value is below 0.05, so the comparison is significant, and the ordinary 95% interval is enough, even though I tested several comparisons.”
- Correction: For a family-wise conclusion, use the adjusted p-value or an interval with the matching simultaneous guarantee. In the Bonferroni example, a raw p-value near \(0.034\) is below \(0.05\), but its adjusted value is about \(0.102\), so it is not significant at family-wise level \(0.05\).
- Wrong reasoning: “An adjusted p-value of \(0.04\) means there is a 4% probability the null is true.”
- Correction: It means the specified procedure would reject that comparison at family-wise significance levels of \(0.04\) or greater (with the usual interpretation of adjusted p-values). It is not a probability assigned to the null hypothesis.
Check Your Understanding
Answer each question using the family and adjustment method named in the question.
- A family has five comparisons and a raw p-value of \(0.012\) for one comparison. What is its Bonferroni-adjusted p-value?
- For Holm’s procedure, why must the raw p-values be sorted before applying the step-down calculations?
- A simultaneous 95% family of intervals includes zero for one difference of means. What does that imply about the corresponding two-sided test at family-wise \(\alpha=0.05\), if it uses the same procedure?
- In a Tukey pairwise analysis, what family does the procedure address, and what does its joint coverage statement concern?
- Explain why an adjusted p-value of \(0.03\) is not a 3% probability that the null hypothesis is true.