Why the Comparison Plan Matters
After an ANOVA, the next question is often not simply whether any group means differ, but which means or combinations of means differ. The answer depends on the family of comparisons: the complete set of claims for which you want a stated level of error protection. A procedure suited to comparing every pair of groups may be unnecessarily broad when the only question is how several treatments compare with one control. Conversely, a procedure for a short list of planned comparisons does not automatically protect every comparison you might decide to report later.
This tutorial focuses on choosing among Tukey, Bonferroni, Holm, Scheffé, and Dunnett procedures. The choice should follow the scientific question and comparison plan, not whichever method happens to produce the smallest p-value.
Start With the ANOVA Model and the Question
These procedures are commonly used within the one-way ANOVA model. For each group \(i\), the observations are modeled as independent measurements from a population with mean \(\mu_i\) and a common variance \(\sigma^2\). The usual model also assumes normally distributed errors within groups. The pooled ANOVA error mean square, \(MSE\), estimates \(\sigma^2\), and its degrees of freedom are \(df_E\). The comparisons below rely on that pooled estimate and the model assumptions; a multiple-comparison adjustment does not repair a poor model.
Before choosing a procedure, state what is to be compared. Are all pairs of means of interest? Are only treatment means compared with a control? Is there a short, scientifically motivated list of comparisons fixed before examining results? Or might any contrast—a weighted combination of means whose coefficients sum to zero—be examined? These questions distinguish the procedures.
| Procedure | Best match to the comparison family | Key feature |
|---|---|---|
| Tukey (or Tukey-Kramer) | Every pair of group means | Designed for all pairwise differences |
| Bonferroni | A specified finite set of tests or intervals | Simple protection using the number of comparisons |
| Holm | A specified finite set of tests | Step-down adjustment; usually at least as powerful as Bonferroni |
| Scheffé | All possible contrasts among the means | Protects even contrasts selected after seeing the data |
| Dunnett | Several treatment-versus-control comparisons | Uses the shared control structure |
What Each Procedure Protects
Tukey’s procedure is the natural choice when the family consists of all pairwise mean differences. For equal group sizes, Tukey HSD uses the studentized-range distribution; for unequal group sizes, Tukey-Kramer adjusts the standard error for each pair. It is usually more efficient for all-pairs questions than applying a general-purpose correction to every pair separately. It does not provide the same protection for every possible contrast.
Bonferroni is flexible: it can be used for any finite family of \(m\) tests. For a familywise significance level \(\alpha\), each test can be conducted at level \(\alpha/m\). For two-sided intervals, a Bonferroni simultaneous interval for each comparison uses a \(t\) critical value with tail area \(\alpha/(2m)\). Its simplicity is useful when the family is short and specified, but the adjustment can become conservative as \(m\) grows.
Holm’s procedure also applies to a finite family of \(m\) tests. Sort the unadjusted p-values from smallest to largest. Compare the smallest with \(\alpha/m\), the next with \(\alpha/(m-1)\), and continue, stopping at the first failure to reject. Holm controls the familywise error rate under arbitrary dependence among the tests and is never less powerful than the ordinary Bonferroni procedure for the same family. It is a testing procedure; intervals require a corresponding simultaneous-interval method if interval estimates are also wanted.
Scheffé’s procedure protects all contrasts among the means, not just a list of pairwise differences. This broad protection is valuable when the contrast of interest may be chosen after inspecting the data, but it generally costs power for a small set of planned comparisons. For \(k\) groups, a Scheffé simultaneous interval for a contrast has a multiplier based on \(\sqrt{(k-1)F_{k-1,df_E;1-\alpha}}\), where \(F_{k-1,df_E;1-\alpha}\) is the \(1-\alpha\) quantile of the \(F\) distribution.
Dunnett’s procedure is tailored to comparing several treatment means with one control mean. The comparisons share the control group, so their test statistics are dependent. Dunnett’s critical value accounts for that structure and is typically more efficient than treating the comparisons as unrelated tests with a generic correction.
A Short Justification for Bonferroni Protection
The Bonferroni rule follows from a probability bound and does not require the tests to be independent. Suppose each of \(m\) true-null tests has probability at most \(\alpha/m\) of a false rejection. Let \(A_j\) be the event that test \(j\) falsely rejects. The event of at least one false rejection is the union of those events. Since the probability of a union is no greater than the sum of the individual probabilities,
Thus, the probability of one or more false rejections in the family is no greater than \(\alpha\). The bound may be conservative because false-rejection events can overlap. Holm improves on the Bonferroni step-down rule by using less stringent thresholds as the ordered p-values increase, while retaining familywise control.
Worked Examples: Matching Procedure to Purpose
Worked Example: Every Pair of Four Group Means
Suppose a hypothetical balanced one-way ANOVA compares four methods, with \(n=8\) observations per method, \(MSE=16.0\), and \(df_E=28\). The sample means are 12, 17, 20, and 27 units. The research question is to identify which of the six possible pairs differ. The observations are assumed independent, the group errors are approximately normal, and the groups share a common variance, so the pooled ANOVA error term is appropriate for these comparisons.
Because the family is all pairwise differences, Tukey HSD is a direct match. Using a hypothetical software Tukey critical value \(q=3.86\) for these settings, the minimum significant difference is
The six absolute differences are 5, 8, 15, 3, 10, and 7 units. Differences greater than 5.46 are significant by this procedure: 8, 15, 10, and 7 units. Differences of 5 and 3 units are not. The conclusion is about these four pairwise differences, with Tukey familywise protection across all six pairs. If group sizes were unequal, Tukey-Kramer rather than this balanced-design formula would be used.
Worked Example: A Short List of Selected Tests
In a separate hypothetical ANOVA, investigators specify three comparisons before examining the results. Assume the model conditions for the pooled ANOVA error term are appropriate. The unadjusted two-sided p-values are 0.008, 0.021, and 0.041, and the desired familywise level is \(\alpha=0.05\). Bonferroni compares each p-value with
Only 0.008 is below 0.0167, so Bonferroni rejects only the first null hypothesis. For Holm, order the p-values as shown and compare sequentially:
All three comparisons pass their Holm thresholds, so Holm rejects all three null hypotheses at familywise level 0.05. For clarity, the Holm adjusted p-values are 0.024, 0.042, and 0.042: the ordered values are multiplied by 3, 2, and 1, respectively, and adjusted to be nondecreasing. Holm is the preferable of these two tests here if the family truly contains just these three comparisons. If the comparisons were selected only after looking for small p-values, however, treating them as a prespecified three-test family would understate multiplicity.
Worked Example: A Contrast That May Be Chosen Flexibly
Consider four hypothetical group means of 12, 17, 20, and 27 units, with \(n=8\) per group, \(MSE=16.0\), and \(df_E=28\). Suppose the question is whether the average of the first two means differs from the average of the last two. The contrast coefficients are \(c=(1/2,1/2,-1/2,-1/2)\), which sum to zero. Its estimated value is
If this is one contrast fixed in advance, a more focused method such as Bonferroni for a prespecified family may be appropriate. If analysts want protection for any contrast they might examine among the four means, Scheffé is designed for that broader family. For the equal-size groups, the standard error of this contrast is
Using \(F_{3,28;0.95}\approx 2.95\), the Scheffé multiplier is \(\sqrt{3(2.95)}\approx2.975\). The simultaneous interval is therefore
This interval excludes zero, so the contrast is statistically distinguishable from zero under the Scheffé procedure. It describes the difference between the average of the first two means and the average of the last two, not a causal effect.
Worked Example: Three Treatments Compared With a Control
A hypothetical experiment has one control group and three treatment groups. Suppose the control has \(n_C=10\), each treatment has \(n_i=9\), \(MSE=25.0\), \(df_E=36\), and the respective sample means are 50, 54, 57, and 52 units. The question, specified in advance, is whether each treatment differs from the control. Assume independent observations, approximately normal errors, and a common group variance.
There are three treatment-control comparisons, so Dunnett is the direct match. The standard error for each difference is the same here:
For three two-sided treatment-versus-control comparisons, \(df_E=36\), and familywise \(\alpha=0.05\), use the Dunnett critical value, approximately 2.45 for these settings. The simultaneous margin is \(2.45(2.297)\approx5.63\) units. The treatment-minus-control estimates and intervals are:
- Treatment 1: \(54-50=4\), giving \(4\pm5.63=(-1.63,9.63)\) units.
- Treatment 2: \(57-50=7\), giving \(7\pm5.63=(1.37,12.63)\) units.
- Treatment 3: \(52-50=2\), giving \(2\pm5.63=(-3.63,7.63)\) units.
Only the second interval excludes zero. At familywise level 0.05, the results support a difference between Treatment 2 and the control; they do not establish a difference for Treatments 1 or 3. Using the Dunnett critical value matters: substituting an unrelated cutoff would change the interval margins and would no longer be the stated Dunnett calculation.
A Practical Decision Process
List the hypotheses or intervals for which familywise protection is required. Decide this from the scientific question, not from which results look promising.
For every pair, use Tukey or Tukey-Kramer. For multiple comparisons to one control, use Dunnett. For a short selected family, consider Bonferroni or Holm. For any contrast, consider Scheffé.
Use the one-way ANOVA pooled error term only when the independence, approximate normality, and common-variance assumptions are reasonable for the analysis.
Name the family, procedure, familywise level, and comparisons supported by the results. Interpret estimates in the original measurement units.
Why Procedure Choice Matters
Different procedures answer different questions and spend their error protection differently. A method that covers every possible contrast has a broader obligation than one covering a short planned list, so its intervals can be wider. A method that exploits the shared-control structure can be more suitable than a general adjustment for the same treatment-control question. The goal is not to find one universally best method; it is to choose a method whose protected family matches the claims that will be made.
A frequent mistake is to choose the comparisons after inspecting the data, then describe them as a small prespecified family. For example, searching all pairwise results and applying Bonferroni only to the two most interesting ones does not control error across the original search. Define the family to include the comparisons actually searched, or use a procedure whose protection covers that broader search. Similarly, Tukey is not the default for any collection of means, and Dunnett is not meant for all pairs among treatments.
Check Your Understanding
For each situation, identify a suitable procedure and explain what family it protects.
- A study has six group means and the investigators want every pairwise comparison. Which procedure is the natural match?
- Three treatments are each compared with the same control, with no treatment-to-treatment comparisons planned. Which procedure is designed for this structure?
- Five scientifically motivated tests were specified before data collection. Name two procedures that can control familywise error for this finite family.
- Why can Scheffé intervals be wider than intervals for a short prespecified set of contrasts?
- An analyst searches all pairwise differences, then applies Bonferroni only to the two comparisons with the smallest p-values. Explain the problem with that reasoning.