Begin With the Comparison Question
After a one-way ANOVA, the relevant question is not simply whether some group means differ. Researchers often want to know which means differ, or whether a particular combination of means differs. Tukey, Dunnett, and Scheffé procedures all provide simultaneous inference, but they protect different families of comparisons. Choosing among them begins by stating which comparisons answer the scientific question.
Use Tukey when the family consists of every pair of group means. Use Dunnett when the family consists of comparisons of several treatments with one designated control. Use Scheffé when inference must cover any contrast among the group means, including a contrast selected after looking at the data. These procedures are not interchangeable: a procedure tailored to a narrower family can generally use a less conservative adjustment than one protecting a much broader family.
Model and Shared Assumptions
The procedures in this tutorial use the fixed-effects one-way ANOVA model. For group \(i\) and observation \(j\), write \(Y_{ij}=\mu_i+\varepsilon_{ij}\), where \(\mu_i\) is the population mean for group \(i\). The errors are assumed independent across observations, normally distributed with mean zero, and to have a common variance \(\sigma^2\). Thus \(Y_{ij}\sim N(\mu_i,\sigma^2)\). The pooled within-group mean square, \(MSE\), estimates \(\sigma^2\), with error degrees of freedom \(df_E=N-k\), where \(N\) is the total sample size.
The procedures also presume that the comparison family is clearly defined. Independence comes from the study design, such as assigning different independent experimental units to treatments. Normality and similar within-group variability are assessed using the data and study context; they cannot be established just by choosing a multiple-comparison method. Serious departures can make the usual ANOVA-based inferences unreliable.
The number and type of comparisons determine the family. All pairwise comparisons among \(k\) means comprise \(k(k-1)/2\) differences. If one group is a control, the treatment-versus-control family has \(k-1\) differences. The family of all contrasts is much broader: it includes every coefficient vector summing to zero, not just the pairwise differences.
What Each Procedure Protects
| Procedure | Comparison family | Typical use |
|---|---|---|
| Tukey HSD or Tukey–Kramer | All pairwise differences among the group means | Every pair is scientifically relevant |
| Dunnett | Each treatment compared with one designated control | The control is the reference for every treatment question |
| Scheffé | All contrasts among the group means | Any contrast may be of interest, including one selected after inspecting the data |
Tukey HSD uses the studentized-range distribution for equal group sizes. Tukey–Kramer extends the method to unequal group sizes. Dunnett uses the joint distribution of treatment-versus-control statistics, accounting for their shared control. Scheffé uses an \(F\)-based critical value that protects all contrasts. The procedures’ critical values depend on \(\alpha\), the number of groups, and the relevant degrees of freedom; for Dunnett and Tukey, the group sizes also matter.
Worked Example: When Every Pair Matters
Worked Example: When Every Pair Matters
Suppose a hypothetical randomized experiment compares four treatments, A, B, C, and D. Each group has \(n_i=6\) independent units. The observed group means are 10, 14, 22, and 16 outcome units, respectively. The total within-group sum of squares is 180, so \(df_E=24-4=20\) and \(MSE=180/20=9\) outcome-units-squared. Assume the design supports independence and that residuals show no serious nonnormality, extreme outliers, or large differences in variability. Since the question concerns every pair, use Tukey HSD.
For equal group sizes, the simultaneous margin for each pair is \(q_{1-\alpha;k,df_E}\sqrt{MSE/n}\). At family-wise \(\alpha=0.05\), suppose the appropriate studentized-range critical value is \(q=3.958\). The margin is
For example, the estimated difference for A minus C is \(10-22=-12\) outcome units, giving the simultaneous interval \(-12\pm4.848=(-16.848,-7.152)\). Applying the same margin to all six differences gives:
- A minus B: \(-4\pm4.848=(-8.848,0.848)\).
- A minus C: \(-12\pm4.848=(-16.848,-7.152)\).
- A minus D: \(-6\pm4.848=(-10.848,-1.152)\).
- B minus C: \(-8\pm4.848=(-12.848,-3.152)\).
- B minus D: \(-2\pm4.848=(-6.848,2.848)\).
- C minus D: \(6\pm4.848=(1.152,10.848)\).
The intervals exclude zero for A versus C, A versus D, B versus C, and C versus D. Under this family-wise procedure, those four pairwise differences are statistically distinguishable at the stated simultaneous level. The intervals for A versus B and B versus D include zero, so they do not establish a difference for those pairs. Each interval is in outcome units, and the collection—not each interval considered in isolation—is the family for which simultaneous protection is intended.
Worked Example: When One Group Is the Control
Worked Example: When One Group Is the Control
Use the same hypothetical means and ANOVA error estimate, but suppose A is a control and the scientific questions concern only B versus A, C versus A, and D versus A. The randomization, independence, residual, and common-variance assumptions are as described in the experiment setup. The family has three comparisons, so Dunnett is tailored to the question; Tukey would protect the larger family of six pairs.
Each control comparison has standard error \(\sqrt{MSE(1/n_A+1/n_i)}\). Here, all groups have six observations, so the standard error is
For a two-sided family of three comparisons with \(df_E=20\) and \(\alpha=0.05\), suppose the Dunnett critical value from the appropriate table or software is approximately 2.54. The common margin is \(2.54\sqrt{3}\approx4.399\) outcome units. The treatment-minus-control estimates and simultaneous intervals are:
- B minus A: \(14-10=4\); interval \(4\pm4.399=(-0.399,8.399)\).
- C minus A: \(22-10=12\); interval \(12\pm4.399=(7.601,16.399)\).
- D minus A: \(16-10=6\); interval \(6\pm4.399=(1.601,10.399)\).
The intervals for C minus A and D minus A exclude zero, while the interval for B minus A includes zero. The supported conclusions concern differences from A, not differences between B, C, and D. A treatment-versus-control analysis does not answer every-pair questions.
Worked Example: When an Arbitrary Contrast Matters
Worked Example: When an Arbitrary Contrast Matters
Suppose the question is whether the average of treatments B, C, and D differs from control A. This is one contrast, \(L=(\mu_B+\mu_C+\mu_D)/3-\mu_A\), with coefficients \((-1,1/3,1/3,1/3)\) in the order A, B, C, D. The coefficients sum to zero. The same hypothetical study’s assumptions and \(MSE=9\) apply.
The estimate is \(\hat L=(14+22+16)/3-10=7.333\) outcome units. Its standard error is \(\sqrt{MSE\sum_i c_i^2/n_i}\), so
Scheffé’s simultaneous critical multiplier for all contrasts is \(\sqrt{(k-1)F_{1-\alpha;k-1,df_E}}\). With \(k=4\), \(\alpha=0.05\), and \(df_E=20\), use \(F_{0.95;3,20}\approx3.098\). The multiplier is \(\sqrt{3(3.098)}\approx3.049\), giving a margin of \(3.049\sqrt{2}\approx4.312\) outcome units. The simultaneous interval is \(7.333\pm4.312=(3.021,11.645)\) outcome units. It excludes zero, supporting a difference between the average of the three treatments and the control under the all-contrasts procedure.
Why Scheffé Covers All Contrasts
Scheffé’s wider protection has a direct justification. Write \(e_i=\bar{x}_i-\mu_i\) for the estimation error in group \(i\), and let \(\bar e_w=\sum_i n_i e_i/N\) be the weighted mean of those errors. For any contrast, \(\sum_i c_i=0\), so its estimation error can be written as \(\hat L-L=\sum_i c_i(e_i-\bar e_w)\). The weighted Cauchy–Schwarz inequality gives
The right side involves the estimation errors, not the observed means’ deviations from the null hypothesis. Under the normal one-way model, \(\sum_i n_i(e_i-\bar e_w)^2/\sigma^2\) has a chi-square distribution with \(k-1\) degrees of freedom and is independent of \(MSE/\sigma^2\). Consequently, the right side divided by \(k-1\) has an \(F\) distribution with \(k-1\) and \(df_E\) degrees of freedom. With probability \(1-\alpha\), it is no greater than \(F_{1-\alpha;k-1,df_E}\). The inequality then bounds the standardized estimation error for every contrast at once, yielding Scheffé’s simultaneous critical multiplier \(\sqrt{(k-1)F_{1-\alpha;k-1,df_E}}\).
A Practical Choice and a Common Pitfall
State whether it asks about all pairs, treatment-versus-control differences, or a general linear combination of means.
List the comparisons that will be interpreted together. The family determines which false-positive risk must be controlled.
Choose Tukey for all pairs, Dunnett for comparisons with one control, and Scheffé when inference must cover all contrasts.
Use the appropriate group-size and degrees-of-freedom inputs, then state which intervals exclude zero and what those differences mean in the outcome’s units.
A common mistake is to run whichever procedure produces the most significant result, then describe its intervals as if they covered every question. For example, Dunnett intervals for three control comparisons do not simultaneously protect all six pairs among four groups. The correction is to define the question and comparison family before choosing the procedure, and to interpret conclusions only within that family. Scheffé is a natural choice when the contrast itself may be selected after inspecting the data; its broader coverage generally comes at the cost of wider intervals for a narrowly specified question.
Check Your Understanding
Answer each question by identifying the comparison family and explaining the relevant choice.
- A study compares five diets, and researchers want estimates for every pair of diet means. Which procedure is designed for this family?
- A drug is tested at several doses against placebo, and only dose-versus-placebo comparisons are of interest. Which procedure is tailored to that question?
- Explain what makes a linear combination of group means a contrast.
- Why is Scheffé useful when the contrast of interest may be chosen after examining the observed means?
- Four groups have six observations each, \(MSE=9\), and the Tukey critical value is 3.958. Write the equal-size Tukey margin formula and calculate its value in outcome units.
- Why do intervals from a Dunnett analysis not justify conclusions about differences between two treatments?