Tukey HSD for Balanced Pairwise Comparisons

Why Use Tukey HSD?

After comparing several group means, the practical question is often which groups differ. Testing every pair separately with ordinary t tests can make the chance of at least one false positive larger than the nominal significance level. Tukey’s honestly significant difference procedure, or Tukey HSD, addresses this by calibrating all pairwise comparisons together.

This tutorial treats the balanced case: every group has the same sample size. The procedure uses the pooled error estimate from a one-way ANOVA and a critical value from the studentized range distribution. It produces simultaneous confidence intervals and a matching set of pairwise tests.

Model and assumptions: There are \(k\) independent groups, each with \(n\) observations. The observation in group \(j\), replicate \(r\), follows \(X_{jr}=\mu_j+\varepsilon_{jr}\), where all errors are independent and normally distributed with mean 0 and common variance \(\sigma^2\). Thus, \(X_{jr}\sim N(\mu_j,\sigma^2)\). The equal group size, independence, normality, and common-variance assumptions are required for the exact balanced Tukey procedure. No interest-rate or mortality assumptions are involved.

Definitions and the Tukey HSD Formula

Write \(\bar X_j\) for the sample mean in group \(j\). Let \(\mathrm{SSE}\) be the within-group error sum of squares from the one-way ANOVA. The error degrees of freedom are \(\nu=k(n-1)\), and the pooled error mean square is \(\mathrm{MSE}=\mathrm{SSE}/\nu\). The pooled estimate uses within-group variation rather than the observed differences between group means.

Definition: The studentized range statistic for \(k\) independent standard normal variables \(Z_1,\ldots,Z_k\), and an independent \(U\sim\chi^2_\nu\), is \(Q=(\max_j Z_j-\min_j Z_j)/\sqrt{U/\nu}\). The value \(q_{1-\alpha;k,\nu}\) is its quantile with cumulative probability \(1-\alpha\). Here \(\alpha\) is the desired familywise error level, \(k\) is the number of groups, and \(\nu\) is the error degrees of freedom.

For a balanced design, the Tukey critical difference is

\mathrm{HSD}=q_{1-\alpha;k,\nu}\sqrt{\frac{\mathrm{MSE}}{n}}.

For each pair of groups \(i,j\), the simultaneous confidence interval for \(\mu_i-\mu_j\) is

(\bar X_i-\bar X_j)\ \mathbin{\pm}\ q_{1-\alpha;k,\nu}\sqrt{\frac{\mathrm{MSE}}{n}}.

The interval has zero in it exactly when \(|\bar X_i-\bar X_j|\leq\mathrm{HSD}\). Equivalently, at familywise level \(\alpha\), reject \(H_0:\mu_i=\mu_j\) in favor of a difference when \(|\bar X_i-\bar X_j|>\mathrm{HSD}\). The procedure considers all \(k(k-1)/2\) distinct pairs. Use the same critical difference for every pair because the design is balanced.

Why the Standard Error and Critical Difference Have This Form

The standard error of a difference between two independent group means is \(\sigma\sqrt{2/n}\). Tukey’s formula can look surprising because it uses \(\sqrt{\mathrm{MSE}/n}\), not that pairwise standard error. The distinction comes from the definition of the studentized range: its numerator is the range of all \(k\) standardized group means, which already incorporates the scale factor \(\sqrt{2}\).

\frac{\max_j\bar X_j-\min_j\bar X_j}{\sqrt{\mathrm{MSE}/n}} = \frac{(\max_j\bar X_j-\min_j\bar X_j)/(\sigma\sqrt{2/n})} {\sqrt{\mathrm{MSE}/\sigma^2}}\,\sqrt{2}.

The standardized group means have the normal structure used in the studentized range, while \(\nu\,\mathrm{MSE}/\sigma^2\) has a chi-square distribution with \(\nu\) degrees of freedom and is independent of the group means under the model. Consequently, the range divided by \(\sqrt{\mathrm{MSE}/n}\) has the studentized-range scaling. Choosing its \(1-\alpha\) quantile makes the critical difference appropriate for the largest separation across the groups, rather than calibrating each pair in isolation. The exact simultaneous-coverage argument is a separate result; for calculation here, the relevant implication is to use the studentized-range quantile, not a separate t critical value for each pair.

Worked Comparisons

Worked Example: Comparing Three Calibration Methods

A hypothetical laboratory compares the mean calibration time, in minutes, for three methods. Each method is tested on six independent units. The sample means are 18, 22, and 29 minutes, and the ANOVA error sum of squares is 75. Assume independent normal observations with a common variance and equal sample sizes. At familywise level \(\alpha=0.05\), use \(q_{0.95;3,15}=3.67\). Identify the significant pairwise differences and give the simultaneous intervals.

Method. Apply balanced Tukey HSD. There are \(k=3\) groups and \(n=6\) observations per group. The error degrees of freedom and pooled mean square are

\nu=3(6-1)=15,\qquad \mathrm{MSE}=\frac{75}{15}=5.

The standard-error factor is \(\sqrt{5/6}=0.9129\). Therefore,

\mathrm{HSD}=3.67\sqrt{\frac{5}{6}} =3.67(0.9129)=3.35.

Now calculate all three absolute mean differences: \(22-18=4\), \(29-22=7\), and \(29-18=11\). Each exceeds 3.35, so all three pairwise null hypotheses are rejected. The corresponding intervals, rounding endpoints to two decimals, are:

For \(\mu_2-\mu_1\): \(4\pm3.35=[0.65,7.35]\).
For \(\mu_3-\mu_2\): \(7\pm3.35=[3.65,10.35]\).
For \(\mu_3-\mu_1\): \(11\pm3.35=[7.65,14.35]\).

Each interval excludes zero, matching the test decisions. The common mistake this avoids is using a separate, unadjusted pairwise t test for each comparison.

Worked Example: Checking All Six Pairs in a Four-Group Study

A hypothetical study compares the mean time, in minutes, for four procedures to complete a task. Each procedure is tested on five independent units. The sample means are 42, 49, 54, and 58 minutes, and the ANOVA error sum of squares is 192. Assume independent normal observations with a common variance and equal sample sizes. At \(\alpha=0.05\), use \(q_{0.95;4,16}=4.05\). Which pairwise differences are significant?

Method. With \(k=4\) and \(n=5\), the error degrees of freedom are

\nu=4(5-1)=16,\qquad \mathrm{MSE}=\frac{192}{16}=12.

The standard-error factor is \(\sqrt{12/5}=\sqrt{2.4}=1.5492\), so the critical difference is

\mathrm{HSD}=4.05\sqrt{\frac{12}{5}} =4.05(1.5492)=6.27.

The four means generate six pairs. Compare each absolute difference with 6.27:

49 versus 42: difference \(7\), significant; interval \(7\pm6.27=[0.73,13.27]\).
54 versus 42: difference \(12\), significant; interval \(12\pm6.27=[5.73,18.27]\).
58 versus 42: difference \(16\), significant; interval \(16\pm6.27=[9.73,22.27]\).
54 versus 49: difference \(5\), not significant; interval \(5\pm6.27=[-1.27,11.27]\).
58 versus 49: difference \(9\), significant; interval \(9\pm6.27=[2.73,15.27]\).
58 versus 54: difference \(4\), not significant; interval \(4\pm6.27=[-2.27,10.27]\).

Thus, four of the six pairs are significant: 49 versus 42, 54 versus 42, 58 versus 42, and 58 versus 49. The last two listed intervals contain zero, consistent with their nonsignificant tests. In particular, four groups require \(4(5-1)=16\) error degrees of freedom and all six pairs must be checked; using three-group degrees of freedom or omitting a group would not analyze the stated design.

Worked Example: Interpreting an Adjusted Interval Near the Cutoff

A hypothetical materials experiment compares the mean response for three treatments, with eight independent specimens per treatment. The observed means are 12, 16, and 21 response units, and \(\mathrm{SSE}=126\). Assume independent normal observations with common variance and equal group sizes. At \(\alpha=0.05\), use \(q_{0.95;3,21}=3.58\). Determine the decisions and interpret the interval for the first two treatments.

Method. The degrees of freedom are \(\nu=3(8-1)=21\), and \(\mathrm{MSE}=126/21=6\). The standard-error factor is \(\sqrt{6/8}=\sqrt{0.75}=0.8660\). Thus,

\mathrm{HSD}=3.58(0.8660)=3.10.

The absolute differences are \(16-12=4\), \(21-16=5\), and \(21-12=9\). All exceed 3.10, so all three pairwise differences are significant. For the first two treatments, the estimated difference is \(16-12=4\), and the simultaneous interval is

4\pm3.10=[0.90,7.10].

Because the full interval is positive, the data support a higher mean response for treatment 2 than treatment 1, at the stated simultaneous confidence level. The common mistake this avoids is reporting only that the observed means are ordered; the adjusted interval quantifies uncertainty while accounting for the family of comparisons.

Scope, Interpretation, and Common Traps

Tukey HSD is designed for all pairwise comparisons in a balanced one-way ANOVA under the stated model. A significant pair indicates evidence of a difference in population means; it does not show that the difference is practically important or explain its cause. A nonsignificant pair does not establish that the means are equal. Report the estimated difference, its interval, and the comparison in context.

The studentized-range critical value depends on three inputs: the familywise level \(\alpha\), the number of groups \(k\), and the error degrees of freedom \(\nu\). The pooled error mean square and the common sample size determine the scale of the critical difference. A table or software output may use a different quantile convention, so check whether its subscript denotes cumulative probability \(1-\alpha\) or upper-tail probability \(\alpha\).

Do not use a t critical value in place of the range quantile. Ordinary t intervals address an individual contrast; Tukey HSD calibrates the collection of pairwise comparisons.
Do not use \(k-1\) as the error degrees of freedom. In the balanced one-way model, \(\nu=k(n-1)\).
Do not change the standard-error factor to \(\sqrt{2\mathrm{MSE}/n}\) while retaining the studentized-range quantile. The Tukey formula already uses the studentized-range scale.
Do not omit pairs. The procedure’s family consists of all \(k(k-1)/2\) pairwise mean comparisons.
Do not apply the balanced formula to unequal sample sizes. A different procedure is needed for that setting.

The method is useful when the scientific or operational question genuinely concerns every pair. It is not a universal adjustment for arbitrary collections of contrasts. When the design is unbalanced, or when only selected comparisons are of interest, the method and assumptions should match that different question.

Key takeaway: For a balanced one-way ANOVA, compute \(\nu=k(n-1)\) and \(\mathrm{MSE}=\mathrm{SSE}/\nu\), then use \(q_{1-\alpha;k,\nu}\sqrt{\mathrm{MSE}/n}\) as the common Tukey critical difference. Reject a pairwise equality hypothesis exactly when the absolute difference of sample means exceeds that value; equivalently, its simultaneous interval excludes zero.

Check Your Understanding

For each question, use the studentized-range value supplied. Round interval endpoints and critical differences to two decimals.

A balanced study has three groups of four observations each, \(\mathrm{SSE}=36\), and sample means 10, 13, and 17. At \(\alpha=0.05\), use \(q_{0.95;3,9}=3.95\). Find the HSD and identify the significant pairs. (Answer: \(\nu=9\), \(\mathrm{MSE}=4\), HSD \(=3.95\); the pairs with differences 4 and 7 are significant, while the pair with difference 3 is not.)
Four groups each have six observations, \(\mathrm{MSE}=4\), and sample means 20, 23, 25, and 31. At \(\alpha=0.05\), use \(q_{0.95;4,20}=3.96\). Find the critical difference and classify all six pairs. (Answer: HSD \(=3.96\sqrt{4/6}=3.23\); significant differences are 5, 11, 8, and 6, while differences 3 and 2 are not.)
Three groups each have five observations, \(\mathrm{MSE}=10\), and means 34, 40, and 43. At \(\alpha=0.05\), use \(q_{0.95;3,12}=3.77\). Give the simultaneous interval for the difference between the third and second means and decide whether it is significant. (Answer: HSD \(=3.77\sqrt{10/5}=5.33\); interval \(3\pm5.33=[-2.33,8.33]\), not significant.)
In a balanced one-way ANOVA with four groups and seven observations per group, \(\mathrm{SSE}=168\). At \(\alpha=0.05\), use \(q_{0.95;4,24}=3.90\). If two group means differ by 4.8, determine the Tukey decision and give the interval for the signed difference, assuming the first mean exceeds the second by 4.8. (Answer: \(\nu=24\), \(\mathrm{MSE}=7\), HSD \(=3.90\sqrt{7/7}=3.90\); the difference is significant, and the interval is \([0.90,8.70]\).)

What You'll Learn