Tukey HSD for Balanced Designs

Why Use Tukey HSD?

After a one-way ANOVA, the overall test can indicate evidence that not all population means are equal, but it does not identify which means differ. Comparing every pair separately with unadjusted tests can inflate the chance of making at least one false claim across the collection of comparisons. Tukey’s honestly significant difference procedure, usually called Tukey HSD, is designed to compare all pairs while controlling that familywise error rate.

This tutorial treats the balanced case: every group has the same sample size. The procedure uses the pooled error variation from the one-way ANOVA and a critical value from the studentized-range distribution. Its cutoff applies to the whole set of pairwise comparisons, not to each comparison as if it were the only one.

Definition: The familywise error rate is the probability of making at least one false rejection among a specified family of hypothesis tests. For Tukey HSD, the family consists of all pairwise hypotheses $H_0:\mu_i=\mu_j$ among the $k$ group means.

Model, Assumptions, and the HSD Rule

Suppose there are $k$ groups, each with $n$ observations. The one-way ANOVA model is

Y_{ij}=\mu_i+\varepsilon_{ij},\qquad i=1,\ldots,k,\quad j=1,\ldots,n.

Here $\mu_i$ is the population mean for group $i$. For the standard Tukey HSD procedure, errors are independent and normally distributed with mean zero and a common variance $\sigma^2$: $\varepsilon_{ij}\sim N(0,\sigma^2)$. Thus observations are independent within and across groups, and the population variances are equal. Equal group sizes are an additional requirement for the version developed here. As with the one-way ANOVA model, independence should be supported by the study design, and the normality and common-variance assumptions should be reasonable for the data.

The pooled within-group mean square error, denoted $MSE$, estimates the common error variance. In a balanced design the residual degrees of freedom are $df=k(n-1)$. Let $q_{1-\alpha;k,df}$ be the $1-\alpha$ quantile of the studentized-range distribution with $k$ groups and $df$ error degrees of freedom. It is an upper-tail critical value: a proportion $\alpha$ of that distribution lies above it.

Definition: For equal group sizes, Tukey HSD declares groups $i$ and $j$ significantly different at familywise level $\alpha$ when $$ |\bar{x}_i-\bar{x}_j|>HSD,\qquad HSD=q_{1-\alpha;k,df}\sqrt{\frac{MSE}{n}}. $$ The quantity $HSD$ is the common cutoff for every pair of sample means.

The sample means are in the response variable’s units, and so is the HSD cutoff. A difference larger than the cutoff is evidence against that pair’s equality hypothesis under the procedure. A difference at or below the cutoff is not declared significant; this does not establish that the two population means are equal.

Why the Studentized Range Gives Familywise Control

The key feature is that the critical value is based on a range across all $k$ group means, rather than on a separate unadjusted cutoff for each pair. The argument must be made using the estimation errors of the means. The range of the observed means themselves also contains the actual differences among population means, so it does not have a studentized-range distribution in general.

Write $e_i=\bar{x}_i-\mu_i$ for the estimation error in group $i$'s sample mean. Under the model, the $e_i$ are independent $N(0,\sigma^2/n)$ variables. Define $Z_i=\sqrt{n}e_i/\sigma$, so the $Z_i$ are independent standard normal variables. Under the normal equal-variance model, $U=df\cdot MSE/\sigma^2$ has a chi-square distribution with $df$ degrees of freedom and is independent of the group means. Therefore,

\frac{\max_i e_i-\min_i e_i}{\sqrt{MSE/n}} = \frac{\max_i Z_i-\min_i Z_i}{\sqrt{U/df}}.

The right-hand side has the studentized-range distribution with $k$ groups and $df$ degrees of freedom. In particular, the probability that this full error range exceeds $q_{1-\alpha;k,df}\sqrt{MSE/n}$ is $\alpha$.

To see why this controls the familywise error rate, consider any pair $i,j$ for which the null hypothesis is true. Then $\mu_i=\mu_j$, and

\bar{x}_i-\bar{x}_j=e_i-e_j.

For every pair of errors, $|e_i-e_j|\leq\max_r e_r-\min_r e_r$. Thus, if Tukey HSD falsely rejects even one true pairwise null, the full error range must exceed the HSD cutoff. The probability of that event is at most $\alpha$. This reasoning applies whether all pairwise nulls are true or only a subset is true: the chance of at least one false rejection is no greater than $\alpha$. When all means are equal, all pairwise nulls are true and the familywise error probability reaches $\alpha$ under the continuous model.

Applying Tukey HSD

The procedure needs the group means, the common sample size, $MSE$, its error degrees of freedom, the chosen significance level, and the appropriate studentized-range critical value. A table or statistical software can supply $q_{1-\alpha;k,df}$. Check that the table or software uses the intended $k$, $df$, and upper-tail probability $1-\alpha$.

Worked Example: Comparing Four Balanced Groups

Suppose a hypothetical experiment compares the mean response, measured in points, for four treatments. There are $n=6$ observations per treatment. The one-way ANOVA gives $MSE=9.00$ points squared and $df=4(6-1)=20$. Assume the observations are independent, the group errors are reasonably normal, and their variances are equal. At familywise level $\alpha=0.05$, a studentized-range table or software gives $q_{0.95;4,20}\approx3.96$.

First calculate the common HSD cutoff:

HSD=3.96\sqrt{\frac{9.00}{6}} =3.96\sqrt{1.50} \approx3.96(1.225) \approx4.85\text{ points}.

Suppose the treatment means are $72.1$, $75.0$, $78.2$, and $81.0$ points. The six absolute pairwise differences are: $2.9$ for groups 1 and 2, $6.1$ for groups 1 and 3, $8.9$ for groups 1 and 4, $3.2$ for groups 2 and 3, $6.0$ for groups 2 and 4, and $2.8$ for groups 3 and 4.

The differences $6.1$, $8.9$, and $6.0$ exceed $4.85$. Tukey HSD therefore identifies groups 1 and 3, groups 1 and 4, and groups 2 and 4 as significantly different at the 5% familywise level. The other three pairs are not declared significantly different by this procedure. These conclusions concern differences in mean response in points, under the model assumptions.

Worked Example: Using the Error Degrees of Freedom

A hypothetical balanced study compares three production methods using a response measured in millimetres. Each method has $n=8$ independent observations. The pooled ANOVA error mean square is $MSE=16.0$ square millimetres. Assume independent, approximately normal errors with a common variance. For $\alpha=0.05$, use $df=3(8-1)=21$ and the table value $q_{0.95;3,21}\approx3.56$.

The cutoff is

HSD=3.56\sqrt{\frac{16.0}{8}} =3.56\sqrt{2} \approx3.56(1.414) \approx5.03\text{ mm}.

Suppose the sample means are $14.2$, $18.0$, and $20.1$ mm. Their absolute differences are $18.0-14.2=3.8$ mm, $20.1-18.0=2.1$ mm, and $20.1-14.2=5.9$ mm. Only $5.9$ mm exceeds the $5.03$ mm cutoff. The method therefore identifies the first and third group means as significantly different; it does not identify either of the other two pairs as different at the specified familywise level.

Worked Example: A Larger Family of Comparisons

A hypothetical balanced study compares five settings, with $n=4$ independent observations at each setting. The measured outcome is in seconds. Suppose $MSE=25.0$ seconds squared, and assume the error distributions are reasonably normal with a shared variance and independent observations. The error degrees of freedom are $5(4-1)=15$. For familywise $\alpha=0.05$, suppose a studentized-range table gives $q_{0.95;5,15}\approx4.37$.

Then

HSD=4.37\sqrt{\frac{25.0}{4}} =4.37(2.50) =10.925\text{ seconds} \approx10.9\text{ seconds}.

Suppose the five sample means, in order, are $102$, $109$, $113$, $121$, and $127$ seconds. The pairwise differences exceeding $10.925$ seconds are: $113-102=11$, $121-102=19$, $127-102=25$, $121-109=12$, $127-109=18$, and $127-113=14$. Those six pairs are declared significantly different. The remaining four differences are $7$, $4$, $8$, and $6$ seconds, all below the cutoff. The procedure assesses all ten pairs using one familywise adjustment; it does not treat the ten comparisons as ten unrelated tests.

Interpreting the Results and Avoiding Mistakes

Tukey HSD is useful when the scientific question calls for every pairwise comparison and the design has equal group sizes. The pooled $MSE$ makes use of the within-group variability estimated across the full ANOVA, while the studentized-range critical value accounts for examining multiple pairs. A significant omnibus ANOVA is often a reason to investigate which groups differ, but the HSD calculation itself uses its own familywise cutoff for the set of pairwise tests.

A common mistake is to use the range of the observed sample means as though it were a pure measure of sampling error. In general, $\max_i\bar{x}_i-\min_i\bar{x}_i$ reflects both real differences among the population means and estimation error. The studentized-range probability statement applies to the range of the mean errors, $\max_i(\bar{x}_i-\mu_i)-\min_i(\bar{x}_i-\mu_i)$, not to the observed range under arbitrary unequal population means. The error-range argument is what justifies familywise control even when only some pairwise null hypotheses are true.

Do not compare each pair to an unadjusted cutoff. That ignores the family of tests. Use the common Tukey HSD cutoff for every pair.
Do not omit the group count or error degrees of freedom. Both determine the studentized-range critical value.
Do not interpret a nonsignificant pair as proof of equality. It means the observed difference did not exceed this procedure’s cutoff.
Do not use this balanced-design formula when sample sizes differ. The equal-size cutoff here assumes the same $n$ in every group.

Key takeaway: For $k$ equally sized groups, Tukey HSD compares every absolute difference in sample means with $q_{1-\alpha;k,df}\sqrt{MSE/n}$. Under the independent, normal, common-variance one-way ANOVA model, the studentized range of the mean errors ensures that the probability of one or more false pairwise rejections is no greater than $\alpha$.

Check Your Understanding

Use the model assumptions and the balanced-design rule to answer these questions.

Four groups each have $n=7$ observations. What are the residual degrees of freedom for the one-way ANOVA error mean square?
In the HSD formula, what determines the studentized-range critical value, and what determines the common cutoff’s units?
Why does the range of the observed sample means not generally have the studentized-range distribution when population means differ?
Suppose one true pairwise null is falsely rejected. Explain why the full range of the group-mean errors must exceed the HSD cutoff.
A pairwise difference is smaller than the HSD cutoff. What can be concluded, and what cannot be concluded, about that pair of population means?

What You'll Learn