Deriving Tukey's Simultaneous Coverage

Why a Range Controls All Pairwise Comparisons

When several group means are compared at once, it is not enough to choose a method that gives the desired error rate for each comparison separately. The goal of Tukey’s procedure is stronger: to control the probability that any of the pairwise intervals misses its true mean difference. For equal-size groups with normal observations and a common variance, that probability can be described exactly using the studentized range.

The key fact is that the largest absolute difference among a set of numbers is its range: the maximum minus the minimum. Consequently, the simultaneous coverage problem for all pairwise mean differences reduces to the distribution of the range of standardized sample means, with the shared pooled error estimate in the denominator. This result explains why Tukey’s procedure uses a studentized-range critical value rather than an ordinary t critical value.

Model and assumptions: There are $k$ independent groups, each with the same sample size $n$. For group $i$, the observations $X_{i1},\ldots,X_{in}$ are independent and normally distributed with mean $\mu_i$ and variance $\sigma^2$. The common variance $\sigma^2$ is unknown. Observations are independent both within and between groups. The pooled mean square error from the one-way ANOVA is denoted by $MSE$, with $\nu=k(n-1)$ error degrees of freedom. The exact result below relies on all of these assumptions, including equal group sizes.

From Pairwise Errors to a Studentized Range

Write $\bar X_i$ for the sample mean in group $i$. For any two different groups $i$ and $j$, the estimated difference is $\bar X_i-\bar X_j$, and its target is $\mu_i-\mu_j$. Tukey’s procedure considers all $k(k-1)/2$ distinct pairs together.

Definition: Let $Z_1,\ldots,Z_k$ be independent standard normal random variables, and let $W$ be an independent chi-square random variable with $\nu$ degrees of freedom. The studentized range statistic is $$ Q=\frac{\max_i Z_i-\min_i Z_i}{\sqrt{W/\nu}}. $$ The $1-\alpha$ quantile of this distribution is written $q_{1-\alpha;k,\nu}$. Its distribution depends on the number of groups $k$ and the error degrees of freedom $\nu$.

This definition is useful because the same form arises from the sample means and the pooled error estimate. Define $$ Z_i=\frac{\sqrt n(\bar X_i-\mu_i)}{\sigma}. $$ Under the stated model, the $Z_i$ are independent standard normal random variables. Also, the pooled error sum of squares divided by $\sigma^2$ has a chi-square distribution with $\nu=k(n-1)$ degrees of freedom. Thus $W=\nu MSE/\sigma^2$ has the required chi-square distribution and is independent of the sample means.

\frac{\max_{i<j}\left|(\bar X_i-\bar X_j)-(\mu_i-\mu_j)\right|} {\sqrt{MSE/n}} = \frac{\max_i Z_i-\min_i Z_i}{\sqrt{W/\nu}} \sim Q.

The numerator on the left is the largest absolute estimation error over all pairs. To see why it becomes a range, put $e_i=\bar X_i-\mu_i$. Then $$ \max_{i<j}|e_i-e_j|=\max_i e_i-\min_i e_i. $$ Indeed, no difference between two values can exceed the maximum minus the minimum, and the difference between an index attaining the maximum and one attaining the minimum equals that range. Standardizing each $e_i$ gives the $Z_i$; the common denominator becomes $\sqrt{W/\nu}$. This establishes the displayed distributional identity.

Theorem: Under the balanced normal model, simultaneous $1-\alpha$ confidence intervals for every pairwise mean difference are $$ (\bar X_i-\bar X_j)\ \pm\ q_{1-\alpha;k,\nu}\sqrt{\frac{MSE}{n}}, \qquad 1\le i<j\le k,\quad \nu=k(n-1). $$ The probability that all these intervals cover their corresponding true differences is exactly $1-\alpha$.

The coverage statement follows directly from the range result. All intervals cover precisely when every pairwise estimation error is at most $q_{1-\alpha;k,\nu}\sqrt{MSE/n}$ in absolute value. That event is equivalent to $Q\le q_{1-\alpha;k,\nu}$, which has probability $1-\alpha$ by the definition of the quantile. Therefore the probability that at least one interval fails to cover is $\alpha$. This is familywise error control for the full set of pairwise comparisons, not merely a separate guarantee for each interval.

Constructing the Simultaneous Intervals

Once the model and degrees of freedom are established, the calculation has a consistent sequence: find the pooled error estimate, determine the studentized-range critical value for the specified $k$ and $\nu$, calculate the common half-width, and apply it to each estimated difference. The same half-width is used for every pair because all groups have equal sample size.

Set the model dimensions.
Record the number of groups $k$, common group size $n$, and error degrees of freedom $\nu=k(n-1)$.

Find the familywise critical value.
Use $q_{1-\alpha;k,\nu}$, not a t critical value for a single comparison.

Calculate the common half-width.
Multiply the range critical value by $\sqrt{MSE/n}$.

Apply it to every pair.
For each pair, add and subtract the half-width from $\bar X_i-\bar X_j$.

Worked Example: Simultaneous Intervals for Three Balanced Groups

A hypothetical study compares a continuous response in three independent groups. Each group has $n=5$ normal observations, with a common but unknown variance. The sample means are $12.4$, $10.9$, and $11.7$, and the ANOVA pooled $MSE$ is $7.5$. Construct 95% simultaneous intervals for all three pairwise mean differences. For $k=3$ and $\nu=12$, use the table value $q_{0.95;3,12}=3.773$.

Method and degrees of freedom. The equal group size is $5$, so the error degrees of freedom are $$ \nu=k(n-1)=3(5-1)=12. $$ The common standard-error factor is $$ \sqrt{\frac{MSE}{n}}=\sqrt{\frac{7.5}{5}}=\sqrt{1.5}=1.224744871\ldots. $$

Common half-width. Multiply by the studentized-range critical value: $$ 3.773\sqrt{1.5}=3.773(1.224744871\ldots)=4.620962\ldots. $$ The half-width is $4.621$, rounded to three decimal places.

Pairwise estimates and intervals. The three estimated differences are $12.4-10.9=1.5$, $12.4-11.7=0.7$, and $10.9-11.7=-0.8$. Applying the same half-width gives: $$ \begin{aligned} \mu_1-\mu_2 &: 1.5\pm4.621=(-3.121,\ 6.121),\\ \mu_1-\mu_3 &: 0.7\pm4.621=(-3.921,\ 5.321),\\ \mu_2-\mu_3 &: -0.8\pm4.621=(-5.421,\ 3.821). \end{aligned} $$

Each interval is part of one family of intervals with 95% simultaneous coverage under the model. The intervals all contain zero, so none of the three pairwise differences is distinguished from zero by these intervals. Check: the largest observed sample-mean difference is $12.4-10.9=1.5$, which is less than the common half-width $4.621$; all three intervals must therefore contain zero. The method avoids treating the three comparisons as if each had its own independent 95% guarantee.

Worked Example: Finding the Pair That Determines the Range

A balanced experiment has three groups of five observations each. Their sample means, in order, are $1.4$, $0.1$, and $-0.8$, and $MSE=7.5$. At the 5% familywise level, use $q_{0.95;3,12}=3.773$. Identify the pair with the largest absolute observed difference and determine whether any pairwise interval excludes zero.

Method. Compute all three differences, then use the largest absolute difference to obtain the observed studentized range. The degrees of freedom are $3(5-1)=12$, and the common standard-error factor is $\sqrt{7.5/5}=\sqrt{1.5}=1.224744871\ldots$.

The differences are $$ 1.4-0.1=1.3,\qquad 1.4-(-0.8)=2.2,\qquad 0.1-(-0.8)=0.9. $$ In particular, $0.1-(-0.8)=0.1+0.8=0.9$, not $0.7$. The largest absolute difference is $2.2$, between the first and third groups. Thus the observed studentized range is $$ \frac{2.2}{\sqrt{1.5}}=\frac{2.2}{1.224744871\ldots}=1.796292\ldots, $$ or $1.796$ to three decimal places. Since $1.796<3.773$, the simultaneous procedure does not reject any pairwise equality at the 5% familywise level.

For a direct interval check, the half-width is again $3.773\sqrt{1.5}=4.621$, rounded to three decimals. The intervals are $1.3\pm4.621=(-3.321,5.921)$, $2.2\pm4.621=(-2.421,6.821)$, and $0.9\pm4.621=(-3.721,5.521)$. Each contains zero. Check: the range of the three means is $1.4-(-0.8)=2.2$, matching the largest absolute pairwise difference used in the statistic. The method avoids both a sign error and overlooking a pair when identifying the maximum.

Worked Example: Applying the Familywise Critical Value to Four Groups

A hypothetical assay is measured on four independent groups, with six observations per group. Assume normal observations and a common variance. The sample means are $18.0$, $15.5$, $16.4$, and $14.8$, and $MSE=9$. Construct the simultaneous 95% interval for the most widely separated pair and assess whether any pair is significant. Use $q_{0.95;4,20}=3.958$.

Method and scale. Here $k=4$, $n=6$, and $$ \nu=4(6-1)=20,\qquad \sqrt{\frac{MSE}{n}}=\sqrt{\frac{9}{6}}=\sqrt{1.5}=1.224744871\ldots. $$ The common half-width is $$ 3.958\sqrt{1.5}=3.958(1.224744871\ldots)=4.847528\ldots, $$ which rounds to $4.848$ to three decimals.

The largest sample mean is $18.0$ and the smallest is $14.8$. Their difference is $3.2$, so this is the pair that sets the range. Its simultaneous interval is $$ (18.0-14.8)\pm4.848=3.2\pm4.848=(-1.648,\ 8.048). $$ Because this interval contains zero, the pair is not significant. Every other pairwise absolute difference is at most $3.2$, so its interval also contains zero when the same half-width is applied. Check: the observed studentized range is $3.2/\sqrt{1.5}=2.612$ to three decimals, below the supplied critical value $3.958$. The conclusion agrees with the interval calculation. This example illustrates why the maximum and minimum group means suffice to identify the most extreme comparison, even though all six pairs are included in the family.

Why an Ordinary t Critical Value Is Not Enough

A common mistake is to construct each pairwise interval with a two-sided t critical value and then describe the entire collection as having the same confidence level. A t interval for one prespecified pair controls that pair’s error probability. It does not, by itself, control the probability of at least one failure across all pairs. Tukey’s critical value is calibrated to the maximum over the full set, which is why it can provide the simultaneous guarantee.

For example, in the three-group setting of the first worked example, a two-sided 95% t critical value for 12 degrees of freedom is about $2.179$. Using it would give a half-width of $$ 2.179\sqrt{1.5}=2.669 $$ to three decimal places. The Tukey half-width is $4.621$, using $q_{0.95;3,12}=3.773$. The smaller t-based half-width is appropriate for a single 95% comparison under the model, but substituting it for the studentized-range cutoff does not establish 95% simultaneous coverage of all three comparisons. The correction is to use the critical value for the number of groups and the ANOVA error degrees of freedom.

The result also has a clear boundary. The derivation assumes equal sample sizes, normality, independent observations, and a common variance. If group sizes differ, the common standard-error factor and the exact balanced studentized-range identity above no longer apply in this form. The equal-size result should not be carried over by simply using one group’s sample size for every pair.

Key takeaway: In a balanced normal one-way model, the maximum absolute pairwise estimation error is the range of the standardized group-mean errors. Studentizing that range by the independent pooled variance estimate gives $Q$. Its quantile therefore yields simultaneous intervals whose joint coverage is exactly the stated confidence level.

Check Your Understanding

For each problem, use the balanced normal model where it is specified. Answers are included so that the calculations can be checked.

Three groups each have $n=4$ observations, with sample means $7.2$, $6.5$, and $8.1$, and $MSE=4$. Find the error degrees of freedom, the largest absolute sample-mean difference, and its observed studentized range. (Answer: $\nu=9$; largest difference $=8.1-6.5=1.6$; $Q_{\text{obs}}=1.6/\sqrt{4/4}=1.600$.)
Four groups each have $n=5$, and $MSE=5$. What is $\nu$, and what is the common standard-error factor $\sqrt{MSE/n}$? (Answer: $\nu=4(5-1)=16$; $\sqrt{5/5}=1.000$.)
For three balanced groups, suppose $n=4$, $MSE=16$, and the supplied studentized-range critical value is $q_{0.95;3,9}=3.95$. Find the simultaneous interval for an estimated pairwise difference of $-2.5$. (Answer: half-width $=3.95\sqrt{16/4}=7.90$; interval $-2.5\pm7.90=(-10.40,\ 5.40)$.)
In a balanced normal experiment, all pairwise intervals use $q_{0.99;k,\nu}\sqrt{MSE/n}$. State their joint coverage and the probability that at least one interval fails to cover. (Answer: joint coverage $=0.99$; familywise noncoverage probability $=0.01$, under the stated model assumptions.)
Four groups have different sample sizes. Can the exact balanced studentized-range derivation in this tutorial be applied by using one common $\sqrt{MSE/n}$ factor for every pair? Explain briefly. (Answer: No. There is no single common group size $n$, so the equal-size standard-error factor and the stated exact identity do not apply in that form.)

What You'll Learn