Tukey Intervals for All Pairwise Mean Differences

Why Pairwise Comparisons Need a Joint Critical Value

When several population means are compared, separate confidence intervals do not automatically give a confidence statement for all the differences together. The familywise target is that every pairwise interval covers its corresponding difference at once. In the balanced one-way normal model, the dependence among the pairwise differences has a particularly useful form: their largest absolute estimation error is the range of the group-mean errors. Studentizing that range produces Tukey’s critical value.

The result is exact under a specific model, not a general-purpose adjustment for any collection of means. In particular, balance, normality, independence, and a common variance are essential to the derivation below.

Model and Studentized-Range Distribution

Definition: In the balanced one-way normal model, there are $a\geq 2$ groups, each with $n\geq 2$ independent observations: $$ Y_{ij}=\mu_i+\varepsilon_{ij},\qquad \varepsilon_{ij}\ \text{independent }N(0,\sigma^2), \qquad i=1,\ldots,a,\quad j=1,\ldots,n. $$ The group means $\mu_i$ are fixed, and the error variance $\sigma^2$ is common to all groups. Write $\bar Y_i=n^{-1}\sum_{j=1}^nY_{ij}$, $\nu=a(n-1)$, and $$ \operatorname{MSE}=\frac{1}{\nu}\sum_{i=1}^a\sum_{j=1}^n(Y_{ij}-\bar Y_i)^2. $$

The studentized-range distribution combines the range of independent standard normal variables with an independent estimate of their scale.

Definition: Let $Z_1,\ldots,Z_a$ be independent $N(0,1)$ variables, and let $W$ be independent of them with a chi-square distribution on $\nu$ degrees of freedom. The studentized range is $$ Q_{a,\nu}= \frac{\max_i Z_i-\min_i Z_i}{\sqrt{W/\nu}}. $$ Write $q_{1-\alpha;a,\nu}$ for its $1-\alpha$ quantile, so that $P(Q_{a,\nu}\leq q_{1-\alpha;a,\nu})=1-\alpha$.

The denominator is a single common estimate of scale. That feature is what lets one cutoff serve all pairwise comparisons in the balanced model.

Deriving the Tukey Intervals

For each pair $i<j$, the target is $\mu_i-\mu_j$, estimated by $\bar Y_i-\bar Y_j$. Tukey’s interval uses the same half-width for every pair:

Theorem: Under the balanced one-way normal model, for $0<\alpha<1$, the intervals $$ (\bar Y_i-\bar Y_j)\ \pm\ q_{1-\alpha;a,\nu}\sqrt{\frac{\operatorname{MSE}}{n}}, \qquad 1\leq i<j\leq a, $$ have exact simultaneous coverage $1-\alpha$: $$ P\left( \mu_i-\mu_j\text{ lies in its interval for every }i<j \right)=1-\alpha. $$

The proof has two ingredients: the centered group-mean errors are independent normal variables, and the pooled residual variance estimate is independent of all group means.

Standardize the group-mean errors.
Set $Z_i=(\bar Y_i-\mu_i)/(\sigma/\sqrt n)$. Independence of groups and the normal model imply that the $Z_i$ are independent $N(0,1)$ variables.

Identify the independent variance estimate.
Within each group, the sample mean and the residual vector are orthogonal linear transformations of a jointly normal observation vector. They are therefore independent. Across groups the observation vectors are independent, so $\operatorname{MSE}$ is independent of all the $\bar Y_i$. The pooled residual sum of squares divided by $\sigma^2$ has a chi-square distribution with $\nu=a(n-1)$ degrees of freedom. Thus $W=\nu\operatorname{MSE}/\sigma^2$ is independent of the $Z_i$ and has that chi-square distribution.

Express all errors through one range.
For any pair, the estimation error is $(\bar Y_i-\bar Y_j)-(\mu_i-\mu_j)=(\sigma/\sqrt n)(Z_i-Z_j)$. The largest absolute pairwise error is therefore $(\sigma/\sqrt n)(\max_i Z_i-\min_i Z_i)$.

Studentize and apply the quantile.
Since $\sqrt{\operatorname{MSE}/n}=(\sigma/\sqrt n)\sqrt{W/\nu}$, the largest absolute pairwise error divided by this common standard-error factor has distribution $Q_{a,\nu}$. It is at most $q_{1-\alpha;a,\nu}$ with probability $1-\alpha$, which is exactly the event that every stated interval covers. $\square$

This proof also explains why the degrees of freedom are $\nu=a(n-1)$: they come from the within-group residual sums of squares used in $\operatorname{MSE}$, not from the number of pairwise comparisons.

Worked Intervals and Their Interpretation

Worked Example: Three Groups With a Shared Half-Width

Suppose $a=3$, $n=9$, the observed group means are $12$, $15$, and $20$, and $\operatorname{MSE}=16$. For a 95% simultaneous confidence level, $\nu=3(9-1)=24$. The studentized-range critical value is $q_{0.95;3,24}$, approximately $3.53$ (the precise value depends on rounding in the table or software used).

The common standard-error factor and half-width are $$ \sqrt{\frac{\operatorname{MSE}}{n}}=\sqrt{\frac{16}{9}}=\frac{4}{3}, \qquad h=3.53\left(\frac{4}{3}\right)\approx4.71. $$

For groups 2 and 1, the estimated difference is $15-12=3$, giving $3\pm4.71=[-1.71,7.71]$. For groups 3 and 2, it is $20-15=5$, giving $5\pm4.71=[0.29,9.71]$. For groups 3 and 1, it is $20-12=8$, giving $8\pm4.71=[3.29,12.71]$.

These three intervals have 95% simultaneous coverage under the model: the probability that all three contain their respective population differences is 0.95. The statement is joint; it does not claim that each interval has an individual 95% coverage probability.

Worked Example: The Corresponding Familywise Test

Use the same model and data. Consider the pairwise null hypotheses $H_{ij}:\mu_i=\mu_j$, with two-sided alternatives. The interval for a pair excludes zero precisely when the absolute estimated difference exceeds the common half-width.

Here the threshold is approximately $4.71$. The absolute differences are $|15-12|=3$, $|20-15|=5$, and $|20-12|=8$. Thus the interval for groups 2 and 1 includes zero, while those for groups 3 and 2 and for groups 3 and 1 exclude zero.

Rejecting exactly when the corresponding Tukey interval excludes zero gives a familywise procedure: under the model, the probability of one or more false rejections among the pairwise hypotheses is at most $0.05$, whatever the values of the group means. This follows because simultaneous coverage of every true difference implies that none of the true zero differences is excluded. The procedure is not an assertion that all three null hypotheses are simultaneously true or false.

Why the Raw Range Is Not the Pivot

A critical distinction is between the range of observed group means and the range of their centered errors. The raw range $\max_i\bar Y_i-\min_i\bar Y_i$ reflects both sampling noise and actual differences among the $\mu_i$. It does not generally have a central studentized-range distribution.

Worked Example: Unequal True Means Shift the Raw Range

Take two groups with $n=4$, $\sigma^2=4$, $\mu_1=0$, and $\mu_2=10$. The sample-mean difference satisfies $$ \bar Y_2-\bar Y_1\sim N\left(10,\frac{4}{4}+\frac{4}{4}\right)=N(10,2). $$

With two groups, the raw range of the sample means is $|\bar Y_2-\bar Y_1|$. This is the absolute value of a normal variable centered at 10, not at zero. In contrast, the centered errors satisfy $$ (\bar Y_2-\mu_2)-(\bar Y_1-\mu_1)\sim N(0,2). $$ The studentized-range pivot is built from the range of these centered errors, divided by the independent pooled scale estimate. Substituting the raw sample-mean range would wrongly leave the true separation of 10 in the numerator and would not produce the stated central pivot.

In the coverage proof, the relevant identity is the maximum absolute pairwise estimation error: $$ \max_{i<j}\left|(\bar Y_i-\bar Y_j)-(\mu_i-\mu_j)\right| =\max_i(\bar Y_i-\mu_i)-\min_i(\bar Y_i-\mu_i). $$ It is not an identity between that error range and the raw range of $\bar Y_i$ when the means differ.

Assumptions, Scope, and Common Pitfalls

Exactness relies on the model conditions used in the derivation: independent normal observations, a common variance, equal group sample sizes, and at least two observations per group so that the pooled residual variance can be estimated. The means are treated as fixed parameters. If these conditions do not hold, the distributional argument above does not establish exact coverage. In particular, unequal sample sizes change the standard errors of the pairwise differences, so the single common factor and range pivot derived here no longer apply as written.

Wrong: “The observed range of the sample means, divided by the estimated standard error, has a central studentized-range distribution.” This ignores the true mean differences. Correction: The pivot uses the range of the centered errors $\bar Y_i-\mu_i$, whose standardized values are independent standard normals.
Wrong: “There are $a(a-1)/2$ intervals, so the studentized-range degrees of freedom equal that number.” Correction: The degrees of freedom are $\nu=a(n-1)$, from the pooled within-group residual variation.
Wrong: “Each interval is an ordinary $t$ interval with a 95% individual level.” Correction: Tukey’s critical value is chosen for the joint range and gives the stated simultaneous familywise coverage.

The practical value of Tukey’s method is that it targets precisely the family of all pairwise mean differences. Its common critical value accounts for the dependence among those comparisons through their shared group-mean errors, rather than treating each comparison as unrelated.

Check Your Understanding

Use the model conditions and the centered-error argument to answer these questions.

In the balanced one-way normal model, what are the degrees of freedom used in the studentized-range critical value, and where do they come from?
Why does the maximum absolute pairwise estimation error equal the range of the centered group-mean errors?
Which random quantity has the studentized-range distribution: the raw range of observed group means or the studentized range of centered errors?
State the conditions needed for the exact simultaneous coverage result.
How does excluding zero from a Tukey interval relate to testing the corresponding equality of two population means?

===