Tukey-Kramer Comparisons for Unequal Groups

Why Unequal Group Sizes Need Pair-Specific Comparisons

After a one-way analysis of variance, it is often useful to identify which population means differ. When every group has the same sample size, Tukey’s HSD uses one standard error for every pair. With unequal group sizes, the precision of a difference depends on the sizes of the two groups being compared. Tukey-Kramer retains the simultaneous-comparison logic of Tukey’s procedure while adjusting the standard error for each pair.

The method uses the ANOVA pooled error estimate, rather than estimating a separate variance for each pair. This pools information across groups and adjusts for the fact that many pairwise comparisons are being made. It is aligned with the SOA VEE Mathematical Statistics syllabus.

Model and assumptions: For \(k\) independent groups, assume observations follow the one-way fixed-effects model \(Y_{ij}=\mu_i+\varepsilon_{ij}\), where group \(i\) has \(n_i\) observations. The errors are independent and normally distributed with mean zero and a common variance \(\sigma^2\). Thus, group \(i\) has population mean \(\mu_i\), and all groups share the same within-group variance. The method does not have its stated ANOVA-based justification if observations are dependent, errors are substantially nonnormal in small samples, or group variances are unequal.

The Tukey-Kramer Interval

Let \(N=\sum_{i=1}^k n_i\) be the total sample size, and let \(SSE\) be the within-group error sum of squares from the one-way ANOVA. The error degrees of freedom and pooled mean square error are

\nu=N-k,\qquad MSE=\frac{SSE}{N-k}.

For groups \(i\) and \(j\), the estimated mean difference is \(\bar{Y}_i-\bar{Y}_j\). The Tukey-Kramer simultaneous confidence interval for that difference, with familywise confidence level \(1-\alpha\), is

(\bar{Y}_i-\bar{Y}_j)\ \pm\ q_{1-\alpha;k,\nu} \sqrt{\frac{MSE}{2}\left(\frac{1}{n_i}+\frac{1}{n_j}\right)}.

Here \(q_{1-\alpha;k,\nu}\) is the \(1-\alpha\) quantile of the studentized-range distribution for \(k\) groups and \(\nu\) error degrees of freedom. In a calculation, obtain it from an appropriate table or software, or use the value supplied in the problem. The expression under the square root is the estimated variance of the pairwise difference divided by two; the studentized-range critical value incorporates the simultaneous comparison adjustment.

Interpretation: If a simultaneous interval for \(\mu_i-\mu_j\) excludes zero, the Tukey-Kramer procedure identifies a difference between those two population means at the familywise level \(\alpha\). If the interval contains zero, it does not identify that pair as different. A positive interval location indicates that the first mean exceeds the second; reversing the order reverses the signs but not the conclusion.

Why the Pair-Specific Standard Error Has This Form

The pooled-variance model explains both the role of \(MSE\) and the need to use the sizes of the two groups in each comparison. For independent observations, the variance of a sample mean in group \(i\) is \(\sigma^2/n_i\). The two group means are independent, so the variance of their difference is the sum of those variances.

\operatorname{Var}(\bar{Y}_i-\bar{Y}_j) =\operatorname{Var}(\bar{Y}_i)+\operatorname{Var}(\bar{Y}_j) =\sigma^2\left(\frac{1}{n_i}+\frac{1}{n_j}\right).

The common variance \(\sigma^2\) is unknown, and the one-way ANOVA estimates it with \(MSE\), which has \(\nu=N-k\) degrees of freedom under the model. Thus, the estimated standard error for the difference is \(\sqrt{MSE(1/n_i+1/n_j)}\). Tukey-Kramer multiplies that standard error by \(q_{1-\alpha;k,\nu}/\sqrt{2}\), giving the interval formula above. The factor involving \(n_i\) and \(n_j\) changes from pair to pair when group sizes are unequal; substituting one common sample size would therefore misstate the uncertainty.

For equal group sizes \(n_i=n_j=n\), the standard error in the interval becomes \(\sqrt{MSE/n}\), matching the Tukey HSD form for balanced comparisons. For unequal sizes, Tukey-Kramer provides simultaneous pairwise intervals with familywise error controlled at no more than the nominal level under the stated model; it is exact in the balanced case and can be conservative in unbalanced cases.

Worked Comparisons

Worked Example: One Pair From an Unbalanced ANOVA

A hypothetical study compares mean response in three independent groups. The group sizes are 4, 7, and 9, and the within-group error sum of squares is \(SSE=120\). The sample means for the groups of sizes 4 and 9 are 28 and 38. At the 5% familywise level, use \(q_{0.95;3,17}=3.60\) to form the Tukey-Kramer interval for the difference between those two population means.

Method and degrees of freedom. The total sample size is \(N=4+7+9=20\), and the number of groups is \(k=3\). Therefore, the error degrees of freedom are \(\nu=N-k=20-3=17\). The pooled error estimate is

MSE=\frac{120}{17}=7.0588235\ldots.

Pair-specific standard error. The comparison uses the groups of sizes 4 and 9, so

\frac{MSE}{2}\left(\frac{1}{4}+\frac{1}{9}\right) =\frac{120}{17}\cdot\frac{1}{2}\cdot\frac{13}{36} =\frac{65}{51} =1.2745098\ldots.

The standard error is \(\sqrt{65/51}=1.12894\ldots\). The estimated difference, in the stated order, is \(28-38=-10\). The margin of error is \(3.60(1.12894\ldots)=4.0642\ldots\). Thus, the interval is \(-10\pm4.0642\ldots\), or approximately \((-14.06,-5.94)\), rounded to two decimals. Equivalently, the interval for the reverse difference, mean of the size-9 group minus mean of the size-4 group, is \((5.94,14.06)\). Zero is excluded, so this pair is identified as different at the stated familywise level.

Check and common mistake. The pooled error degrees of freedom use all three groups, so they are 17, not the size of the two groups being compared minus two. Also, the pair-specific factor uses sizes 4 and 9; the group of size 7 contributes to \(MSE\) but not directly to this pair’s standard error.

Worked Example: Forming All Pairwise Intervals

A hypothetical materials study records measurements in three independent groups. Their sample sizes are \(5,8,12\), their sample means are \(20,23,31\), and the ANOVA error sum of squares is \(132\). Use \(q_{0.95;3,22}=3.50\) to make all pairwise Tukey-Kramer comparisons.

Pool the error. Here \(N=5+8+12=25\), so \(\nu=25-3=22\), and \(MSE=132/22=6\). For each pair, use the same \(q\) and \(MSE\), but that pair’s two sample sizes.

Groups 1 and 2. The estimated difference is \(20-23=-3\). The standard error is \(\sqrt{(6/2)(1/5+1/8)}=\sqrt{0.975}=0.98742\). The margin is \(3.50(0.98742)=3.45597\). The interval is \(-3\pm3.45597=(-6.46,0.46)\), rounded to two decimals. It contains zero.

Groups 1 and 3. The estimated difference is \(20-31=-11\). The standard error is \(\sqrt{(6/2)(1/5+1/12)}=\sqrt{0.85}=0.92195\). The margin is \(3.50(0.92195)=3.22684\). The interval is \(-11\pm3.22684=(-14.23,-7.77)\), rounded to two decimals. It excludes zero.

Groups 2 and 3. The estimated difference is \(23-31=-8\). The standard error is \(\sqrt{(6/2)(1/8+1/12)}=\sqrt{0.625}=0.79057\). The margin is \(3.50(0.79057)=2.76699\). The interval is \(-8\pm2.76699=(-10.77,-5.23)\), rounded to two decimals. It excludes zero.

The simultaneous comparisons identify differences for groups 1 and 3, and for groups 2 and 3, but not for groups 1 and 2. The conclusion is based on the adjusted intervals, not on three separate unadjusted tests. As a check, the comparisons involving the largest observed difference have intervals far from zero, while the estimated difference of 3 is small relative to its adjusted margin.

Worked Example: How the Smallest Group Affects Precision

A hypothetical evaluation compares four group means: 42, 46, 49, and 55. The respective sample sizes are 4, 6, 8, and 10. The pooled error sum of squares is 252, and use \(q_{0.95;4,24}=3.90\). Calculate and interpret the simultaneous interval for the difference between the largest and smallest sample means.

Find the pooled quantities. The total size is \(N=4+6+8+10=28\). Therefore, \(\nu=28-4=24\), and \(MSE=252/24=10.5\). The requested difference, largest mean minus smallest mean, is \(55-42=13\), comparing groups of sizes 10 and 4.

Calculate the pair-specific margin. For these groups,

SE=\sqrt{\frac{10.5}{2}\left(\frac{1}{10}+\frac{1}{4}\right)} =\sqrt{5.25(0.35)} =\sqrt{1.8375} =1.35555\ldots.

The margin is \(3.90(1.35555\ldots)=5.2866\ldots\). The simultaneous interval is \(13\pm5.2866\ldots=(7.71,18.29)\), rounded to two decimals. Since zero is not in the interval, the procedure identifies a difference between these population means at the 5% familywise level.

Check and common mistake. The smaller group makes this pair less precise than a comparison of two larger groups would be. Using only \(n=10\), or replacing both group sizes by their average, would produce an incorrect standard error. The correct factor is specifically \(\tfrac12(1/10+1/4)=0.175\).

A Reliable Calculation Sequence

For a full set of comparisons, apply the same sequence to each unordered pair. There are \(k(k-1)/2\) such pairs. Keeping the sample-size factor visible helps prevent accidental use of the wrong standard error.

Check the model setup.
Identify the independent groups and confirm that the common-variance one-way ANOVA model is the intended one.

Calculate pooled error quantities.
Find \(N\), then \(\nu=N-k\), and calculate \(MSE=SSE/\nu\).

Obtain the studentized-range critical value.
Use the specified familywise level, number of groups \(k\), and error degrees of freedom \(\nu\) to obtain \(q_{1-\alpha;k,\nu}\).

Calculate each pair separately.
For each \(i,j\), use \(\sqrt{(MSE/2)(1/n_i+1/n_j)}\) and the signed difference \(\bar{Y}_i-\bar{Y}_j\).

Interpret intervals in context.
Report which intervals exclude zero, preserving the direction of the difference and the familywise confidence level.

Common Exam Traps

Using the wrong degrees of freedom. The pooled ANOVA error degrees of freedom are \(N-k\), where \(N\) includes every group, not just the pair under consideration.
Using the same standard error for every pair. With unequal group sizes, calculate \(\frac12(1/n_i+1/n_j)\) separately for each pair.
Dropping the factor of one-half. The Tukey-Kramer margin uses \(q\sqrt{(MSE/2)(1/n_i+1/n_j)}\). Omitting the one-half makes the interval too wide; moving it outside the square root also changes the result.
Using a t critical value instead of a studentized-range value. The \(q\) critical value accounts for the family of pairwise comparisons, whereas an ordinary t interval for one contrast does not provide that same adjustment.
Reversing the conclusion when reversing the subtraction. Changing \(\bar{Y}_i-\bar{Y}_j\) to \(\bar{Y}_j-\bar{Y}_i\) changes the signs of both interval endpoints, not whether zero is included.

Tukey-Kramer is useful when all pairwise mean comparisons are of interest and group sizes differ. Its central practical feature is simple: the ANOVA provides one pooled estimate of error, while each pair receives a standard error that reflects the sizes of precisely the two groups being compared. Keep those two roles separate, and check every interval against zero to reach the adjusted conclusion.

Check Your Understanding

For each problem, use the stated critical value and round interval endpoints to two decimals.

Two groups have sizes 5 and 10, sample means 18 and 24, \(MSE=8\), and \(q=3.20\). Find the Tukey-Kramer interval for the first mean minus the second. (Answer: \((-9.51,-2.49)\))
Three groups have sizes 6, 8, and 10, \(SSE=126\), and sample means 12, 16, and 19. For the comparison of the first and second means, use \(q=3.55\). Find the interval and state whether it excludes zero. (Answer: \((-7.32,-0.68)\); it excludes zero.)
Three groups have sizes 4, 9, and 12, sample means 30, 35, and 40, \(MSE=9\), and \(q=3.60\). Find the interval for the second mean minus the third mean and decide whether the pair is identified as different. (Answer: \((-8.37,-1.63)\); yes, it is identified as different.)
Three groups have sizes 5, 7, and 8, sample means 18, 22, and 25, \(SSE=102\), and \(q=3.50\). Find the interval for the first mean minus the second mean. (Answer: approximately \((-7.55,-0.45)\).)
A one-way ANOVA has four groups and total sample size \(N=31\). Its \(SSE\) is 135. A pair being compared has sizes 6 and 10, and the critical value is \(q=3.80\). Find the error degrees of freedom and \(MSE\), then give the pair-specific standard error before applying \(q\). (Answer: \(\nu=27\), \(MSE=5\), and \(SE=\sqrt{(5/2)(1/6+1/10)}=\sqrt{2/3}\approx0.8165\).)

What You'll Learn