Interpreting Tukey Simultaneous Confidence Intervals

Why Interpret the Whole Interval?

An omnibus ANOVA F test can indicate evidence that not all population means are equal, but it does not identify which groups differ or in what direction. Tukey simultaneous confidence intervals address those pairwise questions while accounting for the fact that many comparisons are being considered together. To interpret them, focus on two features: which mean is subtracted from which, and whether the interval includes zero.

A Tukey interval is not just a range of plausible values for one pair considered in isolation. The intervals are constructed as a family so that, under the model assumptions, the collection has the stated simultaneous confidence level. This is what makes their coverage appropriate when examining all pairwise differences.

What a Tukey Interval Estimates

Definition: For groups \(i\) and \(j\), a Tukey simultaneous confidence interval estimates the population mean difference \(\mu_i-\mu_j\). Its center is the observed difference \(\bar{x}_i-\bar{x}_j\). A family of Tukey intervals is constructed to have simultaneous confidence level at least \(1-\alpha\) for all pairwise differences among the groups (exactly \(1-\alpha\) when group sizes are equal, and conservative with unequal sizes, where the Tukey–Kramer version is used).

The direction comes directly from the subtraction order. An interval for \(\mu_A-\mu_B\) describes the mean for A relative to the mean for B; reversing the order gives the negative of the difference and reverses the interval’s endpoints. For example, an interval entirely below zero for \(\mu_A-\mu_B\) supports a lower population mean in group A than in group B.

For a one-way ANOVA with \(k\) independent groups, a common-variance model assumes observations in group \(i\) come from a normal population with mean \(\mu_i\) and the same variance \(\sigma^2\). The errors are independent, and the variance estimate \(MSE\) is the pooled within-group estimate from the ANOVA. Under these assumptions, Tukey’s method gives the following interval for every pair \(i,j\):

(\bar{x}_i-\bar{x}_j)\ \pm\ q_{1-\alpha;k,df}\sqrt{\frac{MSE}{2}\left(\frac{1}{n_i}+\frac{1}{n_j}\right)}

Here \(n_i\) and \(n_j\) are the two group sample sizes, and \(df\) is the ANOVA error degrees of freedom. The critical value \(q_{1-\alpha;k,df}\) comes from the studentized range distribution and depends on the family confidence level, number of groups, and error degrees of freedom. The square-root term is the standard error used for that pair. When all groups have equal sample size, this is Tukey’s HSD form; with unequal sample sizes, it is the Tukey-Kramer form.

A Tukey interval should be used only in a setting where the one-way ANOVA model is appropriate: observations are independent, within-group errors are reasonably normal, and the groups share a common variance. The independence and variance conditions arise from the study design and model; the normality condition concerns the errors within groups. The formula does not make these assumptions true. It uses them to obtain its stated simultaneous coverage.

How Zero Determines the Pairwise Conclusion

The null value for a difference in population means is zero: \(\mu_i-\mu_j=0\) says that the two population means are equal. The interval’s position relative to zero therefore determines whether the data support a nonzero difference at the familywise level used to construct the intervals.

Interpretation rule: If a Tukey simultaneous interval for \(\mu_i-\mu_j\) excludes zero, it supports a difference between those two population means at the corresponding familywise significance level. If the whole interval is positive, the mean for \(i\) is higher; if the whole interval is negative, the mean for \(i\) is lower. If the interval includes zero, the comparison does not establish a difference at that level.

This rule follows from the interval’s endpoints. Write the estimate as \(d=\bar{x}_i-\bar{x}_j\), and the positive margin of error as \(M\). The interval is \([d-M,d+M]\). If \(d-M>0\), every value in the interval is positive, so zero is excluded on the positive side. If \(d+M<0\), every value is negative, so zero is excluded on the negative side. If \(d-M\leq 0\leq d+M\), zero is among the values covered by the interval.

Under the same model and Tukey procedure, excluding zero corresponds to rejecting the pairwise equality hypothesis using the Tukey familywise comparison at level \(\alpha\). Including zero means there is not sufficient evidence to reject that equality for this comparison at that level. It does not prove the population means are equal; the interval may also include meaningful positive and negative differences.

Worked Example: Interpreting All Pairwise Intervals

Suppose a hypothetical balanced one-way ANOVA compares the mean response, measured in points, for groups A, B, and C. Each group has \(n=10\) observations, the ANOVA gives \(MSE=10.125\) with \(df=27\), and the 95% Tukey critical value is \(q=3.506\). The sample means are 20.0 points for A, 24.0 for B, and 31.0 for C. Assume the independent, normal, common-variance model is appropriate.

With equal sample sizes, the standard error is \(\sqrt{MSE/n}\), so the common margin of error is:

M=3.506\sqrt{\frac{10.125}{10}} =3.506\sqrt{1.0125} \approx 3.5278\text{ points}

For A minus B, the estimated difference is \(20.0-24.0=-4.0\) points. Its interval is:

-4.0\pm 3.5278=[-7.5278,\,-0.4722]\text{ points}

The entire interval is negative. Thus, the familywise 95% Tukey results support a lower population mean response for A than B; the estimated difference is 4.0 points lower for A.

For A minus C, the estimate is \(20.0-31.0=-11.0\) points, giving:

-11.0\pm 3.5278=[-14.5278,\,-7.4722]\text{ points}

Zero is excluded, and the interval is negative: the population mean for A is supported as lower than that for C. For B minus C, the estimate is \(24.0-31.0=-7.0\), and:

-7.0\pm 3.5278=[-10.5278,\,-3.4722]\text{ points}

This interval is also entirely negative. All three pairwise comparisons support differences at the familywise 5% level, with C higher than both A and B, and B higher than A. Each conclusion is about population means, not every individual observation.

Unequal Sample Sizes and Interval Direction

For unequal group sizes, the Tukey-Kramer standard error depends on both sample sizes. In general, comparisons involving smaller groups have larger standard errors, all else equal. Interpret the resulting interval in exactly the same way: identify its subtraction order, then check whether zero is inside it.

Worked Example: A Tukey-Kramer Interval That Includes Zero

Consider a hypothetical comparison of mean recovery time, in days, for three groups. The sample sizes are \(n_A=6\), \(n_B=10\), and \(n_C=15\); the sample means for A and B are 50 and 54 days. Suppose \(MSE=16\), \(df=28\), and the reported 95% Tukey critical value is \(q=3.50\). Assume the one-way model conditions are reasonable. Calculate the interval for \(\mu_A-\mu_B\).

First compute the Tukey-Kramer standard error:

SE=\sqrt{\frac{16}{2}\left(\frac{1}{6}+\frac{1}{10}\right)} =\sqrt{8(0.2666667)} =\sqrt{2.133333} \approx 1.4606\text{ days}

The estimated difference is \(50-54=-4\) days. The margin of error is \(3.50(1.4606)\approx5.1121\) days, so the interval is:

-4\pm5.1121=[-9.1121,\,1.1121]\text{ days}

Because this interval includes zero, it does not establish a difference between the population mean recovery times for A and B at the familywise 5% level. It remains compatible with A having a mean up to about 9.1 days lower than B, no difference, or a mean up to about 1.1 days higher. The data do not support choosing one direction at this level.

Report the Direction and the Size

An interval communicates more than a yes-or-no decision. Its endpoints give a range of differences compatible with the data under the procedure, in the outcome’s original units. An interval wholly above zero gives both evidence of a difference and a range for how much higher the first mean may be. An interval wholly below zero gives evidence that the first mean is lower, with the negative endpoints retaining that subtraction convention.

A statistically supported difference need not be important in practice. For example, a narrow interval that excludes zero might indicate a difference of only a small fraction of a clinically meaningful unit. Conversely, an interval including zero may still span differences that matter in practice. Statistical support and practical importance are distinct questions; Tukey intervals directly address uncertainty in mean differences, not whether those differences matter for a particular decision.

Worked Example: A Positive Interval in Context

In a hypothetical study of a four-group program, the outcome is a skill score measured in points. Suppose the sample mean is 18.4 points in group A and 12.2 points in group C. There are \(n_A=8\) and \(n_C=12\) observations, the ANOVA has \(MSE=12\) and \(df=20\), and software reports a 95% Tukey critical value of \(q=3.958\). Assume the model conditions are suitable. Interpret the interval for \(\mu_A-\mu_C\).

The estimated difference is \(18.4-12.2=6.2\) points. The unequal-size standard error is:

SE=\sqrt{\frac{12}{2}\left(\frac{1}{8}+\frac{1}{12}\right)} =\sqrt{6\left(\frac{5}{24}\right)} =\sqrt{1.25} \approx1.1180\text{ points}

The margin is \(3.958\sqrt{1.25}\approx4.4252\) points. Therefore:

6.2\pm4.4252=[1.7748,\,10.6252]\text{ points}

The interval is entirely positive, so it supports a higher population mean score for A than C at the familywise 5% level. The interval estimates that A’s mean is about 1.77 to 10.63 points higher. Whether that range is important for the program’s purpose requires a substantive judgment beyond the zero-inclusion rule.

Simultaneous Confidence Is About the Family

If there are \(k\) groups, the number of distinct pairwise comparisons is \(k(k-1)/2\). Looking at many ordinary, unadjusted intervals would make it increasingly likely that at least one interval misses its true difference. Tukey’s adjustment controls this familywise issue: under the model, the probability that all intervals in the family contain their corresponding population differences is \(1-\alpha\).

For example, a family of 95% Tukey intervals has at least 95% simultaneous coverage (exactly 95% with equal group sizes), not merely 95% coverage for each interval considered separately. The confidence statement concerns the method over repeated sampling: in repeated comparable studies, at least 95% of such interval families would cover all the pairwise population mean differences. It is not a statement that the fixed population differences have a 95% probability of lying in the particular intervals already computed.

Common Mistakes

Ignoring the subtraction order. An interval below zero for \(\mu_A-\mu_B\) means A’s mean is lower than B’s, not higher. Always name the difference in the same order as the interval.
Calling an interval that includes zero proof of equality. Including zero means a difference is not established at the chosen familywise level; it does not show the population means are exactly equal.
Interpreting one interval as if it were an unadjusted comparison. Tukey intervals are simultaneous across the family of pairwise comparisons. Their confidence level accounts for examining all those pairs.
Equating statistical support with practical importance. Excluding zero supports a nonzero difference, but the size and real-world relevance of that difference must also be considered.
Dropping units or context. State which group mean is higher or lower and give the difference in the response variable’s units.

Key takeaway: For a Tukey interval written for \(\mu_i-\mu_j\), an interval above zero supports a higher mean for group \(i\), and an interval below zero supports a lower mean for group \(i\). If zero is included, the comparison does not establish a difference at the familywise level. In every case, report the interval’s size and units as well as its direction.

Check Your Understanding

For each question, focus on the named subtraction order and the interval’s relationship to zero.

A Tukey interval for \(\mu_R-\mu_S\) is \([2.3,\,6.8]\) milligrams. Which group has the higher supported population mean, and what is the estimated range of differences?
An interval for \(\mu_A-\mu_B\) is \([-4.1,\,0.7]\) hours. What conclusion is supported about the pairwise difference at the procedure’s familywise level?
Why does an interval for \(\mu_C-\mu_D\) that lies entirely below zero indicate that group C has the lower mean?
What does 95% simultaneous confidence mean for a set of Tukey intervals covering all pairs among several groups?
Can an interval that excludes zero establish that the difference is practically important? Explain.