Why Use Methods That Do Not Pool Variances?
The usual one-way ANOVA and familiar pooled-variance pairwise procedures rely on a shared within-group variance. When group spreads differ substantially, especially alongside unequal sample sizes, that shared-variance assumption can make standard errors and conclusions unreliable. Welch’s one-way ANOVA provides an omnibus test of equal population means without assuming equal variances. If pairwise follow-up comparisons are needed, Games-Howell uses a separate variance estimate for each pair and adjusts for examining multiple pairs.
These methods address unequal variances; they do not remove the need to examine the study design and the shape of the data. The groups should be independent, and the response should be quantitative. In the standard model, observations within each group are independent and approximately normal. The Welch and Games-Howell approximations are often useful when variances differ, but they are not a general remedy for strong nonnormality or very small samples.
Welch’s One-Way ANOVA
Welch’s test gives greater weight to groups whose sample means are estimated more precisely. Define the weight \(w_i=n_i/s_i^2\), the total weight \(W=\sum_i w_i\), and the weighted mean \(\bar{x}_w=\sum_i w_i\bar{x}_i/W\). The test statistic is an adjusted version of the weighted variation among the group means.
The numerator measures weighted differences among group means. The adjustment in the denominator and the approximate denominator degrees of freedom account for unequal variances and group sizes. As with other omnibus ANOVA tests, a small p-value is evidence against equality of all population means; it does not identify which means differ.
Worked Example: Testing Three Means With Welch ANOVA
Suppose three independent hypothetical groups have the following summaries for a response measured in points: group A has \(n_1=8\), \(\bar{x}_1=10\), and \(s_1^2=4\); group B has \(n_2=12\), \(\bar{x}_2=13\), and \(s_2^2=16\); group C has \(n_3=20\), \(\bar{x}_3=15\), and \(s_3^2=36\). The variances differ notably, so a test that assumes a common variance is not the appropriate choice here. Assume the observations are independent and each group’s response is reasonably modeled by a normal distribution.
Calculate the weights and weighted mean. The weights are \(w_1=8/4=2\), \(w_2=12/16=0.75\), and \(w_3=20/36=0.5556\), so \(W=3.3056\). Thus, \(\bar{x}_w=(2(10)+0.75(13)+0.5556(15))/3.3056\approx11.521\) points.
Calculate the adjustment and degrees of freedom. Using the unrounded weights, \(A\approx(1-2/3.3056)^2/7+(1-0.75/3.3056)^2/11+(1-0.5556/3.3056)^2/19\approx0.1130\). Therefore \(df_1=3-1=2\) and \(df_2=8/(3(0.1130))\approx23.59\).
Calculate the test statistic. The weighted sum of squared deviations is approximately \(2(10-11.521)^2+0.75(13-11.521)^2+0.5556(15-11.521)^2=12.99\). The numerator mean square is \(12.99/2=6.496\), and the correction factor is \(1+0.25(0.1130)=1.0283\). Hence \(F_W=6.496/1.0283\approx6.317\).
For \(F_W=6.317\) with \(df_1=2\) and \(df_2\approx23.59\), the upper-tail p-value is approximately \(0.0064\). At significance level \(\alpha=0.05\), reject \(H_0\): the data provide evidence that the three population means are not all equal. This result alone does not say which pair differs; pairwise comparisons require a procedure that accounts for multiplicity.
Games-Howell Pairwise Comparisons
For a particular pair of groups \(i\) and \(j\), Games-Howell estimates the variance of the difference in sample means as \(s_i^2/n_i+s_j^2/n_j\). It uses a pair-specific Welch-Satterthwaite degrees of freedom, so each comparison can have a different degrees of freedom. The studentized-range statistic accounts for considering all \(k\) means rather than treating each pair as an isolated test.
The interval and test give matching decisions when they use the same familywise level: the interval excludes zero precisely when the adjusted two-sided comparison is significant at that level. The interval estimates the direction and size of the population mean difference in the response’s units. The adjusted p-value measures how surprising a statistic at least this extreme would be under equal means, using the studentized-range reference distribution; it is not the probability that the null hypothesis is true.
Worked Example: Comparing Three Means With Games-Howell
Consider hypothetical independent groups measuring a response in points. Group A has \(n_A=10\), mean \(42\), and standard deviation \(4\); group B has \(n_B=18\), mean \(48\), and standard deviation \(9\); group C has \(n_C=12\), mean \(51\), and standard deviation \(5\). The variances and sample sizes are not equal. Assume independent observations and approximately normal responses within groups. We compare all three pairs using Games-Howell at familywise level \(0.05\).
Compare A with B. The estimated difference is \(\bar{x}_A-\bar{x}_B=42-48=-6\) points. The standard error is \(SE_{AB}=\sqrt{4^2/10+9^2/18}=\sqrt{6.1}\approx2.470\) points. The degrees of freedom are \(df_{AB}=6.1^2/\{1.6^2/9+4.5^2/17\}\approx25.22\). Thus \(q_{AB}=\sqrt{2}|-6|/2.470\approx3.435\). For three groups and about 25 degrees of freedom, the 0.05 studentized-range critical value is about \(3.52\). Since \(3.435<3.52\), this pair is not statistically significant at the familywise 0.05 level.
Compare A with C. The estimated difference is \(42-51=-9\) points. Here \(SE_{AC}=\sqrt{4^2/10+5^2/12}=\sqrt{3.6833}\approx1.919\) points, and \(df_{AC}=3.6833^2/\{1.6^2/9+(25/12)^2/11\}\approx19.98\). The statistic is \(q_{AC}=\sqrt{2}|-9|/1.919\approx6.632\), which exceeds the critical value of about \(3.58\) for three groups and 20 degrees of freedom. This comparison is significant. The simultaneous interval for \(\mu_A-\mu_C\) is approximately \(-9\pm(3.58/\sqrt{2})(1.919)\), or \((-13.86,-4.14)\) points. It indicates that group A’s population mean is estimated to be between about 4.14 and 13.86 points lower than group C’s, using a 95% familywise procedure.
Compare B with C. The difference is \(48-51=-3\) points, and \(SE_{BC}=\sqrt{9^2/18+5^2/12}=\sqrt{6.5833}\approx2.566\) points. The pair-specific degrees of freedom are \(df_{BC}=6.5833^2/\{4.5^2/17+(25/12)^2/11\}\approx27.33\), giving \(q_{BC}=\sqrt{2}(3)/2.566\approx1.653\). This is below the relevant critical value, so the pair is not significant at familywise level \(0.05\). Among these comparisons, only A and C show evidence of different population means under the stated procedure.
Why Pooling Can Mislead
Pooling replaces different group variances with one common estimate. When sample sizes differ, this can give too much influence to the variance from a large group and too little to the uncertainty in a small, high-variance group. Welch and Games-Howell instead retain the separate variance contributions in each standard error. Unequal variances alone do not guarantee that pooled and unpooled methods will reach different conclusions, but they can produce materially different uncertainty estimates.
Worked Example: How Unequal Spreads Change a Pairwise Result
Suppose two hypothetical independent groups have a response measured in units: group A has \(n_A=6\), mean \(20\), and standard deviation \(2\); group B has \(n_B=24\), mean \(28\), and standard deviation \(10\). A third group is also part of the analysis, so the family of comparisons contains \(k=3\) groups. Assume independent observations and approximately normal responses. Consider the A-versus-B comparison.
The difference is \(\bar{x}_A-\bar{x}_B=-8\) units. Games-Howell gives \(SE_{AB}=\sqrt{2^2/6+10^2/24}=\sqrt{4.8333}\approx2.1985\) units, and \(df_{AB}=4.8333^2/\{(4/6)^2/5+(100/24)^2/23\}\approx27.69\). Therefore \(q_{AB}=\sqrt{2}(8)/2.1985\approx5.146\). This exceeds the studentized-range critical value of about \(3.50\), so the Games-Howell comparison is significant at familywise level \(0.05\). The interval is approximately \(-8\pm(3.50/\sqrt{2})(2.198)=(-13.44,-2.56)\) units.
For contrast, a pooled variance estimate for these two groups would be \(s_p^2=\{5(2^2)+23(10^2)\}/28\approx82.86\), giving a pooled standard error \(\sqrt{82.86(1/6+1/24)}\approx4.155\) units. That estimate is much larger than the Games-Howell standard error, because the large variance from group B is treated as if it also described group A. The corresponding pooled studentized-range statistic would be only \(\sqrt{2}(8)/4.155\approx2.72\), below a three-group critical value near \(3.50\). This contrast illustrates why it matters to match the method to the variance structure; it is not a reason to choose whichever result is preferred.
A Short Justification for Pair-Specific Degrees of Freedom
The pairwise standard error follows from independence: the variance of a difference of independent sample means is the sum of their variances. Since the sample variance \(s_i^2\) estimates the population variance in group \(i\), the estimated variance of \(\bar{x}_i-\bar{x}_j\) is \(s_i^2/n_i+s_j^2/n_j\). Each term has its own sampling uncertainty, so one common degrees-of-freedom value is not generally appropriate.
The Welch-Satterthwaite approximation matches the variance of this sum to the variance of a scaled chi-square estimate. In general, if independent variance estimates contribute terms \(V_r\), the approximate degrees of freedom are the square of their sum divided by the sum of their squared contributions divided by their respective degrees of freedom. For this pair, those contributions are \(s_i^2/n_i\) and \(s_j^2/n_j\); substituting them yields the \(df_{ij}\) formula above. Games-Howell then uses the resulting pair-specific reference distribution, while its studentized-range adjustment accounts for the collection of pairwise comparisons.
Confirm independent groups and observations, identify the quantitative response, and examine whether within-group distributions make the normal approximation reasonable.
When variances differ substantially, calculate Welch’s statistic and approximate degrees of freedom to test whether all population means are equal.
Calculate pair-specific standard errors and degrees of freedom, then use the studentized-range distribution to adjust comparisons and form simultaneous intervals.
Common Mistakes
- Using pooled-variance post hoc comparisons automatically. If group variances and sample sizes differ substantially, the pooled standard error may not represent the uncertainty for a particular pair. Use Games-Howell when its assumptions are appropriate.
- Assuming Welch’s significant omnibus test identifies a pair. It only supports the conclusion that not all population means are equal. Use an appropriate multiple-comparison procedure to investigate which pairs differ.
- Ignoring multiplicity. Several unadjusted pairwise tests increase the chance of at least one false positive across the family. Games-Howell’s studentized-range procedure accounts for examining all pairs.
- Reading an interval as a probability statement about a fixed mean. A 95% familywise interval procedure does not mean there is a 95% probability that this particular interval contains the fixed population difference. The stated coverage describes the long-run performance of the procedure.
- Treating unequal-variance methods as a cure for every assumption problem. Welch and Games-Howell address variance heterogeneity; independence and an adequate distributional approximation still matter.
Check Your Understanding
Try these questions before reviewing the worked calculations.
- What null hypothesis does Welch’s one-way ANOVA test for \(k\) groups?
- Why can unequal variances combined with unequal sample sizes make a pooled-variance standard error inappropriate?
- For a Games-Howell comparison, what quantities determine the pair-specific standard error and degrees of freedom?
- If the Welch omnibus test is significant, can you conclude that every pair of population means differs? Explain.
- What does it mean if a Games-Howell simultaneous confidence interval for \(\mu_i-\mu_j\) excludes zero?