Common Misreadings of Welch and Games-Howell Results

Two Different Questions: Omnibus and Pairwise Results

Welch’s one-way test and Games–Howell comparisons answer related but distinct questions. Welch’s test assesses whether the population means across several groups are all equal. Games–Howell comparisons assess particular pairs of means while allowing group variances and sample sizes to differ. Confusing these roles can turn a valid analysis into an overconfident conclusion about which groups differ.

The distinction matters when results are reported. A small p-value for Welch’s test is evidence against the hypothesis that all population means are equal; it does not name a differing pair. Conversely, a table of pairwise results must be read with its multiplicity adjustment in mind. A pairwise p-value that has not been adjusted is not interchangeable with the Games–Howell adjusted p-value.

Definition: For \(k\) independent groups, let \(\mu_i\), \(\bar{x}_i\), \(s_i\), and \(n_i\) denote the population mean, sample mean, sample standard deviation, and sample size in group \(i\). Welch’s omnibus test assesses \(H_0:\mu_1=\mu_2=\cdots=\mu_k\) against the alternative that not all population means are equal. A Games–Howell comparison evaluates a selected pair, such as \(H_0:\mu_i=\mu_j\), using a standard error and degrees of freedom that do not require equal population variances.

These procedures assume independent observations within and across groups. Welch’s test and Games–Howell comparisons are approximate procedures; their usual reference distributions are most reliable when the within-group distributions are reasonably well behaved, particularly when sample sizes are small. Unequal variances are allowed, not ignored. Before interpreting output, consider the study design, inspect groupwise distributions and variability, and check for influential observations using the diagnostic ideas established earlier in the course. Neither procedure repairs dependence between observations or severe problems in the data.

What the Welch Omnibus Result Can—and Cannot—Say

Welch’s test produces an approximate \(F\) statistic with numerator and denominator degrees of freedom. Its p-value is calculated under the null hypothesis that all \(k\) population means are equal. A small p-value indicates that the observed test statistic, or a more extreme one, would be unusual under that null model. It is not the probability that the null hypothesis is true.

If the test is significant at a chosen \(\alpha\), the conclusion is that the data provide evidence that not all population means are equal. The omnibus result alone does not establish that every pair differs, identify the pair or pairs responsible, or show the direction of any particular difference. Those questions require suitable follow-up comparisons, such as Games–Howell comparisons.

Worked Example: Interpreting an Omnibus Result Without Overreaching

Suppose a hypothetical analysis compares the mean change in a laboratory measurement across three independent treatment groups. Assume the observations are independent, the group distributions show no severe outliers or extreme skew, and the group standard deviations differ enough that a pooled-variance test is not the chosen method. A software report gives Welch \(F(2,12)=4.00\), \(p=0.0467\). For an \(F\) distribution with numerator \(df=2\) and denominator \(df=12\), the upper-tail probability at 4.00 is \((12/(12+2\cdot4))^6=0.046656\), which rounds to 0.0467.

At \(\alpha=0.05\), the p-value is below the significance level, so reject the hypothesis that the three population means are all equal. In context, the results provide evidence that mean change is not the same across all three treatment groups.

It would be incorrect to report that all three groups differ, or to name a particular pair based on this omnibus statistic. The statistic and p-value contain no pair-specific result. To identify which comparisons are supported, examine Games–Howell results, including their adjustment for the set of pairwise comparisons.

Why Games–Howell Does Not Use a Pooled Standard Error

For a comparison of groups \(i\) and \(j\), the ordinary estimated standard error of the difference in sample means is \(\sqrt{s_i^2/n_i+s_j^2/n_j}\). Games–Howell uses the same group-specific variance contributions, but expresses the comparison on the studentized-range scale. Its statistic is

q_{ij}=\frac{|\bar{x}_i-\bar{x}_j|}{\sqrt{\left(s_i^2/n_i+s_j^2/n_j\right)/2}}.

The denominator here is \(\sqrt{(s_i^2/n_i+s_j^2/n_j)/2}\), not a pooled-variance estimate. The pair-specific degrees of freedom are approximated by the Welch–Satterthwaite formula:

\nu_{ij}= \frac{(s_i^2/n_i+s_j^2/n_j)^2} {\dfrac{(s_i^2/n_i)^2}{n_i-1}+\dfrac{(s_j^2/n_j)^2}{n_j-1}}.

The absolute difference makes \(q_{ij}\) nonnegative; the sign and direction are recovered from \(\bar{x}_i-\bar{x}_j\). The adjusted p-value is calibrated using the studentized-range distribution with \(k\) groups and the pair’s approximate degrees of freedom \(\nu_{ij}\). This calibration accounts for examining multiple pairs in the family of comparisons. It is not the same calculation as a two-sided \(t\) test for one pair.

Justification: Write \(V_i=s_i^2/n_i\) and \(V_j=s_j^2/n_j\). The ordinary standard error is \(\sqrt{V_i+V_j}\), while the Games–Howell denominator is \(\sqrt{(V_i+V_j)/2}\). Therefore \(q_{ij}=\sqrt{2}\,|\bar{x}_i-\bar{x}_j|/\sqrt{V_i+V_j}\). The factor \(\sqrt{2}\) puts the pairwise difference on the studentized-range scale. Replacing \(V_i+V_j\) with an equal-variance pooled estimate would change the method and would not be a Games–Howell comparison.

Worked Example: Checking the Standard Error and Statistic

Consider hypothetical data from three independent groups, with \(n_i=n_j=10\), \(s_i=s_j=2\) measurement units, and sample means 10 and 12 units for the pair being compared. Assume the observations within these groups are reasonably well behaved, with no severe outliers, and that independence is supported by the study design. The equal sample standard deviations here are just features of this example; the calculation does not require equal variances.

The variance contributions are \(s_i^2/n_i=4/10=0.4\) and \(s_j^2/n_j=4/10=0.4\). Thus the ordinary standard error of the difference is \(\sqrt{0.4+0.4}=\sqrt{0.8}\approx0.8944\) units. The Games–Howell denominator is different:

\sqrt{\frac{0.4+0.4}{2}}=\sqrt{0.4}\approx0.6325\text{ units}.

The absolute difference is \(|10-12|=2\) units, so \(q_{ij}=2/\sqrt{0.4}\approx3.162\). Equivalently, \(q_{ij}=\sqrt{2}(2/0.8944)\approx3.162\). The pair degrees of freedom are

\nu_{ij}= \frac{(0.4+0.4)^2}{0.4^2/9+0.4^2/9} =\frac{0.64}{0.03556}=18.

Using \(0.8944\) as the denominator in the displayed Games–Howell formula would give the wrong \(q\). It is the ordinary standard error; converting the difference to the studentized-range scale requires the factor \(\sqrt{2}\). The adjusted p-value must then be obtained from the studentized-range calibration for three groups and 18 degrees of freedom, not from a \(t\) table.

Multiplicity: Read the Adjusted Result

With \(k\) groups, there are \(k(k-1)/2\) distinct pairs. Looking at many unadjusted tests increases the chance of at least one false positive across the set. Games–Howell addresses this multiplicity by calibrating the set of pairwise comparisons using the studentized-range distribution. Its adjusted p-value is the quantity to compare with \(\alpha\) when the reported family is all pairwise comparisons.

An unadjusted p-value answers a different question: how unusual is the pair’s statistic under its null hypothesis when considered as a single comparison? It does not account for searching across the other pairs. It is therefore possible for an unadjusted pairwise result to be below 0.05 while the Games–Howell adjusted result is not. Do not select whichever version gives the preferred conclusion.

Worked Example: Why an Unadjusted Pairwise p-Value Is Not the Games–Howell Result

Use the three-group summaries from the preceding calculation: each group has \(n=10\), \(s=2\) units, and the means are 10, 12, and 14 units. For the 10-versus-12 comparison, a conventional two-sided \(t\) test considered alone uses the ordinary standard error \(0.8944\) and 18 degrees of freedom. Its statistic is \(t=2/0.8944\approx2.236\), with unadjusted \(p\approx0.03824\).

That p-value is not the Games–Howell adjusted p-value. For this pair, the Games–Howell statistic is \(q\approx3.162\), and its reference distribution accounts for all three groups’ pairwise comparisons. The 5% studentized-range critical value for three groups and 18 degrees of freedom is approximately 3.61. Since \(3.162<3.61\), this pair does not meet the 5% Games–Howell criterion. The ordinary single-pair \(t\) result and the multiplicity-adjusted conclusion differ because they answer different questions.

For the 10-versus-14 comparison, the difference is 4 units, and the same Games–Howell denominator is \(\sqrt{0.4}\approx0.6325\). Hence \(q=4/\sqrt{0.4}\approx6.325\), which exceeds 3.61. This pair does meet the 5% Games–Howell criterion. The direction is that the sample mean in the group with mean 14 is higher by 4 units. This is evidence about the population means, not proof that the group assignment caused the difference.

Interpreting Pairwise Confidence Intervals

Games–Howell output may include confidence intervals for differences in means. Their endpoints are in the measurement units of the response variable, and their direction depends on the subtraction order. For the ordered difference \(\bar{x}_i-\bar{x}_j\), a Games–Howell interval has the form

(\bar{x}_i-\bar{x}_j)\ \pm\ q_{\text{crit}}\sqrt{\frac{s_i^2/n_i+s_j^2/n_j}{2}},

where \(q_{\text{crit}}\) uses the studentized-range calibration for the family of \(k\) groups and the pair’s approximate degrees of freedom. Intervals constructed this way provide approximate simultaneous coverage across the family, rather than an unadjusted confidence level for each pair separately. An interval containing zero is consistent with no difference at the corresponding familywise level; an interval entirely above or below zero indicates a difference in the indicated direction.

Worked Example: Reading a Games–Howell Interval in Context

For the same hypothetical 10-versus-12 comparison, the estimated difference in the order “group with mean 10 minus group with mean 12” is \(-2\) units. The Games–Howell standard-error factor is \(\sqrt{0.4}\approx0.6325\). Using \(q_{\text{crit}}\approx3.61\), the margin is \(3.61(0.6325)\approx2.283\) units, giving the interval

-2\pm2.283=(-4.283,\ 0.283)\text{ units}.

Because this interval includes zero, it does not establish a difference between these two population means at the approximate 5% familywise level. The negative point estimate means the first group’s sample mean is 2 units lower, but the interval also allows a small positive difference. Reversing the subtraction order would reverse the signs of both endpoints, not change the comparison’s conclusion.

Common Misreadings and a Reliable Reporting Habit

“The Welch test is significant, so every group differs.” The omnibus result rejects equality of all means; it does not show that all pairs differ. Use pairwise comparisons to make pair-specific claims.
“The omnibus result tells us which pair is different.” It does not locate differences. Report the relevant Games–Howell comparisons, with their adjusted p-values or intervals.
“The pooled standard error is close enough.” Games–Howell uses the two groups’ separate variance contributions. Use its formula and pair-specific degrees of freedom rather than substituting a pooled variance estimate.
“The unadjusted p-value is below 0.05, so the pair is significant.” If the question concerns the family of all pairwise comparisons, use the adjusted result. An unadjusted p-value does not control error across that family.
“A non-significant pair proves the means are equal.” Failure to find evidence of a difference is not proof of equality. Consider the interval’s range and the precision of the estimates.

A clear report separates the overall and pairwise conclusions. State the Welch statistic, its degrees of freedom, and its p-value, then describe its implication for the equality of all means. For pairwise claims, name the groups, give the estimated difference in a stated subtraction order and in measurement units, and report the Games–Howell adjusted p-value or interval. This keeps the conclusion aligned with the question each result actually answers.

Check Your Understanding

Answer each question by identifying exactly what the reported result supports.

A Welch test is significant for four groups. What can you conclude about the four population means, and what can you not conclude from this test alone?
For two groups, \(s_i^2/n_i=0.25\) and \(s_j^2/n_j=0.64\). Write the Games–Howell denominator and the ordinary standard error. Why are they not the same?
A pairwise \(t\) test has an unadjusted p-value of 0.03, while the corresponding Games–Howell adjusted p-value is 0.08. If the family is all pairwise comparisons and \(\alpha=0.05\), which result guides the familywise conclusion?
A Games–Howell interval for \(\mu_i-\mu_j\) is entirely below zero. Explain its direction in context, and state what reversing the subtraction order would do.
Name two conditions that should be considered before relying on Welch and Games–Howell results.