An End-to-End Unequal-Variance ANOVA Case

Introduction

An applied one-way ANOVA analysis involves more than choosing a test and reading its p-value. Group sizes, means, and variances help show whether a common-variance analysis is a reasonable description of the data. When spreads differ, Welch’s omnibus test and Games–Howell pairwise comparisons provide alternatives that do not pool the group variances in the same way. This tutorial works through those steps for one hypothetical dataset and shows how to report the results without claiming more than the analysis supports.

The example concerns a hypothetical laboratory exercise comparing the time, in minutes, required to complete a task under three procedures. We will use group summaries rather than list every observation. Assume that observations are independent within and across groups, and that diagnostic checks of the underlying observations show no severe outliers or departures from approximate within-group normality. These assumptions are important: summary statistics alone cannot establish independence or assess the shape of the observations.

Group Summaries and the Variability Question

Let \(k\) be the number of groups. For group \(i\), let \(n_i\) be its sample size, \(\bar{x}_i\) its sample mean, and \(s_i^2\) its sample variance. A one-way analysis compares the group means for one categorical factor and one quantitative response. The classical one-way F test assumes independent observations, approximate normality within each group, and a common population variance across groups. Welch’s one-way test instead allows the population variances to differ; it still relies on independent observations and approximate within-group normality for its usual calibration.

The summaries below are hypothetical. They are chosen to illustrate unequal group sizes and unequal sample variances. The units of the means and standard deviations are minutes; variances are in squared minutes.

The largest sample variance is about \(2.374/1.394=1.70\) times the smallest. The spreads are not identical, although their differences are not extreme in this example. The sample sizes also differ, with group C having half as many observations as group B. Unequal variances combined with unequal allocation can affect the behavior of a pooled-variance test, so it is useful to examine a Welch analysis as well as the classical result.

Classical and Welch Omnibus Tests

Both omnibus tests address the null hypothesis that all population means are equal: \(H_0:\mu_1=\mu_2=\mu_3\). The alternative is that at least one population mean differs. Neither test, by itself, identifies which means differ.

For the classical test, the between-group sum of squares is \(\sum_i n_i(\bar{x}_i-\bar{x}_{\cdot})^2\), where \(\bar{x}_{\cdot}\) is the overall sample mean. Dividing this sum by \(k-1\) gives the between-group mean square. The classical F statistic is the between-group mean square divided by the pooled within-group mean square.

Welch’s test uses inverse-variance weights. For each group, define \(w_i=n_i/s_i^2\), and let \(W=\sum_i w_i\). The weighted mean is \(\bar{x}_w=\sum_iw_i\bar{x}_i/W\). The statistic also uses a correction factor and an estimated denominator degrees of freedom. In this case, these adjustments account for the group-specific variances and sample sizes instead of treating the pooled variance as common.

Games–Howell Pairwise Comparisons

An omnibus result indicates whether the set of means is inconsistent with equality, but it does not identify the pairs that differ. Games–Howell comparisons assess pairs while allowing unequal variances and unequal sample sizes. For groups \(i\) and \(j\), the standard error of their mean difference is \(\sqrt{s_i^2/n_i+s_j^2/n_j}\). The pair-specific Welch–Satterthwaite degrees of freedom are

Games–Howell expresses the absolute mean difference on a studentized-range scale: \(q_{ij}=|\bar{x}_i-\bar{x}_j|/\sqrt{\{s_i^2/n_i+s_j^2/n_j\}/2}\). Each pair is assessed using the studentized-range distribution with \(k\) groups and that pair’s estimated degrees of freedom. This calibration accounts for making multiple pairwise comparisons; it does not turn each comparison into an unadjusted two-sample test.

The same calculation is repeated for each remaining pair. The following table gives rounded results; adjusted p-values are reported as bounds rather than as overly precise estimates. They are obtained from the studentized-range calibration with three groups and the pair-specific degrees of freedom.

All three comparisons exceed the relevant 0.05 studentized-range critical values. The table supports evidence of differences for each pair, with the sample means ordered A, then B, then C. It does not show that every individual using procedure C will take longer than every individual using A or B; these are comparisons of population means, not predictions about individual outcomes.

Putting the Results Together

For these hypothetical data, the classical and Welch omnibus tests both provide strong evidence against equal population means. Their statistics and denominator degrees of freedom differ because their variance assumptions and calculations differ. The classical test uses the pooled within-group mean square, while Welch’s test uses group-specific inverse-variance weights and an adjusted reference distribution. Since the sample variances and sample sizes are not identical, reporting Welch’s result alongside the classical result makes the analysis more transparent.

The Games–Howell follow-ups add information the omnibus tests do not provide: which pairs show evidence of differing means under a multiple-comparison procedure. A concise report could state: “In a hypothetical comparison of task completion times under three procedures, group variances ranged from 1.394 to 2.374 minutes squared, and group sizes ranged from 6 to 12. Welch’s one-way test found evidence that not all population means were equal, \(F_W(2,39.91)=32.97\), \(p\approx3.5\times10^{-9}\). Games–Howell comparisons found evidence of differences for all three pairs (each adjusted \(p<0.01\)); sample means were 10.000, 12.976, and 15.952 minutes for A, B, and C, respectively.” This report identifies the response units and the procedure used for follow-up comparisons.

These results depend on the assumptions stated at the outset. In particular, independence should follow from the study design, not merely from a statistical test. Approximate normality and the absence of severe outliers should be checked using the observations and diagnostic plots; group summaries cannot show these features. If the data are strongly non-normal or the design creates dependence, the reported reference distributions may not be appropriate. Also, statistical evidence of different means does not by itself establish practical importance or a causal effect.

Common Mistake

A frequent error is to calculate \(A\) using the weight proportions themselves, for example \((128/209)^2/7\) for group A. That is not the term in Welch’s correction. The formula uses \(1-w_i/W\), so the correct group A numerator is \((1-128/209)^2=(81/209)^2\). Using the weight proportion in place of its complement changes \(A\), which then changes the correction factor, Welch statistic, denominator degrees of freedom, and p-value. Check each term against the definition before combining them.

A second interpretive error is to treat a significant omnibus test as proof that every pair differs. The omnibus test only addresses whether all population means are equal. Pairwise claims require follow-up comparisons, and those comparisons should account for multiplicity. Here, the Games–Howell results supply that separate pairwise evidence.