Choosing Welch ANOVA for Unequal Clinic Variability

Why Unequal Clinic Spreads Matter

A one-way ANOVA compares the means of several independent groups. Its usual F test relies on a common within-group variance. If clinic outcomes have substantially different spreads, that assumption may be implausible. The problem can become especially important when the clinics also have different sample sizes: a small clinic with highly variable outcomes can contribute a different amount of uncertainty from a large clinic with tightly clustered outcomes.

Welch’s one-way ANOVA is an alternative for comparing group means when the population variances are not assumed equal. It uses weights based on each group’s estimated variance and sample size, and it approximates the null distribution of its test statistic with an F distribution. For follow-up comparisons, the Games–Howell procedure provides pairwise tests and intervals that likewise allow unequal variances and sample sizes.

Definition: Welch’s one-way ANOVA tests whether \(k\) independent groups have the same population mean, without assuming a common population variance. Its null hypothesis is \(H_0:\mu_1=\mu_2=\cdots=\mu_k\), and its alternative is that not all group means are equal. The test uses an approximate F distribution with \(k-1\) numerator degrees of freedom and an estimated denominator degrees of freedom.

Model and Conditions

Let \(X_{ij}\) be the outcome for observation \(j\) in group \(i\), where group \(i\) has sample size \(n_i\), sample mean \(\bar{x}_i\), and sample standard deviation \(s_i\). Welch’s procedure does not require the group variances to be equal. It does, however, rely on the following conditions for its usual approximate inference:

Independent observations. Outcomes from different individuals should be independent, and the groups should consist of independent samples. A design with repeated measurements or matched individuals needs a method that accounts for that dependence.
Quantitative outcome. The group means and their variability should be meaningful summaries of the measured outcome.
Reasonably well-behaved group distributions. Within each group, the outcome should be approximately normal, particularly when sample sizes are small. Strong skewness or influential outliers can make the approximation unreliable.
Unequal variances are allowed. The population variances may differ. This is the reason to choose Welch’s method rather than the ordinary equal-variance F test.

A residual plot can help reveal unequal spreads: for example, residuals may fan out as their fitted values change, or one clinic’s residuals may be much more dispersed than another’s. A residual plot does not prove that the population variances differ, but it can show that a common-variance model is questionable. Choosing Welch’s procedure addresses unequal variances; it does not repair dependence, severe outliers, or an unsuitable sampling design.

Worked Example: Deciding Whether to Use Welch’s Test

Suppose a hypothetical clinic comparison records the number of days until a patient experiences a specified level of symptom improvement. The clinics have independent patients, and the outcome is measured in days. Residual plots show a wider spread for one clinic than for the others, without an obvious extreme outlier; within-clinic distributions are reasonably symmetric. Should the usual equal-variance ANOVA or Welch’s ANOVA be the primary mean comparison?

The observations are independent across patients and clinics, and the quantitative outcome has an interpretable mean. The plots do not reveal a major departure from approximate within-clinic normality, although small samples would make that assessment less reassuring. The notably different residual spreads argue against relying on a common-variance model. Welch’s one-way ANOVA is therefore a suitable mean-comparison procedure. It does not imply that the clinics have equal variances; it allows their variances to differ.

Welch’s Test Statistic

For each group, calculate the estimated weight \(w_i=n_i/s_i^2\), and let \(W=\sum_{i=1}^k w_i\). These weights are larger for groups with more precise sample means. The weighted mean used in the calculation is \(\bar{x}_w=\sum_i w_i\bar{x}_i/W\). It is a reference value for the test, not necessarily the ordinary overall sample mean.

w_i=\frac{n_i}{s_i^2},\qquad W=\sum_{i=1}^k w_i,\qquad \bar{x}_w=\frac{\sum_{i=1}^k w_i\bar{x}_i}{W}

Define \(A=\sum_{i=1}^k (1-w_i/W)^2/(n_i-1)\). Welch’s statistic and its approximate degrees of freedom are

F_W= \frac{\displaystyle\sum_{i=1}^k w_i(\bar{x}_i-\bar{x}_w)^2/(k-1)} {\displaystyle 1+\frac{2(k-2)}{k^2-1}A}, \qquad df_1=k-1,\qquad df_2=\frac{k^2-1}{3A}

Under \(H_0\), the reference distribution is approximately \(F\) with \(df_1\) and \(df_2\) degrees of freedom. The p-value is the upper-tail probability \(P(F_{df_1,df_2}\ge F_W)\). Because the denominator degrees of freedom are estimated and may be non-integer, the reference distribution is an approximation, not an exact finite-sample distribution.

Why the Weights Are Reasonable

The weights reflect the precision of the individual group means. To see why, suppose a weighted estimate of a common mean has the form \(\sum_i c_i\bar{x}_i\), with \(\sum_i c_i=1\). If observations are independent and group \(i\) has variance \(\sigma_i^2\), then the variance of this estimate is \(\sum_i c_i^2\sigma_i^2/n_i\). Minimizing this expression subject to the weights summing to one gives coefficients proportional to \(n_i/\sigma_i^2\).

\frac{\partial}{\partial c_i} \left(\sum_i c_i^2\frac{\sigma_i^2}{n_i} -\lambda\left(\sum_i c_i-1\right)\right)=0 \quad\Longrightarrow\quad c_i=\frac{\lambda n_i}{2\sigma_i^2}

The constraint \(\sum_i c_i=1\) makes these coefficients proportional to \(n_i/\sigma_i^2\), as claimed. Since the population variances are unknown, Welch’s method estimates these weights with \(n_i/s_i^2\). Its correction factor and denominator degrees of freedom account approximately for the additional uncertainty from estimating different variances. This weighting argument motivates the procedure; the formula for the approximate F calibration is the one stated above.

Worked Example: Calculating Welch’s Omnibus Test

Consider hypothetical times to symptom improvement from three clinics. The residual diagnostics in this constructed example show unequal spreads, approximately symmetric within-clinic distributions, and no prominent outliers. The groups are independent. The table gives the sample sizes, means, and sample standard deviations.

Clinic	\(n_i\)	\(\bar{x}_i\) (days)	\(s_i\) (days)
A	8	12	\(\sqrt{10}\approx3.162\)
B	12	15	4
C	20	18	3

The group variances are \(10\), \(16\), and \(9\) days squared, so they are not treated as a common variance. The estimated weights are \(w_A=8/10=0.8000\), \(w_B=12/16=0.7500\), and \(w_C=20/9=2.2222\). Thus \(W=3.7722\), and

\bar{x}_w= \frac{0.8000(12)+0.7500(15)+2.2222(18)}{3.7722} \approx16.131\text{ days}

The weighted sum of squared deviations is approximately \(22.3741\) days squared. For the correction term,

A= \frac{(1-0.8000/3.7722)^2}{7} +\frac{(1-0.7500/3.7722)^2}{11} +\frac{(1-2.2222/3.7722)^2}{19} \approx0.15593

Here \(k=3\), so \(df_1=2\). Substitution gives

F_W= \frac{22.3741/2}{1+\{2(3-2)/(3^2-1)\}(0.15593)} = \frac{11.1871}{1.03898} \approx10.77, \qquad df_2=\frac{3^2-1}{3(0.15593)} \approx17.10

The upper-tail probability is approximately \(p=0.00094\). At significance level \(\alpha=0.05\), reject \(H_0\): the data provide evidence that the three population mean times are not all equal. This omnibus result does not establish that every pair of clinics differs, nor does it establish why their means differ.

Pairwise Follow-Up: Games–Howell Comparisons

After a meaningful omnibus result, a researcher may need to identify which clinic means differ. Ordinary pooled-variance comparisons are not a natural follow-up when spreads differ. Games–Howell compares each pair using the two groups’ separate estimated variances and a pair-specific degrees-of-freedom estimate. It uses the studentized-range distribution to account for examining multiple pairs.

For groups \(i\) and \(j\), write \(v_i=s_i^2/n_i\) and \(v_j=s_j^2/n_j\). The pairwise degrees of freedom use the Welch–Satterthwaite approximation. The Games–Howell statistic divides the absolute difference in sample means by the square root of half the estimated variance of that difference.

df_{ij}= \frac{(v_i+v_j)^2} {\displaystyle v_i^2/(n_i-1)+v_j^2/(n_j-1)}, \qquad q_{ij}= \frac{|\bar{x}_i-\bar{x}_j|} {\sqrt{(v_i+v_j)/2}}

For \(k\) groups, compare \(q_{ij}\) with the upper-tail critical value from the studentized-range distribution with \(k\) groups and \(df_{ij}\) degrees of freedom. A simultaneous interval for \(\mu_i-\mu_j\), using familywise confidence level approximately \(1-\alpha\), is

(\bar{x}_i-\bar{x}_j) \ \mathbin{\pm}\ q_{1-\alpha;\,k,\,df_{ij}} \sqrt{\frac{v_i+v_j}{2}}

The critical value must match the chosen familywise level, number of groups, and pair-specific degrees of freedom. The Games–Howell procedure is an approximate method; its multiple-comparison control is not based on assuming one shared variance.

Worked Example: Comparing the Three Clinic Pairs

Use the clinic summaries from the preceding calculation and compare all three pairs with Games–Howell at familywise \(\alpha=0.05\). The differences below are expressed as the first clinic’s mean minus the second clinic’s mean, in days.

Clinics A and B. Here \(v_A=10/8=1.2500\) and \(v_B=16/12=1.3333\), in days squared. Therefore,

\sqrt{\frac{v_A+v_B}{2}} =\sqrt{\frac{1.2500+1.3333}{2}} =\sqrt{\frac{2.5833}{2}} \approx1.137\text{ days}, \qquad q_{AB}=\frac{|12-15|}{1.1365}\approx2.64

The pair-specific degrees of freedom are \((2.5833)^2/\{1.2500^2/7+1.3333^2/11\}\approx17.34\). A studentized-range critical value for three groups and about 17 degrees of freedom is approximately \(3.62\). Since \(2.64<3.62\), the A–B difference is not statistically significant at the familywise 5% level. The corresponding approximate simultaneous interval for \(\mu_A-\mu_B\) is \( -3\mathbin{\pm}3.62(1.1365)\), or about \((-7.11,1.11)\) days.

Clinics B and C. Here \(v_B=1.3333\) and \(v_C=9/20=0.4500\), so the standardizer is \(\sqrt{(1.3333+0.4500)/2}\approx0.9443\) days. The degrees of freedom are \((1.7833)^2/\{1.3333^2/11+0.4500^2/19\}\approx18.45\), and \(q_{BC}=|15-18|/0.9443\approx3.18\). Using a critical value of about \(3.60\), this comparison is not significant. The approximate simultaneous interval for \(\mu_B-\mu_C\) is \( -3\mathbin{\pm}3.60(0.9443)\), or about \((-6.40,0.40)\) days.

Clinics A and C. Here \(v_A=1.2500\) and \(v_C=0.4500\), giving a standardizer of \(\sqrt{(1.2500+0.4500)/2}=\sqrt{0.8500}\approx0.9220\) days. The degrees of freedom are \((1.7000)^2/\{1.2500^2/7+0.4500^2/19\}\approx12.36\), and \(q_{AC}=|12-18|/0.9220\approx6.51\). The critical value with about 12 degrees of freedom is approximately \(3.77\), so this comparison is significant. Its approximate simultaneous interval for \(\mu_A-\mu_C\) is \( -6\mathbin{\pm}3.77(0.9220)\), or about \((-9.48,-2.52)\) days.

In this hypothetical example, the pairwise analysis identifies evidence of a difference between clinics A and C, but not between A and B or between B and C under the familywise procedure. The negative A-minus-C interval indicates that the mean time for clinic A is lower than the mean time for clinic C; it does not by itself explain the difference or establish a causal effect.

How Imbalance Affects Interpretation

Unequal sample sizes affect the precision of the group means and therefore the test’s weights and pairwise standard errors. In the example, clinic C has the largest sample and a relatively modest sample variance, so it receives the largest Welch weight. Clinic A has only eight observations and a variance of 10 days squared, so its mean is estimated less precisely. The weighted mean \(\bar{x}_w\) consequently differs from the simple average of the three clinic means. That difference is not an error: the weighted mean is part of Welch’s test calculation, not a claim that one clinic is more important clinically.

Unequal sample sizes do not change the meaning of \(\mu_i-\mu_j\): it remains the population mean difference for the two specified clinics, measured in days here. They do change the uncertainty attached to that estimated difference through \(s_i^2/n_i+s_j^2/n_j\). When a small group also has a large variance, its mean can be particularly imprecise. Report the group sizes, means, and standard deviations so readers can see the pattern rather than relying on a p-value alone.

Common Mistakes

Using ordinary ANOVA after seeing clearly unequal spreads. The usual pooled-variance F test assumes a common within-group variance. If that assumption is doubtful, Welch’s test is a direct mean-comparison alternative.
Reading an omnibus rejection as proof that every pair differs. Welch’s test establishes evidence that not all means are equal; pairwise comparisons are needed to identify which differences are supported.
Using pooled standard errors for Games–Howell. Each pair has its own variance estimate and degrees of freedom. Use \(s_i^2/n_i\) and \(s_j^2/n_j\), not a single pooled variance.
Treating unequal group sizes as a reason to interpret the weighted mean as the overall clinical target. Welch’s weighted mean is a calculation device. The scientific quantities of interest are usually the individual clinic means and their pairwise differences.
Claiming that a small p-value proves a treatment caused the difference. The analysis compares means. Causal interpretation depends on the study design, such as whether patients were randomly assigned to clinics or treatments.

Key takeaway: When independent groups have unequal spreads, Welch’s one-way ANOVA compares their means without imposing a common variance. Use Games–Howell for unequal-variance pairwise follow-up, report the group summaries and sample sizes, and interpret the omnibus and pairwise conclusions separately.

Check Your Understanding

Try these questions before reviewing the worked calculations.

What assumptions does Welch’s one-way ANOVA relax compared with the ordinary one-way F test, and what important assumptions does it still require?
Why does a group with a larger sample size and a smaller variance tend to receive a larger Welch weight?
In the clinic example, what does rejecting the omnibus null hypothesis establish, and what does it not establish?
Why do Games–Howell comparisons use pair-specific degrees of freedom?
If the mean difference is reported in days, what units do the standard error and confidence interval have?