Why Calculate Welch’s Test by Hand?
When group variances differ, the classical one-way ANOVA test can be unreliable, particularly when group sizes also differ. Welch’s one-way test addresses this concern by weighting groups according to their sample sizes and variances, then adjusting the test statistic and its reference degrees of freedom. A hand calculation makes those adjustments visible: it shows how the group summaries contribute to the result rather than hiding them behind a software output.
This tutorial works from group means, sample sizes, and sample variances. The numerical summaries in the examples are hypothetical. They are enough to calculate the test, but they cannot by themselves establish that observations were independent or that the group distributions were approximately normal; those conditions must be justified from the study design and data.
Model, Hypotheses, and Calculation Formula
A standard model for this setting is \(Y_{ij}=\mu_i+\varepsilon_{ij}\), where observations are independent within and across groups and the errors in group \(i\) have mean zero and variance \(\sigma_i^2\). For the usual small-sample F calibration, the observations in each group are assumed to come from a normal population. The variances \(\sigma_i^2\) may differ from group to group. Each group must have at least two observations so that its sample variance can be calculated.
For each group, calculate the weight \(w_i=n_i/s_i^2\), and let \(W=\sum_{i=1}^k w_i\). The weighted grand mean is \(\bar{x}_w=\sum_{i=1}^k w_i\bar{x}_i/W\). Then calculate the correction quantity \(C\), the Welch statistic \(F_W\), and its approximate denominator degrees of freedom as follows:
The numerator measures the weighted separation of the sample means. The denominator correction and approximate \(df_2\) account for uncertainty associated with unequal group variances. The omnibus p-value is the upper-tail probability \(P(F_{df_1,df_2}\geq F_W)\) under the approximate F reference distribution. In practice, software or an F distribution calculator supplies this probability; the examples report rounded approximations.
A Useful Check on the Weighted Mean
The weighted mean is not an arbitrary center. It is the value \(c\) that minimizes the weighted squared deviations \(\sum_i w_i(\bar{x}_i-c)^2\). To see this, write each difference as \(\bar{x}_i-c=(\bar{x}_i-\bar{x}_w)+(\bar{x}_w-c)\), square, and sum with weights. The cross term is zero because \(\sum_i w_i(\bar{x}_i-\bar{x}_w)=0\). Therefore:
Equality holds when \(c=\bar{x}_w\). This identity also gives a practical arithmetic check: after calculating the weighted mean, the weighted deviations from it should sum to zero, apart from rounding.
Worked Calculations
Worked Example: Three Groups With Different Variances
Suppose three independent groups have the hypothetical summaries below. Assume they come from independent observations and that each group’s population distribution is reasonably modeled as normal. The group variances differ, so use Welch’s test rather than pooling them.
| Group | \(n_i\) | \(\bar{x}_i\) (units) | \(s_i^2\) (units squared) |
|---|---|---|---|
| 1 | 8 | 10 | 4 |
| 2 | 12 | 13 | 9 |
| 3 | 10 | 15 | 16 |
First calculate the weights and their total:
The weighted mean is \((2(10)+1.3333(13)+0.625(15))/3.9583=11.800\) units. The weighted squared deviations are \(2(10-11.8)^2=6.48\), \(1.3333(13-11.8)^2=1.92\), and \(0.625(15-11.8)^2=6.40\). Their sum is \(14.80\), so the numerator before correction is \(14.80/(3-1)=7.400\).
Next calculate \(C\), using the group-specific \(n_i-1\) denominators:
With \(k=3\), the correction factor is \(1+\{2(3-2)/(3^2-1)\}C=1+0.25(0.1537)=1.0384\). Thus \(F_W=7.400/1.0384=7.126\). The numerator degrees of freedom are \(df_1=2\); the approximate denominator degrees of freedom are \(df_2=8/(3(0.1537))=17.35\). The upper-tail probability for \(F_{2,17.35}\) at 7.126 is approximately \(p=0.0055\).
At significance level \(\alpha=0.05\), this result is evidence against equal population means: the observed group-mean separation is unusual under the equal-means null model and this approximate F calibration. The test is omnibus; it does not by itself identify which means differ.
Worked Example: A Small Statistic and a Larger P-Value
Consider another hypothetical set of independent groups, with approximately normal observations within each group. The sample sizes are unequal, and the variances differ; the summaries are in arbitrary measurement units.
| Group | \(n_i\) | \(\bar{x}_i\) | \(s_i^2\) |
|---|---|---|---|
| 1 | 6 | 20 | 1 |
| 2 | 10 | 20.5 | 4 |
| 3 | 14 | 21 | 9 |
The weights are \(6/1=6\), \(10/4=2.5\), and \(14/9=1.5556\), giving \(W=10.0556\). The weighted mean is \(\{6(20)+2.5(20.5)+1.5556(21)\}/10.0556\approx20.279\). The weighted squared-deviation sum is approximately \(6(20-20.279)^2+2.5(20.5-20.279)^2+1.5556(21-20.279)^2=1.398\). Dividing by \(k-1=2\) gives a numerator of approximately \(0.6989\).
The correction quantity is:
The correction factor is \(1+0.25(0.1502)=1.0376\), so \(F_W=0.6989/1.0376=0.6736\). The degrees of freedom are \(df_1=2\) and \(df_2=8/(3(0.1502))\approx17.75\). The approximate upper-tail p-value is \(p\approx0.52\).
At \(\alpha=0.05\), this test does not provide sufficient evidence to reject equal population means. It does not establish that the population means are equal: a large p-value is not proof of the null hypothesis.
Worked Example: A Result Near the 0.05 Threshold
A final hypothetical example uses three independent, approximately normal groups with six observations each. The equal sample sizes do not remove the need to account for the unequal variances.
| Group | \(n_i\) | \(\bar{x}_i\) (units) | \(s_i^2\) (units squared) |
|---|---|---|---|
| 1 | 6 | 5 | 1 |
| 2 | 6 | 7 | 4 |
| 3 | 6 | 8 | 9 |
The weights are \(6\), \(1.5\), and \(6/9=0.6667\), so \(W=8.1667\). The weighted mean is \(\{6(5)+1.5(7)+0.6667(8)\}/8.1667\approx5.612\) units. The weighted squared-deviation sum is approximately \(6(5-5.612)^2+1.5(7-5.612)^2+0.6667(8-5.612)^2=8.939\). Dividing by two gives \(4.470\).
Here,
The correction factor is \(1+0.25(0.3160)=1.0790\), giving \(F_W=4.470/1.0790\approx4.142\). The degrees of freedom are \(df_1=2\) and \(df_2=8/(3(0.3160))\approx8.438\). The upper-tail p-value is approximately \(0.056\).
At \(\alpha=0.05\), the result does not cross the chosen significance threshold. The p-value is close to that threshold, but it is not a measure of the size or practical importance of the mean differences. Report the statistic, approximate degrees of freedom, p-value, and group summaries rather than describing this result as proof of no group effect.
A Reliable Hand-Calculation Sequence
For every group, record \(n_i\), \(\bar{x}_i\), and \(s_i^2\). Confirm that each sample variance is positive and that every \(n_i\) is at least 2.
Compute \(w_i=n_i/s_i^2\), \(W=\sum_iw_i\), and \(\bar{x}_w=\sum_iw_i\bar{x}_i/W\). Retain enough digits to limit rounding error.
Find the weighted squared-deviation sum, \(C\), the correction factor, \(F_W\), \(df_1=k-1\), and approximate \(df_2\).
Use the approximate \(F_{df_1,df_2}\) distribution. State the conclusion in context and compare the p-value with the prespecified \(\alpha\).
Common Mistakes and Why the Calculation Matters
A frequent error is to calculate \(w_i=s_i^2/n_i\) rather than \(n_i/s_i^2\). Welch’s weights are larger for groups with greater precision: increasing \(n_i\) increases the weight, while increasing \(s_i^2\) decreases it. Check that the weight formula follows that logic before proceeding.
Another mistake is to stop after finding the weighted numerator and call it an ordinary ANOVA F statistic. Welch’s test also requires the correction factor and its approximate denominator degrees of freedom. Using \(df_2\) as though it were the pooled ANOVA residual degrees of freedom changes the reference distribution and the p-value.
Finally, do not treat the p-value as the probability that \(H_0\) is true. It is the probability, under the equal-means null model and the approximate F calibration, of obtaining a statistic at least as large as the one observed. Welch’s test matters because it provides an omnibus comparison of means when equal population variances are not a reasonable assumption, while still requiring attention to independence, distributional shape, and the limits of a single summary test.
Check Your Understanding
Use the formulas and reasoning above to answer these questions.
- For a group with \(n_i=12\) and \(s_i^2=6\), calculate its Welch weight. What happens to the weight if the sample size stays fixed but the variance doubles?
- Why is the weighted mean used in the weighted squared-deviation calculation?
- With three groups and \(C=0.20\), calculate the Welch correction factor and approximate denominator degrees of freedom.
- A Welch test has \(p=0.18\) and \(\alpha=0.05\). State the decision and explain why it does not prove that the population means are equal.
- Can group means, sample sizes, and variances alone confirm independence and normality? Explain what additional information would be needed.