Balanced Samples Under Unequal Variances

Why Balance Helps, but Does Not Settle the Variance Question

Equal group sizes are often a useful feature of a one-way ANOVA design. They prevent the group with the largest variance from also being systematically given the largest or smallest weight solely because it has more observations. That can lessen some distortions caused by unequal variances and unequal allocation.

But balance is not the same as equal variability. If every group has the same sample size, the groups can still have very different population variances. The classical one-way F test pools within-group variation and relies on a common-variance model. Welch’s one-way test instead allows group variances to differ. Comparing the two tests in balanced examples clarifies what balance can protect against—and what it cannot.

Definition: A one-way design with \(k\) groups is balanced when every group has the same sample size, \(n_i=n\) for all \(i\). It is heteroscedastic when the group population variances are not all equal. A design can be balanced and heteroscedastic at the same time.

Models and Tests Being Compared

Suppose independent observations are collected from \(k\) groups. In group \(i\), let the population mean be \(\mu_i\), the population variance be \(\sigma_i^2\), the sample mean be \(\bar{x}_i\), the sample variance be \(s_i^2\), and the sample size be \(n_i\). Both tests address \(H_0:\mu_1=\cdots=\mu_k\) against the alternative that not all group means are equal.

For the classical one-way ANOVA F test, the standard model assumes independent observations, normally distributed observations within each group, and one common population variance: \(\sigma_1^2=\cdots=\sigma_k^2=\sigma^2\). The test pools the within-group variation into \(MS_{\text{within}}\) and compares it with \(MS_{\text{between}}\). Under \(H_0\) and these assumptions, the statistic has an \(F\) distribution with \(k-1\) and \(N-k\) degrees of freedom, where \(N=\sum_i n_i\).

Welch’s one-way test also assumes independent observations and, for its standard approximation, normally distributed observations within each group. It does not require a common population variance. It weights group means inversely according to their estimated variances and uses an approximate F reference distribution with a Welch–Satterthwaite denominator degrees of freedom.

Key distinction: Balance concerns the \(n_i\). Equal variances concern the \(\sigma_i^2\). Setting all \(n_i\) equal does not imply, estimate, or establish that the \(\sigma_i^2\) are equal.

For calculations below, the classical sums of squares are \(SS_{\text{between}}=\sum_i n_i(\bar{x}_i-\bar{x})^2\) and \(SS_{\text{within}}=\sum_i(n_i-1)s_i^2\), where \(\bar{x}\) is the grand mean. Then \(F=MS_{\text{between}}/MS_{\text{within}}\), with \(MS_{\text{between}}=SS_{\text{between}}/(k-1)\) and \(MS_{\text{within}}=SS_{\text{within}}/(N-k)\).

For Welch’s test, define \(w_i=n_i/s_i^2\), \(W=\sum_i w_i\), and the weighted mean \(\bar{x}_w=\sum_iw_i\bar{x}_i/W\). Its statistic and approximate denominator degrees of freedom are:

F_W = \frac{\displaystyle \frac{1}{k-1}\sum_{i=1}^k w_i(\bar{x}_i-\bar{x}_w)^2} {\displaystyle 1+\frac{2(k-2)}{k^2-1}\sum_{i=1}^k \frac{(1-w_i/W)^2}{n_i-1}}, \qquad df_2 \approx \frac{k^2-1}{3\displaystyle\sum_{i=1}^k\frac{(1-w_i/W)^2}{n_i-1}}.

The numerator gives greater weight to means estimated more precisely. The denominator correction and approximate degrees of freedom account for the fact that the group variances are not assumed equal. These formulae are included so the examples can be checked directly; the central issue here is how their results compare in balanced designs.

A Short Justification: What Equal Allocation Changes

Under \(H_0\), suppose the group means are equal and the observations are independent. The sample mean in group \(i\) has variance \(\sigma_i^2/n_i\). When all group sizes equal \(n\), the expected between-group sum of squares is \((k-1)\sum_i\sigma_i^2/k\). The expected within-group mean square is also \(\sum_i\sigma_i^2/k\), because \(E[SS_{\text{within}}]=\sum_i(n-1)\sigma_i^2\) and \(N-k=k(n-1)\). Thus, with equal allocation, both mean squares have the same expectation under the equal-means null, even when the variances differ.

This helps explain why balance can lessen some problems: unlike an unbalanced design, it does not make the larger group sizes systematically amplify the contribution of particular group variances. But matching expectations does not make the classical F statistic exactly follow its usual F distribution. With unequal variances, the numerator and denominator need not have the distributions or independence properties required for that exact reference distribution. Equal sample sizes therefore do not repair the classical test’s equal-variance assumption.

Worked Example: Correcting the Classical Test in a Balanced Design

Consider hypothetical measurements in points from three independent groups, each with \(n_i=5\). Suppose the sample means are \(0,0,5\) points and the sample variances are \(4,4,27\) points squared. Assume normal observations within each group for this illustration. The large difference in sample variances is a reason to question the common-variance model; balance alone does not remove that concern.

Classical calculation. The grand mean is \(\bar{x}=(5(0)+5(0)+5(5))/15=5/3\) points. Therefore,

SS_{\text{between}} =5\left[2(0-5/3)^2+(5-5/3)^2\right] =5\left(\frac{50}{9}+\frac{100}{9}\right) =\frac{250}{3} \approx 83.33\text{ points}^2.

The within-group sum of squares uses \(n_i-1=4\) for each sample variance:

SS_{\text{within}}=4(4)+4(4)+4(27)=140\text{ points}^2.

Here \(df_{\text{between}}=2\), \(df_{\text{within}}=15-3=12\), \(MS_{\text{between}}=83.333/2=41.667\), and \(MS_{\text{within}}=140/12=11.667\). Thus,

F=\frac{83.333/2}{140/12} =\frac{25}{7} \approx 3.571,\qquad df=(2,12).

The upper-tail p-value is approximately \(0.0607\). At \(\alpha=0.05\), this classical test does not reject \(H_0\). This conclusion is conditional on the classical model; the unequal sample variances make its common-variance assumption questionable.

Welch calculation. The weights are \(w_1=5/4=1.25\), \(w_2=1.25\), and \(w_3=5/27\approx0.1852\), giving \(W\approx2.6852\) and \(\bar{x}_w=10/29\approx0.3448\) points. The weighted sum of squared deviations is

\sum_i w_i(\bar{x}_i-\bar{x}_w)^2 =\frac{3625}{841}\approx4.310.

The correction sum is \(\sum_i(1-w_i/W)^2/(n_i-1)\approx0.3595\). Hence the correction factor is \(1+0.25(0.3595)\approx1.0899\), and \(df_2\approx8/[3(0.3595)]=7.417\). The resulting statistic is

F_W=\frac{(4.310/2)}{1.0899}\approx1.977,\qquad df\approx(2,7.417).

Its approximate upper-tail p-value is \(0.205\). At the 0.05 level, Welch’s test also does not reject equal means. The p-values differ because the tests use different variance assumptions and reference distributions. Neither test result establishes that the means or variances are equal.

Variance Differences Remain in a Balanced Design

The next example isolates the variance issue without needing a test calculation. It also shows why “equal sample size” should not be paraphrased as “equal precision.” For a group mean, the sampling variance is \(\sigma_i^2/n_i\). With the same \(n_i\), a group with a larger population variance still has a less precise mean.

Worked Example: Equal Sample Sizes, Different Precision

Suppose three hypothetical groups each have six independent observations. Their population variances are \(1\), \(4\), and \(25\) units squared. For a comparison of group 1 with group 3, the variance of the difference in sample means is the sum of the two sample-mean variances:

\operatorname{Var}(\bar{x}_1-\bar{x}_3) =\frac{\sigma_1^2}{n_1}+\frac{\sigma_3^2}{n_3} =\frac{1}{6}+\frac{25}{6} =\frac{26}{6} \approx4.333\text{ units}^2.

The common-variance model would replace both variances with one pooled \(\sigma^2\). If that common value were represented here by the simple average \((1+4+25)/3=10\), the corresponding variance calculation for two means would be \(10(1/6+1/6)=3.333\) units squared, not \(4.333\). This is an illustration of the mismatch, not a claim that the average is always the correct estimate or that this comparison alone determines an ANOVA result. The key point is that equal \(n_i\) does not make the group means equally precise when the variances differ.

In an actual analysis, the population variances are unknown. The sample variances summarize observed variability, but they do not prove the population variances are equal or unequal. The design’s balance cannot answer that question by itself.

A Nonsignificant Test Does Not Test the Variances

A one-way test of means addresses differences among \(\mu_i\); it is not a test that the \(\sigma_i^2\) are equal. In particular, a small mean difference can produce little evidence against equal means even when sample spreads look quite different. Conversely, a significant mean test does not by itself diagnose heteroscedasticity.

Worked Example: Equal Sample Means Do Not Imply Equal Variances

Imagine three hypothetical independent groups, each with six observations, measuring an outcome in millimetres. Their sample means are all \(12\) mm, while their sample variances are \(1\), \(9\), and \(25\) mm squared. Assume normal observations within groups for applying the standard test procedures.

For classical ANOVA, \(SS_{\text{between}}=\sum_i6(12-12)^2=0\). The within-group sum of squares is \(5(1)+5(9)+5(25)=175\) mm squared, so \(MS_{\text{within}}=175/(18-3)=11.667\) mm squared. Therefore \(F=0/(11.667)=0\), with \(df=(2,15)\), and the upper-tail p-value is \(1\).

For Welch’s test, all three weighted deviations from \(\bar{x}_w\) are zero because the sample means are identical. Its numerator is consequently zero, so \(F_W=0\) and its upper-tail p-value is also \(1\). These results provide no evidence against equal population means. The substantially different sample variances remain a separate feature of the data; neither p-value establishes equal population variances.

Practical Interpretation and Common Pitfall

Balance is valuable because it prevents unequal group sizes from interacting with unequal variances in the same way they can in an unbalanced design. It can make the classical F test less sensitive to some forms of heteroscedasticity, especially when group sizes are moderate and distributions are reasonably well behaved. It is not a guarantee of a well-calibrated test in every balanced design, and it does not make Welch’s unequal-variance adjustment unnecessary by definition.

A common mistake is: “All groups have the same sample size, so the variance assumption is satisfied and the classical test is safe.” That reasoning confuses allocation with spread. Equal allocation makes the \(n_i\) equal; the classical assumption concerns the population variances \(\sigma_i^2\). The correction is to examine the groupwise variability and use the assumptions of the intended test, rather than treating balance as evidence of homoscedasticity.

The practical question is not whether balance is good or bad. It is what balance contributes to the comparison. It removes one source of imbalance in how group variances affect the analysis, while leaving the variance differences themselves in place. The choice between classical and Welch procedures should therefore consider the design, the observed groupwise spreads, and the assumptions behind each test.

Key takeaway: Equal group sizes can lessen some distortions associated with unequal allocation and unequal variances. They neither equalize the group variances nor guarantee that the classical pooled F reference distribution is appropriate.

Check Your Understanding

Use the assumptions and calculations above to answer these questions.

Can a design be balanced and heteroscedastic at the same time? State what each term describes.
In a balanced design, what happens to the variance of a sample mean when its group population variance increases but its sample size stays fixed?
For the first worked example, why is the classical p-value not evidence that the equal-variance assumption holds?
What is the main way equal allocation can lessen the effect of variance heterogeneity on the classical test?
In the third worked example, what question do the two ANOVA tests address, and what distinct feature do the sample variances describe?