How Balance and Sample Size Affect Robustness

Why Balance and Sample Size Matter

One-way ANOVA compares the means of several groups by contrasting variation between the group means with variation within the groups. Its usual F test is derived under a model with independent, normally distributed errors having the same variance in every group. Real data can depart from those assumptions. The practical question is not simply whether a diagnostic plot looks perfectly ideal, but whether a departure is likely to make the test's conclusions unreliable.

Two features affect that sensitivity: balance, meaning that groups have equal or similar sample sizes, and the number of observations within each group. Adequate group sizes often make the F test less sensitive to moderate nonnormality. Balance can reduce the effect of unequal variances. Neither feature guarantees a valid test: severe skew, influential outliers, dependence, or a strong combination of imbalance and unequal variances can still be consequential.

The Model and the Role of Group Sizes

Write an observation in group \(i\) as \(Y_{ij}=\mu_i+\varepsilon_{ij}\), where \(\mu_i\) is that group's population mean and \(\varepsilon_{ij}\) is its error. The usual one-way ANOVA model assumes independent errors with mean zero, a common variance \(\sigma^2\), and a normal distribution. Under the null hypothesis \(H_0:\mu_1=\cdots=\mu_k\), these assumptions give the usual F reference distribution. Independence is primarily justified by how the observations were collected; balance cannot repair dependence.

The standard ANOVA statistic is \(F=MS_{\text{between}}/MS_{\text{within}}\). The numerator measures variation among the observed group means, and the denominator estimates variation within groups. When variances differ, the group means have different sampling variances: approximately \(\sigma_i^2/n_i\) if group \(i\)'s population variance is \(\sigma_i^2\). Thus both the variance and the sample size of each group affect how much its mean fluctuates. The way those features line up matters.

A useful justification comes from comparing the expected mean squares under equal population means but possibly unequal variances. Assume independent observations with common mean \(\mu\) under \(H_0\), while group \(i\) has variance \(\sigma_i^2\). The grand mean is \(\bar{Y}_{..}=\sum_i n_i\bar{Y}_{i.}/N\), and the between-groups mean square is \(MS_{\text{between}}=\sum_i n_i(\bar{Y}_{i.}-\bar{Y}_{..})^2/(k-1)\). Independence gives the following expectation:

The pooled within-groups mean square averages the group variances with weights determined by their residual degrees of freedom:

If every group has size \(n\), then \(N=kn\). Substitution shows that both expectations equal the simple average of the group variances:

This matching helps explain why balance often reduces sensitivity to unequal variances: the two parts of the F ratio are, on average, on a compatible scale under the null. It is a justification for a robustness tendency, not a proof that the statistic has an exact F distribution when variances differ. Matching expectations alone does not determine the full distribution of a ratio.

Nonnormality: What Larger Group Sizes Can—and Cannot—Do

The normality condition concerns the errors within each group, not whether the combined raw outcomes look normal. When group sizes are adequate, the sampling distributions of the group means often become closer to normal even if individual outcomes are moderately skewed, provided the observations are independent and the population variances are finite. This can make inference based on the group means less sensitive to moderate nonnormality.

There is no universal sample-size cutoff that makes a distribution harmless. The group sizes should be considered separately: a large \(N\) can conceal one small group if the other groups are much larger. Strong skew, heavy tails, or an influential observation can have a substantial effect on a mean and on the within-group variance, especially when a group is small. Increasing sample size also does not fix dependence or automatically remove unequal-variance problems caused by imbalance.

Why Imbalance and Unequal Variances Can Conflict

With unequal sizes, the pooled within-groups variance estimate gives more weight to groups with more residual degrees of freedom. But the variability of a difference between two group means depends on each group's variance divided by its own sample size. If a small group has a large variance, that group mean can be especially unstable while contributing relatively little weight to the pooled variance estimate. The resulting F statistic may be too large too often under \(H_0\), making the test anti-conservative: its actual Type I error rate can exceed \(\alpha\). If the larger variances are instead in the larger groups, the usual test can be conservative, with an actual Type I error rate below \(\alpha\).

A two-group comparison makes the mismatch visible. For two groups, the ordinary one-way ANOVA F statistic is the square of the pooled-variance two-sample t statistic. Under equal means but unequal variances, the actual variance of the difference in sample means is \(\sigma_1^2/n_1+\sigma_2^2/n_2\). The pooled variance estimate has expected value \(\{(n_1-1)\sigma_1^2+(n_2-1)\sigma_2^2\}/(n_1+n_2-2)\). If the design is unbalanced, the usual test's estimated uncertainty can therefore disagree with the actual uncertainty, depending on which group has the larger variance.

Using the Diagnostics in Context

Residual Q-Q plots and residual-versus-fitted plots help assess distributional shape and changing spread; study design informs whether independence is plausible. These diagnostics are useful alongside sample sizes, not substitutes for them. In particular, inspect each group’s size and residual pattern rather than relying on the total \(N\) or on a single pooled plot. A variance test can add evidence about spread differences, but no diagnostic test turns an assumption into a certainty.

A practical assessment should consider the combination of evidence. Similar group sizes and adequate observations per group are reassuring when departures from normality are moderate and no influential residuals are apparent. Strongly unequal spreads combined with substantial imbalance warrant much greater caution, especially when the smallest group is the most variable. Robustness is about the reliability of the procedure for the situation at hand, not a label that can be assigned from balance alone.