Why Unequal Variances Matter

Why the Variance Assumption Matters

When comparing two population means, the uncertainty in the difference between the sample means depends on variability in both groups and on how many observations each group contributes. A calculation that treats the two populations as having one common variance may seem reasonable when the sample standard deviations look similar. But if the population variances differ, that common-variance calculation can misrepresent the standard error, particularly when the sample sizes are unequal.

The practical issue is not simply whether one group is more variable than the other. It is which group is more variable and how large that group’s sample is. A high-variance group with a small sample contributes substantially to the actual uncertainty in its sample mean, but may contribute relatively little to a pooled variance estimate. Reversing which group has the large sample can reverse the direction of the error.

Here \(n_1\) and \(n_2\) are the sample sizes, while \(s_1^2\) and \(s_2^2\) are the sample variances. The term “standard error” refers to the estimated standard deviation of the sampling distribution of the difference in sample means. Both formulas have the units of the original measurement; the variances themselves have squared units.

The Two Variance Calculations

For independent samples, the variance of a difference of sample means is the sum of the variances of the two sample means. Since the variance of a sample mean is the population variance divided by its sample size, the actual sampling variance is

The sample variances replace the unknown population variances to estimate this quantity, giving the Welch standard error. This calculation lets each group’s variance contribute in proportion to its own sample size.

Pooling follows a different idea. It estimates a single variance by averaging the two sample variances with weights based on their degrees of freedom:

This weighted average is suitable as an estimate of a common population variance when the two populations truly share that variance and the samples are independent. It is then inserted into \(SE_{\text{pooled}}=\sqrt{s_p^2(1/n_1+1/n_2)}\). If the population variances are unequal, however, the weighted average is not generally the right variance estimate for both sample means. In particular, its weights are based on sample degrees of freedom, whereas the contribution of a group to the variance of a mean difference depends on its variance divided by its sample size.

A Useful Special Case: Equal Sample Sizes

If the sample sizes are equal, the two standard-error calculations have the same numerical value, even if the sample variances differ. This is an algebraic coincidence that can obscure the importance of the common-variance assumption.

To see why, set \(n_1=n_2=n\). The pooled variance is the average of the two sample variances, with equal degrees-of-freedom weights. Therefore,

The numerical match does not show that the population variances are equal. Nor does it establish that every part of a pooled inferential procedure is appropriate. It only shows that, with equal sample sizes, these two formulas for the estimated standard error agree.

Unequal Sample Sizes Can Change the Result

When sample sizes differ, the pooled calculation gives more weight to the variance from the larger sample. That may work well if the variance assumption is reasonable. But if the larger sample happens to come from the lower-variance population, pooling can give too little weight to the high variance that most affects the smaller group’s sample mean. If the larger sample instead comes from the higher-variance population, pooling can push the standard error in the other direction.

What the Examples Establish

Under unequal population variances, the sampling variance of the difference is \(\sigma_1^2/n_1+\sigma_2^2/n_2\). Each group’s contribution is its variance divided by its own sample size. By contrast, the pooled calculation first combines the sample variances into one number and then applies that number to both sample sizes. If the variances differ and the sample sizes are unequal, this substitution can distort the estimated sampling variance.

The examples also show why the direction of distortion is not fixed. A high variance in the smaller group can be muted by pooling, as in the second example. A high variance in the larger group can inflate the pooled result, as in the third. When sample sizes are equal, the standard errors match algebraically, but the common-variance assumption remains a separate question.

These calculations presume two independent groups and independent observations within each sample. The pooled calculation adds the assumption of equal population variances. The Welch calculation does not require that equality assumption. As with other two-sample t inference, the sampling design and the shapes of the group distributions also matter for reliable inference; a standard-error formula alone cannot repair dependence, severe outliers, or an inappropriate sampling design.

Common Mistake

Wrong reasoning: “The larger sample always makes the standard error smaller, so it is safe to pool the variances.”

Correction: A larger sample reduces the contribution of its own variance through division by its sample size. But pooling combines variances before using the sample-size factors, and the resulting standard error can be too small or too large if the population variances differ. Check how variability and sample size line up; do not treat a pooled estimate as harmless just because one group has many observations.

A related mistake is to conclude that unequal variances never matter when the sample sizes are equal. Equal sample sizes make the two standard-error formulas match, but do not make the population variances equal or validate the pooled model. Keep the arithmetic result separate from the assumptions that justify a procedure.