Unequal Group Sizes Under Covariance Heterogeneity

Why Group-Size Imbalance Matters

A pooled covariance estimate combines information across groups. When group covariances are equal, this pooling is natural: each group estimates the same matrix. When covariances differ, however, the pooled estimate is a weighted average, while the uncertainty in a difference of group means depends on each group’s covariance divided by its own sample size. If sample sizes are unequal, those two weightings need not agree.

The practical consequence is not simply that a pooled estimate is “too small” or “too large.” Its error can vary by direction: it may understate uncertainty for one linear combination of outcomes and overstate it for another. This matters directly to multivariate mean tests, which combine information across directions.

Two Sources of Uncertainty, Two Weighting Schemes

Definition: Suppose two independent groups provide observations \(X_{gi}\in\mathbb{R}^p\), where \(g=1,2\) and \(i=1,\ldots,n_g\). Within group \(g\), assume the observations are identically distributed with mean vector \(\mu_g\) and covariance matrix \(\Sigma_g\). Write \(\bar X_g\) for the sample mean and \(S_g\) for the usual unbiased sample covariance matrix. The mean-difference estimand is \(\delta=\mu_1-\mu_2\), estimated by \(\bar X_1-\bar X_2\).

Independence between groups gives the covariance of the estimated difference without requiring normality:

\operatorname{Cov}(\bar X_1-\bar X_2) =\frac{\Sigma_1}{n_1}+\frac{\Sigma_2}{n_2}.

The pooled within-group covariance estimate is

S_p=\frac{(n_1-1)S_1+(n_2-1)S_2}{n_1+n_2-2}.

Under a common-covariance model, the usual pooled estimate of the covariance of the mean difference is \(S_p(1/n_1+1/n_2)\). Under heterogeneity, its expectation is instead

E\left[S_p\left(\frac{1}{n_1}+\frac{1}{n_2}\right)\right] = \left(\frac{1}{n_1}+\frac{1}{n_2}\right) \frac{(n_1-1)\Sigma_1+(n_2-1)\Sigma_2}{n_1+n_2-2}.

This expression generally differs from the true covariance \(\Sigma_1/n_1+\Sigma_2/n_2\). The distinction is between the weights used to combine within-group covariances and the weights that determine sampling variability of the two means.

Worked Example: A High-Variance Smaller Group

Suppose \(n_1=80\), \(n_2=20\), and the two outcome coordinates have within-group covariance matrices \( \Sigma_1=\operatorname{diag}(1,4)\) and \( \Sigma_2=\operatorname{diag}(9,1)\). Calculate the true covariance of the mean difference and the expected covariance estimate obtained by pooling.

The true covariance is

\frac{\Sigma_1}{80}+\frac{\Sigma_2}{20} = \operatorname{diag}\left(\frac{1}{80}+\frac{9}{20}, \frac{4}{80}+\frac{1}{20}\right) = \operatorname{diag}(0.4625,0.10).

The expected pooled covariance is

E[S_p] = \frac{79\Sigma_1+19\Sigma_2}{98} = \operatorname{diag}\left(\frac{250}{98},\frac{335}{98}\right).

Multiplying by \(1/80+1/20=0.0625\), the expected pooled estimate of the mean-difference covariance is approximately \(\operatorname{diag}(0.1594,0.2136)\). It is far below the true variance in the first coordinate and above it in the second. The smaller group has the larger variance in the first coordinate, but pooling gives its covariance relatively little weight.

What Imbalance Does to the Pooled Approximation

The weighting pattern is especially clear when both sample sizes grow with a fixed positive ratio. Let \(N=n_1+n_2\), with \(n_1/N\to\pi_1\) and \(n_2/N\to\pi_2\), where \(\pi_1,\pi_2>0\) and \(\pi_1+\pi_2=1\). The expected pooled covariance approaches \(\pi_1\Sigma_1+\pi_2\Sigma_2\). Thus the pooled covariance-based estimate has leading form

\left(\frac{1}{n_1}+\frac{1}{n_2}\right) (\pi_1\Sigma_1+\pi_2\Sigma_2) \ \approx\ \frac{\Sigma_1}{n_2}+\frac{\Sigma_2}{n_1}.

The true covariance is \(\Sigma_1/n_1+\Sigma_2/n_2\): the denominators attached to the two covariance matrices are interchanged in the pooled approximation. Their leading difference is

\left(\frac{1}{n_2}-\frac{1}{n_1}\right)(\Sigma_1-\Sigma_2).

This matrix difference need not be positive or negative semidefinite. Consequently, there is no general claim that pooling always underestimates, or always overestimates, the covariance matrix. The direction of the discrepancy depends on how the covariance matrices differ.

Result: Under covariance heterogeneity, the leading pooled covariance-based estimate is \(\Sigma_1/n_2+\Sigma_2/n_1\), whereas the actual covariance of the mean difference is \(\Sigma_1/n_1+\Sigma_2/n_2\). If the group sizes are equal, these expressions agree. If the covariance matrices are equal, they also agree, regardless of group sizes.

The balanced-design statement concerns the covariance calculation, not exact test calibration. Equal sample sizes make the expected pooled estimate of the mean-difference covariance agree with the true covariance even if \(\Sigma_1\ne\Sigma_2\). They do not make the pooled covariance matrix an estimate of a common covariance, nor do they establish the exact pooled \(F\) distribution. The standard exact pooled two-sample calibration requires multivariate normal observations with a common covariance matrix.

Worked Example: Balanced Groups With Different Covariances

Let \(n_1=n_2=30\), with \(\Sigma_1=\operatorname{diag}(1,4)\) and \(\Sigma_2=\operatorname{diag}(9,1)\). The actual covariance of the mean difference is

\frac{\Sigma_1+\Sigma_2}{30} =\operatorname{diag}\left(\frac{10}{30},\frac{5}{30}\right) =\operatorname{diag}(0.3333,0.1667).

Because the group sizes and covariance degrees of freedom match, \(E[S_p]=(\Sigma_1+\Sigma_2)/2=\operatorname{diag}(5,2.5)\). Therefore

E\left[S_p\left(\frac{1}{30}+\frac{1}{30}\right)\right] = \operatorname{diag}(5,2.5)\frac{2}{30} = \operatorname{diag}(0.3333,0.1667).

The expected covariance estimates match, but the groups still have different covariance matrices. The equality of these expectations is not a license to use an exact common-covariance \(F\) calibration under heterogeneity.

Separate-Covariance Wald Inference

A direct way to respect the group-specific covariance structure is to estimate the covariance of the mean difference separately:

\widehat V_{\mathrm{sep}}=\frac{S_1}{n_1}+\frac{S_2}{n_2}.

When this matrix is nonsingular, the separate-covariance Wald statistic for \(H_0:\mu_1-\mu_2=0\) is

W_{\mathrm{sep}}=(\bar X_1-\bar X_2)^\mathsf{T} \widehat V_{\mathrm{sep}}^{-1} (\bar X_1-\bar X_2).

This statistic uses each group’s estimated covariance with its own sample-size divisor. In finite samples it does not generally have the exact pooled Hotelling \(F\) distribution under heterogeneity. Welch–James-type approximations address finite-sample calibration; the following result explains the separate Wald statistic’s large-sample reference distribution.

Theorem: Suppose \(p\) is fixed, the two groups are independent, and observations within each group are identically distributed with covariance matrices \(\Sigma_g\). Assume finite second moments, \(n_g/N\to\pi_g\in(0,1)\), and \(\Omega=\Sigma_1/\pi_1+\Sigma_2/\pi_2\) is positive definite. Under \(H_0:\mu_1=\mu_2\), \(W_{\mathrm{sep}}\xrightarrow{d}\chi^2_p\).

Scale the difference of sample means.
The multivariate central limit theorem for independent groups gives \(\sqrt N(\bar X_1-\bar X_2)\xrightarrow{d}N(0,\Omega)\) under the null, because \(N/n_g\to1/\pi_g\).

Scale the covariance estimate.
Finite second moments imply \(S_g\xrightarrow{p}\Sigma_g\). Hence \(N\widehat V_{\mathrm{sep}}=(N/n_1)S_1+(N/n_2)S_2\xrightarrow{p}\Omega\).

Apply Slutsky’s theorem.
Since \(\Omega\) is positive definite, inversion is continuous at \(\Omega\). The quadratic form using the convergent scaled difference and covariance estimate therefore converges to the squared norm of a \(p\)-variate standard normal vector, which has a \(\chi^2_p\) distribution. \(\square\)

Worked Example: Comparing the Two Covariance Calculations

Use the \(n_1=80,n_2=20\) covariance matrices from the first example as observed sample covariance matrices, and suppose the observed mean difference is \((1,1)^\mathsf{T}\). The separate estimate is \(\operatorname{diag}(0.4625,0.10)\), giving

W_{\mathrm{sep}}=\frac{1^2}{0.4625}+\frac{1^2}{0.10} \approx 2.1622+10=12.1622.

The pooled estimate of the mean-difference covariance is \(\operatorname{diag}(0.1594,0.2136)\), so the corresponding quadratic form is approximately

W_{\mathrm{pool}}=\frac{1^2}{0.1594}+\frac{1^2}{0.2136} \approx 6.272+4.681=10.953.

The statistics differ because the covariance estimates allocate uncertainty differently across coordinates. These values alone do not supply a valid p-value: calibration must also reflect the covariance assumptions and finite-sample method.

Design Implications and a Common Pitfall

When group sizes are fixed by design, the imbalance is not merely a cosmetic feature of the data table. It determines how strongly each group’s covariance contributes to the uncertainty of its mean. In a highly imbalanced study, the smaller group’s covariance can be particularly consequential because its contribution is divided by the smaller sample size. A pooled estimate may obscure that contribution by weighting covariance matrices largely according to group size.

This is one reason to plan group sizes with plausible covariance differences in mind, and to report group-specific covariance estimates when heterogeneity is scientifically credible. Separate-covariance methods avoid imposing a common covariance structure in the estimated variance, although small samples can make group-specific estimates unstable. The choice of test and calibration should reflect both the design and the covariance assumptions.

Common mistake: “The pooled covariance is a weighted average, so it must give a reasonable variance estimate for the mean difference.” A weighted average by itself does not ensure the right weighting for that difference.
Correction: Compare the target covariance \(\Sigma_1/n_1+\Sigma_2/n_2\) with the pooled construction. Under imbalance and heterogeneity, the latter can mismatch in different directions.
Common mistake: “Balanced group sizes make the pooled test exact.” They align the expected covariance calculations, but exact pooled calibration still requires the common-covariance normal model.

Check Your Understanding

Use the covariance formulas and assumptions in this tutorial to answer the following.

Write the covariance of \(\bar X_1-\bar X_2\) for independent groups with unequal covariance matrices.
In the large-sample pooled approximation, which group covariance is divided by \(n_1\), and which is divided by \(n_2\)?
Why can imbalance cause underestimation in one outcome direction and overestimation in another?
What covariance assumption is needed, in addition to multivariate normality, for the exact pooled two-sample calibration?
Under what conditions does the separate-covariance Wald statistic have an asymptotic \(\chi^2_p\) null distribution?