Estimating a Common Covariance From Two Groups

Why Pool Within-Group Covariance?

When two populations have different mean vectors but share a covariance matrix, each group provides information about that common covariance. The central question is how to combine the groups’ observed within-group variation. The usual pooled estimator adds their within-group scatter matrices and divides by the total residual degrees of freedom. This weighting reflects how much within-group information each sample contributes.

The common-covariance assumption is substantive: it says the groups may differ in location, but their variability and cross-variable relationships are the same. Here we derive the estimator and prove its unbiasedness under independent Gaussian sampling. Gaussianity is useful for exact finite-sample inference, but the expectation calculation itself needs only finite second moments.

Setup and Definition

Suppose group \(g\), for \(g=1,2\), contains \(n_g\) independent observations \(X_{g1},\ldots,X_{gn_g}\), each a \(p\)-dimensional column vector. Assume the two groups are independent and \(X_{gi}\sim N_p(\mu_g,\Sigma)\), where the mean vectors \(\mu_1\) and \(\mu_2\) may differ, while the covariance matrix \(\Sigma\) is common. Assume \(n_1,n_2\geq 2\). Write \(\bar X_g=n_g^{-1}\sum_{i=1}^{n_g}X_{gi}\).

Definition: The within-group scatter matrix for group \(g\) is \(W_g=\sum_{i=1}^{n_g}(X_{gi}-\bar X_g)(X_{gi}-\bar X_g)^\top\). Its sample covariance matrix is \(S_g=W_g/(n_g-1)\). The degrees-of-freedom-weighted pooled covariance estimator is \(\displaystyle S_p=\frac{W_1+W_2}{n_1+n_2-2}=\frac{(n_1-1)S_1+(n_2-1)S_2}{n_1+n_2-2}\).

Each group mean is estimated from its own observations, so the within-group residuals sum to zero. The resulting residual degrees of freedom are \(n_g-1\) for group \(g\), giving \(n_1+n_2-2\) in total. The weights in \(S_p\) are those degrees of freedom, not the group sample sizes themselves.

Why the Scatter Has \(n_g-1\) Degrees of Freedom

The reason for the denominator can be seen from an exact matrix identity. Fix a group and temporarily write its observations as \(X_i\), its mean as \(\mu\), and its sample mean as \(\bar X\). Since \(X_i-\mu=(X_i-\bar X)+(\bar X-\mu)\), expansion of the outer products gives

\sum_{i=1}^{n}(X_i-\mu)(X_i-\mu)^\top = \sum_{i=1}^{n}(X_i-\bar X)(X_i-\bar X)^\top +n(\bar X-\mu)(\bar X-\mu)^\top.

The cross-product terms vanish because \(\sum_{i=1}^{n}(X_i-\bar X)=0\). Rearranging shows that centering at the estimated mean removes the contribution \(n(\bar X-\mu)(\bar X-\mu)^\top\) from the total variation around the true mean. The expectation calculation below makes the resulting \(n-1\) factor precise.

Result: If \(X_1,\ldots,X_n\) are independent with common mean \(\mu\), common covariance \(\Sigma\), and finite second moments, then \(\displaystyle E\!\left[\sum_{i=1}^{n}(X_i-\bar X)(X_i-\bar X)^\top\right]=(n-1)\Sigma\).

Proof. Take expectations in the identity above. Each term in the sum on the left has expectation \(\Sigma\), so the expected left side is \(n\Sigma\). Independence gives \(\operatorname{Cov}(\bar X)=\Sigma/n\), and \(E[\bar X]=\mu\). Therefore

E\!\left[n(\bar X-\mu)(\bar X-\mu)^\top\right] =n\operatorname{Cov}(\bar X) =n\frac{\Sigma}{n} =\Sigma.

Subtracting this expectation from \(n\Sigma\) yields \((n-1)\Sigma\), as claimed. \(\square\)

In particular, \(E[S_g]=\Sigma\) for each group under the stated common-covariance model. This expectation statement does not require Gaussian observations; the independence and finite-second-moment conditions suffice.

Unbiasedness of the Pooled Estimator

Theorem: Under the independent sampling model above, with a common covariance matrix \(\Sigma\) and \(n_1,n_2\geq 2\), the pooled estimator satisfies \(E[S_p]=\Sigma\).

Proof. Apply the within-group scatter result to each group separately. It gives \(E[W_1]=(n_1-1)\Sigma\) and \(E[W_2]=(n_2-1)\Sigma\). By linearity of expectation,

E[S_p] =\frac{E[W_1]+E[W_2]}{n_1+n_2-2} =\frac{(n_1-1)\Sigma+(n_2-1)\Sigma}{n_1+n_2-2} =\Sigma.

This proof does not need independence between the two scatter matrices; independence between groups is part of the sampling model, but linearity alone gives the displayed expectation. The Gaussian assumption is likewise not needed for unbiasedness. It matters for the exact distributional theory used in subsequent inference. \(\square\)

If the group covariances are instead \(\Sigma_1\) and \(\Sigma_2\), the same calculation gives \(\displaystyle E[S_p]=\frac{(n_1-1)\Sigma_1+(n_2-1)\Sigma_2}{n_1+n_2-2}\). That is a weighted average of two different matrices, not an estimate that is unbiased for a single common covariance unless the common-covariance assumption holds.

Worked Calculations

Worked Example: Combining Two Reported Covariance Matrices

Suppose \(n_1=8\), \(n_2=13\), and the groups’ sample covariance matrices are \(S_1=\begin{pmatrix}4&1\\1&2\end{pmatrix}\) and \(S_2=\begin{pmatrix}9&3\\3&5\end{pmatrix}\). Find the pooled estimate.

The residual degrees of freedom are \(7\) and \(12\), for a total of \(19\). Multiply each sample covariance by its degrees of freedom, add, and divide by \(19\):

S_p=\frac{7S_1+12S_2}{19} =\frac{1}{19} \left[ \begin{pmatrix}28&7\\7&14\end{pmatrix} + \begin{pmatrix}108&36\\36&60\end{pmatrix} \right] = \begin{pmatrix} 136/19&43/19\\ 43/19&74/19 \end{pmatrix} \approx \begin{pmatrix} 7.158&2.263\\ 2.263&3.895 \end{pmatrix}.

For a check, the off-diagonal entry is \((7\cdot1+12\cdot3)/19=43/19\). The result is symmetric, as a covariance matrix must be. Its determinant is \((136\cdot74-43^2)/19^2=14055/361>0\), and its first diagonal entry is positive, so this particular estimate is positive definite.

Worked Example: Why the Weights Matter

Let \(n_1=4\), \(n_2=10\), with \(S_1=\begin{pmatrix}1&0.2\\0.2&2\end{pmatrix}\) and \(S_2=\begin{pmatrix}5&1\\1&4\end{pmatrix}\). The degrees of freedom are \(3\) and \(9\), so

S_p =\frac{3S_1+9S_2}{12} =\frac{1}{12} \begin{pmatrix}48&9.6\\9.6&42\end{pmatrix} =\begin{pmatrix}4&0.8\\0.8&3.5\end{pmatrix}.

An unweighted average would be \(\frac{1}{2}(S_1+S_2)=\begin{pmatrix}3&0.6\\0.6&3\end{pmatrix}\). That average is also unbiased when both group covariance matrices estimate the same \(\Sigma\), because it is an average of two unbiased estimators. It is not, however, the pooled within-group scatter divided by its total degrees of freedom: it gives equal influence to groups with 3 and 9 residual degrees of freedom. The conventional pooled estimator combines each observed within-group residual contribution on the same scale.

Worked Example: Computing the Estimate From Observations

In group 1, let the two-dimensional observations be \((0,0)\), \((2,0)\), and \((1,2)\). In group 2, let them be \((0,0)\), \((0,2)\), \((2,0)\), and \((2,2)\). Compute \(S_p\) directly from the observations.

For group 1, \(\bar X_1=(1,2/3)^\top\). Its centered observations are \((-1,-2/3)^\top\), \((1,-2/3)^\top\), and \((0,4/3)^\top\). Summing their outer products gives \(W_1=\begin{pmatrix}2&0\\0&8/3\end{pmatrix}\).

For group 2, \(\bar X_2=(1,1)^\top\). The centered observations are \((-1,-1)^\top\), \((-1,1)^\top\), \((1,-1)^\top\), and \((1,1)^\top\). The diagonal sums are each \(4\), and the off-diagonal products sum to \(1-1-1+1=0\), so \(W_2=\begin{pmatrix}4&0\\0&4\end{pmatrix}\).

The total residual degrees of freedom are \((3-1)+(4-1)=5\). Therefore

S_p=\frac{W_1+W_2}{5} =\frac{1}{5} \begin{pmatrix}6&0\\0&20/3\end{pmatrix} = \begin{pmatrix}6/5&0\\0&4/3\end{pmatrix}.

The calculation uses deviations from each group’s own mean. Centering both groups at one overall mean would mix between-group mean differences into the covariance estimate, which is not the within-group pooled estimator.

Common Pitfalls and Interpretation

Using sample sizes instead of residual degrees of freedom. The weights are \(n_g-1\), because each group’s mean is estimated before its within-group scatter is formed.
Confusing pooled covariance with covariance around a grand mean. The pooled estimator uses each group’s own mean. Between-group separation is not within-group variation.
Assuming the unweighted average is necessarily biased. If both \(S_1\) and \(S_2\) are unbiased for the same \(\Sigma\), their unweighted average is unbiased too. The degrees-of-freedom-weighted estimator is the one formed by aggregating all within-group scatter and dividing by total residual degrees of freedom.
Treating unbiasedness as proof of the common-covariance assumption. The expectation result relies on that assumption. It does not establish that the two populations actually have equal covariance matrices.

The pooled estimate is useful because it summarizes within-group variation under a shared covariance model while allowing different group means. Its role is therefore both computational and interpretive: the estimate is meaningful as a common covariance only when pooling is appropriate for the scientific model. Unbiasedness describes its average over repeated samples under that model; it does not guarantee that any one observed estimate equals \(\Sigma\).

Check Your Understanding

Use the definitions and derivation above to check your understanding.

For group sizes \(n_1=6\) and \(n_2=11\), what are the two residual degrees of freedom and the denominator of the pooled estimator?
State the within-group scatter expectation result and identify the assumptions it needs for unbiasedness.
Why does centering observations at their group sample mean lead to \(n_g-1\), rather than \(n_g\), in the expectation of within-group scatter?
If \(E[S_1]=E[S_2]=\Sigma\), is their unweighted average unbiased? Explain how it differs from the pooled estimator.
What does the pooled estimator estimate on average if the two groups instead have different covariance matrices \(\Sigma_1\) and \(\Sigma_2\)?