Tutorials › Multivariate Statistics (Graduate) › Pooling Covariance Estimates Across Classes

Home  /  Tutorials  /  Discriminant Analysis Theory

Discriminant Analysis Theory · Tutorial 586 of 975

Pooling Covariance Estimates Across Classes

Learn how class scatter matrices combine into a shared covariance estimate, why two standard divisors differ, and how that choice enters LDA scores.

Intermediate 10 min read

What You'll Learn

  • Define within-class scatter and the pooled covariance estimator for multiple classes.
  • Derive the maximum-likelihood divisor and distinguish it from the unbiased divisor.
  • Prove the expectation of pooled within-class scatter under a common covariance model.
  • Combine class sample covariances using their degrees of freedom rather than equal weights.
  • Insert a pooled covariance estimate into Gaussian LDA scores and interpret the result.
  • Identify when a pooled estimate cannot be inverted for LDA.

Why Pool Within-Class Covariation?

In linear discriminant analysis, the class distributions are modeled with a shared covariance matrix. The training data, however, are observed class by class, so each class has its own sample scatter. Pooling combines these within-class scatters into one estimate of the common covariance. This uses information from all classes while respecting the shared-covariance assumption.

Pooling is not the same as taking an unweighted average of the class covariance estimates. Classes may have different sample sizes, and each class covariance has a different number of degrees of freedom. There is also a distinction between the maximum-likelihood estimate and the usual degrees-of-freedom-corrected estimate. Both are useful, but they answer different estimation questions.

Definition: Suppose there are \(K\) classes. Class \(k\) contains \(n_k\) independent observations \(X_{ki}\in\mathbb{R}^p\), for \(i=1,\ldots,n_k\), with sample mean \(\bar X_k=n_k^{-1}\sum_{i=1}^{n_k}X_{ki}\). The within-class scatter matrix is $$ W_k=\sum_{i=1}^{n_k}(X_{ki}-\bar X_k)(X_{ki}-\bar X_k)'. $$ Let \(N=\sum_{k=1}^K n_k\) and \(W=\sum_{k=1}^K W_k\). Under a common covariance model, the pooled degrees-of-freedom-corrected covariance estimate is \(S_{\mathrm{pool}}=W/(N-K)\), provided \(N>K\). The pooled maximum-likelihood estimate is \(\hat\Sigma_{\mathrm{ML}}=W/N\), when the likelihood has a positive-definite maximizer.

A prime denotes transpose. Each \(W_k\) records variation around the mean of its own class, not variation of the class means around a grand mean. The pooled estimate therefore describes within-class covariance; it does not include between-class separation.

How the Class Estimates Are Weighted

When \(n_k>1\), the usual sample covariance within class \(k\) is \(S_k=W_k/(n_k-1)\). Rewriting the pooled estimate gives

$$ S_{\mathrm{pool}} =\frac{\sum_{k=1}^K W_k}{N-K} =\frac{\sum_{k=1}^K(n_k-1)S_k}{\sum_{k=1}^K(n_k-1)}. $$

Thus, class \(k\)'s covariance estimate receives weight proportional to \(n_k-1\), not equal weight across classes. The weighting follows from combining the scatter matrices: a class with more observations contributes more within-class information. If a class contains only one observation, its scatter is zero; its individual sample covariance is undefined, but the pooled scatter remains well-defined.

Worked Example: Pooling Three Class Covariances

Suppose \(K=3\), with sample sizes \(n_1=4\), \(n_2=6\), and \(n_3=5\). The observed within-class scatter matrices are

$$ W_1=\begin{pmatrix}3&1\\1&2\end{pmatrix},\qquad W_2=\begin{pmatrix}8&2\\2&6\end{pmatrix},\qquad W_3=\begin{pmatrix}4&-1\\-1&3\end{pmatrix}. $$

The total sample size is \(N=4+6+5=15\), and the total scatter is

$$ W=W_1+W_2+W_3 =\begin{pmatrix}15&2\\2&11\end{pmatrix}. $$

There are \(N-K=15-3=12\) within-class degrees of freedom. Therefore,

$$ S_{\mathrm{pool}} =\frac{1}{12}\begin{pmatrix}15&2\\2&11\end{pmatrix} =\begin{pmatrix}1.25&0.1667\\0.1667&0.9167\end{pmatrix}. $$

The maximum-likelihood estimate instead divides by \(N=15\):

$$ \hat\Sigma_{\mathrm{ML}} =\begin{pmatrix}1&0.1333\\0.1333&0.7333\end{pmatrix}. $$

For comparison, the class sample covariances are \(W_1/3\), \(W_2/5\), and \(W_3/4\). Weighting these by \(3,5,4\) and dividing by \(3+5+4=12\) recovers \(S_{\mathrm{pool}}\). Giving the three class estimates equal weight would not.

Why the Degrees-of-Freedom Divisor Is \(N-K\)

Assume that, conditional on their class labels, observations are independent and \(X_{ki}\sim N(\mu_k,\Sigma)\), with a potentially different mean \(\mu_k\) for each class but one common positive-definite covariance \(\Sigma\). The following calculation establishes the expectation of the pooled scatter.

Result: Under the independent common-covariance model above, with \(n_k\geq 1\), $$ E[W_k]=(n_k-1)\Sigma,\qquad E[W]=(N-K)\Sigma. $$ Consequently, if \(N>K\), \(S_{\mathrm{pool}}=W/(N-K)\) is unbiased for \(\Sigma\).

To prove the first identity, write \(\varepsilon_{ki}=X_{ki}-\mu_k\) and \(\bar\varepsilon_k=n_k^{-1}\sum_i\varepsilon_{ki}\). Expanding the scatter around the class mean gives the exact matrix identity

$$ W_k=\sum_{i=1}^{n_k}\varepsilon_{ki}\varepsilon_{ki}'- n_k\bar\varepsilon_k\bar\varepsilon_k'. $$

Indeed, \(X_{ki}-\bar X_k=\varepsilon_{ki}-\bar\varepsilon_k\); expanding the sum of outer products yields two cross terms, each equal to \(n_k\bar\varepsilon_k\bar\varepsilon_k'\), and one added term equal to \(n_k\bar\varepsilon_k\bar\varepsilon_k'\). Their net contribution is the subtracted term shown. Since each error has mean zero and covariance \(\Sigma\), the first sum has expectation \(n_k\Sigma\). Independence gives \(\operatorname{Cov}(\bar\varepsilon_k)=\Sigma/n_k\), so \(E[n_k\bar\varepsilon_k\bar\varepsilon_k']=\Sigma\). Subtracting proves \(E[W_k]=(n_k-1)\Sigma\). Summing over the \(K\) classes gives \(E[W]=\sum_k(n_k-1)\Sigma=(N-K)\Sigma\), which proves the result. \(\square\)

The subtraction of \(K\) reflects the estimation of one mean in each class. The correction does not depend on the number of variables \(p\); it counts the within-class residual degrees of freedom.

Maximum Likelihood Uses a Different Divisor

The degrees-of-freedom correction gives an unbiased estimate, but it is not the maximum-likelihood estimate when the class means are unknown and estimated from the same data. For the Gaussian model, the maximum-likelihood covariance estimate divides the within-class scatter by \(N\).

Result: Suppose the observations are independent with \(X_{ki}\sim N(\mu_k,\Sigma)\), where each \(\mu_k\) is unrestricted and \(\Sigma\) is common and positive definite. If \(W\) is positive definite, the joint maximum-likelihood estimates are \(\hat\mu_k=\bar X_k\) and \(\hat\Sigma_{\mathrm{ML}}=W/N\).

For fixed \(\Sigma\), the part of the log likelihood involving \(\mu_k\) is proportional to minus the sum of squared Mahalanobis residuals in class \(k\). The identity

$$ \sum_{i=1}^{n_k}(X_{ki}-\mu_k)(X_{ki}-\mu_k)' =W_k+n_k(\bar X_k-\mu_k)(\bar X_k-\mu_k)' $$

shows that this sum is minimized at \(\mu_k=\bar X_k\). Substituting all class means into the joint log likelihood leaves, up to a constant,

$$ \ell(\Sigma)=-\frac{N}{2}\log|\Sigma|-\frac{1}{2}\operatorname{tr}(\Sigma^{-1}W). $$

Differentiating with respect to the symmetric matrix \(\Sigma\) gives the likelihood equation

$$ -\frac{N}{2}\Sigma^{-1} +\frac{1}{2}\Sigma^{-1}W\Sigma^{-1}=0, $$

which implies \(W=N\Sigma\), and hence the stated estimate. To verify that this stationary point is the maximum when \(W\) is positive definite, write \(\Sigma=W^{1/2}BW^{1/2}\), where \(B\) is positive definite. Apart from a constant, the log likelihood becomes \(-\frac{N}{2}\log|B|-\frac{1}{2}\operatorname{tr}(B^{-1})\). If \(\lambda_1,\ldots,\lambda_p\) are the positive eigenvalues of \(B\), this expression is a sum of terms \(-\frac{N}{2}\log\lambda_j-\frac{1}{2\lambda_j}\), each uniquely maximized at \(\lambda_j=1/N\). Thus \(B=I/N\), giving \(\Sigma=W/N\). \(\square\)

Since \(E[W]=(N-K)\Sigma\), the maximum-likelihood estimate has expectation \(E[\hat\Sigma_{\mathrm{ML}}]=\frac{N-K}{N}\Sigma\), so it is downward biased when \(K>0\). The distinction is therefore not a disagreement about the observed scatter: it is a difference in the divisor and in the estimation criterion. If \(W\) is singular, the ordinary positive-definite covariance maximum need not exist; one should not treat \(W/N\) as an invertible covariance estimate.

Worked Example: Checking Bias With Two Classes

Suppose \(K=2\), \(n_1=3\), \(n_2=4\), and the common covariance is

$$ \Sigma=\begin{pmatrix}4&1\\1&2\end{pmatrix}. $$

The expectation result gives \(E[W_1]=2\Sigma\) and \(E[W_2]=3\Sigma\). Hence \(E[W]=5\Sigma\), since \(N-K=7-2=5\). The pooled corrected estimator has expectation

$$ E[S_{\mathrm{pool}}]=\frac{E[W]}{5}=\Sigma =\begin{pmatrix}4&1\\1&2\end{pmatrix}. $$

The maximum-likelihood estimator divides by \(N=7\), so

$$ E[\hat\Sigma_{\mathrm{ML}}] =\frac{5}{7}\Sigma =\begin{pmatrix}20/7&5/7\\5/7&10/7\end{pmatrix}. $$

This verifies the bias factor directly: the ML estimate is not unbiased, even though it is the likelihood maximizer under the stated model.

Using the Pooled Estimate in LDA

In Gaussian LDA, the common covariance in the class log-density is replaced by a pooled estimate when the covariance is unknown. For class \(k\), with estimated mean \(\bar X_k\), prior probability \(\pi_k>0\), and a chosen positive-definite pooled estimate \(S\), the class score, up to terms common to all classes, is

$$ \delta_k(x)=x'S^{-1}\bar X_k-\frac{1}{2}\bar X_k'S^{-1}\bar X_k+\log\pi_k. $$

Here \(S\) may be \(S_{\mathrm{pool}}\) or \(\hat\Sigma_{\mathrm{ML}}\), depending on the estimation convention. The same covariance estimate must be used across classes for this shared-covariance LDA model. The score makes clear why invertibility matters: if \(S\) is singular, the displayed inverse does not exist. The choice of divisor can affect the scores and resulting decisions, particularly when priors or class means make the comparison sensitive to the scale of the covariance.

Worked Example: A Divisor Can Change an LDA Decision

Consider two classes with sample sizes \(n_1=n_2=4\), observed within-class scatters \(W_1=4I\) and \(W_2=4I\), means \(\bar X_1=(0,0)'\) and \(\bar X_2=(2,0)'\), and prior probabilities \(\pi_1=0.8\) and \(\pi_2=0.2\). The total scatter is \(W=8I\), \(N=8\), and \(N-K=6\). Thus the corrected pooled estimate is \(S_{\mathrm{pool}}=(4/3)I\), whereas the ML estimate is \(\hat\Sigma_{\mathrm{ML}}=I\). Both are positive definite.

For an observation \(x=(1.8,0)'\), compare class 2 with class 1. Their score difference is

$$ \delta_2(x)-\delta_1(x) =x'S^{-1}(\bar X_2-\bar X_1) -\frac12\left(\bar X_2'S^{-1}\bar X_2-\bar X_1'S^{-1}\bar X_1\right) +\log\frac{\pi_2}{\pi_1}. $$

Using \(S=S_{\mathrm{pool}}\), we have \(S^{-1}=(3/4)I\). The data-dependent terms are \((1.8,0)(3/4)(2,0)'=2.7\) and \(\frac12(2,0)(3/4)(2,0)'=1.5\). Also, \(\log(0.2/0.8)=\log(0.25)\approx-1.3863\). Therefore,

$$ \delta_2(x)-\delta_1(x)=2.7-1.5-1.3863=-0.1863. $$

The pooled corrected estimate gives the larger score to class 1. With \(S=\hat\Sigma_{\mathrm{ML}}=I\), the corresponding terms are \(3.6\) and \(2\), so

$$ \delta_2(x)-\delta_1(x)=3.6-2-1.3863=0.2137. $$

Now class 2 has the larger score. This example does not imply that either divisor is universally preferable for every classification goal. It shows that the estimation convention should be stated: the two estimates differ by a scale factor, and LDA scores use the inverse covariance.

Common Mistake and Practical Checks

  • Wrong reasoning: “There are \(K\) class covariance matrices, so the pooled estimate is their unweighted average.” This gives a small class as much influence as a large class and does not generally equal the combined within-class estimate. Correction: sum the scatter matrices \(W_k\), or weight each \(S_k\) by \(n_k-1\).
  • Wrong reasoning: “The unbiased estimate and the ML estimate both use the same data, so they must have the same divisor.” They do not: their divisors are \(N-K\) and \(N\), respectively. Correction: state whether the goal is unbiased estimation or likelihood maximization with class means estimated.
  • Wrong reasoning: “A positive-semidefinite pooled scatter is automatically usable in ordinary LDA.” A singular matrix has no inverse. Correction: check that the selected pooled covariance estimate is positive definite before using the usual inverse-based score.
  • Wrong reasoning: “Pooling includes the variation among class means.” The scatter \(W\) is centered separately within each class. Correction: distinguish within-class covariance estimation from between-class variation.
Key takeaway: Combine within-class scatter matrices by addition. With unknown class means, their total residual degrees of freedom are \(N-K\), giving the unbiased pooled estimate \(W/(N-K)\); the common-covariance Gaussian maximum-likelihood estimate is \(W/N\). Use the chosen estimate consistently in LDA, and verify that it can be inverted.

Check Your Understanding

Use the definitions and results above to check your reasoning.

  1. For \(K=4\) classes with total sample size \(N=50\), what are the divisors for the pooled unbiased estimate and the pooled ML estimate?
  2. If class sample covariances are \(S_k\), what weights recover the pooled unbiased covariance estimate?
  3. Why does the within-class degrees-of-freedom correction subtract \(K\), rather than subtracting one?
  4. Under what condition in the ML derivation does \(W/N\) give an ordinary positive-definite covariance maximizer?
  5. Why might using the ML divisor rather than the degrees-of-freedom divisor change an LDA classification?

Previous

Next

Back to Tutorials
Clinical Biostats