Shared Versus Component-Specific Covariances

Why Decide Whether Covariances Are Shared?

In a Gaussian clustering model, each component has a mean and a covariance matrix. The mean locates a component; its covariance describes the spread and dependence of observations around that location. A modeling decision is whether components should have one covariance matrix in common or a different covariance matrix for each component. This choice affects both the shapes the model can represent and how much information is used to estimate each shape.

Sharing can stabilize estimates when components have limited data and similar dispersion patterns. But it also imposes a substantive restriction: if the components truly differ in spread, shape, or orientation, a common covariance cannot represent those differences. Conversely, estimating separate matrices offers flexibility but asks each component to supply enough information to estimate its own covariance reliably.

The corresponding mixture density is \(f(x)=\sum_{k=1}^K\pi_k\phi_d(x;\mu_k,\Sigma_k)\), where \(\pi_k>0\), \(\sum_k\pi_k=1\), and \(\phi_d\) is the \(d\)-variate normal density. The shared model sets all the \(\Sigma_k\) equal; the component-specific model does not. This tutorial compares these two full-matrix choices, not the additional restrictions imposed by spherical, diagonal, or other covariance families.

What Covariance Sharing Means Geometrically

A Gaussian component’s equal-density contours are ellipsoids centered at its mean. The covariance eigenvectors give their principal directions, and the eigenvalues determine their squared axis lengths at a fixed density level. If all components share \(\Sigma\), their ellipsoids have the same shape, size, and orientation: they are translated copies centered at different means. They may overlap, and their mixture weights may differ, but the covariance matrix does not allow one component to be more elongated or differently oriented than another.

With component-specific matrices, each component can have its own ellipsoid. One cluster can be broad and nearly circular while another is narrow and elongated; their principal directions need not agree. That flexibility can describe genuine heterogeneity, but it also makes each estimate depend on its component’s own observations or effective membership.

Estimating Covariance When Component Memberships Are Known

Suppose observations \(x_1,\ldots,x_n\in\mathbb{R}^d\) have known component labels \(z_i\in\{1,\ldots,K\}\). Write \(n_k\) for the number assigned to component \(k\), \(\bar{x}_k\) for its sample mean, and \(W_k=\sum_{i:z_i=k}(x_i-\bar{x}_k)(x_i-\bar{x}_k)^\mathsf{T}\) for its within-component scatter matrix. Assume each \(n_k>0\). The formulas below use maximum likelihood denominators, not the unbiased sample-covariance denominators.

For separate covariances, each component’s likelihood depends on its own observations and its own covariance. For the shared model, all components contribute within-component scatter to one estimate. The pooled estimate is not, in general, an unweighted average of the separate estimates: it weights each by its component sample size.

Why the Pooled Estimate Uses Within-Component Scatter

A shared covariance describes variation around each component’s own mean. It does not describe how far apart the component means are. This distinction is essential: a data set can have widely separated cluster centers even when observations are tightly concentrated around each center.

To prove the decomposition, for an observation in component \(k\), write \(x_i-\bar{x}=(x_i-\bar{x}_k)+(\bar{x}_k-\bar{x})\). Expand the outer product and sum within component \(k\). The two cross-product sums vanish because \(\sum_{i:z_i=k}(x_i-\bar{x}_k)=0\). The remaining terms are \(W_k\) and \(n_k(\bar{x}_k-\bar{x})(\bar{x}_k-\bar{x})^\mathsf{T}\). Summing over \(k\) gives the result. \(\square\)

Derivation of the Shared Maximum-Likelihood Estimate

For fixed labels and component means at their Gaussian maximum-likelihood estimates \(\bar{x}_k\), the part of the log-likelihood depending on a shared positive-definite \(\Sigma\) is \( \ell(\Sigma)=-\frac{n}{2}\log|\Sigma|-\frac{1}{2}\operatorname{tr}(\Sigma^{-1}W)+C \), where \(W=\sum_k W_k\) and \(C\) does not depend on \(\Sigma\). Assume \(W\) is positive definite, so \(\bar{W}=W/n\) is positive definite. Set \(C_\Sigma=\bar{W}^{-1/2}\Sigma\bar{W}^{-1/2}\), which is positive definite. Determinant and trace identities give \( \log|\Sigma|=\log|\bar{W}|+\log|C_\Sigma| \) and \( \operatorname{tr}(\Sigma^{-1}W)=n\operatorname{tr}(C_\Sigma^{-1}) \).

Let \(\lambda_1,\ldots,\lambda_d\) be the positive eigenvalues of \(C_\Sigma\). Up to constants independent of \(\Sigma\), maximizing the log-likelihood is equivalent to minimizing \(\sum_{j=1}^d(\log\lambda_j+1/\lambda_j)\). For every \(\lambda>0\), the function \(g(\lambda)=\log\lambda+1/\lambda\) has derivative \(g'(\lambda)=(\lambda-1)/\lambda^2\); it decreases up to \(1\) and increases after \(1\). Its unique minimum is \(g(1)=1\). Consequently the likelihood is maximized only when every \(\lambda_j=1\), which makes \(C_\Sigma=I\) and \(\Sigma=\bar{W}=W/n\). This proves the shared estimate under the stated positive-definiteness condition. \(\square\)

The same argument applied to the observations in one component, replacing \(n\) and \(W\) by \(n_k\) and \(W_k\), gives \(\hat{\Sigma}_k=W_k/n_k\) when \(W_k\) is positive definite. If a scatter matrix is singular, the unrestricted Gaussian covariance likelihood need not have a finite positive-definite maximizer; the formula should not be treated as a valid positive-definite estimate in that case.

Unknown Memberships and the Pooling Trade-Off

In a mixture model, component labels are not observed. Given current parameter values, let \(r_{ik}\) be the posterior responsibility of component \(k\) for observation \(x_i\), with \(r_{ik}\geq0\) and \(\sum_k r_{ik}=1\). In an expectation-maximization update, define \(N_k=\sum_i r_{ik}\) and \(\bar{x}_k=N_k^{-1}\sum_i r_{ik}x_i\), assuming \(N_k>0\). Maximizing the expected complete-data Gaussian log-likelihood gives the shared update \( \hat{\Sigma}=\frac{1}{n}\sum_k\sum_i r_{ik}(x_i-\bar{x}_k)(x_i-\bar{x}_k)^\mathsf{T} \), or, with component-specific covariances, \( \hat{\Sigma}_k=\frac{1}{N_k}\sum_i r_{ik}(x_i-\bar{x}_k)(x_i-\bar{x}_k)^\mathsf{T} \). These are maximizers of the EM covariance-update objective for the current responsibilities; they are not a claim that the observed mixture likelihood has a closed-form covariance maximizer.

The statistical trade-off follows directly. A shared estimate collects within-component information from all \(K\) groups, which can reduce the instability of estimating a full covariance from a small component. But if true component covariances differ, the pooled matrix compromises across them and may represent none of them well. Separate estimates can capture such differences, but each relies on fewer effective observations; a small or poorly dispersed component can yield an unstable or singular estimate. Sharing is therefore a modeling assumption, not merely a computational shortcut.

Common Mistake

A common error is to say that a shared covariance means the clusters have the same mean or occupy the same region. It means only that their covariance matrices are constrained to agree. Another error is to pool total scatter around the grand mean. The decomposition above shows why that adds between-component separation, although the covariance in the component Gaussian model describes within-component dispersion. Before interpreting a shared estimate, ask both what is pooled and what quantity the covariance is intended to describe.