Worked Covariance-Model Comparisons in Two Dimensions

Why Compare Covariance Structures?

In model-based clustering, each Gaussian component describes both a cluster’s location and its spread. In two dimensions, that spread has a direct geometric interpretation: a component’s equal-density contours are ellipses, and its covariance matrix determines their sizes, elongations, and directions. A covariance model controls which of those shapes different components may have.

The choice is a trade-off in representation and parsimony. A spherical model permits only circular contours, while an unrestricted model permits each component to have its own ellipse. Intermediate structures impose more specific geometric constraints. Comparing them means checking both the contours those constraints allow and the number of free covariance parameters they require.

Here \(I_2\) is the two-dimensional identity matrix and \(R(\phi)\) is the rotation matrix \( \begin{pmatrix}\cos\phi&-\sin\phi\\ \sin\phi&\cos\phi\end{pmatrix} \). A diagonal structure aligns every component’s principal axes with the two coordinate axes. A shared-orientation structure instead permits those axes to be rotated, but requires the same pair of principal directions for every component. It does not require the components to have the same covariance matrix: their eigenvalues can differ.

How Covariance Determines a Contour

For a component with mean vector \(\mu_g\), a Gaussian equal-density contour is specified by a constant \(c>0\). The contour consists of points \(x\) satisfying a squared Mahalanobis distance of \(c\). If \(v_1,v_2\) are orthonormal eigenvectors of \(\Sigma_g\), with corresponding eigenvalues \(\lambda_1,\lambda_2\), the contour’s axes lie in those eigenvector directions.

To derive the result, decompose the covariance as \(\Sigma_g=V\Lambda V^\mathsf{T}\), where \(V\) has columns \(v_1,v_2\) and \(\Lambda=\operatorname{diag}(\lambda_1,\lambda_2)\). Set \(y=V^\mathsf{T}(x-\mu_g)\). Since \(V^\mathsf{T}V=I_2\), substitution into the contour equation gives

This is the equation of an ellipse with semiaxis lengths \(\sqrt{c\lambda_1}\) and \(\sqrt{c\lambda_2}\). The associated axes are the columns of \(V\), so they are the covariance eigenvector directions. This also explains why a spherical covariance gives a circle: both eigenvalues are equal, and the contour has the same radius in every direction.

Counting the Covariance Parameters

The covariance-only count records the number of free continuous parameters in the covariance matrices under a structure. Counts below are generic: they describe a model away from degenerate cases such as an exactly spherical covariance, where an orientation is not uniquely determined.

The count follows by adding the free entries in each representation. A spherical component has one positive scale \(\lambda_g\), giving \(G\) parameters. A diagonal component has two positive variances, giving \(2G\). An unrestricted symmetric \(2\times2\) covariance has two diagonal entries and one off-diagonal entry, giving three parameters per component, or \(3G\) altogether. For shared orientation, there are two eigenvalues for each component and one common angle, giving \(2G+1\).

For a Gaussian mixture with \(G\) components, each component also has a two-dimensional mean, and the \(G\) mixing proportions sum to one. Thus there are \(2G\) mean parameters and \(G-1\) free mixing proportions. Adding these to the covariance counts gives the following totals:

The shared-orientation count uses one angle, not one angle per component. If every component had its own angle, the covariance count would be \(3G\), the unrestricted count in two dimensions. When all covariances happen to be spherical, the common angle is unidentifiable because rotating a circle changes nothing; the generic count does not describe that special degeneracy.

Which Structures Can Represent Which Covariances?

The constraints form a nested sequence: every spherical pair is diagonal, every diagonal pair is representable with a common orientation, and every shared-orientation pair is unrestricted. The inclusions follow directly from the definitions. A spherical matrix is diagonal; a diagonal pair is a shared-orientation pair with \(\phi=0\); and every shared-orientation covariance is symmetric positive definite, so it is allowed by the unrestricted structure.

For \(G\geq2\), these inclusions are strict. For spherical versus diagonal, a matrix such as \(\operatorname{diag}(4,1)\) is diagonal but not spherical. For diagonal versus shared orientation, a pair of matrices obtained by rotating \(\operatorname{diag}(4,1)\) by \(30^\circ\) is shared-orientation but not diagonal. For shared orientation versus unrestricted, use \(\Sigma_1=\operatorname{diag}(4,1)\) and \(\Sigma_2=\begin{pmatrix}5&4\\4&5\end{pmatrix}\): the first has principal directions along the coordinate axes, while the second has directions at \(45^\circ\), so no single orientation represents both. These examples establish strictness without relying only on parameter counts. \(\square\)

Common Mistake and Practical Interpretation

Mistake: “Shared orientation means the components share a covariance matrix, so they must have the same ellipse.” That conclusion confuses orientation with the full covariance. Under shared orientation, the eigenvectors are common, but the eigenvalues may differ by component. The ellipses can therefore have different sizes and different elongations while retaining the same principal directions.

A useful comparison proceeds in two checks. First, inspect the eigenvalue pattern and eigenvectors: are the contours circular, coordinate-aligned, aligned to common rotated axes, or allowed to point in different directions? Second, record the parameter cost. For example, in two dimensions a shared-orientation model uses one common angle, whereas an unrestricted model can adjust each component’s orientation independently.

These checks clarify what each covariance structure permits. They do not by themselves determine which model should be selected: that requires evaluating fitted models and their fit or selection criteria. The central geometric distinction is precise, however: spherical restricts shape, diagonal fixes axes to the coordinates, shared orientation ties principal directions across components, and unrestricted removes that cross-component orientation constraint.