Geometry of Gaussian Cluster Covariances

Why Covariance Has a Geometry

A Gaussian component in a clustering model is described not only by its centre but also by how its probability spreads around that centre. In two dimensions, a component with equal spread in every direction has circular contours. If its spread is greater in one direction than another, the contours are elongated ellipses. If the direction of greater spread is tilted, the ellipse is tilted too.

The covariance matrix encodes these features. Its eigenvectors specify the directions in which the component spreads without cross-covariance, and its eigenvalues give the variances in those directions. This tutorial connects that interpretation to the geometry of Gaussian density contours. All statements about a component’s contours concern its own Gaussian distribution; contours of a mixture density need not themselves be ellipses.

The multivariate Gaussian density is proportional to \(\exp\{-\tfrac12(x-\mu)^\mathsf{T}\Sigma^{-1}(x-\mu)\}\). Thus, on a fixed density contour, the quadratic form is constant. The contour at \(q\) has density \(\exp(-q/2)\) times the density at the centre. To read its shape, use the spectral theorem for real symmetric matrices: \(\Sigma\) has an orthonormal basis of eigenvectors \(u_1,\ldots,u_p\), with positive eigenvalues \(\lambda_1,\ldots,\lambda_p\). The eigenvectors are mutually perpendicular, and the covariance acts on each by \(\Sigma u_j=\lambda_j u_j\).

Proof. Write any displacement from the centre in the orthonormal eigenvector basis as \(x-\mu=\sum_{j=1}^p z_j u_j\). Since \(\Sigma u_j=\lambda_j u_j\), the inverse matrix satisfies \(\Sigma^{-1}u_j=\lambda_j^{-1}u_j\). Orthonormality therefore gives

The contour equation is consequently \(\sum_j z_j^2/\lambda_j=q\), or \(\sum_j z_j^2/(q\lambda_j)=1\). This is the equation of an ellipsoid in the eigenvector coordinates. Setting all coordinates except \(z_j\) to zero gives \(z_j^2=q\lambda_j\), so the two endpoints along \(u_j\) are at distances \(\sqrt{q\lambda_j}\) from \(\mu\). This proves the claimed directions and lengths. \(\square\)

The theorem separates two roles. The eigenvector tells us which way an axis points; the eigenvalue tells us the variance along that axis. For a fixed contour level \(q\), a larger eigenvalue produces a longer semiaxis. The contour’s axis length is not the eigenvalue itself: it is \(\sqrt{q\lambda_j}\). In particular, at \(q=1\), the semiaxis length is the standard deviation \(\sqrt{\lambda_j}\) along that direction.

Worked Two-Dimensional Contours

Contour Levels and Probability Content

A contour level describes relative density, but it can also be selected to enclose a specified probability. Under a positive-definite Gaussian model, transform the displacement into its eigenvector coordinates and divide each coordinate by its standard deviation. The resulting coordinates are independent standard normal variables. The sum of their squares has a chi-square distribution with \(p\) degrees of freedom.

Proof. Let \(u_j\) and \(\lambda_j\) be the orthonormal eigenvectors and positive eigenvalues of \(\Sigma\). Define \(Y_j=u_j^\mathsf{T}(X-\mu)/\sqrt{\lambda_j}\). In eigenvector coordinates, the Gaussian density factors into the product of \(p\) standard normal densities: the orthogonal change of coordinates preserves lengths and has absolute Jacobian \(1\), and the subsequent scaling by \(\sqrt{\lambda_j}\) yields independent \(N(0,1)\) coordinates \(Y_j\). The quadratic form becomes \(Q=\sum_{j=1}^p Y_j^2\). By the definition of the chi-square distribution, a sum of squares of \(p\) independent standard normal variables has distribution \(\chi_p^2\). The interior of the contour is exactly the event \(Q\le q\), proving the probability statement. \(\square\)

Interpreting Covariance Geometry in Clustering

In a model-based clustering analysis, each Gaussian component has its own centre and covariance. The covariance contour gives a geometric summary of that component’s spread: its orientation identifies the directions of principal variation, and the relative eigenvalue sizes indicate how elongated the contour is. Two components can have differently oriented contours even if their centres are close, and a component’s longest axis need not align with either measured variable.

A component contour is not a hard boundary for cluster membership. A point outside a chosen contour still has positive Gaussian density, and points can receive appreciable membership support from more than one component. A contour’s probability statement is about repeated draws from that one Gaussian component, not about the correctness of assigning every point inside it to a cluster.

If two or more eigenvalues are equal, there is no unique principal direction within the associated eigenspace. For example, if \(\Sigma=\sigma^2 I\), then all directions have the same variance and every contour is a sphere (a circle when \(p=2\)). One may choose different orthonormal bases of eigenvectors, but none describes a uniquely preferred orientation. This is a feature of the geometry, not a numerical failure.

Common Mistakes

• Treating an eigenvalue as an axis length. An eigenvalue is a variance along its eigenvector. The semiaxis length at level \(q\) is \(\sqrt{q\lambda_j}\), not \(\lambda_j\).
• Calling an eigenvector the spread. An eigenvector specifies a direction; the associated eigenvalue quantifies variance in that direction.
• Reading the contour level as probability directly. The equation with level \(q\) does not enclose probability \(q\). Its probability content is \(F_{\chi_p^2}(q)\).
• Assuming the axes follow the measured variables. That is true for a diagonal covariance matrix in those coordinates, but a covariance with off-diagonal entries can have tilted principal directions.
• Confusing one component’s ellipse with the mixture’s shape. A mixture combines several component densities; its overall contours are not generally the individual component ellipses.