Volume, Shape, and Orientation Constraints

Why Separate Covariance Constraints?

A Gaussian cluster is described not only by its center but also by how its observations spread around that center. A covariance matrix determines the size of that spread, whether it is elongated, and which directions its principal axes follow. In a mixture with several clusters, each of these features may be shared across components or allowed to vary.

Separating covariance into volume, shape, and orientation makes those modeling choices explicit. It also gives a direct geometric interpretation: volume controls overall scale, shape controls relative axis lengths, and orientation controls the directions of the axes. The eigenvalue decomposition of a cluster covariance matrix provides the starting point.

Definition: For a $p$-dimensional Gaussian component $g$ with positive-definite covariance matrix $\Sigma_g$, write $$ \Sigma_g=\lambda_g D_g A_g D_g^{T}. $$ Here $\lambda_g=|\Sigma_g|^{1/p}>0$ is the volume scale, $D_g$ is an orthogonal matrix of eigenvectors, and $A_g$ is a diagonal shape matrix with positive entries and $|A_g|=1$. Thus $\lambda_g$ sets overall scale, $A_g$ sets relative axis lengths, and $D_g$ sets orientation. The word “volume” refers to this scale parameter; the geometric volume of a fixed-probability ellipsoid is proportional to $\lambda_g^{p/2}$.

One convenient convention is to order the eigenvalues from largest to smallest and pair the entries of $A_g$ with the corresponding columns of $D_g$. Since $|A_g|=1$, its entries describe relative rather than absolute spread. If eigenvalues are repeated, the associated eigenvectors are not unique; rotating a basis within the repeated-eigenvalue subspace leaves $\Sigma_g$ unchanged. Sign changes to eigenvectors also leave the covariance unchanged.

What the Three Parts Control

For a component with mean $\mu_g$, a constant-density contour is an ellipsoid. More generally, the set of points with squared Mahalanobis distance $c$ is $$ (x-\mu_g)^T\Sigma_g^{-1}(x-\mu_g)=c,\qquad c>0. $$ In the eigenvector coordinates, its principal-axis directions are the columns of $D_g$, and its semiaxis lengths are $\sqrt{c\lambda_g a_{gj}}$, where $a_{gj}$ is the $j$th diagonal entry of $A_g$. Thus multiplying $\lambda_g$ changes every squared axis length by the same factor, changing $A_g$ changes the relative axis lengths, and changing $D_g$ rotates the axes.

The determinant gives a useful check. Since $D_g$ is orthogonal and $|A_g|=1$, $|\Sigma_g|=\lambda_g^p$. Consequently, components with equal $\lambda_g$ have equal covariance determinant, and their fixed-$c$ ellipsoids have equal geometric volume. Equal volume does not imply equal shape or orientation.

Constraint choice across components	What is shared	What may differ	Geometric reading
Volume equal	Overall covariance scale	Shape and orientation	Equal-volume contours can still have different axis ratios or directions
Shape equal	Relative axis lengths	Volume and orientation	Contours have the same eccentricity, but may be differently sized or rotated
Orientation equal	Principal-axis directions	Volume and shape	Contours are aligned, but can differ in size and eccentricity
All three equal	Volume, shape, and orientation	Nothing in the covariance geometry	All covariance contours have the same size, shape, and alignment

These properties are separate modeling decisions. If volume varies but shape and orientation are shared, contours are scaled copies. If shape varies while volume and orientation are shared, contours have different eccentricities but the same determinant and aligned axes. If orientation varies while volume and shape are shared, the contours have the same axis lengths and volume but point in different directions.

Schematic Contour Diagrams

The sketches below represent two-dimensional contours, not data points. The symbols are schematic: a circle represents equal axis lengths, an elongated oval represents unequal axis lengths, and a diamond-like oval represents a rotated contour. The component centers have been omitted so that the covariance geometry is easier to compare.

Representative constraint	Component 1 contour	Component 2 contour	What the sketch illustrates
Common spherical shape and orientation; varying volume	○	◎	Both contours are circular; one is larger
Equal volume and common orientation; varying shape	⬭	⭖	Aligned contours have different axis ratios
Equal volume and shape; varying orientation	⬭	⧄	Equal axis lengths and area, but different directions
All three equal	⬭	⬭	Matching covariance contours

The schematic glyphs are intentionally simple; the equations and calculations below determine the exact contours. In particular, matching-looking symbols do not imply that the component means are equal. Means locate the contours, while the covariance decomposition describes their geometry.

Worked Covariance Decompositions

Worked Example: Equal Shape and Orientation, Different Volume

Consider two bivariate components with covariance matrices $\Sigma_1=I_2$ and $\Sigma_2=4I_2$. Find the volume, shape, and orientation parts for each.

For $\Sigma_1$, the determinant is $1$, so $\lambda_1=|\Sigma_1|^{1/2}=1$. Its two eigenvalues are both $1$; hence $A_1=I_2$, and we can take $D_1=I_2$. Therefore $\Sigma_1=1\cdot I_2\cdot I_2\cdot I_2^T$.

For $\Sigma_2$, the determinant is $16$, so $\lambda_2=|\Sigma_2|^{1/2}=4$. Its eigenvalues are both $4$, which again gives $A_2=I_2$; take $D_2=I_2$. Thus $\Sigma_2=4\cdot I_2\cdot I_2\cdot I_2^T$.

The shape and orientation are shared, while volume varies. At any fixed $c$, the first contour is $x_1^2+x_2^2=c$, with radius $\sqrt{c}$. The second is $(x_1^2+x_2^2)/4=c$, with radius $2\sqrt{c}$. The second contour is twice as wide in each direction, and its area is four times as large.

Worked Example: Equal Volume and Orientation, Different Shape

Let $$ \Sigma_1=\begin{pmatrix}4&0\\0&1\end{pmatrix}, \qquad \Sigma_2=\begin{pmatrix}6&0\\0&2/3\end{pmatrix}. $$ Both matrices are diagonal with the same principal-axis directions. Determine whether they have equal volume, and compare their shapes.

For the first matrix, $|\Sigma_1|=4$, so $\lambda_1=\sqrt{4}=2$. Its normalized shape matrix is $A_1=\operatorname{diag}(4/2,1/2)=\operatorname{diag}(2,1/2)$, whose determinant is $1$. The orientation is $D_1=I_2$.

For the second, $|\Sigma_2|=6(2/3)=4$, so $\lambda_2=2$ as well. Its shape matrix is $A_2=\operatorname{diag}(6/2,(2/3)/2)=\operatorname{diag}(3,1/3)$, also with determinant $1$, and $D_2=I_2$.

The components therefore have equal volume and common orientation, but different shape. Their axis-length ratios are $\sqrt{4/1}=2$ and $\sqrt{6/(2/3)}=3$, respectively. The second contour is more elongated, even though both covariance determinants are $4$.

Worked Example: Equal Volume and Shape, Different Orientation

Take $\Sigma_1=\operatorname{diag}(4,1)$ and construct a second covariance by rotating its principal axes through $45^\circ$, without changing its eigenvalues. Use $$ D_2=\frac{1}{\sqrt{2}}\begin{pmatrix}1&-1\\1&1\end{pmatrix}, \qquad \Sigma_2=D_2\begin{pmatrix}4&0\\0&1\end{pmatrix}D_2^T. $$

For $\Sigma_1$, the determinant is $4$, so $\lambda_1=2$, $A_1=\operatorname{diag}(2,1/2)$, and $D_1=I_2$. For $\Sigma_2$, rotation preserves the determinant and eigenvalues, so $\lambda_2=2$ and $A_2=\operatorname{diag}(2,1/2)$. Its orientation matrix is $D_2$.

To calculate $\Sigma_2$, the diagonal entries after rotation are $(4+1)/2=5/2$, and the off-diagonal entries are $(4-1)/2=3/2$. Hence $$ \Sigma_2=\begin{pmatrix}5/2&3/2\\3/2&5/2\end{pmatrix}. $$ Its determinant is $25/4-9/4=4$, confirming the equal volume. Both components have the same axis lengths, but the second component's axes are rotated by $45^\circ$. This is orientation variation without volume or shape variation.

Two Consequences of the Decomposition

Theorem: For a positive-definite covariance matrix $\Sigma_g=\lambda_gD_gA_gD_g^T$, the level set $(x-\mu_g)^T\Sigma_g^{-1}(x-\mu_g)=c$ is an ellipsoid with principal-axis directions given by the columns of $D_g$ and semiaxis lengths $\sqrt{c\lambda_ga_{gj}}$.

Proof. Because $D_g$ is orthogonal, $D_g^{-1}=D_g^T$. Set $y=D_g^T(x-\mu_g)$. The inverse covariance is $\Sigma_g^{-1}=\lambda_g^{-1}D_gA_g^{-1}D_g^T$, so the level-set equation becomes $$ \lambda_g^{-1}y^TA_g^{-1}y=c, \qquad \sum_{j=1}^p\frac{y_j^2}{\lambda_ga_{gj}}=c. $$ Equivalently, $\sum_j y_j^2/(c\lambda_ga_{gj})=1$, the equation of an ellipsoid in the $y$-coordinates. Its semiaxis along coordinate $j$ has length $\sqrt{c\lambda_ga_{gj}}$. Transforming back by $D_g$ gives the stated directions. $\square$

Result: The determinant of $\Sigma_g$ is $\lambda_g^p$. For any fixed $c>0$, the volume of its ellipsoidal level set is a constant depending only on $p$ and $c$, multiplied by $\lambda_g^{p/2}$.

Derivation. Determinants multiply, so $$ |\Sigma_g|=|\lambda_g I_p|\,|D_g|\,|A_g|\,|D_g^T| =\lambda_g^p |D_g|^2 =\lambda_g^p. $$ Here $|A_g|=1$, and orthogonality implies $|D_g|^2=|D_g^TD_g|=1$. For the level set, the linear transformation from the unit ball is $x=\mu_g+\sqrt{c}\,\Sigma_g^{1/2}u$, where $u^Tu\leq 1$. Its volume is the unit-ball volume multiplied by the absolute determinant of $\sqrt{c}\,\Sigma_g^{1/2}$, namely $c^{p/2}|\Sigma_g|^{1/2}$. Substituting $|\Sigma_g|=\lambda_g^p$ gives the claimed proportionality. $\square$

Reading Constraints Carefully

A statement that covariance matrices are “the same size” is ambiguous unless it specifies which feature is held fixed. Equal determinants mean equal volume scale, not equal axis lengths. Equal normalized shape matrices mean equal relative axis lengths, not equal overall scale. Equal orientation matrices mean common principal-axis directions, not equal covariance matrices. In particular, equal volume alone places no restriction on orientation.

These distinctions matter because covariance restrictions encode assumptions about the cluster geometry. A model that forces common orientation may be unsuitable when clusters have differently directed elongations. A model that forces equal volume may be unsuitable when one group is much more dispersed. Conversely, allowing every component to have unrelated volume, shape, and orientation uses a more flexible covariance description. The decomposition clarifies the assumptions before any model comparison is made.

Common Mistake

Wrong reasoning: “The determinants are equal, so the covariance matrices—and therefore the contours—are equal.” Equal determinants establish only equal volume. For example, $\operatorname{diag}(4,1)$ and $\operatorname{diag}(6,2/3)$ both have determinant $4$, but their axis ratios are $2$ and $3$, so their shapes differ.

Correction: Compare the three parts separately. Check $\lambda_g$ for volume, $A_g$ for relative axis lengths, and $D_g$ for axis directions. Equality of all three gives the same covariance geometry; equality of only one does not.

Key takeaway: A Gaussian covariance constraint is a geometric statement about scale, relative spread, and alignment. The decomposition $\Sigma_g=\lambda_gD_gA_gD_g^T$ makes those three statements inspectable, while the contour equation shows exactly how each affects a cluster's ellipsoid.

Check Your Understanding

Use the decomposition and contour interpretation to answer the following.

If two components have equal $\lambda_g$ but different $D_g$, what geometric feature is shared and what may differ?
For $p=3$, if $|\Sigma_g|=27$, what is $\lambda_g$?
Can two covariance matrices have equal volume but different shape? Give a numerical example or explain why not.
What happens to every semiaxis length when $\lambda_g$ is multiplied by $4$, with $A_g$, $D_g$, and $c$ held fixed?
Why does equality of orientation not imply equality of covariance matrices?