Parameter Counts Under Covariance Constraints

Why Covariance Parameter Counts Depend on Constraints

A Gaussian clustering model assigns each cluster its own mean and covariance structure, or imposes selected constraints to reduce complexity. Counting the covariance parameters makes those constraints precise: it tells us how many independent quantities must be specified, not how many entries happen to appear in a matrix. This distinction matters when describing a model and when assessing its overall dimension.

Let \(p\) be the dimension of each observation and \(G\) the number of clusters. Write \(\Sigma_g\) for the positive-definite covariance matrix of cluster \(g\). Unless a constraint is stated, a covariance may differ across clusters. We will count covariance parameters first, then show how to add the means and mixing proportions in a finite Gaussian mixture.

Three Basic Dimensions

A symmetric \(p\times p\) matrix has \(p\) diagonal entries and \(p(p-1)/2\) distinct off-diagonal entries. Thus an unrestricted symmetric matrix has \(p(p+1)/2\) free entries. Positive definiteness restricts which values those entries may take, but does not change this dimension: positive-definite matrices form an open subset of the vector space of symmetric matrices.

A diagonal covariance matrix has only its \(p\) diagonal entries free. A spherical covariance matrix is more constrained still: it is a positive scalar times the identity, so it has just one scale parameter.

The word shared changes the count. If all clusters share one diagonal covariance, there are \(p\) covariance parameters total, not \(Gp\). If they share one spherical covariance, there is one parameter total, not \(G\).

Counting Orientation and Shape

For covariance models described through volume, shape, and orientation, a useful parameterization is \(\Sigma_g=\lambda_g D_g A_g D_g^{\mathsf T}\). Here \(\lambda_g>0\) is a volume or scale factor, \(D_g\) is an orthogonal orientation matrix, and \(A_g\) is diagonal with positive entries and determinant \(1\). The determinant constraint separates scale from shape: \(\det(\Sigma_g)=\lambda_g^p\). The shape matrix therefore has \(p-1\) free entries.

For a generic covariance with distinct eigenvalues, the orientation has \(p(p-1)/2\) continuous parameters. The reason is that the orthogonal matrices form a space of that dimension. Although \(D_g\) has \(p^2\) entries, its columns must be orthonormal, imposing \(p(p+1)/2\) independent local constraints.

Signs and permutations of eigenvectors can give different matrix representations of the same covariance. These are discrete ambiguities, not additional continuous degrees of freedom. When eigenvalues are repeated, rotations within the repeated-eigenvalue subspace also leave the covariance unchanged; the orientation is then not fully identifiable. The generic orientation count applies on the distinct-eigenvalue part of the model.

Shared Shape and Unrestricted Component Covariances

A shared-shape model needs its orientation constraint stated explicitly. In the version considered here, every cluster has the same normalized shape \(A\), but each cluster may have its own scale and orientation: \(\Sigma_g=\lambda_g D_g A D_g^{\mathsf T}\). The scales contribute \(G\), the common shape contributes \(p-1\), and the \(G\) orientations contribute \(G p(p-1)/2\). Therefore the count is \(G+(p-1)+G p(p-1)/2\).

If orientation is shared as well as shape, all clusters use the same \(D\). Then the count is \(G+(p-1)+p(p-1)/2\). This is a different, more constrained model. If both scale and orientation are shared, only one common covariance shape remains, with \(p(p+1)/2\) parameters in total. Always translate a verbal constraint into an explicit equation before counting.

With unrestricted component covariances, each \(\Sigma_g\) is a free positive-definite symmetric matrix. The total covariance count is \(G p(p+1)/2\).

From Covariance Counts to Full Mixture Counts

In a \(G\)-component Gaussian mixture, each component also has a \(p\)-dimensional mean, giving \(Gp\) mean parameters. The mixing proportions are positive and sum to one, so they contribute \(G-1\) free parameters. Consequently, if the covariance count under a chosen structure is \(K\), the usual full parameter count is \(Gp+(G-1)+K\).

This addition assumes the ordinary interior parameterization: all mixture proportions are positive, and the component means and covariance parameters are counted as distinct continuous parameters. It is a bookkeeping identity, not a claim that every parameter is identifiable in every fitted mixture. For instance, coincident components can create non-identifiability even though the nominal dimension is unchanged.

Common Mistakes and a Reliable Counting Procedure

A frequent mistake is to count every matrix entry as free. An orthogonal matrix is not an arbitrary \(p\times p\) array, and a symmetric covariance does not have \(p^2\) distinct entries. Another is to count a shared parameter once per cluster: if the model specifies one common shape or covariance, that quantity is counted once.

A subtler error is to count \(p\) shape parameters after imposing \(\det(A)=1\). The determinant constraint removes one degree of freedom, leaving \(p-1\). Conversely, if the scale has not been separated from shape by a normalization, counting both a free scale and an unconstrained shape can count the same degree of freedom twice.

Parameter counts make the meaning of a covariance constraint concrete. They also prevent misleading comparisons between models whose labels sound similar but whose equations impose different sharing patterns. The count is most reliable when derived from the covariance parameterization itself, with each equality, normalization, and cluster-specific quantity accounted for explicitly.