Why Component Labels Are Not Identifiable
A finite Gaussian mixture describes a distribution as a weighted sum of component distributions. The component labels help write the model, but the labels themselves do not appear in the resulting distribution: exchanging labels and exchanging their associated weights and parameters leaves the mixture unchanged. The central question is therefore not whether a particular label is recoverable, but whether the component parameters are determined up to a permutation.
Identifiability is a population-level property of the model: it asks whether two parameter values can produce exactly the same distribution. It does not say that a finite data set determines the parameters exactly, or that an estimation procedure will find a unique solution.
Finite Gaussian Mixture and Identifiability
Let \(X\) take values in \(\mathbb{R}^p\). A \(K\)-component Gaussian mixture has density
where \(\phi_p(x;\mu_k,\Sigma_k)\) is the \(p\)-variate normal density with mean vector \(\mu_k\) and positive-definite covariance matrix \(\Sigma_k\). The parameter is \(\theta=((\pi_k,\mu_k,\Sigma_k):k=1,\ldots,K)\). A component is specified by the pair \((\mu_k,\Sigma_k)\); assume for now that these pairs are distinct.
The restriction to positive weights excludes components that contribute nothing. Requiring distinct component distributions rules out a different problem: if two components have exactly the same mean and covariance, their weights can be combined or divided in many ways without changing the mixture. That is genuine nonidentifiability, not merely a relabeling.
Worked Example: Permuting Two Components
Consider a two-dimensional mixture with weights \(0.3\) and \(0.7\), means \((0,1)^\mathsf{T}\) and \((2,-1)^\mathsf{T}\), and covariance matrices \(I_2\) and \(\begin{pmatrix}2&0\\0&1\end{pmatrix}\). Its density is
Now exchange the two labels, including each label’s weight and Gaussian parameters. The resulting expression is
Addition is commutative, so \(\tilde g(x)=g(x)\) for every \(x\). The component parameters have not changed as an unordered collection; only their names have changed. Thus the labeled parameter vector is not unique, even when the mixture is identifiable up to permutations.
Worked Example: Duplicate Components Create More Than Label Switching
Compare the one-dimensional mixtures
and
Because the first two terms in \(g_1\) have the same density, their coefficients add:
The first representation has three components and the second has two, so no permutation can make their parameter lists match. In fact, for any \(a\) with \(0<a<0.5\), the weights \(a\) and \(0.5-a\) on two copies of \(\phi_1(x;0,1)\) give the same density. Positive weights alone are therefore not enough for identifiability: distinct component distributions are also needed.
The Identifiability Result
The key fact behind the theorem is that distinct multivariate Gaussian characteristic functions are linearly independent. A characteristic function of a random vector \(X\) is \(E[\exp(i t^\mathsf{T}X)]\), for \(t\in\mathbb{R}^p\). For a Gaussian component it is
then \(c_j=0\) for every \(j\).
Proof. Choose a vector \(u\in\mathbb{R}^p\) such that the projected pairs \((u^\mathsf{T}\mu_j,u^\mathsf{T}\Sigma_j u)\) are distinct. Such a choice exists: for each pair of distinct components, either their means differ or their covariance matrices differ. In the first case, \(u^\mathsf{T}(\mu_j-\mu_\ell)\) is a nonzero polynomial in \(u\); in the second case, \(u^\mathsf{T}(\Sigma_j-\Sigma_\ell)u\) is a nonzero polynomial. Avoiding the zero set of one such polynomial for each of the finitely many component pairs ensures that every projected pair is distinct. A finite union of zero sets of nonzero polynomials cannot cover \(\mathbb{R}^p\).
Set \(a_j=u^\mathsf{T}\mu_j\) and \(b_j=u^\mathsf{T}\Sigma_j u\). Positive definiteness gives \(b_j>0\). Substituting \(t=su\) into the assumed identity yields
Let \(b_0\) be the smallest value among the \(b_j\), and multiply the identity by \(e^{b_0s^2/2}\). The terms with \(b_j=b_0\) form the trigonometric polynomial \(q(s)=\sum_{j:b_j=b_0}c_j e^{ia_js}\). All remaining terms tend to zero as \(s\to+\infty\), so \(q(s)\to0\). The frequencies \(a_j\) in this sum are distinct, because the projected pairs are distinct and their second coordinates are all \(b_0\). Direct integration gives
Indeed, the diagonal terms contribute \(|c_j|^2\), and each cross term has average proportional to \(T^{-1}\int_0^T e^{i(a_j-a_\ell)s}\,ds\), which tends to zero when \(a_j\ne a_\ell\). Since \(q(s)\to0\), its squared absolute value is bounded and tends to zero, so its average over \([0,T]\) tends to zero. Hence every coefficient with \(b_j=b_0\) is zero. Remove those terms and repeat the argument with the smallest remaining \(b_j\). The finite repetition proves \(c_j=0\) for all \(j\). \(\square\)
Proof of the theorem. Equality of distributions implies equality of their characteristic functions. Take the union of all distinct component pairs appearing in either mixture. For each pair in this union, subtract its total weight in the second mixture from its total weight in the first. The difference of the characteristic functions is a finite linear combination of the Gaussian characteristic functions for these distinct pairs, equal to zero for every \(t\). By the lemma, each coefficient is zero. Thus every component pair has the same total weight on both sides. Since the component pairs are distinct within each mixture and their weights are positive, each pair appears once in each mixture with the same weight. The mixtures consequently have the same number of components and match after a permutation. \(\square
Worked Example: Using an Ordering Convention
Suppose a one-dimensional two-component mixture has weights \((0.65,0.35)\), means \((1,-2)\), and variances \((0.5,1)\), in that order. Sorting components by increasing mean rewrites it as weights \((0.35,0.65)\), means \((-2,1)\), and variances \((1,0.5)\). The density is unchanged because the two weighted terms have merely exchanged positions.
This convention selects one written representative from the two labelings when the means are unequal. It does not add information to the distribution or make the labels intrinsically meaningful. If two means are equal, ordering by mean alone cannot distinguish their order; if the full component distributions are identical, no ordering convention can resolve the underlying nonidentifiability.
What Identifiability Does—and Does Not—Say
The theorem establishes that a finite mixture of distinct, nonsingular Gaussian components has no alternative finite representation except one obtained by relabeling. It applies to mixtures in any finite dimension, and it allows different component covariance matrices. Its conclusion concerns equality of the full distributions, not equality of a few summaries such as the overall mean or covariance.
A common mistake is to treat a label as if it identified a scientifically distinguished subgroup. The reasoning might be: “component 1 is the first component in the parameter vector, so it is the same group in every fit.” The correction is that the likelihood and the model density are invariant under a joint permutation of labels and component parameters. A label such as “1” has no distributional meaning unless an external convention or substantive definition supplies one.
A second mistake is to infer that identifiability guarantees a unique estimate from observed data. It does not. Identifiability compares the population distributions represented by parameter values; finite samples and numerical estimation raise separate questions. Likewise, sorting means can make output more consistently formatted, but it is a naming convention, not a stronger identifiability result.
Check Your Understanding
Use the definitions and theorem above to answer the following questions.
- What must be permuted together when the labels of mixture components are exchanged?
- Why does requiring every weight to be positive fail to guarantee identifiability when two component distributions are identical?
- In the theorem’s proof, why can a direction be chosen so that all projected component pairs are distinct?
- What does an ordering convention accomplish when all component means are different, and what does it not accomplish?
- Does identifiability up to permutations imply that an estimate based on a finite sample is unique? Explain.