Latent Allocations and Complete-Data Likelihood

Why Introduce Latent Allocations?

A finite Gaussian mixture describes each observation as coming from one of several Gaussian components, but the component that generated an observed vector is usually unknown. Introducing an explicit variable for that membership gives a precise way to describe the joint model for the observations and their unobserved allocations. The resulting complete-data likelihood is the likelihood we would write if both were available.

This representation is more than bookkeeping. It makes clear which likelihood terms belong to the mixing weights and which belong to the component-specific Gaussian distributions. It also prevents a common conceptual error: treating an unknown allocation as though it were a known class label. Here the goal is to define and derive the complete-data likelihood, not to estimate the parameters or infer allocations.

One-Hot Indicators for Component Membership

Suppose \(X_1,\ldots,X_n\) are independent observations in \(\mathbb{R}^p\) from a \(K\)-component Gaussian mixture. Write the parameter as \(\theta=(\pi_1,\ldots,\pi_K,\mu_1,\ldots,\mu_K,\Sigma_1,\ldots,\Sigma_K)\). The mixing weights satisfy \(\pi_k>0\) and \(\sum_{k=1}^K\pi_k=1\); each mean \(\mu_k\) is in \(\mathbb{R}^p\), and each covariance matrix \(\Sigma_k\) is symmetric positive definite.

For observation \(i\), let \(Z_i=(Z_{i1},\ldots,Z_{iK})\) record its component. The entries are indicators: \(Z_{ik}=1\) means observation \(i\) belongs to component \(k\), and \(Z_{ik}=0\) otherwise. Exactly one entry equals 1.

The observed vectors are \(x_1,\ldots,x_n\). If their allocation vectors \(z_1,\ldots,z_n\) were also given, the pair \((x_i,z_i)\) would be the complete observation for case \(i\). In the usual mixture problem only \(x_i\) is observed, so the allocations are latent. The complete-data likelihood is the joint density or mass-density of both parts, evaluated at a particular allocation configuration.

Deriving the Complete-Data Likelihood

Let \(\phi_p(x;\mu,\Sigma)\) denote the \(p\)-variate Gaussian density,

For a single case, the probability of its allocation and the conditional density of its vector multiply. Because \(z_i\) is one-hot, this joint contribution can be written with indicator exponents: the active component contributes its weight and Gaussian density, while inactive components contribute a factor of 1.

For strictly positive weights and positive-definite covariance matrices, its log-likelihood is

Proof. For a single observation, the allocation probability is \(P(Z_i=z_i)=\prod_{k=1}^K\pi_k^{z_{ik}}\), since exactly one indicator is 1. Conditional on that allocation, the density of \(X_i=x_i\) is \(\prod_{k=1}^K\phi_p(x_i;\mu_k,\Sigma_k)^{z_{ik}}\), for the same reason. Multiplying these expressions gives the joint contribution for case \(i\). Independence across cases gives the product over \(i\), which proves the likelihood formula. Taking its logarithm turns the product into a sum. Substitution of the Gaussian log-density gives the displayed log-likelihood. \(\square\)

The expression can be regrouped using the component counts \(n_k=\sum_{i=1}^n z_{ik}\). Since each case has exactly one allocation, \(\sum_k n_k=n\). Collecting terms gives

Thus the allocation pattern enters through which observations contribute to each component’s quadratic terms, as well as through the counts multiplying the log weights and log determinants. This is an exact rewriting, not a parameter estimate or an assumption that the allocations are observed in practice.

Multivariate Terms and Their Interpretation

In dimension \(p>1\), the structure is unchanged, but each component contributes a log-determinant and a Mahalanobis quadratic form. The determinant measures covariance volume, while the quadratic form measures squared distance from the component mean in the geometry defined by that covariance matrix. Positive definiteness ensures that the determinant is positive and the inverse exists, so these terms are well-defined.

How the Complete Likelihood Relates to Observed Data

For fixed observed vectors \(x_1,\ldots,x_n\), there are \(K^n\) possible allocation configurations. The observed-data likelihood is obtained by summing the joint likelihood over all of them. This follows directly from the rule that the probability density of an observed event is the sum of the joint probabilities or densities over its mutually exclusive latent states.

Proof. For each \(i\), the sum over its \(K\) possible one-hot vectors selects one of the \(K\) component contributions. Therefore,

because the allocation choices for different observations range independently over their \(K\) possibilities. The right side is exactly the density of the observed vectors under independent sampling from the mixture. \(\square\)

Common Pitfall: Treating Unknown Membership as a Known Label

A frequent mistake is to insert a fractional allocation, such as \(z_{i1}=0.7\) and \(z_{i2}=0.3\), into the complete-data likelihood and call it a possible latent state. That is not a valid one-hot allocation: the definition requires each indicator to be either 0 or 1, with exactly one equal to 1. Fractional values may arise in other calculations, but they are not realizations of \(Z_i\) and do not define a complete-data likelihood at an allocation.

A related error is to use only the Gaussian density for the assigned component and omit its mixing weight. The joint contribution for an observation assigned to component \(k\) is \(\pi_k\phi_p(x_i;\mu_k,\Sigma_k)\), not merely \(\phi_p(x_i;\mu_k,\Sigma_k)\). The weight is part of the allocation probability and therefore belongs in the complete-data likelihood.

The distinction matters whenever a derivation uses latent membership: the complete-data expression is a joint model for vectors and allocations; the observed-data likelihood is obtained by summing out allocations. Keeping those two objects separate makes the role of every factor explicit.