Initialization with Moment and Clustering-Based Starts

Why Initialization Matters

For a Gaussian mixture, the EM algorithm begins from a set of component weights, means, and covariance matrices. These starting values matter because the likelihood can have multiple stationary points, and different starts can lead to substantially different fitted likelihoods. An initialization is not itself evidence that the data contain the proposed groups: it is a way to place the algorithm in a particular part of the parameter space.

Write the observations as $x_1,\ldots,x_n\in\mathbb{R}^p$, and let the mixture have $K$ components, with weights $\pi_k$, means $\mu_k$, and positive-definite covariance matrices $\Sigma_k$. The observed-data log-likelihood at a candidate starting parameter is

\ell(\theta)=\sum_{i=1}^n\log\left\{\sum_{k=1}^K \pi_k\,\phi_p(x_i;\mu_k,\Sigma_k)\right\}.

Here $\phi_p(\,\cdot\,;\mu,\Sigma)$ is the $p$-variate Gaussian density. A usable start has $\pi_k>0$, $\sum_k\pi_k=1$, and positive-definite $\Sigma_k$. In practice, covariance constraints or regularization may be needed to avoid singular or nearly singular estimates, as in constrained or penalized mixture estimation.

Starts From Global Moments

Definition: The empirical global mean and covariance, using divisor $n$, are $$ \bar x=\frac{1}{n}\sum_{i=1}^n x_i,\qquad S=\frac{1}{n}\sum_{i=1}^n(x_i-\bar x)(x_i-\bar x)^{\mathsf T}. $$ A moment-based start uses $\bar x$, $S$, or their directions and scales to choose initial component parameters. For example, one may begin with equal weights, a shared covariance based on $S$, and means displaced from $\bar x$ along selected directions.

The global moments do not determine a unique $K$-component mixture. They summarize the full data, not the latent component distributions. A split along a leading eigenvector of $S$ can give distinct initial means, but the displacement size and the component covariance scale remain choices. Setting every mean equal to $\bar x$ and every covariance equal to $S$ creates identical components; it is a valid parameter value when $S$ is positive definite, but it gives the algorithm no initial separation.

The relationship between global and partition-based moments clarifies what a clustering start preserves.

Theorem (within- and between-group covariance decomposition): Partition the observations into nonempty groups $C_1,\ldots,C_K$. Let $n_k$ be the size of group $C_k$, $\bar x_k$ its mean, and $$ S_k=\frac{1}{n_k}\sum_{i\in C_k}(x_i-\bar x_k)(x_i-\bar x_k)^{\mathsf T}. $$ Then $$ S=\sum_{k=1}^K\frac{n_k}{n}\left[ S_k+(\bar x_k-\bar x)(\bar x_k-\bar x)^{\mathsf T}\right]. $$

Proof. For an observation in group $C_k$, write $x_i-\bar x=(x_i-\bar x_k)+(\bar x_k-\bar x)$. Expand the outer product and sum within that group. The two cross-product sums vanish because $\sum_{i\in C_k}(x_i-\bar x_k)=0$. The remaining terms are $n_kS_k$ and $n_k(\bar x_k-\bar x)(\bar x_k-\bar x)^{\mathsf T}$. Summing over groups and dividing by $n$ gives the identity. $\square$

Thus group means and within-group covariances describe two contributions to global spread: variation among group centers and variation within groups. In a mixture start, setting $\pi_k=n_k/n$, $\mu_k=\bar x_k$, and $\Sigma_k=S_k$ reproduces the global mean and covariance when the $S_k$ are defined as above. However, a group covariance may be singular, particularly when a group is small or its points lie in a lower-dimensional subspace. A positive-definite adjustment or covariance constraint is then essential.

Worked Example: A Two-Dimensional Moment-Based Start

Take four observations $(0,0),(2,0),(0,2),(2,2)$, and use two components. The global mean is $\bar x=(1,1)^{\mathsf T}$. Each coordinate has empirical variance $1$, and the empirical cross-covariance is $0$, so $S=I_2$.

Choose equal weights, shared covariance $I_2$, and means displaced along the first coordinate: $\mu_1=(0,1)^{\mathsf T}$ and $\mu_2=(2,1)^{\mathsf T}$. Their weighted mean is $(1,1)^{\mathsf T}$, the global mean. This is a directional split of the global moments, not a claim that the data have already been classified.

For any corner observation, one component mean is at squared Mahalanobis distance $1$, and the other is at squared distance $5$, since the covariance is $I_2$. Therefore its mixture density is \[ \frac{1}{2}\frac{1}{2\pi}e^{-1/2} +\frac{1}{2}\frac{1}{2\pi}e^{-5/2} =\frac{e^{-1/2}+e^{-5/2}}{4\pi} \approx 0.05480. \] All four observations have this density by symmetry, so the initial log-likelihood is \[ 4\log(0.05480)\approx -11.616. \] This calculation gives a reproducible baseline for this particular start; it does not establish that the start will lead to the best fit.

Random Allocations and Their Risks

A random-allocation start assigns each observation to one of $K$ groups, then computes group sizes and summaries. The assignments can be generated subject to minimum group sizes, or sampled from provisional allocation probabilities. Group proportions suggest weights $n_k/n$, and group means suggest component means. Covariances may be estimated within groups, pooled across groups, or regularized. These choices must be stated: a hard allocation alone does not specify a unique mixture parameter.

Random assignments can produce empty or tiny groups, indistinguishable component means, or covariance estimates that are singular. Randomness does not remove these structural problems. A valid implementation checks group sizes and enforces the covariance constraints required by the model.

Worked Example: A Random Allocation With Identical Components

Consider the univariate observations $0,1,4,5$ and two components. Suppose a random allocation places $0,5$ in the first group and $1,4$ in the second. Both group means are $2.5$, so this allocation cannot provide separated initial centers.

To keep the example nonsingular and isolate the effect of the allocation, use equal weights and a chosen shared component variance of $6.25$ for both components. This is positive, and both component means and covariances are identical; the resulting mixture is simply $N(2.5,6.25)$. Its density is \[ f(x)=\frac{1}{2.5\sqrt{2\pi}}\exp\left\{-\frac{(x-2.5)^2}{2(6.25)}\right\}. \] Let $c=1/(2.5\sqrt{2\pi})\approx0.159577$. The observations $0$ and $5$ are one standard deviation from the mean, so each has density $c e^{-1/2}$. The observations $1$ and $4$ are $0.6$ standard deviations away, so each has density $c e^{-0.18}$. Hence \[ \ell=2\log(c e^{-1/2})+2\log(c e^{-0.18}) =4\log(c)-1.36\approx -8.701. \] The allocation has supplied no component separation, despite assigning observations to two groups. This is one reason to compare starts by their actual mixture likelihoods and parameter values, rather than treating the allocation labels as evidence of a useful initialization.

Clustering-Based Starts and Likelihood Comparisons

A clustering algorithm can provide a more structured hard partition than an unconstrained random allocation. Its centers can initialize mixture means, and its group proportions can initialize weights. The cluster geometry is not automatically Gaussian, however, and clustering objectives need not agree with the mixture likelihood. Covariance choices still matter, and a cluster with too few observations may require a covariance floor or a shared covariance.

Worked Example: Clustering Centers Versus a Diffuse Start

Use observations $1,1,2,2$ and two equally weighted univariate Gaussian components. A clustering partition separates the two values, giving means $1$ and $2$. Use variance $0.25$ for each component; this is a positive variance rather than the zero within-cluster variance, and can be interpreted as a chosen covariance floor.

At $x=1$, the component densities are \[ \phi(1;1,0.25)=\frac{1}{\sqrt{2\pi(0.25)}}\approx0.797885,\qquad \phi(1;2,0.25)=0.797885e^{-2}\approx0.107982. \] Thus the mixture density is $0.5(0.797885+0.107982)\approx0.452933$. By symmetry, the mixture density at $x=2$ is the same. Each value occurs twice, so \[ \ell_{\mathrm{cluster}}=4\log(0.452933)\approx -3.168. \]

For comparison, take a diffuse start with weights $0.5,0.5$, means $-0.5,0.5$, and common standard deviation $1.5$ (variance $2.25$). At $x=1$, \[ f(1)=\tfrac12\phi(1;-0.5,2.25)+\tfrac12\phi(1;0.5,2.25) \approx \tfrac12(0.161314+0.251589)=0.206451. \] At $x=2$, \[ f(2)=\tfrac12\phi(2;-0.5,2.25)+\tfrac12\phi(2;0.5,2.25) \approx \tfrac12(0.066318+0.161314)=0.113816. \] Consequently, \[ \ell_{\mathrm{diffuse}}=2\log(0.20645)+2\log(0.11382)\approx -7.502. \] The clustering-based start has the higher initial log-likelihood for these data and these specified parameters. This comparison is about initial values only; it does not show what either run will attain after EM iterations.

Likelihood comparisons also depend on measurement scale. If features are rescaled, a density changes by a Jacobian factor. The following identity makes that dependence explicit.

Result (likelihood under an invertible affine transformation): Let $y_i=Ax_i+b$, where $A$ is a nonsingular $p\times p$ matrix. Transform each mixture mean and covariance to $A\mu_k+b$ and $A\Sigma_kA^{\mathsf T}$, retaining the weights. Then the transformed-data log-likelihood is $\ell_y=\ell_x-n\log|\det A|$.

Proof. The multivariate Gaussian change-of-variables formula gives \[ \phi_p(Ax+b;A\mu+b,A\Sigma A^{\mathsf T}) =|\det A|^{-1}\phi_p(x;\mu,\Sigma). \] Because this same factor applies to every component, the transformed mixture density at observation $y_i$ is $|\det A|^{-1}$ times the original mixture density at $x_i$. Taking logarithms and summing over $n$ observations gives $\ell_y=\ell_x-n\log|\det A|$. $\square$

For a fixed data set and a fixed feature scale, candidate starts can be compared by their observed-data log-likelihoods. Across differently scaled data, the numerical likelihood values have a known offset and should not be compared without accounting for it. Standardizing variables can also change clustering distances and hence change the proposed partition; scaling is part of the initialization decision, not a cosmetic afterthought.

Practical Choices and a Common Pitfall

A disciplined initialization specifies the weights, means, covariance structure, and any constraints explicitly. Moment-based starts are quick and reproducible, and are useful when the data have a meaningful low-dimensional direction for separating means. Random allocations explore partitions but can waste starts on empty groups or near-duplicate components. Clustering-based starts can give separated centers, but their implied within-group covariances may be unusable without regularization.

Common pitfall: “This start has a high initial likelihood, so it must be the best mixture fit.” The reasoning confuses the likelihood at one parameter value with the likelihood reached by an iterative fit. A high initial value is informative, but it does not guarantee that a run reaches a global maximum, and a lower initial value does not establish that its subsequent fit will remain inferior. Compare starts under the same model, data scale, and covariance constraints, and evaluate their fitted likelihoods after fitting.

Also avoid interpreting arbitrary initial labels as final component identities. Mixture likelihood is unchanged by permuting component labels, so the labels “1” and “2” have no intrinsic meaning. The practical goal of initialization is to provide numerically valid and meaningfully varied starting parameters, not to settle the interpretation of components in advance.

Check Your Understanding

Use the definitions and calculations above to answer these questions.

Why can initializing every component mean at the global mean be a poor choice even when the parameters are valid?
In the covariance decomposition, which term represents variation among group centers, and which represents variation within groups?
What problem can arise if a clustering-based start uses an unregularized covariance from a very small group?
For the random allocation in the example, why did two different assigned groups lead to identical component parameters?
What likelihood adjustment follows when every observation is transformed by a nonsingular matrix $A$?
Why does a larger initial observed-data log-likelihood not guarantee a larger fitted likelihood after EM?