Worked EM Iterations for a Univariate Mixture

Following an EM Fit One Iteration at a Time

For a mixture model, EM alternates between estimating how likely each observation is to belong to each component and updating the component parameters using those probabilities. The steps are straightforward to state, but a numerical fit is easier to audit when the responsibilities, parameter updates, and observed likelihood are all displayed. The likelihood is observed because it sums over the unobserved component labels rather than treating them as known.

This tutorial carries out two updates for a two-component univariate Gaussian mixture. The example is deliberately small enough to check by hand. Its symmetry makes the arithmetic compact, but the same weighted calculations apply to less symmetric data.

Worked Example: Two Updates for Symmetric Data

Take the observations \(x_1=-1\) and \(x_2=1\). Start with equal weights, component means \(-0.5\) and \(0.5\), and both component variances equal to \(1\). At every iteration in this example, symmetry keeps the weights equal and the two means and variances paired. Write the parameters at iteration \(t\) as $$ \pi_1^{(t)}=\pi_2^{(t)}=\frac12,\qquad \mu_1^{(t)}=-a_t,\quad \mu_2^{(t)}=a_t,\qquad (\sigma_1^2)^{(t)}=(\sigma_2^2)^{(t)}=v_t. $$ The starting values are \(a_0=0.5\) and \(v_0=1\).

E-step at iteration 0. For the observation \(x=1\), the ratio of the component-2 density to the component-1 density is $$ \frac{\phi(1;a_0,v_0)}{\phi(1;-a_0,v_0)} =\exp\left(\frac{(1+a_0)^2-(1-a_0)^2}{2v_0}\right) =\exp\left(\frac{2a_0}{v_0}\right)=e. $$ The equal weights cancel in the responsibility ratio. Thus the responsibility for the positive-mean component at \(x=1\) is \(e/(1+e)=0.731059\). By symmetry, the responsibility for the positive-mean component at \(x=-1\) is \(0.268941\). The other component responsibilities are the complements, so each observation’s two responsibilities sum to one.

M-step to iteration 1. Each component has total responsibility \(1\), so its updated weight is \(1/2\). The updated positive mean is $$ \mu_2^{(1)} =\frac{0.268941(-1)+0.731059(1)}{0.268941+0.731059} =0.462117. $$ The negative mean is \(-0.462117\). For the positive component, the updated variance is the responsibility-weighted squared deviation from its updated mean: $$ (\sigma_2^2)^{(1)} =0.268941(-1-0.462117)^2+0.731059(1-0.462117)^2 =0.786448. $$ The negative component has the same variance. The denominator in the variance update is the component’s total responsibility, which here equals one.

Observed likelihood at iteration 1. Each observation has the same mixture density by symmetry. In general, for either \(x=1\) or \(x=-1\), $$ f(x;\theta^{(t)}) =\frac{1}{\sqrt{2\pi v_t}} \exp\left(-\frac{1+a_t^2}{2v_t}\right) \cosh\left(\frac{a_t}{v_t}\right). $$ At iteration 1, \(a_1=0.462117\) and \(v_1=0.786448\), so $$ f(1;\theta^{(1)})\approx 0.244917,\qquad \ell(\theta^{(1)})=2\log f(1;\theta^{(1)})\approx -2.8137. $$ The factor of two appears because there are two observations, each with this same density.

E-step at iteration 1. For \(x=1\), the positive-component responsibility is $$ \tau_{2,2}^{(1)} =\frac{1}{1+\exp(-2a_1/v_1)} =\frac{1}{1+\exp(-1.175201)} \approx 0.764084. $$ For \(x=-1\), the positive-component responsibility is \(1-0.764084=0.235916\). The negative-component responsibilities are again the complements.

M-step to iteration 2. The updated positive mean is \(2(0.764084)-1=0.528168\); the negative mean is its opposite. The updated variance for either component is $$ 4(0.764084)(1-0.764084)\approx 0.721039. $$ The weights remain \(1/2\). Substituting \(a_2=0.528168\) and \(v_2=0.721039\) into the density formula gives $$ f(1;\theta^{(2)})\approx 0.2479,\qquad \ell(\theta^{(2)})\approx -2.7901. $$ For comparison, the starting density is \(f(1;\theta^{(0)})\approx 0.2408\), giving \(\ell(\theta^{(0)})\approx-2.8476\). Thus the observed log likelihood increases across these two updates. The table collects the parameters and likelihood values; responsibilities are for the component whose mean has the same sign as the observation.

Why the Symmetric Updates Have This Form

The preceding calculations can be expressed as a simple recurrence. Suppose the current means are \(-a\) and \(a\), the common variance is \(v>0\), and the weights are equal. Let \(r\) be the responsibility of the positive component for \(x=1\). The density ratio gives \(r=1/(1+\exp(-2a/v))\). The positive component has responsibility \(r\) at \(1\) and \(1-r\) at \(-1\), so its updated mean is \(r-(1-r)=2r-1\). The negative component is updated to the negative of that value.

For the variance identity, the positive component’s responsibility-weighted squared deviations from its updated mean are \(r(1-a_{\mathrm{new}})^2+(1-r)(-1-a_{\mathrm{new}})^2\). Since \(a_{\mathrm{new}}=2r-1\), we have \(1-a_{\mathrm{new}}=2(1-r)\) and \(-1-a_{\mathrm{new}}=-2r\). Substitution gives \(4r(1-r)^2+4(1-r)r^2=4r(1-r)\). Also \(4r(1-r)=1-(2r-1)^2\), proving the stated variance update. Finally, \(2r-1=\tanh(a/v)\) follows by rewriting \(2/(1+\exp(-2a/v))-1\) using the definition of the hyperbolic tangent. These identities explain the updates; for other data, use the general responsibility-weighted M-step rather than this special recurrence.

Checking the Observed Likelihood

For any observation, the mixture density is a weighted sum of component densities. The log likelihood is therefore the sum of logarithms of those sums. It is not the sum of log densities evaluated only at the component with the largest responsibility. That latter calculation would correspond to replacing soft responsibilities with hard classifications.

Why an Exact EM Update Does Not Lower Likelihood

Likelihood tracking is a useful numerical check. Its nondecrease under an exact EM update is not a promise that each iteration finds a global maximum; it is a statement about the objective value at consecutive iterates. Here is the basic argument for a finite latent-label space, as in a finite mixture. Assume the observed-data probabilities or densities are positive and finite at the current and proposed parameters, and that the conditional distribution of the latent labels at the current parameter is well defined.

To prove the result, hold the observed data \(X\) fixed and write \(q(z)=p_{\theta^{(t)}}(z\mid X)\). For any candidate parameter \(\theta'\) with positive joint densities on the support of \(q\), split the likelihood sum into states where \(q(z)>0\) and where \(q(z)=0\): $$ \frac{p_{\theta'}(X)}{p_{\theta^{(t)}}(X)} =\sum_{z:q(z)>0}q(z)\frac{p_{\theta'}(X,z)}{p_{\theta^{(t)}}(X,z)} +\frac{\sum_{z:q(z)=0}p_{\theta'}(X,z)}{p_{\theta^{(t)}}(X)} \geq \sum_{z:q(z)>0}q(z)\frac{p_{\theta'}(X,z)}{p_{\theta^{(t)}}(X,z)}. $$ The extra contribution is nonnegative. Taking logarithms and applying Jensen’s inequality on the support of \(q\) gives $$ \ell(\theta')-\ell(\theta^{(t)}) \geq \log\left(\sum_{z:q(z)>0}q(z)\frac{p_{\theta'}(X,z)}{p_{\theta^{(t)}}(X,z)}\right) \geq \sum_{z:q(z)>0}q(z)\log\frac{p_{\theta'}(X,z)}{p_{\theta^{(t)}}(X,z)} =Q(\theta'\mid\theta^{(t)})-Q(\theta^{(t)}\mid\theta^{(t)}). $$ Taking \(\theta'=\theta^{(t+1)}\), the right side is nonnegative because the update maximizes \(Q\) and \(\theta^{(t)}\) was itself an admissible candidate. This proves the claim. In a Gaussian mixture with positive weights and variances, the finite latent-label probabilities satisfy the stated support condition. \(\square\)

Common Pitfalls and Practical Checks

• Using hard assignments in the M-step. The update uses each observation’s fractional responsibility, not a label chosen by whichever component has the larger responsibility.
• Forgetting to normalize the variance update. In general, the weighted squared deviations must be divided by the component’s total responsibility. That denominator happened to be one in the symmetric example.
• Computing likelihood from assigned components only. Evaluate the mixture density as a sum over components for every observation, then take its logarithm.
• Expecting a global optimum from monotonicity. EM’s nondecreasing likelihood property does not rule out convergence to a local stationary point or slow progress.
• Ignoring nearly zero variances. The displayed example has positive variances at both updates, but Gaussian-mixture likelihoods can have problematic behavior as a variance approaches zero. A numerically increasing likelihood is not by itself evidence that a fit is well behaved.

In practice, record the observed log likelihood after each completed M-step, retain sufficient precision during calculations, and check that every set of responsibilities sums to one for each observation. Responsibilities summarize uncertainty in the latent labels; they are not direct evidence that the labels are known.