Conditional Expectations in Mixture Models

Mixture Models: Update First, Then Average

A mixture model describes a population assembled from several component distributions. A latent label \(Z\) records which component generated the observation, but \(Z\) may not be observed. Once an observation \(X=x\) is available, the chance of each component generally changes: components that make \(x\) more likely receive greater posterior weight. A conditional mean in a mixture must use these updated weights, not simply the original proportions.

Work on a probability space \((\Omega,\mathcal{F},\mathbb{P})\). Let \(Z\) take values in \(\{1,\ldots,k\}\), with \(\mathbb{P}(Z=i)=\pi_i\), where \(\pi_i\ge0\) and \(\sum_{i=1}^k\pi_i=1\). The variable \(Z\) is the component label. Suppose \(X\) is an observation and \(Y\) is a real-valued integrable response. Conditional expectations are equalities \(\mathbb{P}\)-almost surely; formulas indexed by a continuous observation \(x\) specify versions and are meaningful up to sets of zero marginal probability.

Finite Observations and Posterior Means

When \(X\) takes values in a finite or countable set, write \(q_i(x)=\mathbb{P}(X=x\mid Z=i)\). For an observation \(x\) with \(\mathbb{P}(X=x)>0\), the mixture formula gives \(\mathbb{P}(X=x)=\sum_j\pi_jq_j(x)\). Bayes’ rule therefore gives the posterior weight \(\mathbb{P}(Z=i\mid X=x)=\pi_iq_i(x)/\sum_j\pi_jq_j(x)\). If \(m_i(x)=\mathbb{E}[Y\mid X=x,Z=i]\) is defined for the component and observation in question, then the conditional mean given \(X=x\) is the posterior-weighted average of these component means.

Proof. The event \(\{X=x\}\) is the disjoint union of the events \(\{X=x,Z=i\}\). By the definition of conditional expectation on an event of positive probability, \(\mathbb{E}[Y\mid X=x]=\mathbb{E}[Y\chi_{\{X=x\}}]/\mathbb{P}(X=x)\). Finite additivity of the integral over the disjoint union makes the numerator \(\sum_i\mathbb{E}[Y\chi_{\{X=x,Z=i\}}]\). For each term with positive event probability, this equals \(m_i(x)\mathbb{P}(X=x,Z=i)\). Divide by \(\mathbb{P}(X=x)\), and use \(\mathbb{P}(X=x,Z=i)/\mathbb{P}(X=x)=\mathbb{P}(Z=i\mid X=x)\). A zero-probability event contributes zero to the numerator. This proves the formula. \(\square\)

Continuous Observations and Density Weights

For continuous observations, the probability of a single value \(X=x\) is often zero, so conditional expectation on the event \(\{X=x\}\) is not defined by dividing event probabilities. Instead, express component distributions by densities. Let \(\lambda\) be a sigma-finite measure on the observation space, and suppose the conditional law of \(X\) given \(Z=i\) has a nonnegative measurable density \(f_i\) with respect to \(\lambda\), with \(\int f_i\,d\lambda=1\). The marginal density is \(f(x)=\sum_i\pi_i f_i(x)\). Where \(f(x)>0\), the posterior component weights are \(\pi_i f_i(x)/f(x)\).

Proof. The function \(g\) is measurable on \(\{f>0\}\), as it is a quotient of measurable functions with positive denominator there. To verify the conditional expectation, it suffices to check its defining integral identity on every event \(\{X\in A\}\), for measurable \(A\). The marginal law of \(X\) has density \(f\), so \(\mathbb{E}[g(X)\chi_{\{X\in A\}}]=\int_A g(x)f(x)\,d\lambda(x)=\int_A\sum_i\pi_i f_i(x)m_i(x)\,d\lambda(x)\). The last integral equals \(\sum_i\mathbb{E}[Y\chi_{\{X\in A,Z=i\}}]\): in component \(i\), the conditional mean given \(X\) is \(m_i(X)\), and the joint contribution of \(X\) and \(Z=i\) has density \(\pi_i f_i\). Summing over the finite set of labels gives \(\mathbb{E}[Y\chi_{\{X\in A\}}]\). The identity holds for all measurable \(A\), and \(g(X)\) is integrable because the absolute integral is bounded by \(\mathbb{E}[|Y|]\), using the same component identity with \(|m_i(X)|\le\mathbb{E}[|Y|\mid X,Z=i]\). Thus \(g(X)\) satisfies the defining integral characterization of conditional expectation. \(\square\)

A Common Pitfall: Using Prior Weights After Observing Data

In the binary example, averaging component means with the prior weights gives \((2/5)2+(3/5)6=22/5\). That is the unconditional mean of \(Y\), not \(\mathbb{E}[Y\mid X=1]\). After observing \(X=1\), the likelihoods \(4/5\) and \(1/4\) change the weights to \(32/47\) and \(15/47\). The correct conditional mean is \((32/47)2+(15/47)6=154/47\), not \(22/5\). The distinction is the same whether component distributions are discrete or continuous: condition on the observation by updating the weights, then average the component-specific conditional means.

These calculations are useful whenever an unobserved group, class, or regime generates data—for example, when combining populations with different response patterns. They also provide an internal consistency check: averaging \(\mathbb{E}[Y\mid X]\) over the marginal law of \(X\) must recover \(\mathbb{E}[Y]\), by the tower property of conditional expectation.