Why Mixture Weights Change After an Observation
A mixture model describes a population with several latent classes. A class is selected first, and the observed data are then generated according to that class. Before seeing data, the class probabilities are the mixture weights. After observing data, those weights usually change: classes that make the observation more likely receive greater posterior weight. Conditional expectations in a mixture must use these updated weights, not simply average the class-specific means using the original mixture proportions.
Let \((\Omega,\mathcal{F},\mathbb{P})\) be a probability space. Let \(Z\) be a latent class taking values in \(\{1,\ldots,m\}\), with \(\mathbb{P}(Z=z)=\alpha_z\), where \(\alpha_z>0\) and \(\sum_{z=1}^m\alpha_z=1\). Let \(Y\) be the observed data. Suppose the conditional distribution of \(Y\) given \(Z=z\) has density or probability mass function \(f_z(y)\), relative to a common measure. For discrete data, this means \(f_z(y)=\mathbb{P}(Y=y\mid Z=z)\). The marginal density or mass function of \(Y\) is the mixture \(\sum_z\alpha_z f_z(y)\).
This is Bayes' rule applied to the latent class and the observation. The denominator is the marginal likelihood of the data. Each posterior weight is nonnegative, and the weights sum to one whenever the denominator is positive.
Now let \(V\) be an integrable quantity whose conditional mean given both the data and the class is available. Write \(r_z(y)\) for a version of \(\mathbb{E}[V\mid Y=y,Z=z]\). For example, \(V\) could be a future measurement, an unobserved outcome, or an indicator of a class-related event. When the class-specific conditional distributions exist, the mixture calculation is:
The formula has two distinct ingredients. The likelihood \(f_z(y)\) updates how plausible each class is; the conditional mean \(r_z(y)\) describes the target within that class after accounting for the data. If the target is independent of \(Y\) conditional on \(Z\), then \(r_z(y)=\mathbb{E}[V\mid Z=z]\). Without that conditional independence, the within-class mean may itself depend on the observed value.
Why the Posterior-Weighted Formula Holds
First, the posterior weights follow directly from the joint distribution. For any measurable set \(A\) of possible observations, the probability that the class is \(z\) and the observation falls in \(A\) is \(\alpha_z\int_A f_z(y)\,d\nu(y)\), where \(\nu\) is the common reference measure. Summing over classes gives the marginal density \(f_Y(y)=\sum_k\alpha_k f_k(y)\).
Proof. Define \(w_z(y)=\alpha_z f_z(y)/f_Y(y)\) wherever \(f_Y(y)>0\). For every measurable \(A\), the joint probability is \(\mathbb{P}(Z=z,Y\in A)=\alpha_z\int_A f_z(y)\,d\nu(y)\). On the set where \(f_Y(y)>0\), the proposed posterior satisfies \(\int_A w_z(y)f_Y(y)\,d\nu(y)=\int_A\alpha_z f_z(y)\,d\nu(y)\). The part of the observation space where \(f_Y=0\) has marginal probability zero and contributes nothing to either probability. Thus the proposed function gives the conditional probability of class \(z\) given the observation. Also, summing the numerators over \(z\) gives \(f_Y(y)\), so \(\sum_z w_z(y)=1\). \(\square\)
To prove the conditional-mean formula, use the defining integral property of conditional expectation: a random variable \(H\) is a version of \(\mathbb{E}[V\mid Y]\) if it is measurable with respect to \(Y\) and satisfies \(\mathbb{E}[V\mathbf{1}_{\{Y\in A\}}]=\mathbb{E}[H\mathbf{1}_{\{Y\in A\}}]\) for every measurable set \(A\). The candidate value at \(y\) is \(h(y)=\sum_z w_z(y)r_z(y)\).
Proof. The function \(h(Y)\) is measurable with respect to \(Y\). For any measurable set \(A\), split the expectation according to the finite classes. The defining property of each class-specific conditional mean gives \(\mathbb{E}[V\mathbf{1}_{\{Y\in A\}}\mathbf{1}_{\{Z=z\}}]=\alpha_z\int_A r_z(y)f_z(y)\,d\nu(y)\). Summing these equalities over \(z\) yields \(\mathbb{E}[V\mathbf{1}_{\{Y\in A\}}]=\int_A\sum_z\alpha_z f_z(y)r_z(y)\,d\nu(y)\). Since \(\alpha_z f_z(y)=w_z(y)f_Y(y)\) wherever \(f_Y(y)>0\), this is \(\int_A h(y)f_Y(y)\,d\nu(y)=\mathbb{E}[h(Y)\mathbf{1}_{\{Y\in A\}}]\). Thus \(h(Y)\) satisfies the defining property of \(\mathbb{E}[V\mid Y]\), proving the result up to equality almost surely. \(\square\)
Worked Mixture Calculations
Worked Example: A Two-Class Discrete Mixture
Suppose \(\mathbb{P}(Z=0)=\tfrac13\) and \(\mathbb{P}(Z=1)=\tfrac23\). Conditional on \(Z=0\), an observed binary variable \(Y\) has success probability \(\tfrac14\); conditional on \(Z=1\), its success probability is \(\tfrac34\). A second binary variable \(V\) is conditionally independent of \(Y\) given \(Z\), with \(\mathbb{P}(V=1\mid Z=0)=\tfrac25\) and \(\mathbb{P}(V=1\mid Z=1)=\tfrac45\). Find \(\mathbb{E}[V\mid Y=1]\).
The observation \(Y=1\) has positive probability. Its class-specific likelihoods are \(\tfrac14\) and \(\tfrac34\), so its marginal probability is \(\tfrac13\cdot\tfrac14+\tfrac23\cdot\tfrac34=\tfrac1{12}+\tfrac12=\tfrac7{12}\). The posterior class weights are therefore \(w_0(1)=(\tfrac13\cdot\tfrac14)/(\tfrac7{12})=\tfrac17\) and \(w_1(1)=(\tfrac23\cdot\tfrac34)/(\tfrac7{12})=\tfrac67\). They are nonnegative and sum to \(\tfrac17+\tfrac67=1\).
Conditional independence means the class-specific mean of \(V\) is unchanged by observing \(Y\): \(r_0(1)=\tfrac25\) and \(r_1(1)=\tfrac45\). Hence \(\mathbb{E}[V\mid Y=1]=\tfrac17\cdot\tfrac25+\tfrac67\cdot\tfrac45=\tfrac2{35}+\tfrac{24}{35}=\tfrac{26}{35}\).
Check by a joint probability. The probability of \(Y=1,V=1\) is \(\tfrac13\cdot\tfrac14\cdot\tfrac25+\tfrac23\cdot\tfrac34\cdot\tfrac45=\tfrac1{30}+\tfrac25=\tfrac{13}{30}\). Dividing by \(\mathbb{P}(Y=1)=\tfrac7{12}\) gives \((\tfrac{13}{30})/(\tfrac7{12})=\tfrac{26}{35}\), as above.
Worked Example: Posterior Weight in a Continuous Mixture
Let \(Z\in\{0,1\}\), with \(\mathbb{P}(Z=0)=\tfrac23\) and \(\mathbb{P}(Z=1)=\tfrac13\). Conditional on \(Z=0\), let \(Y\) have a normal density with mean \(-1\) and variance \(1\); conditional on \(Z=1\), let it have a normal density with mean \(1\) and variance \(1\). Find the posterior probability of class \(1\) at observation \(y=1\), which is also \(\mathbb{E}[Z\mid Y=1]\).
Write \(\phi(u)=(2\pi)^{-1/2}e^{-u^2/2}\) for the standard normal density. The two component densities at \(y=1\) are \(f_0(1)=\phi(2)\) and \(f_1(1)=\phi(0)\). Since \(\phi(2)=\phi(0)e^{-2}\), the marginal density at \(1\) is \(\tfrac23\phi(0)e^{-2}+\tfrac13\phi(0)=\tfrac13\phi(0)(1+2e^{-2})>0\). Thus the posterior weight of class \(1\) is \[ w_1(1)=\frac{\tfrac13\phi(0)}{\tfrac13\phi(0)(1+2e^{-2})} =\frac{1}{1+2e^{-2}}. \] The other posterior weight is \(w_0(1)=\frac{2e^{-2}}{1+2e^{-2}}\), and the two sum to one.
Because \(Z\) equals one in class \(1\) and zero in class \(0\), its conditional mean is the posterior probability: \(\mathbb{E}[Z\mid Y=1]=w_1(1)\). As a check, the posterior odds are \(w_1(1)/w_0(1)=1/(2e^{-2})=e^2/2\), which also equals the prior odds \((\tfrac13)/(\tfrac23)=\tfrac12\) multiplied by the likelihood ratio \(f_1(1)/f_0(1)=e^2\).
Since \(Y\) is continuous, the event \(\{Y=1\}\) has probability zero; this calculation uses the conditional-density version of the posterior, not conditioning on a positive-probability event. The density formula gives the stated value at \(1\) and specifies the conditional probability almost everywhere under the mixture distribution.
Worked Example: Prediction in a Hierarchical Binomial Model
A latent probability \(p\) is selected by a class \(Z\): \(p=\tfrac14\) when \(Z=0\), and \(p=\tfrac34\) when \(Z=1\). The classes have equal prior probabilities. Given \(Z\), three observations \(X_1,X_2,X_3\) and a future observation \(W\) are independent Bernoulli variables with success probability \(p\). Given that \(K=X_1+X_2+X_3=2\), find \(\mathbb{E}[W\mid K=2]\).
The likelihood of \(K=2\) in a class with success probability \(p\) is \(\binom32p^2(1-p)\). For \(p=\tfrac14\), this is \(3\cdot\tfrac1{16}\cdot\tfrac34=\tfrac9{64}\). For \(p=\tfrac34\), it is \(3\cdot\tfrac9{16}\cdot\tfrac14=\tfrac{27}{64}\). Including the equal prior probabilities, the joint probabilities of the class and the observation are \(\tfrac12\cdot\tfrac9{64}=\tfrac9{128}\) and \(\tfrac12\cdot\tfrac{27}{64}=\tfrac{27}{128}\). Their sum is \(\tfrac{36}{128}=\tfrac9{32}>0\).
Consequently, the posterior weights are \(w_0(2)=\tfrac9{36}=\tfrac14\) and \(w_1(2)=\tfrac{27}{36}=\tfrac34\). Given each class and the observed sample, the conditional mean of \(W\) remains its class-specific success probability. Therefore \(\mathbb{E}[W\mid K=2]=\tfrac14\cdot\tfrac14+\tfrac34\cdot\tfrac34=\tfrac1{16}+\tfrac9{16}=\tfrac58\).
Check by the joint probability. The probability of \(K=2,W=1\) is \(\tfrac12\cdot\tfrac9{64}\cdot\tfrac14+\tfrac12\cdot\tfrac{27}{64}\cdot\tfrac34=\tfrac9{512}+\tfrac{81}{512}=\tfrac{90}{512}\). The probability of \(K=2\) is \(\tfrac9{32}=\tfrac{144}{512}\), so the conditional probability of \(W=1\) is \(\tfrac{90}{144}=\tfrac58\). Since \(W\) is binary, this is its conditional expectation, confirming the result.
Common Mistake: Keeping the Prior Weights
A common error is to average the component means using the prior class probabilities even after observing data. In the first example, that incorrect calculation would give \(\tfrac13\cdot\tfrac25+\tfrac23\cdot\tfrac45=\tfrac2{15}+\tfrac8{15}=\tfrac23\). But after observing \(Y=1\), the correct weights are \(\tfrac17\) and \(\tfrac67\), not \(\tfrac13\) and \(\tfrac23\). The correct answer is \(\tfrac{26}{35}\), as verified by the joint probability calculation. The prior-weighted mean would apply before seeing \(Y\), or in a setting where the observation leaves the class probabilities unchanged; it is not the general conditional calculation.
A second point to check is whether the target’s class-specific mean depends on the observed data. In the examples with \(V\) and \(W\), conditional independence made the mean within each class constant in the observation. In general, use \(\mathbb{E}[V\mid Y=y,Z=z]\), not automatically \(\mathbb{E}[V\mid Z=z]\). Finally, posterior weights are only defined by the displayed ratio where the marginal likelihood is positive; an observation impossible under every class cannot be used in that formula.
Check Your Understanding
In each problem, identify the observation likelihoods before calculating the posterior-weighted mean.
- A two-class model has prior probabilities \(\tfrac12,\tfrac12\), and the observed binary outcome has success probabilities \(\tfrac13,\tfrac23\). Given an observed success, find both posterior weights.
- In the first worked example, suppose the class-specific means of \(V\) are instead \(\tfrac12\) and \(\tfrac13\), while the observation model is unchanged. Find \(\mathbb{E}[V\mid Y=1]\).
- Why must the conditional-density version be used in the normal-mixture example rather than the elementary event formula for \(\mathbb{P}(Z=1\mid Y=1)\)?
- In a finite mixture, state the condition under which the component conditional mean \(\mathbb{E}[V\mid Y=y,Z=z]\) can be replaced by \(\mathbb{E}[V\mid Z=z]\).
- Give the posterior-weighted prediction for a future Bernoulli observation when posterior class weights are \(\tfrac25,\tfrac35\) and the class-specific success probabilities are \(\tfrac14,\tfrac34\).