From Posterior Weights to Conditional Moments
In a mixture model, an unobserved variable selects a component, and observations change the probabilities assigned to those components. Calculations can then involve several levels of conditioning: first on the component, then on observed data, and finally on a coarser event. The central discipline is to use posterior—not prior—weights after observing data, and to condition only on events with positive probability.
Throughout, let \((\Omega,\mathcal{F},\mathbb{P})\) be a probability space. The examples use discrete random variables with finite ranges. For an event \(A\) with \(\mathbb{P}(A)>0\), conditional probability means \(\mathbb{P}(B\mid A)=\mathbb{P}(B\cap A)/\mathbb{P}(A)\), and conditional expectation means \(\mathbb{E}[X\mid A]=\mathbb{E}[X\mathbf{1}_A]/\mathbb{P}(A)\), when the expectation exists. Conditional quantities given a variable are defined at values of that variable having positive probability. Values assigned at probability-zero outcomes do not affect an unconditional expectation, but a conditional probability given a probability-zero event is not defined by this ratio.
For nested conditioning, knowing both \(Z\) and \(Y\) gives at least as much information as knowing \(Y\) alone. In finite models, the tower property says that the conditional mean given \(Y=y\) is the weighted average of the conditional means given \(Z=z,Y=y\), using the posterior weights. The conditional variance has a matching decomposition into variation within components and variation between their conditional means.
Proof. For each included \(z\), the event \(\{Z=z,Y=y\}\) has positive probability, so its conditional mean and variance are defined. These events partition \(\{Y=y\}\) apart from components with zero joint probability. By summing the conditional expectation over that partition, \(\mathbb{E}[X\mid Y=y]=\sum_z w_zm_z\). For the variance, use \(\operatorname{Var}(X\mid Y=y)=\mathbb{E}[X^2\mid Y=y]-\mathbb{E}[X\mid Y=y]^2\). On each included component, \(\mathbb{E}[X^2\mid Z=z,Y=y]=v_z+m_z^2\). Thus the variance is \(\sum_z w_z(v_z+m_z^2)-(\sum_z w_zm_z)^2\). Expanding the final sum in the theorem gives \(\sum_z w_zv_z+\sum_z w_zm_z^2-(\sum_z w_zm_z)^2\), because \(\sum_z w_z=1\). This is the same expression, proving both identities. \(\square\)
Worked Calculations
Worked Example: A Test Updates a Two-Component Mixture
Let \(Z\in\{H,L\}\) have prior probabilities \(\mathbb{P}(Z=H)=\tfrac13\) and \(\mathbb{P}(Z=L)=\tfrac23\). A test result \(Y\in\{+,-\}\) satisfies \(\mathbb{P}(Y=+\mid Z=H)=\tfrac34\) and \(\mathbb{P}(Y=+\mid Z=L)=\tfrac14\). Given \(Z=H\), let \(X\) equal \(1\) or \(3\), each with probability \(\tfrac12\); given \(Z=L\), let \(X=0\). Assume \(X\) and \(Y\) are conditionally independent given \(Z\). Find the posterior component probabilities and the conditional mean and variance of \(X\) given a positive test.
First, the probability of a positive test is the sum of the two disjoint component contributions: \(\mathbb{P}(Y=+)=\tfrac13\cdot\tfrac34+\tfrac23\cdot\tfrac14=\tfrac14+\tfrac16=\tfrac5{12}\). This is positive, so conditioning on \(Y=+\) is meaningful. Bayes’ rule gives \(\mathbb{P}(H\mid +)=(\tfrac13\cdot\tfrac34)/(\tfrac5{12})=(\tfrac14)/(\tfrac5{12})=\tfrac35\), and therefore \(\mathbb{P}(L\mid +)=1-\tfrac35=\tfrac25\).
By conditional independence, the distribution of \(X\) within either component is unchanged by the test. Given \(H\), its mean is \((1+3)/2=2\), and its variance is \(((1-2)^2+(3-2)^2)/2=1\). Given \(L\), its mean and variance are both \(0\). The mixture-averaging formula therefore gives \(\mathbb{E}[X\mid +]=\tfrac35\cdot2+\tfrac25\cdot0=\tfrac65\). The conditional variance formula gives \(\operatorname{Var}(X\mid +)=\tfrac35\cdot1+\tfrac25\cdot0+\tfrac35(2-\tfrac65)^2+\tfrac25(0-\tfrac65)^2=\tfrac35+\tfrac{24}{25}+\tfrac{72}{25}=\tfrac{111}{25}\)?
That last line requires care: the between-component terms must be computed with their weights as written. They are \(\tfrac35(\tfrac45)^2=\tfrac{48}{125}\) and \(\tfrac25(\tfrac65)^2=\tfrac{72}{125}\), whose sum is \(\tfrac{120}{125}=\tfrac{24}{25}\). Thus the variance is \(\tfrac35+\tfrac{24}{25}=\tfrac{39}{25}\).
Check by the conditional distribution. Given \(Y=+\), \(X=0\) has probability \(\tfrac25\), while \(X=1\) and \(X=3\) each have probability \(\tfrac35\cdot\tfrac12=\tfrac3{10}\). These probabilities sum to \(\tfrac25+\tfrac3{10}+\tfrac3{10}=1\). The mean is \(0\cdot\tfrac25+1\cdot\tfrac3{10}+3\cdot\tfrac3{10}=\tfrac65\). The second moment is \(0+\tfrac3{10}+9\cdot\tfrac3{10}=3\), so the variance is \(3-(\tfrac65)^2=\tfrac{75}{25}-\tfrac{36}{25}=\tfrac{39}{25}\), as before.
Worked Example: Nested Conditioning Through Two Observations
Let \(Z\in\{A,B\}\) have equal prior probabilities. An observation \(Y\in\{u,v\}\) has \(\mathbb{P}(u\mid A)=\tfrac34\) and \(\mathbb{P}(u\mid B)=\tfrac14\). Conditional distributions of \(X\) are as follows: given \((A,u)\), \(X\) is \(0\) or \(2\) with equal probabilities; given \((A,v)\), \(X=4\); given \((B,u)\), \(X=6\); given \((B,v)\), \(X\) is \(2\) or \(4\) with equal probabilities. Find the conditional means and variances given \(Y\), and verify the tower property and total variance.
The joint probabilities of the component-observation pairs are \(\mathbb{P}(A,u)=\tfrac12\cdot\tfrac34=\tfrac38\), \(\mathbb{P}(B,u)=\tfrac12\cdot\tfrac14=\tfrac18\), \(\mathbb{P}(A,v)=\tfrac12\cdot\tfrac14=\tfrac18\), and \(\mathbb{P}(B,v)=\tfrac12\cdot\tfrac34=\tfrac38\). Consequently \(\mathbb{P}(u)=\tfrac12\) and \(\mathbb{P}(v)=\tfrac12\). All four pairs have positive probability, so all conditional distributions specified here are defined. Bayes’ rule gives \(\mathbb{P}(A\mid u)=\tfrac{3/8}{1/2}=\tfrac34\), \(\mathbb{P}(B\mid u)=\tfrac14\), \(\mathbb{P}(A\mid v)=\tfrac14\), and \(\mathbb{P}(B\mid v)=\tfrac34\).
For \(u\), the component means are \(1\) and \(6\), and the component variances are \(1\) and \(0\). Hence \(\mathbb{E}[X\mid u]=\tfrac34\cdot1+\tfrac14\cdot6=\tfrac94\). Its variance is \(\tfrac34\cdot1+\tfrac14\cdot0+\tfrac34(1-\tfrac94)^2+\tfrac14(6-\tfrac94)^2=\tfrac34+\tfrac{75}{64}+\tfrac{225}{64}=\tfrac{87}{16}\).
For \(v\), the component means are \(4\) and \(3\), and the component variances are \(0\) and \(1\). Therefore \(\mathbb{E}[X\mid v]=\tfrac14\cdot4+\tfrac34\cdot3=\tfrac{13}{4}\). The variance is \(\tfrac14\cdot0+\tfrac34\cdot1+\tfrac14(4-\tfrac{13}{4})^2+\tfrac34(3-\tfrac{13}{4})^2=\tfrac34+\tfrac{9}{64}+\tfrac{3}{64}=\tfrac{15}{16}\).
Tower check. Since \(u,v\) each have probability \(\tfrac12\), averaging the two conditional means gives \(\mathbb{E}[X]=\tfrac12\cdot\tfrac94+\tfrac12\cdot\tfrac{13}{4}=\tfrac{11}{4}\). Averaging the four component means directly gives \(\tfrac38\cdot1+\tfrac18\cdot4+\tfrac18\cdot6+\tfrac38\cdot3=\tfrac{11}{4}\), the same result.
Total-variance check. The average conditional variance given \(Y\) is \(\tfrac12\cdot\tfrac{87}{16}+\tfrac12\cdot\tfrac{15}{16}=\tfrac{51}{16}\). The variance of the conditional means \(\tfrac94,\tfrac{13}{4}\), each with probability \(\tfrac12\), is \(\tfrac14\). Thus \(\operatorname{Var}(X)=\tfrac{51}{16}+\tfrac4{16}=\tfrac{55}{16}\). Directly, \(\mathbb{E}[X^2\mid u]=\tfrac{21}{2}\) and \(\mathbb{E}[X^2\mid v]=\tfrac{23}{2}\): for \(u\), the component second moments are \(2\) and \(36\), giving \(\tfrac34\cdot2+\tfrac14\cdot36=\tfrac{21}{2}\); for \(v\), they are \(16\) and \(10\), giving \(\tfrac14\cdot16+\tfrac34\cdot10=\tfrac{23}{2}\). Hence \(\mathbb{E}[X^2]=\tfrac12(\tfrac{21}{2}+\tfrac{23}{2})=11\), and \(11-(\tfrac{11}{4})^2=\tfrac{55}{16}\), confirming the variance.
Worked Example: A Mixture of Dice After a Successful Event
A fair die is selected with probability \(\tfrac12\); a second die, which has probability \(\tfrac13\) of showing a six on each roll, is selected with probability \(\tfrac12\). Roll the selected die twice. Let \(Z\) identify the die, \(X\) be the number of sixes, and \(Y\) be the event that at least one six appears. Find the posterior die probabilities, \(\mathbb{E}[X\mid Y]\), and \(\operatorname{Var}(X\mid Y)\).
Write \(F\) for the fair die and \(B\) for the second die. Conditional on a die whose six-probability is \(p\), the probability of at least one six in two independent rolls is \(1-(1-p)^2\). Thus \(\mathbb{P}(Y\mid F)=1-(\tfrac56)^2=\tfrac{11}{36}\), and \(\mathbb{P}(Y\mid B)=1-(\tfrac23)^2=\tfrac59\). The event probability is \(\tfrac12\cdot\tfrac{11}{36}+\tfrac12\cdot\tfrac59=\tfrac{11}{72}+\tfrac{20}{72}=\tfrac{31}{72}>0\). Bayes’ rule gives \(\mathbb{P}(B\mid Y)=(\tfrac12\cdot\tfrac59)/(\tfrac{31}{72})=\tfrac{20}{31}\), and \(\mathbb{P}(F\mid Y)=\tfrac{11}{31}\).
Given the die and \(Y\), the count \(X\) can only be \(1\) or \(2\). For six-probability \(p\), the conditional probability of \(X=2\) is \(p^2/[1-(1-p)^2]=p/(2-p)\). Substituting \(p=\tfrac16\) gives \(\mathbb{P}(X=2\mid F,Y)=\tfrac1{11}\); substituting \(p=\tfrac13\) gives \(\mathbb{P}(X=2\mid B,Y)=\tfrac15\). The component means are therefore \(\tfrac{12}{11}\) and \(\tfrac65\). The posterior mixture mean is \(\mathbb{E}[X\mid Y]=\tfrac{11}{31}\cdot\tfrac{12}{11}+\tfrac{20}{31}\cdot\tfrac65=\tfrac{12}{31}+\tfrac{24}{31}=\tfrac{36}{31}\).
For a variable equal to \(1\) or \(2\), with probability \(r\) of being \(2\), its second moment is \(1+3r\), because \(1^2(1-r)+2^2r=1+3r\). Thus the component second moments are \(1+\tfrac3{11}=\tfrac{14}{11}\) and \(1+\tfrac35=\tfrac85\). Their posterior average is \(\mathbb{E}[X^2\mid Y]=\tfrac{11}{31}\cdot\tfrac{14}{11}+\tfrac{20}{31}\cdot\tfrac85=\tfrac{14}{31}+\tfrac{32}{31}=\tfrac{46}{31}\). It follows that \(\operatorname{Var}(X\mid Y)=\tfrac{46}{31}-(\tfrac{36}{31})^2=\tfrac{1426-1296}{961}=\tfrac{130}{961}\).
Check by the conditional variance formula. Within-component variances are \(\tfrac1{11}(1-\tfrac1{11})=\tfrac{10}{121}\) and \(\tfrac15(1-\tfrac15)=\tfrac4{25}\). Their posterior average is \(\tfrac{11}{31}\cdot\tfrac{10}{121}+\tfrac{20}{31}\cdot\tfrac4{25}=\tfrac{10}{341}+\tfrac{16}{155}=\tfrac{226}{1705}\). The between-component term is \(\tfrac{11}{31}\cdot\tfrac{20}{31}\cdot(\tfrac{12}{11}-\tfrac65)^2=\tfrac{220}{961}\cdot\tfrac{36}{3025}=\tfrac{144}{52855}\). Since \(\tfrac{226}{1705}=\tfrac{7006}{52855}\), their sum is \(\tfrac{7150}{52855}=\tfrac{130}{961}\), agreeing with the second-moment calculation.
Checks That Prevent Mixture Errors
A frequent error is to keep the prior weights after seeing data. In the dice example, averaging the component means with prior weights would give \(\tfrac12(\tfrac{12}{11}+\tfrac65)=\tfrac{63}{55}\), not the correct \(\tfrac{36}{31}\). Those component means are conditional on the event \(Y\), so their correct weights are the posterior probabilities \(\tfrac{11}{31}\) and \(\tfrac{20}{31}\), not the prior probabilities \(\tfrac12,\tfrac12\). The direct second-moment check then confirms the resulting variance.
For any finite-mixture calculation, a reliable order is: establish that the conditioning event has positive probability; calculate its probability by summing component contributions; obtain posterior weights; calculate component conditional moments; then average using those posterior weights. When applying the tower property, omit any pair \((z,y)\) with zero joint probability from sums that use conditional means or variances. Such a pair has zero posterior weight given \(y\), but its conditional mean given that null event is not defined by the ordinary event-conditioning formula.
Check Your Understanding
For each problem, identify the conditioning events with positive probability before using conditional quantities.
- A mixture has \(\mathbb{P}(Z=A)=\tfrac14\), \(\mathbb{P}(Z=B)=\tfrac34\), \(\mathbb{P}(Y=+\mid A)=\tfrac12\), and \(\mathbb{P}(Y=+\mid B)=\tfrac16\). Find \(\mathbb{P}(A\mid Y=+)\).
- If the posterior weights are \(\tfrac25,\tfrac35\) and the component means are \(1,4\), compute the mixture mean.
- For a two-component conditional mixture, explain which two terms make up the conditional variance.
- In Worked Example 2, compute \(\mathbb{E}[X\mid v]\) directly from the conditional distribution of \(X\) given \(v\).
- Why must a sum involving \(\mathbb{E}[X\mid Z=z,Y=y]\) omit values \(z\) for which \(\mathbb{P}(Z=z,Y=y)=0\)?