Posterior Component Probabilities in Mixture Models

Why Posterior Component Probabilities Matter

A mixture distribution describes a population in which an observation can arise from one of several component distributions. The component label is often unobserved, but learning which component is more plausible after seeing the data is useful in its own right. It also gives a principled way to average component-specific predictions or expectations.

Let $Z$ denote the latent component label and $X$ the observed variable. Before observing $X$, the probabilities of the labels are the mixture weights. After observing $X=x$, the probabilities generally change: values more likely under one component give that component greater posterior weight. The calculation is Bayes’ rule applied to the joint mixture model.

Posterior Weights for a Finite Mixture

Definition: Let $Z$ take values in $\{1,\ldots,k\}$, with $P(Z=j)=\pi_j$, where $\pi_j\geq 0$ and $\sum_{j=1}^k\pi_j=1$. Suppose the conditional distribution of $X$ given $Z=j$ has density or mass function $f_j$, relative to a common dominating measure $\mu$. The marginal density or mass function is $$ f(x)=\sum_{j=1}^k \pi_j f_j(x). $$ At values $x$ for which $f(x)>0$, the posterior component probability is $$ w_j(x)=P(Z=j\mid X=x)=\frac{\pi_j f_j(x)}{\sum_{\ell=1}^k \pi_\ell f_\ell(x)}. $$

For discrete $X$, this formula is the ordinary conditional probability ratio whenever $P(X=x)>0$. For continuous $X$, $P(X=x)=0$, so conditioning on the exact value is understood through a regular conditional distribution. The displayed ratio gives a version of that conditional probability wherever the marginal density is positive; as with conditional distributions generally, versions can differ on sets of marginal probability zero.

The common dominating measure formulation allows the same calculation for a finite mixture of mass functions or densities. In familiar settings, $\mu$ is counting measure for discrete variables and Lebesgue measure for continuous variables. Where $f(x)=0$, all component contributions $\pi_j f_j(x)$ sum to zero, and the ratio is undefined. Such an $x$ is outside the mixture’s support up to sets of probability zero, so the model does not determine posterior weights there.

Why the Formula Gives Conditional Probabilities

Theorem: Under the finite-mixture assumptions above, define $w_j(x)=\pi_j f_j(x)/f(x)$ when $f(x)>0$, and assign any values on $\{x:f(x)=0\}$. Then $w_j(X)$ is a version of $E[\mathbf{1}_{\{Z=j\}}\mid X]$. In particular, it is a version of $P(Z=j\mid X)$.

To verify the conditional probability, it is enough to check the defining integral identity over every measurable set $B$ in the range of $X$. On $\{f>0\}$, $w_j(x)f(x)=\pi_j f_j(x)$. On $\{f=0\}$, every nonnegative term $\pi_\ell f_\ell(x)$ must be zero, so $\pi_j f_j(x)=0$ there as well. Therefore,

\begin{aligned} E[\mathbf{1}_{\{Z=j\}}\mathbf{1}_{\{X\in B\}}] &=P(Z=j,X\in B)\\ &=\pi_j\int_B f_j(x)\,d\mu(x)\\ &=\int_B w_j(x)f(x)\,d\mu(x)\\ &=E[w_j(X)\mathbf{1}_{\{X\in B\}}]. \end{aligned}

This identity holds for every measurable $B$, which is precisely the defining property of the conditional expectation $E[\mathbf{1}_{\{Z=j\}}\mid X]$. Since an indicator’s conditional expectation is the corresponding conditional probability, the result follows. Also, where $f(x)>0$, the weights are nonnegative and sum to one because

\sum_{j=1}^k w_j(x) =\frac{\sum_{j=1}^k\pi_j f_j(x)}{f(x)} =1.

Thus the posterior weights form a probability distribution over the possible labels after observing $X=x$. They combine two factors: the prior probability $\pi_j$ and the component’s relative support for the observed value, $f_j(x)$.

Worked Example: Posterior Weights in an Exponential Mixture

Suppose $P(Z=1)=0.25$ and $P(Z=2)=0.75$. Given $Z=1$, $X$ has an exponential distribution with rate $1$; given $Z=2$, it has an exponential distribution with rate $3$. Find the posterior component probabilities at $x=1$, and the posterior expectation of the component rate.

For an exponential distribution with rate $\lambda$, the density at $x\geq 0$ is $\lambda e^{-\lambda x}$. The two weighted density contributions at $x=1$ are

\begin{aligned} \pi_1 f_1(1)&=0.25e^{-1}\approx 0.09196986,\\ \pi_2 f_2(1)&=0.75(3e^{-3})\approx 0.11202090. \end{aligned}

Their sum, the mixture density at $1$, is approximately $0.20399076$. Hence

\begin{aligned} P(Z=1\mid X=1) &=\frac{0.25e^{-1}}{0.25e^{-1}+0.75(3e^{-3})} \approx 0.450853,\\ P(Z=2\mid X=1) &=\frac{0.75(3e^{-3})}{0.25e^{-1}+0.75(3e^{-3})} \approx 0.549147. \end{aligned}

The probabilities sum to one, up to rounding. If the component rate is denoted by $\Lambda$, then $\Lambda=1$ when $Z=1$ and $\Lambda=3$ when $Z=2$. Its conditional expectation is

E[\Lambda\mid X=1] =1(0.450853)+3(0.549147) \approx 2.098294.

The prior mean rate is $0.25(1)+0.75(3)=2.5$. Observing $X=1$ raises the posterior probability of the lower-rate component, so the posterior mean rate is lower than the prior mean.

Using Posterior Weights for Conditional Expectations

The same weights apply when the target is an unobserved quantity whose conditional expectation depends on the component. Let $Y$ be an integrable random variable, and suppose the conditional expectations $m_j(x)=E[Y\mid X=x,Z=j]$ are defined by regular conditional distributions. Then conditioning first on the component label gives a weighted average of the component-specific expectations.

Theorem: Suppose $Y$ is integrable, $Z$ has finitely many values, and regular conditional distributions for the variables involved exist. For marginal-almost every $x$, $$ E[Y\mid X=x]=\sum_{j=1}^k P(Z=j\mid X=x)\,E[Y\mid X=x,Z=j], $$ where terms for components with zero posterior probability may be assigned arbitrarily. Equivalently, $E[Y\mid X]$ is the corresponding weighted sum evaluated at $X$.

To prove the identity, write $m_j(X)=E[Y\mid X,Z=j]$ on events where that conditional expectation is relevant. For any measurable set $B$ in the range of $X$, split the expectation according to the finitely many possible values of $Z$:

\begin{aligned} E[Y\mathbf{1}_{\{X\in B\}}] &=\sum_{j=1}^k E[Y\mathbf{1}_{\{X\in B\}}\mathbf{1}_{\{Z=j\}}]\\ &=\sum_{j=1}^k E[m_j(X)\mathbf{1}_{\{X\in B\}}\mathbf{1}_{\{Z=j\}}]\\ &=\sum_{j=1}^k E[m_j(X)w_j(X)\mathbf{1}_{\{X\in B\}}]\\ &=E\left[\mathbf{1}_{\{X\in B\}}\sum_{j=1}^k w_j(X)m_j(X)\right]. \end{aligned}

The second equality uses the defining property of the conditional expectation given $(X,Z)$; the third uses $E[\mathbf{1}_{\{Z=j\}}\mid X]=w_j(X)$ and the fact that $m_j(X)\mathbf{1}_{\{X\in B\}}$ is a function of $X$. The resulting identity for every $B$ is the defining property of $E[Y\mid X]$, proving the theorem. $\square$

A particularly simple case is a latent component-specific quantity $a_Z$, where $a_j$ is a fixed value in component $j$. Then

E[a_Z\mid X=x]=\sum_{j=1}^k w_j(x)a_j.

More generally, $Y$ need not be determined by the component. Its within-component conditional mean $m_j(x)$ can depend on the observation, and the posterior weights average those means.

Worked Example: A Poisson Mixture and a Posterior Prediction

Let $P(Z=1)=0.4$ and $P(Z=2)=0.6$. Conditional on $Z=1$, $X$ is Poisson with mean $1$; conditional on $Z=2$, it is Poisson with mean $4$. After observing $X=2$, calculate the posterior probability of component 2. Then suppose a future count $X_{\mathrm{new}}$ is conditionally independent of $X$ given $Z$, with the same Poisson distribution as $X$. Find $E[X_{\mathrm{new}}\mid X=2]$.

The weighted probabilities of observing $2$ are

\begin{aligned} \pi_1P(X=2\mid Z=1) &=0.4\left(e^{-1}\frac{1^2}{2!}\right) =0.2e^{-1}\approx 0.07357589,\\ \pi_2P(X=2\mid Z=2) &=0.6\left(e^{-4}\frac{4^2}{2!}\right) =4.8e^{-4}\approx 0.08791507. \end{aligned}

The marginal probability of $X=2$ is their sum, approximately $0.16149095$. Therefore,

P(Z=2\mid X=2) =\frac{4.8e^{-4}}{0.2e^{-1}+4.8e^{-4}} =\frac{24e^{-3}}{1+24e^{-3}} \approx 0.544396.

The posterior probability of component 1 is $1-0.544396=0.455604$. Conditional independence and the component distributions give $E[X_{\mathrm{new}}\mid X=2,Z=1]=1$ and $E[X_{\mathrm{new}}\mid X=2,Z=2]=4$. Averaging these means with the posterior weights,

E[X_{\mathrm{new}}\mid X=2] =1(0.455604)+4(0.544396) \approx 2.633188.

The posterior predictive mean is not the observed count $2$, nor is it the prior mixture mean $0.4(1)+0.6(4)=2.8$. It is the average future count under the updated probabilities of the two components.

Worked Example: Averaging Component-Specific Conditional Means

Suppose $P(Z=1)=0.3$ and $P(Z=2)=0.7$, and $X$ is binary with $P(X=1\mid Z=1)=0.2$ and $P(X=1\mid Z=2)=0.8$. Given $X=x$ and $Z=1$, suppose $Y$ has mean $10+2x$; given $X=x$ and $Z=2$, suppose $Y$ has mean $4+5x$. Find $E[Y\mid X=1]$.

First calculate the posterior weights using the probabilities of $X=1$:

\begin{aligned} P(Z=1\mid X=1) &=\frac{0.3(0.2)}{0.3(0.2)+0.7(0.8)} =\frac{0.06}{0.62} =\frac{3}{31},\\ P(Z=2\mid X=1) &=\frac{0.7(0.8)}{0.62} =\frac{28}{31}. \end{aligned}

At $x=1$, the component-specific conditional means of $Y$ are $10+2(1)=12$ and $4+5(1)=9$. Thus

E[Y\mid X=1] =\frac{3}{31}(12)+\frac{28}{31}(9) =\frac{36+252}{31} =\frac{288}{31} \approx 9.290323.

The component weights are posterior probabilities, not the original mixture probabilities $0.3$ and $0.7$. Reusing the prior weights would ignore the information about $Z$ carried by the observation $X=1$.

Conditioning on an Event Instead of a Value

Sometimes the information is an event such as $X\in A$, rather than a particular observed value. Provided $P(X\in A)>0$, the posterior probability of component $j$ is

P(Z=j\mid X\in A) =\frac{\pi_j\int_A f_j(x)\,d\mu(x)} {\sum_{\ell=1}^k \pi_\ell\int_A f_\ell(x)\,d\mu(x)}.

This is obtained by calculating $P(Z=j,X\in A)$ in the numerator and $P(X\in A)$ in the denominator. It also equals the average of $w_j(X)$ among observations in $A$, with the distribution of $X$ conditional on that event:

P(Z=j\mid X\in A)=E[w_j(X)\mid X\in A].

It is generally not correct to choose a representative value $x\in A$ and use $w_j(x)$ as the event-level posterior. The posterior weights can vary throughout $A$; event-level conditioning averages them over the conditional distribution of $X$ in that event.

Common Pitfall and Practical Interpretation

A frequent mistake is to report $\pi_j$, the prior mixture weight, as the probability that an observed case belongs to component $j$. That is correct before seeing the observation, but generally not afterward. The posterior probability must account for how likely the observed value is under each component. For example, if $f_j(x)$ is much larger than the other component densities at $x$, its posterior weight can substantially exceed its prior weight.

A second pitfall is treating a continuous observation as if it had positive probability at an exact point. The density-ratio formula is not obtained by dividing $P(Z=j,X=x)$ by $P(X=x)$, since both probabilities are zero in the continuous case. Rather, it is a version of the conditional probability derived from the joint and marginal densities, as the integral proof above shows. It is meaningful where the marginal density is positive, with the usual almost-everywhere qualification.

Key takeaway: Update each component’s prior weight by multiplying it by that component’s density or mass at the observed value, then normalize across components. To obtain a conditional expectation, average the component-specific conditional expectations using these posterior weights.

Check Your Understanding

Try these questions before consulting the worked calculations again.

In a two-component mixture, what conditions make the ratio defining $P(Z=j\mid X=x)$ meaningful?
Why does the continuous case require interpreting the posterior formula as a version of a conditional distribution rather than as a ratio of point probabilities?
A mixture has prior weights $0.2$ and $0.8$, and the component densities at an observation are $0.5$ and $0.1$. Calculate both posterior component probabilities.
State the formula for $E[Y\mid X=x]$ in terms of posterior component weights and $E[Y\mid X=x,Z=j]$.
Explain why conditioning on $X\in A$ generally requires integrating component densities over $A$, rather than evaluating posterior weights at one point in $A$.