Finite Mixture Laws from Latent Component Labels

Finite Mixtures and Latent Labels

A population may contain several subgroups, each following a different probability law. If the subgroup membership of an observation is not recorded, its distribution is not one of the subgroup distributions alone. It is a finite mixture: the component laws combined according to the probabilities of the corresponding subgroup labels. The label provides a precise way to describe which component generated the observation, even when that label is unobserved.

Definition: Let $J$ take values in $\{1,\ldots,k\}$, with $P(J=j)=\pi_j>0$ and $\sum_{j=1}^k\pi_j=1$. Let $X$ be real-valued, and suppose its conditional distribution given $J=j$ is $P_j$, with distribution function $F_j$. The marginal distribution of $X$ is the finite mixture of $P_1,\ldots,P_k$ with weights $\pi_1,\ldots,\pi_k$. The variable $J$ is the latent component label.

The definition separates two stages of the model: first a component label is selected with probabilities $\pi_j$, then an observation is generated according to that component's law. The label is latent when it is part of the probability model but is not observed along with $X$. The weights are nonnegative and sum to one because they are probabilities; in this definition they are taken to be strictly positive so that each stated conditional law is relevant.

The Mixture Law for Events and Distribution Functions

The fundamental formula applies to every measurable event, not just intervals. The distribution function formula follows by taking the event to be $\{X\leq x\}$.

Theorem: For every Borel set $A\subseteq\mathbb{R}$, the mixture law satisfies $$ P(X\in A)=\sum_{j=1}^k \pi_j P_j(A). $$ Consequently its distribution function $F$ is $$ F(x)=P(X\leq x)=\sum_{j=1}^k \pi_j F_j(x), \qquad x\in\mathbb{R}. $$

Proof. The events $\{J=j\}$, for $j=1,\ldots,k$, are disjoint and their union is the whole sample space. Partition the event $\{X\in A\}$ according to these labels and use the law of total probability:

\begin{aligned} P(X\in A) &=\sum_{j=1}^k P(X\in A,\ J=j)\\ &=\sum_{j=1}^k P(X\in A\mid J=j)P(J=j)\\ &=\sum_{j=1}^k P_j(A)\pi_j. \end{aligned}

Setting $A=(-\infty,x]$ gives the stated formula for $F(x)$. $\square$

This argument also verifies that the weighted component laws form a probability law: the mixture assigns nonnegative probability to every event, and for $A=\mathbb{R}$ the formula gives $\sum_j\pi_j=1$. No assumption that the components are all discrete or all continuous is needed.

Worked Example: A Mixture of Two Binomial Laws

Let $P(J=1)=1/4$ and $P(J=2)=3/4$. Given $J=1$, let $X\sim\operatorname{Binomial}(2,0.2)$; given $J=2$, let $X\sim\operatorname{Binomial}(2,0.8)$. Find the probability mass function of $X$.

For $x=0$, the component probabilities are $(1-0.2)^2=0.64$ and $(1-0.8)^2=0.04$. Therefore,

P(X=0)=\frac14(0.64)+\frac34(0.04)=0.16+0.03=0.19.

For $x=1$, the component probabilities are $2(0.2)(0.8)=0.32$ and $2(0.8)(0.2)=0.32$, giving

P(X=1)=\frac14(0.32)+\frac34(0.32)=0.32.

For $x=2$, the probabilities are $0.2^2=0.04$ and $0.8^2=0.64$, so

P(X=2)=\frac14(0.04)+\frac34(0.64)=0.01+0.48=0.49.

The resulting probabilities sum to $0.19+0.32+0.49=1$. Thus the mixture mass function is $0.19$ at 0, $0.32$ at 1, and $0.49$ at 2. Its distribution function takes the values $0.19$ for $0\leq x<1$, $0.51$ for $1\leq x<2$, and 1 for $x\geq2$, and is 0 for $x<0$.

Mass Functions, Densities, and Atoms

The event formula yields the familiar pointwise formulas when component laws have mass functions or densities. For a discrete mixture, take a countable set containing the supports of all components. If $f_j(x)=P_j(X=x)$ denotes component $j$'s mass at $x$, then

P(X=x)=\sum_{j=1}^k \pi_j f_j(x).

For a continuous mixture, if every component is absolutely continuous with respect to Lebesgue measure and has density $f_j$, then the mixture also has a Lebesgue density. This follows by applying the event formula to any Borel set $A$:

P(X\in A) =\sum_{j=1}^k\pi_j\int_A f_j(x)\,dx =\int_A\left(\sum_{j=1}^k\pi_j f_j(x)\right)dx.

Thus $f(x)=\sum_j\pi_j f_j(x)$ is a density of the mixture. The finite sum can be brought inside the integral by linearity. A mixture can also have atoms, including when some components are continuous and others put positive probability at individual points.

Result: For any real number $a$, the probability that a mixture equals $a$ is $$ P(X=a)=\sum_{j=1}^k\pi_j P_j(X=a). $$ In particular, if every component has no atom at $a$, the mixture has no atom there. If a component has an atom at $a$, the mixture's probability at $a$ includes that component's atom multiplied by its weight.

To verify this result, apply the event formula with the Borel set $A=\{a\}$. This also explains the jumps of the mixture distribution function: its jump at $a$ is exactly $F(a)-\lim_{x\uparrow a}F(x)=P(X=a)$, the weighted sum of the component jumps.

There is a density formulation that remains available even when the components are discrete, continuous, or mixed. Let $P_j$ denote the component probability measures and define the finite measure $\nu=\sum_{j=1}^k P_j$. Each $P_j$ is absolutely continuous with respect to $\nu$, so it has a Radon–Nikodym density $g_j=dP_j/d\nu$. The mixture has density $g=\sum_j\pi_jg_j$ with respect to $\nu$: for every Borel set $A$,

\int_A g\,d\nu =\sum_{j=1}^k\pi_j\int_A g_j\,d\nu =\sum_{j=1}^k\pi_jP_j(A) =P(X\in A).

This density is with respect to $\nu$, not necessarily with respect to Lebesgue measure. A point mass cannot be represented by an ordinary Lebesgue density, so when a mixture has atoms the distinction matters.

Worked Example: An Exponential Mixture

Let $P(J=1)=1/4$, $P(J=2)=3/4$, and let $X$ conditional on $J=1$ have an exponential distribution with rate 1, while $X$ conditional on $J=2$ has rate 2. Both component densities are zero for $x<0$. For $x\geq0$, their distribution functions are $F_1(x)=1-e^{-x}$ and $F_2(x)=1-e^{-2x}$. Therefore,

F(x)=\frac14(1-e^{-x})+\frac34(1-e^{-2x}),\qquad x\geq0.

For $x<0$, $F(x)=0$. Differentiating the formula on $x>0$ gives the mixture density

f(x)=\frac14e^{-x}+\frac32e^{-2x},\qquad x>0.

Its integral is $\frac14\int_0^\infty e^{-x}\,dx+\frac32\int_0^\infty e^{-2x}\,dx=\frac14+\frac34=1$, as required. At $x=\log 2$, the component distribution functions are $1/2$ and $3/4$, so $F(\log 2)=\frac14\cdot\frac12+\frac34\cdot\frac34=\frac{11}{16}$. Since both components are continuous, the mixture has no atom at any point.

A Mixture with Both an Atom and a Continuous Part

Worked Example: A Point Mass Mixed with a Uniform Law

Let $P(J=1)=0.4$ and $P(J=2)=0.6$. Conditional on $J=1$, let $X=0$ with probability 1. Conditional on $J=2$, let $X$ be uniform on $(0,2)$. The first component places all its mass at 0; the second has density $1/2$ on $(0,2)$. The mixture has an atom of size

$$ P(X=0)=0.4(1)+0.6(0)=0.4. $$

For $x<0$, the distribution function is 0. At $x=0$, it jumps to 0.4. For $0\leq x<2$, the point-mass component contributes 0.4 and the uniform component contributes $0.6(x/2)$. For $x\geq2$, both components have accumulated all their probability. Hence

F(x)= \begin{cases} 0, & x<0,\\ 0.4+0.3x, & 0\leq x<2,\\ 1, & x\geq2. \end{cases}

On $(0,2)$, the mixture has Lebesgue density $0.3$, and its atom at 0 has probability 0.4. The continuous part has total probability $0.3(2)=0.6$; together with the atom, this gives total probability 1. For instance, $P(0<X\leq1)=0.3$, whereas $P(X\leq1)=0.4+0.3=0.7$. The difference is the atom at 0.

Common Mistakes and Why the Construction Matters

A common mistake is to average component parameters or component descriptions and treat the result as the mixture law. The mixture construction instead averages probabilities of events: $P(X\in A)=\sum_j\pi_jP_j(A)$. The distribution function formula follows from this event identity, and pointwise formulas for masses or densities require their stated conditions.

Forgetting the weights. Adding component mass functions or densities without multiplying each by its weight generally gives a function with the wrong total probability.
Assuming every mixture has a Lebesgue density. A point-mass component contributes an atom, which cannot be represented by a Lebesgue density. State the atom separately or use a dominating measure that includes it.
Using component-specific formulas outside their scope. A weighted sum of Lebesgue densities describes the entire law only when all components are absolutely continuous. The event formula for the distribution function does not require that condition.

The latent-label construction is useful whenever a population is divided into a finite number of types or states but the type is missing, unrecorded, or deliberately treated as random. It gives a single probability law for the observed variable while retaining an explicit account of how its component distributions contribute. The event-based formulation is especially valuable because it continues to apply without change when a model combines discrete, continuous, or atomic components.

Check Your Understanding

Use the latent-label definition and the event formula to answer these questions.

A mixture has three component weights $0.2$, $0.3$, and $0.5$. Express its distribution function in terms of the three component distribution functions.
Suppose two components assign probabilities $0.1$ and $0.4$ to the point $a$, and their weights are $0.6$ and $0.4$. What is the mixture probability $P(X=a)$?
Why does a weighted sum of component Lebesgue densities fail to describe the entire law if one component is a point mass?
A mixture has component densities $f_1$ and $f_2$ and weights $\pi$ and $1-\pi$. Write its Lebesgue density and state the condition under which this formula describes the whole law.
In the point-mass and uniform example, explain why $P(X\leq1)$ differs from $P(0<X\leq1)$.