Total Probability as a Mixture of Conditional Laws

Why Conditional Laws Combine into an Unconditional Law

A population may consist of several groups, or a random outcome may have different distributions under different scenarios. If the group or scenario is known, we can describe the outcome with a conditional distribution. If it is not known, the unconditional distribution combines those conditional laws, with each one weighted by the probability of its group. This is the mixture form of the law of total probability.

The weights matter: a group that occurs more often contributes more to the overall distribution. The mixture operation is a weighted sum of probabilities, not an unweighted average and not a product of the component distributions.

Definition: Let $A_1,A_2,\ldots$ be a finite or countable collection of disjoint events whose union has probability one, and suppose $P(A_j)>0$ for each event included. Write $w_j=P(A_j)$. The conditional law of a random variable $X$ given $A_j$ is the $j$th component law. Their probability-weighted mixture assigns weight $w_j$ to that component. The weights are nonnegative and $\sum_j w_j=1$.

If a partition event has probability zero, the elementary conditional probability $P(B\mid A_j)$ is not defined by division by $P(A_j)$. Such an event can be omitted from the mixture: its weight is zero. We therefore state the formulas using only positive-probability events. The partition may cover the whole sample space, or it may cover it up to an event of probability zero.

The Mixture Formula for Distributions

For any Borel set $B$ on the real line, the unconditional probability that $X$ belongs to $B$ is the sum of the probabilities that $X$ belongs to $B$ and the partition event $A_j$ occurs. Since $P(A_j)>0$, conditional probability gives $P(X\in B,A_j)=P(A_j)P(X\in B\mid A_j)$.

Theorem: If $(A_j)$ is a finite or countable partition up to probability zero, with $P(A_j)>0$ for the events included, then for every Borel set $B$, $$ P(X\in B)=\sum_j P(A_j)P(X\in B\mid A_j). $$ In particular, if $F_j(x)=P(X\le x\mid A_j)$ and $F(x)=P(X\le x)$, then $$ F(x)=\sum_j P(A_j)F_j(x),\qquad x\in\mathbb{R}. $$

Proof. The events $\{X\in B\}\cap A_j$ are disjoint. Their union agrees with $\{X\in B\}$ except possibly on the probability-zero part of the sample space not covered by the partition. Countable additivity therefore gives $$ P(X\in B)=\sum_j P(\{X\in B\}\cap A_j). $$ For each included event, the definition of conditional probability yields $$ P(\{X\in B\}\cap A_j)=P(A_j)P(X\in B\mid A_j). $$ Substituting these equalities gives the stated formula. Taking $B=(-\infty,x]$ gives the CDF formula. $\square$

The formula applies to a distribution as a whole, not only to a particular way of recording it. For a discrete $X$, take $B=\{x\}$ to obtain the pmf formula. For other types of distributions, probabilities of intervals or other Borel sets are obtained by the same weighted sum.

Corollary: If $X$ is discrete, with conditional pmf $f_j(x)=P(X=x\mid A_j)$, then its unconditional pmf is $$ f(x)=P(X=x)=\sum_j P(A_j)f_j(x). $$

This follows directly from the theorem with $B=\{x\}$. The right-hand side is nonnegative. Summing over the possible values of $x$ gives $$ \sum_x f(x)=\sum_x\sum_j P(A_j)f_j(x) =\sum_j P(A_j)\sum_x f_j(x) =\sum_j P(A_j)=1. $$ The interchange is valid because every term is nonnegative; equivalently, the order of summation can be exchanged for a nonnegative double series. Thus the mixture is a properly normalized pmf.

Why a Weighted Sum of CDFs Is a CDF

The mixture formula also guarantees that combining component CDFs in this way produces a valid CDF. This is immediate when the partition events arise from an actual joint model, but it is useful to verify what makes a proposed weighted mixture valid.

Result: Let $F_j$ be CDFs and let $w_j\ge 0$ be finitely or countably many weights with $\sum_j w_j=1$. Then $G(x)=\sum_j w_jF_j(x)$ is a CDF.

Proof. Each $F_j$ is nondecreasing, so for $x\le y$, $F_j(x)\le F_j(y)$. Multiplication by $w_j\ge0$ and summation show $G(x)\le G(y)$. Also, $0\le G(x)\le\sum_jw_j=1$.

For right-continuity, let $x_m\downarrow x$. Then $F_j(x_m)\to F_j(x)$ for each $j$, by right-continuity of each component CDF. Since $0\le F_j(x_m)\le1$, dominated convergence for the probability measure that assigns mass $w_j$ to index $j$ gives $G(x_m)\to G(x)$. Finally, as $x\to-\infty$, every $F_j(x)\to0$; as $x\to+\infty$, every $F_j(x)\to1$. The same dominated-convergence argument gives $G(x)\to0$ and $G(x)\to\sum_jw_j=1$, respectively. Thus $G$ has all the defining properties of a CDF. $\square$

Worked Examples

Worked Example: Combining Two Conditional PMFs

A production line uses machine A for 60% of items and machine B for 40%. Let $X$ be the number of surface flaws on an item. The conditional pmfs are:

For machine A: $P(X=0\mid A)=0.20$, $P(X=1\mid A)=0.50$, and $P(X=2\mid A)=0.30$. For machine B: $P(X=0\mid B)=0.50$, $P(X=1\mid B)=0.25$, and $P(X=2\mid B)=0.25$. Find the unconditional pmf and CDF.

The mixture weights are $P(A)=0.60$ and $P(B)=0.40$. For each possible value, weight each conditional probability and add:

\begin{aligned} P(X=0)&=(0.60)(0.20)+(0.40)(0.50)=0.12+0.20=0.32,\\ P(X=1)&=(0.60)(0.50)+(0.40)(0.25)=0.30+0.10=0.40,\\ P(X=2)&=(0.60)(0.30)+(0.40)(0.25)=0.18+0.10=0.28. \end{aligned}

The probabilities sum to $0.32+0.40+0.28=1$, as required. Since $X$ takes values $0,1,2$, its CDF is

F(x)= \begin{cases} 0, & x<0,\\ 0.32, & 0\le x<1,\\ 0.72, & 1\le x<2,\\ 1, & x\ge2. \end{cases}

For example, $F(1)=P(X\le1)=0.32+0.40=0.72$. The CDF is a step function, with jumps equal to the unconditional point probabilities.

Worked Example: A Piecewise CDF from Uniform Components

Suppose a scenario indicator selects one of two conditional distributions: with probability $1/4$, $X$ is uniform on $(0,1)$, and with probability $3/4$, $X$ is uniform on $(0,2)$. Find the unconditional CDF.

For $0\le x<1$, the component CDFs are $F_1(x)=x$ and $F_2(x)=x/2$. Hence

F(x)=\frac14x+\frac34\frac{x}{2}=\frac{5x}{8},\qquad 0\le x<1.

For $1\le x<2$, the first component CDF has reached one, while the second is still $x/2$. Thus

F(x)=\frac14(1)+\frac34\frac{x}{2} =\frac14+\frac{3x}{8},\qquad 1\le x<2.

Outside the supports, the CDF is zero to the left of zero and one from $x=2$ onward. In full,

F(x)= \begin{cases} 0, & x<0,\\ \frac{5x}{8}, & 0\le x<1,\\ \frac14+\frac{3x}{8}, & 1\le x<2,\\ 1, & x\ge2. \end{cases}

At the join $x=1$, the first expression approaches $5/8$, and the second gives $1/4+3/8=5/8$. At $x=2$, the second expression reaches $1/4+6/8=1$. These checks confirm that the pieces fit together and that there are no jumps at the endpoints, as expected for this continuous mixture.

For instance, $P(X\le1.5)=F(1.5)=1/4+(3/8)(1.5)=1/4+9/16=13/16$.

Worked Example: A Countable Mixture with Stepwise CDF

Let $(A_j)_{j\ge1}$ be a countable partition with $P(A_j)=2^{-j}$. Suppose $X=j$ with probability one conditional on $A_j$. Find the unconditional pmf and calculate $P(X\le3)$.

The weights form a probability distribution because $$ \sum_{j=1}^{\infty}2^{-j}=\frac{1/2}{1-1/2}=1. $$ For any positive integer $k$, the conditional probability $P(X=k\mid A_j)$ equals one if $j=k$, and zero otherwise. The mixture formula therefore gives

P(X=k)=\sum_{j=1}^{\infty}2^{-j}P(X=k\mid A_j)=2^{-k}.

There is no probability at nonpositive or noninteger values. In particular, $$ P(X\le3)=P(X=1)+P(X=2)+P(X=3) =\frac12+\frac14+\frac18=\frac78. $$ More generally, for a positive integer $k$, $F(k)=\sum_{j=1}^{k}2^{-j}=1-2^{-k}$. This example shows that the countable formula works even when there are infinitely many possible partition events.

Common Mistakes and a Practical Check

A frequent error is to average component distributions without using the probabilities of the partition events. In the production example, averaging the two conditional probabilities of zero flaws equally would give $(0.20+0.50)/2=0.35$. But machine A and machine B are not equally likely; the correct value is $0.60(0.20)+0.40(0.50)=0.32$.

Another error is to combine only the conditional probabilities of a selected outcome while forgetting that the weights must come from the partition. The relevant components must be mutually exclusive and collectively exhaustive up to probability zero. Otherwise, simply summing their weighted contributions may omit outcomes or count some outcomes more than once. Before calculating, check that the component weights are nonnegative and sum to one.

Once the weights and conditional laws are correctly specified, the calculation is direct: apply the same weights to the probability of the event, to the CDF value at a chosen point, or to the pmf at a chosen value. For a discrete outcome, a final normalization check should give $\sum_x f(x)=1$; for a CDF, check its limits, monotonicity, and right-continuity.

Check Your Understanding

Use the mixture formula and show enough work to make the role of the weights clear.

A partition has two events with probabilities $0.3$ and $0.7$. For a fixed $x$, the conditional CDF values are $0.2$ and $0.6$. What is the unconditional CDF value at $x$?
Two conditional pmfs assign probabilities $0.4$ and $0.1$ to the value $X=2$. Their partition-event probabilities are $0.25$ and $0.75$. Find $P(X=2)$.
Why can a zero-probability partition event be omitted from the elementary mixture formula?
What properties must the weights in a finite or countable mixture satisfy?
For the countable example, calculate $P(X\le2)$ and compare it with $F(2)$.