Iterated Conditioning for Conditional Distributions

Why Average Over an Intermediate Variable?

A conditional distribution can be easier to describe after conditioning on additional information. For example, a response may have a simple distribution once a latent class or an intermediate measurement is known, even when its distribution given only the observed group is more complicated. The connection between these two descriptions is an averaging identity: first compute the conditional probability at each value of the intermediate variable, then average those probabilities using the intermediate variable’s conditional distribution.

The identity is sometimes described informally as “conditioning twice.” Its precise content is that conditioning on less information averages out the additional information. The averaging distribution matters: when conditioning on $Z=z$, average over the conditional law of $Y$ given $Z=z$, not over the marginal law of $Y$.

Conditional Laws and the Iterated Conditioning Identity

Definition: A conditional law of $X$ given $Z$ is a probability kernel $Q_z(A)$ such that, for each Borel set $A$, $Q_Z(A)$ is a version of $P(X\in A\mid Z)$. A probability kernel is a probability distribution in $A$ for each fixed $z$, and is measurable as a function of $z$ for each fixed $A$. For real-valued random variables, such conditional laws exist; they are determined only up to sets of $z$-values having probability zero under the distribution of $Z$.

Let $X,Y,Z$ be real-valued random variables. Write $K_{y,z}(A)$ for a conditional law of $X$ given $(Y,Z)=(y,z)$, and $H_z(dy)$ for a conditional law of $Y$ given $Z=z$. The iterated conditioning identity says that the law of $X$ given $Z=z$ is obtained by mixing $K_{y,z}$ over $H_z$.

Theorem (Iterated Conditioning): For every Borel set $A$, a version of the conditional probability of $X\in A$ given $Z$ is $$ P(X\in A\mid Z=z)=\int K_{y,z}(A)\,H_z(dy), $$ for $P_Z$-almost every $z$. Equivalently, $$ P(X\in A\mid Z)=E\!\left[P(X\in A\mid Y,Z)\mid Z\right]\quad\text{almost surely}. $$

The equality is an almost-sure statement, not necessarily a claim about every possible value of $z$. In particular, when $Z$ has a continuous distribution, the conditional law specified at any single point can be changed without affecting its status as a version if that point has probability zero.

Proof and Useful Forms

To prove the kernel form, define $Q_z(A)=\int K_{y,z}(A)\,H_z(dy)$. This is a probability kernel: it is a probability measure in $A$, and its measurability in $z$ follows from the measurability of the conditional kernels and integration with respect to $H_z$. We verify that it is a conditional law of $X$ given $Z$.

For any Borel set $C$ in the range of $Z$, the defining property of conditional laws and iterated integration give

\begin{aligned} \int_C Q_z(A)\,P_Z(dz) &=\int_C\int K_{y,z}(A)\,H_z(dy)\,P_Z(dz)\\ &=P(X\in A,\ Z\in C). \end{aligned}

The last equality follows by first integrating the conditional law of $X$ given $(Y,Z)$, then the conditional law of $Y$ given $Z$. This identity for every $C$ is exactly the defining property of a conditional probability given $Z$. Uniqueness of conditional probabilities up to almost-sure equality proves the result. The conditional-expectation form follows by applying the same defining property to the indicator $1_{\{X\in A\}}$. $\square$

For a discrete intermediate variable $Y$, the integral becomes a sum over its possible values:

P(X\in A\mid Z=z) =\sum_y P(X\in A\mid Y=y,Z=z)\,P(Y=y\mid Z=z).

For a continuous $Y$ with conditional density $f_{Y\mid Z}(y\mid z)$, this becomes an integral:

P(X\in A\mid Z=z) =\int P(X\in A\mid Y=y,Z=z)\,f_{Y\mid Z}(y\mid z)\,dy.

If the conditional law of $X$ given $(Y,Z)=(y,z)$ has density $f_{X\mid Y,Z}(x\mid y,z)$, then the conditional density of $X$ given $Z=z$ can be written

f_{X\mid Z}(x\mid z) =\int f_{X\mid Y,Z}(x\mid y,z)\,H_z(dy).

When $H_z$ has density $f_{Y\mid Z}(y\mid z)$, substitute $H_z(dy)=f_{Y\mid Z}(y\mid z)\,dy$. This density formula holds for almost every $x$, for $P_Z$-almost every $z$. It follows by integrating the nonnegative conditional density over $y$; Tonelli’s theorem permits the order of integration, and the resulting function integrates to one because each conditional density does.

Worked Examples

Worked Example: Averaging Over a Discrete Intermediate Variable

Suppose that, within group $Z=1$, an intermediate indicator $Y$ equals 1 with probability 0.30. The probability of an event $A=\{X=1\}$ is 0.80 when $Y=1,Z=1$, and 0.20 when $Y=0,Z=1$. Find $P(X=1\mid Z=1)$.

The two possible values of $Y$ have conditional probabilities $P(Y=1\mid Z=1)=0.30$ and $P(Y=0\mid Z=1)=1-0.30=0.70$. Iterated conditioning gives

\begin{aligned} P(X=1\mid Z=1) &=P(X=1\mid Y=1,Z=1)P(Y=1\mid Z=1)\\ &\quad+P(X=1\mid Y=0,Z=1)P(Y=0\mid Z=1)\\ &=(0.80)(0.30)+(0.20)(0.70)\\ &=0.24+0.14=0.38. \end{aligned}

Thus the event has conditional probability 0.38 in group $Z=1$. The probabilities 0.80 and 0.20 are not averaged equally: their weights are the conditional probabilities of the corresponding $Y$-values within that group.

Worked Example: A Conditional Density from a Continuous Intermediate Variable

Fix $z>0$. Suppose $Y\mid Z=z$ is uniform on $(z,z+1)$, and, given $Y=y,Z=z$, $X$ is uniform on $(0,y)$. Derive $f_{X\mid Z}(x\mid z)$.

The density of $Y\mid Z=z$ is 1 on $(z,z+1)$. The conditional density of $X$ given $Y=y,Z=z$ is $1/y$ for $0<x<y$, and 0 otherwise. Therefore, for $0<x<z+1$,

f_{X\mid Z}(x\mid z) =\int_z^{z+1}\frac{1}{y}\,1_{\{x<y\}}\,dy =\int_{\max(z,x)}^{z+1}\frac{1}{y}\,dy.

Evaluating the integral gives the piecewise density

f_{X\mid Z}(x\mid z)= \begin{cases} \log\!\left(\dfrac{z+1}{z}\right),&0<x<z,\\[6pt] \log\!\left(\dfrac{z+1}{x}\right),&z\le x<z+1,\\[6pt] 0,&\text{otherwise}. \end{cases}

The result is normalized. Using the integral representation and Tonelli’s theorem,

\begin{aligned} \int_0^{z+1} f_{X\mid Z}(x\mid z)\,dx &=\int_z^{z+1}\frac{1}{y}\left(\int_0^y dx\right)dy\\ &=\int_z^{z+1}1\,dy=1. \end{aligned}

This calculation also shows why the conditional support changes with the intermediate value: when $Y=y$, $X$ cannot exceed $y$. The mixture accounts for all $y$-values allowed by $Z=z$.

Worked Example: A Mixed Latent-Class Model

Let $Y$ be a binary latent class. Within a fixed group $Z=z$, suppose $P(Y=1\mid Z=z)=1/4$ and $P(Y=0\mid Z=z)=3/4$. Given $Y=1,Z=z$, let $X$ have an exponential distribution with rate 1; given $Y=0,Z=z$, let it have an exponential distribution with rate 2. Find $P(X>1\mid Z=z)$.

An exponential distribution with rate $\lambda$ has survival probability $P(X>t)=e^{-\lambda t}$ for $t\ge 0$. Applying the discrete mixture identity at $t=1$,

\begin{aligned} P(X>1\mid Z=z) &=P(X>1\mid Y=1,Z=z)P(Y=1\mid Z=z)\\ &\quad+P(X>1\mid Y=0,Z=z)P(Y=0\mid Z=z)\\ &=e^{-1}\left(\frac14\right)+e^{-2}\left(\frac34\right)\\ &=\frac{e^{-1}+3e^{-2}}{4}\approx 0.1935. \end{aligned}

The final probability is a mixture of the two class-specific survival probabilities. It is not the survival probability from an exponential distribution with a rate formed by simply averaging the two rates: mixing distributions generally does not preserve the original distributional family.

Common Pitfalls and Why the Identity Matters

Iterated conditioning is useful whenever a model has a latent class, an intermediate measurement, or a sequence of observations. It turns a complex conditional law into an average of simpler conditional laws, and it provides a direct check on whether probabilities computed under a more detailed conditioning scheme agree with the less detailed model.

Using the wrong weights. To find a probability given $Z=z$, use $P(Y=y\mid Z=z)$, not $P(Y=y)$. These weights can differ when $Y$ and $Z$ are associated.
Replacing an average by one conditional value. In general, $P(A\mid Z)$ is not $P(A\mid Y,Z)$ evaluated at a convenient or typical value of $Y$. It is the conditional average over the full distribution of $Y$ given $Z$.
Ignoring null sets in continuous conditioning. Conditional distributions at continuous values are specified only up to almost-sure equality. A formula may need to be adjusted on a set of conditioning values with probability zero without changing the conditional law.
Assuming a mixture stays in the same family. Averaging conditional densities is valid, but the resulting density need not have the same named distribution as the component densities.

A reliable calculation begins by identifying the information retained in the final conditioning variable, then writing the conditional distribution of the intermediate variable under that information. Each conditional probability or density is weighted by that law and integrated or summed across its support.

Check Your Understanding

Use the identity to reason through each question before calculating.

If $Y$ takes three possible values, what are the weights in the expression for $P(X\in A\mid Z=z)$?
Why does a continuous intermediate variable lead to an integral rather than a sum?
In the conditional-density example, why is the lower integration limit $\max(z,x)$?
Suppose $P(Y=1\mid Z=z)$ differs from $P(Y=1)$. Which probability must weight the $Y=1$ conditional law when computing $P(X\in A\mid Z=z)$?
Why does the iterated conditioning identity not require the conditional distribution specified at every individual continuous value $z$ to be unique?