Conditional Independence and Its Factorization

Why Conditional Independence Needs Its Own Definition

Knowing a third variable can change the relationship between two others. For example, two outcomes may be independent once a group label is known, even though they are associated when the groups are combined. Or two outcomes may be independent overall but dependent within each level of a conditioning variable. Conditional independence describes the relationship after the information in that variable has been taken into account.

The central test is whether the conditional joint law factors into the two conditional marginal laws. This formulation applies to both discrete probability masses and continuous densities, but the factorization must be checked under the conditional law—not merely under the unconditional distribution.

Definition Through Conditional Laws

Let \(X\) and \(Y\) be real-valued random variables, and let \(Z\) be a random element taking values in a standard Borel space. Conditional distributions of \((X,Y)\), \(X\), and \(Y\) given \(Z=z\) can then be represented by probability kernels. Write \(K_{XY}(z;\cdot)\) for the conditional law of \((X,Y)\) given \(Z=z\), and \(K_X(z;\cdot)\) and \(K_Y(z;\cdot)\) for the corresponding marginal conditional laws.

“Almost every” is with respect to the distribution of \(Z\): the factorization is required for conditioning values that occur in the probabilistic sense, not for values outside the support or for exceptional values of probability zero. The conditional probabilities in the second display are random variables determined by \(Z\). The definition says their product gives the conditional probability of the joint event.

When \(Z\) is discrete, this definition can be checked separately at each value \(z\) with \(P(Z=z)>0\). When densities exist, it becomes a factorization of conditional densities, as derived below.

The Discrete Factorization Criterion

Suppose \(X,Y,Z\) are countable-valued random variables. For any \(z\) with \(P(Z=z)>0\), define the conditional probability masses \(p_{xy\mid z}=P(X=x,Y=y\mid Z=z)\), \(p_{x\mid z}=P(X=x\mid Z=z)\), and \(p_{y\mid z}=P(Y=y\mid Z=z)\).

Proof. If \(X\perp Y\mid Z\), apply the definition to the singleton sets \(A=\{x\}\) and \(B=\{y\}\). This gives the stated equality for each \(x,y\) at each positive-probability value \(z\).

Conversely, suppose the equality holds for every \(x,y\) at a fixed \(z\) with positive probability. For any subsets \(A\) and \(B\) of the respective countable state spaces, countable additivity gives $$ \begin{aligned} P(X\in A,Y\in B\mid Z=z) &=\sum_{x\in A}\sum_{y\in B}p_{xy\mid z}\\ &=\sum_{x\in A}\sum_{y\in B}p_{x\mid z}p_{y\mid z}\\ &=\left(\sum_{x\in A}p_{x\mid z}\right) \left(\sum_{y\in B}p_{y\mid z}\right)\\ &=P(X\in A\mid Z=z)P(Y\in B\mid Z=z). \end{aligned} $$ The sums contain nonnegative terms, so the factorization of the double sum is valid, including for infinite countable state spaces. This is exactly the conditional-law definition. \(\square\)

The Density Factorization Criterion

Now suppose \((X,Y,Z)\) has a joint density \(f_{X,Y,Z}(x,y,z)\), and the relevant marginal densities exist. Integrating the joint density gives $$ f_Z(z)=\int_{\mathbb{R}}\int_{\mathbb{R}}f_{X,Y,Z}(x,y,z)\,dx\,dy, $$ $$ f_{X,Z}(x,z)=\int_{\mathbb{R}}f_{X,Y,Z}(x,y,z)\,dy, \qquad f_{Y,Z}(y,z)=\int_{\mathbb{R}}f_{X,Y,Z}(x,y,z)\,dx. $$ For almost every \(z\) with \(f_Z(z)>0\), conditional densities can be chosen as $$ f_{X,Y\mid Z}(x,y\mid z)=\frac{f_{X,Y,Z}(x,y,z)}{f_Z(z)},\quad f_{X\mid Z}(x\mid z)=\frac{f_{X,Z}(x,z)}{f_Z(z)},\quad f_{Y\mid Z}(y\mid z)=\frac{f_{Y,Z}(y,z)}{f_Z(z)}. $$ The ratios are not used where \(f_Z(z)=0\), since those conditioning values do not determine a conditional law through this density formula.

Proof. Fix a \(z\) with \(f_Z(z)>0\) for which the conditional density formulas hold. If the conditional joint density is the product of the two conditional marginal densities almost everywhere, integrating over \(A\times B\) gives $$ \begin{aligned} P(X\in A,Y\in B\mid Z=z) &=\int_A\int_B f_{X,Y\mid Z}(x,y\mid z)\,dy\,dx\\ &=\left(\int_A f_{X\mid Z}(x\mid z)\,dx\right) \left(\int_B f_{Y\mid Z}(y\mid z)\,dy\right)\\ &=P(X\in A\mid Z=z)P(Y\in B\mid Z=z). \end{aligned} $$ The separation of the integrals follows from the product form and Tonelli’s theorem, which applies to these nonnegative densities. Thus the conditional laws factor.

In the other direction, suppose the conditional laws factor. Their conditional joint distribution is then the product of the two conditional marginal distributions. Since each has a density, that product distribution has density \(f_{X\mid Z}(x\mid z)f_{Y\mid Z}(y\mid z)\). Densities of the same distribution agree almost everywhere, so this product equals \(f_{X,Y\mid Z}(x,y\mid z)\) almost everywhere. Finally, substituting the three ratio formulas gives $$ \frac{f_{X,Y,Z}(x,y,z)}{f_Z(z)} =\frac{f_{X,Z}(x,z)}{f_Z(z)} \frac{f_{Y,Z}(y,z)}{f_Z(z)}, $$ which, after multiplication by \(f_Z(z)^2\), is the stated joint-density identity. The reverse algebra recovers the conditional-density product whenever \(f_Z(z)>0\). \(\square\)

Common Mistakes and Why the Distinction Matters

• Replacing conditional independence with marginal independence. A product rule given \(Z\) concerns conditional probabilities or densities. It does not, by itself, give a product rule after averaging over \(Z\), as the first example demonstrates.
• Assuming marginal independence survives conditioning. Conditioning changes the distribution being examined. In the last example, \(X\) and \(Y\) are marginally independent, but knowing their sum links their possible values.
• Using a density identity where the conditioning density is zero. The ratio formulas require \(f_Z(z)>0\). The criterion is an almost-everywhere statement and makes no claim about conditional densities at unsupported values.
• Checking only one pair of outcomes or one event. Conditional independence requires factorization for all relevant events. In countable settings the singleton mass criterion checks all pairs; with densities, the full conditional density must factor almost everywhere.

The factorization criterion is useful because it translates a conceptual claim—no remaining conditional association—into a calculation with the joint and marginal conditional laws. It also keeps two questions separate: whether \(X\) and \(Y\) are independent overall, and whether they are independent after \(Z\) is known. Neither answer can generally be substituted for the other.