Marginal and Conditional Independence Counterexamples

Why Independence Can Change When We Condition

Independence describes a relationship under a particular probability law. Conditioning changes that law: it restricts attention to a stratum or updates probabilities using observed information. As a result, two variables that are independent before conditioning need not remain independent within a stratum. The reverse issue also matters: variables that are independent within every stratum of a third variable need not be independent after those strata are combined.

These are not contradictions. Marginal independence and conditional independence are distinct properties, and neither generally implies the other. The examples below use finite distributions so each claim can be checked directly from the probabilities.

Definition: Random variables \(X\) and \(Y\) are marginally independent if their joint distribution factors into their marginal distributions. For discrete variables, this means \(P(X=x,Y=y)=P(X=x)P(Y=y)\) for every \(x,y\). They are conditionally independent given \(Z\) if, for almost every value of \(Z\), their conditional joint distribution factors into their conditional marginal distributions. In the discrete case, for every \(z\) with \(P(Z=z)>0\), this means \(P(X=x,Y=y\mid Z=z)=P(X=x\mid Z=z)P(Y=y\mid Z=z)\) for every \(x,y\).

A condition with probability zero is not a stratum on which an ordinary conditional probability is defined by the ratio rule. In the finite examples, we therefore check conditional statements only at values of \(Z\) or events that have positive probability.

Marginal Independence Can Disappear After Conditioning

A simple way to produce this first counterexample is to let a third variable record a relationship between two independent variables. Observing that relationship can restrict which pairs of values are possible.

Worked Example: Independent Bits Given Their Exclusive-or

Let \(X\) and \(Y\) be independent Bernoulli random variables with success probability \(1/2\). Define \(Z=X\mathbin{\mathrm{xor}}Y\), so \(Z=0\) when \(X=Y\) and \(Z=1\) when \(X\ne Y\). The four possible pairs \((X,Y)\) each have probability \(1/4\).

First check marginal independence. Since \(P(X=1)=P(Y=1)=1/2\), and \(P(X=1,Y=1)=1/4\), the joint probability equals the product of the marginal probabilities. The same check for all four pairs gives \(P(X=x,Y=y)=1/4=P(X=x)P(Y=y)\).

Now condition on \(Z=0\). This event consists of \((X,Y)=(0,0)\) and \((1,1)\), each with conditional probability \((1/4)/(1/2)=1/2\). Thus \(P(X=1\mid Z=0)=P(Y=1\mid Z=0)=1/2\), but \(P(X=1,Y=1\mid Z=0)=1/2\), not \(1/4\), the product of the conditional marginals. The variables are dependent given \(Z=0\).

Conditioning on \(Z=1\) gives the other pair of outcomes, \((0,1)\) and \((1,0)\), each with probability \(1/2\). In that stratum, \(P(X=1,Y=1\mid Z=1)=0\), while both conditional marginal probabilities are \(1/2\). So \(X\) and \(Y\) are dependent given \(Z=1\) as well, but in the opposite direction: knowing one variable determines the other.

The original independence claim remains true marginally. It is the restriction to a value of \(Z\) that changes the joint law.

Conditioning on a selection event can have the same effect. A selection rule may exclude some combinations of values, even when the variables were independent in the population before selection.

Worked Example: Selection on at Least One Success

Let \(A\) and \(B\) be independent Bernoulli variables with success probability \(1/2\). Let \(C=1\) when at least one of \(A\) and \(B\) equals 1, and \(C=0\) otherwise. Before selection, each pair \((A,B)\) has probability \(1/4\), so \(A\) and \(B\) are independent.

The event \(C=1\) contains \((1,0),(0,1),(1,1)\). Its probability is \(3/4\), so each of these three pairs has conditional probability \((1/4)/(3/4)=1/3\) given \(C=1\). Consequently, \(P(A=1\mid C=1)=2/3\) and \(P(B=1\mid C=1)=2/3\).

If \(A\) and \(B\) were conditionally independent in this selected group, the conditional probability that both equal 1 would be \((2/3)(2/3)=4/9\). The actual probability is \(P(A=1,B=1\mid C=1)=1/3\). Since \(1/3\ne4/9\), they are conditionally dependent given \(C=1\).

The negative association has a direct explanation: among selected pairs, observing \(A=0\) forces \(B=1\), while observing \(A=1\) leaves two possible values of \(B\). Selection has linked the variables even though they were independent before it.

Conditional Independence Can Disappear When We Mix Strata

To see the reverse direction, consider a latent variable \(Z\) that divides the population into groups. If \(X\) and \(Y\) are conditionally independent within each group, their marginal distributions are obtained by mixing the group-specific distributions. When the groups have different success probabilities for both variables, that mixing can make \(X\) and \(Y\) associated.

Worked Example: Independent Within Groups, Associated Overall

Let \(Z\) be Bernoulli with \(P(Z=0)=P(Z=1)=1/2\). Given \(Z\), let \(X\) and \(Y\) be conditionally independent Bernoulli variables with \(P(X=1\mid Z=1)=P(Y=1\mid Z=1)=0.9\) and \(P(X=1\mid Z=0)=P(Y=1\mid Z=0)=0.1\).

Conditional independence holds in both groups. For example, given \(Z=1\), \(P(X=1,Y=1\mid Z=1)=0.9(0.9)=0.81\); given \(Z=0\), that probability is \(0.1(0.1)=0.01\). The same factorization applies to the other pairs of values.

Mix the two groups to calculate the marginal probability that both variables equal 1: \(P(X=1,Y=1)=(1/2)(0.81)+(1/2)(0.01)=0.41\). Each marginal success probability is \(P(X=1)=(1/2)(0.9)+(1/2)(0.1)=0.5\), and likewise \(P(Y=1)=0.5\). If the variables were marginally independent, their joint success probability would be \(0.5(0.5)=0.25\), not \(0.41\).

For a full check, the marginal joint table is

\begin{array}{c|cc} & Y=0 & Y=1\\ \hline X=0 & 0.41 & 0.09\\ X=1 & 0.09 & 0.41 \end{array}

For instance, \(P(X=0,Y=1)=(1/2)(0.9)(0.1)+(1/2)(0.1)(0.9)=0.09\). The entries sum to 1, and each marginal is \(0.5\) for either value. The table confirms that \(X\) and \(Y\) are conditionally independent given \(Z\), but not marginally independent.

Why Mixing Can Create Association

The previous example has a general explanation through the law of total covariance. Suppose \(X\) and \(Y\) have finite second moments. Define conditional covariance by \(\operatorname{Cov}(X,Y\mid Z)=E[XY\mid Z]-E[X\mid Z]E[Y\mid Z]\). Then

Theorem: If \(E[X^2]<\infty\) and \(E[Y^2]<\infty\), then

\operatorname{Cov}(X,Y) = E[\operatorname{Cov}(X,Y\mid Z)] + \operatorname{Cov}(E[X\mid Z],E[Y\mid Z]).

Proof. By the defining property of conditional expectation,

\begin{aligned} \operatorname{Cov}(X,Y) &=E[XY]-E[X]E[Y]\\ &=E\!\left[E[XY\mid Z]\right]-E[X]E[Y]\\ &=E\!\left[\operatorname{Cov}(X,Y\mid Z)+E[X\mid Z]E[Y\mid Z]\right]-E[X]E[Y]\\ &=E[\operatorname{Cov}(X,Y\mid Z)] +\left(E[E[X\mid Z]E[Y\mid Z]]-E[X]E[Y]\right)\\ &=E[\operatorname{Cov}(X,Y\mid Z)] +\operatorname{Cov}(E[X\mid Z],E[Y\mid Z]). \end{aligned}

The last equality uses \(E[E[X\mid Z]]=E[X]\) and \(E[E[Y\mid Z]]=E[Y]\). The second moments ensure the expectations and covariances in the identity are finite. \(\square\)

If \(X\) and \(Y\) are conditionally independent given \(Z\), then their conditional covariance is zero (when the relevant conditional moments are finite). The identity reduces to

\operatorname{Cov}(X,Y) = \operatorname{Cov}(E[X\mid Z],E[Y\mid Z]).

Thus a shared pattern across groups can produce marginal covariance: if groups with higher conditional means of \(X\) also have higher conditional means of \(Y\), the right-hand side can be positive. In the Bernoulli example, \(E[X\mid Z]=E[Y\mid Z]\) equals 0.9 or 0.1 with equal probabilities. Its variance is \((1/2)(0.9-0.5)^2+(1/2)(0.1-0.5)^2=0.16\), so the marginal covariance is 0.16, agreeing with \(P(X=1,Y=1)-P(X=1)P(Y=1)=0.41-0.25=0.16\).

Common Mistake and Practical Check

A common mistake is to treat conditional independence as though it were automatically preserved when strata are combined: “\(X\) and \(Y\) are independent at each value of \(Z\), so they are independent overall.” The latent-group example disproves this reasoning. Conditional probabilities are mixed using the probabilities of the strata, and the resulting joint probabilities need not equal the products of the resulting marginal probabilities.

The converse shortcut is equally unsafe. Marginal independence does not guarantee independence after conditioning, as the exclusive-or example shows. A reliable check is to name the distribution under consideration and verify the relevant factorization there: use the marginal joint distribution for marginal independence, and the conditional joint distribution within each positive-probability stratum for conditional independence.

Key takeaway: Marginal independence and conditional independence are different factorization statements. Conditioning can create dependence by restricting possible pairs; mixing conditionally independent groups can create marginal dependence through differences in their conditional distributions.

Check Your Understanding

For each question, distinguish the distribution being examined before deciding whether independence holds.

In the exclusive-or example, what is \(P(X=1,Y=1\mid Z=0)\), and why does it contradict conditional independence?
For the selection event \(C=1\), calculate the product of the conditional marginal probabilities and compare it with the conditional joint probability that both variables equal 1.
In the latent-group example, calculate \(P(X=1)\), \(P(Y=1)\), and \(P(X=1,Y=1)\). Which factorization fails marginally?
Under conditional independence given \(Z\), what does the law of total covariance say about the source of any marginal covariance, assuming finite second moments?
Can marginal independence alone establish conditional independence given a third variable? Explain using one counterexample from this tutorial.