Latent Variables and Conditional Independence
A latent-variable model describes observable random elements through one or more unobserved variables. Its central mathematical object is not merely a diagram of dependencies but a joint probability law: a measure that specifies how the latent and observable variables are generated together.
Probability kernels make this description precise. When the conditional law of two observations given a latent variable factors into a product, the observations are conditionally independent given that variable. Integrating out the latent variable can nevertheless make them dependent. The factorization of the joint law lets us distinguish these two claims carefully.
Kernel Form of a Latent-Variable Model
Let \(E,F,G\) be standard Borel spaces, and let \(Z,X,Y\) be random elements taking values in \(E,F,G\), respectively. Write \(\mu\) for the law of \(Z\). A probability kernel \(K\) from \(E\) to \(F\) assigns a probability measure \(K(z,\cdot)\) on \(F\) to each \(z\in E\), with \(z\mapsto K(z,A)\) measurable for each measurable \(A\subseteq F\). Let \(L\) be a probability kernel from \(E\) to \(G\).
If, conditional on \(Z=z\), \(X\) is generated according to \(K(z,\cdot)\) and \(Y\) according to \(L(z,\cdot)\), with their conditional joint law given by the product kernel, the resulting joint law is specified by the following formula on measurable rectangles:
Here \(C\subseteq E\), \(A\subseteq F\), and \(B\subseteq G\) are measurable. The product kernel \(K(z,\cdot)\otimes L(z,\cdot)\) gives the conditional joint law of \((X,Y)\) given \(Z=z\); the formula therefore describes the entire joint distribution, not just the separate conditional distributions of \(X\) and \(Y\).
The conditional-independence statement concerns the joint law after conditioning on \(Z\). It does not assert that \(X\) and \(Y\) are independent without conditioning. In fact, their common dependence on \(Z\) can produce marginal dependence.
Worked Example: A Binary Latent Variable
Worked Example: A Binary Latent Variable
Let \(Z\) take the values \(0\) and \(1\), each with probability \(1/2\). Given \(Z=z\), let \(X\) and \(Y\) be conditionally independent Bernoulli random variables with success probabilities \(p_z\) and \(q_z\), where \(p_0=0.1,\ p_1=0.8,\ q_0=0.2,\) and \(q_1=0.9\). Their joint kernel is the product of the two Bernoulli kernels, so \(X\perp\!\!\!\perp Y\mid Z\).
For example, when \(Z=0\), the conditional probability that both equal \(1\) is \(0.1\cdot0.2=0.02\), which is the product of their conditional success probabilities. When \(Z=1\), it is \(0.8\cdot0.9=0.72\). Averaging over \(Z\) gives
The marginal success probabilities are \(\mathbb{P}(X=1)=\frac12(0.1+0.8)=0.45\) and \(\mathbb{P}(Y=1)=\frac12(0.2+0.9)=0.55\). If \(X\) and \(Y\) were marginally independent, their joint success probability would be \(0.45\cdot0.55=0.2475\), not \(0.37\). Thus conditional independence given \(Z\) holds, while marginal independence does not.
The covariance makes the same distinction: \(\operatorname{Cov}(X,Y)=\mathbb{E}[XY]-\mathbb{E}[X]\mathbb{E}[Y]=0.37-0.2475=0.1225\). This nonzero value is consistent with conditional independence because the calculation integrates over the shared latent variable.
The Joint-Law Factorization Criterion
The kernel formulation provides a direct criterion. Suppose \(K\) and \(L\) are versions of the conditional laws of \(X\) and \(Y\) given \(Z\), and \(M\) is a regular conditional law of \((X,Y)\) given \(Z\). The key question is whether \(M(z,\cdot)\) equals \(K(z,\cdot)\otimes L(z,\cdot)\) for almost every latent value.
Proof. Standard Borel spaces have countable generating collections that can be chosen to be closed under finite intersections. Choose such collections \(\mathcal{A}\) for \(F\) and \(\mathcal{B}\) for \(G\). For each \(A\in\mathcal{A}\) and \(B\in\mathcal{B}\), the conditional-independence definition gives
There are only countably many such pairs. The union of their exceptional null sets is therefore a null set \(N\). For every \(z\notin N\), the two probability measures \(M(z,\cdot)\) and \(K(z,\cdot)\otimes L(z,\cdot)\) agree on every rectangle \(A\times B\) with \(A\in\mathcal{A}\) and \(B\in\mathcal{B}\). These rectangles form a generating collection closed under finite intersections, so uniqueness of probability measures implies that the two measures agree on all measurable subsets of \(F\times G\). This proves the factorization almost everywhere.
Conversely, suppose the stated equality of kernels holds outside a \(\mu\)-null set. On every rectangle \(A\times B\), it gives \(M(z,A\times B)=K(z,A)L(z,B)\) for almost every \(z\). By the defining property of the conditional kernels, these are the conditional probabilities of the joint event and of its two marginal events, respectively. The conditional-independence definition follows. \(\square\)
Indeed, the displayed formula identifies the conditional joint law given \(Z=z\) as the product kernel, while the marginal conditional laws are \(K\) and \(L\). The theorem then applies. This is why writing down the joint law is so useful: the product structure is visible directly.
Worked Example: A Gaussian Random-Effects Model
Worked Example: A Gaussian Random-Effects Model
Let \(Z\sim N(0,\tau^2)\), and let \(\varepsilon_1\sim N(0,\sigma_1^2)\) and \(\varepsilon_2\sim N(0,\sigma_2^2)\) be independent of one another and of \(Z\), with \(\tau^2,\sigma_1^2,\sigma_2^2>0\). Set \(X=Z+\varepsilon_1\) and \(Y=Z+\varepsilon_2\). Given \(Z=z\), the conditional laws are \(N(z,\sigma_1^2)\) and \(N(z,\sigma_2^2)\). The independent errors give a product conditional law, hence \(X\perp\!\!\!\perp Y\mid Z\).
Unconditionally, both variables have mean \(0\). Their marginal variances are \(\tau^2+\sigma_1^2\) and \(\tau^2+\sigma_2^2\), while
The cross terms have expectation zero by independence and zero means, and \(\mathbb{E}[\varepsilon_1\varepsilon_2]=0\). Since \(\tau^2>0\), the covariance is nonzero, so \(X\) and \(Y\) are not marginally independent. The shared random effect creates dependence after it is integrated out.
Worked Example: A Beta–Bernoulli Hierarchy
Worked Example: A Beta–Bernoulli Hierarchy
Let \(\Theta\sim\operatorname{Beta}(2,2)\). Given \(\Theta=\theta\), let \(X_1\) and \(X_2\) be conditionally independent Bernoulli random variables with success probability \(\theta\). Their joint conditional probability of both successes is \(\theta^2\), so integrating the product kernel gives
Each marginal success probability is \(\mathbb{E}[\Theta]=2/(2+2)=1/2\). Their product is \(1/4\), which differs from \(3/10\). In particular, their covariance is \(3/10-1/4=1/20\). Thus the observations are conditionally independent given the latent success probability but marginally dependent. Learning that one observation is a success changes the distribution of the shared \(\Theta\), and therefore changes the success probability for the other.
Why the Product Must Be a Joint-Law Statement
Knowing the conditional marginal laws alone does not establish conditional independence. The product must describe the conditional joint law. For a simple check, let \(Z\) be constant and let \(X\) be a fair Bernoulli random variable with \(Y=X\). Both conditional marginal success probabilities are \(1/2\), but
The conditional joint law is not the product of the conditional marginals, so conditional independence fails. This is the practical use of the factorization criterion: specify the full hierarchical law, then check whether the conditional joint kernel factors. Separate descriptions of the two conditional marginals are not enough.
Check Your Understanding
Use the kernel factorization criterion to answer these questions.
- In a model with joint law \(\mu(dz)K(z,dx)L(z,dy)\), what does the product \(K(z,\cdot)\otimes L(z,\cdot)\) represent?
- Why can \(X\) and \(Y\) be conditionally independent given \(Z\) but have nonzero marginal covariance?
- In the Gaussian random-effects example, which term contributes to \(\operatorname{Cov}(X,Y)\), and why do the other terms vanish?
- Why do the conditional marginal distributions of \(X\) and \(Y\) not, by themselves, imply conditional independence?
- In the Beta–Bernoulli example, compare the joint success probability with the product of the marginal success probabilities.