Deriving Conditionals From a Joint Distribution

Why Derive Conditionals From a Joint Distribution?

In chained equations, each variable with missing values is modeled conditionally on other variables. A useful way to understand how those conditional models relate is to begin with a joint distribution: one probability law for the variables together. Factoring that law in either direction produces conditional distributions that must agree with one another because they come from the same source.

This direction of reasoning is important. A joint distribution does not merely offer another way to write two conditional models; it places shared constraints on them. We will use a binary pair to make those constraints explicit. The derivation applies to a joint probability mass function on any finite discrete support, while the central factorization also has a density analogue.

Factoring a Joint Distribution

For any pair of discrete variables, summing the joint probabilities over one variable gives a marginal distribution. Dividing a joint probability by the corresponding marginal then gives a conditional probability. Thus, wherever the relevant marginal probabilities are positive, the same joint law can be factored in either direction:

The equality is not an additional assumption: each product equals \(p(x,y)\) by the definition of conditional probability. The conditions matter, however. If \(p_X(x)=0\), the expression \(p_{Y\mid X}(y\mid x)\) is not determined by the joint law at that value of \(x\), because conditioning on an event of probability zero does not define a unique conditional distribution in this discrete setting.

For binary variables, write \(p_{ij}=P(X=i,Y=j)\), for \(i,j\in\{0,1\}\). If all four cells are positive, their sum is one and both conditional distributions are determined. For example, \(P(Y=1\mid X=0)=p_{01}/(p_{00}+p_{01})\), whereas \(P(X=1\mid Y=0)=p_{10}/(p_{00}+p_{10})\). These are different conditional probabilities, but their numerators and denominators come from the same four joint cells.

Shared Parameters in Binary Conditional Models

The relationship becomes especially clear in a log-linear parameterization of a binary joint distribution. Assign a weight to each cell by \(w(x,y)=\exp(\alpha x+\beta y+\gamma xy)\), then divide each weight by their sum \(Z\) to obtain probabilities. The normalizing constant is \(Z=1+\exp(\alpha)+\exp(\beta)+\exp(\alpha+\beta+\gamma)\), ensuring that the four cell probabilities sum to one.

To derive the conditional law of \(Y\) given \(X=x\), hold \(x\) fixed. The terms that vary with \(y\) are \(\beta y+\gamma xy\), or \((\beta+\gamma x)y\). The two conditional weights for \(y=0\) and \(y=1\) are therefore proportional to \(1\) and \(\exp(\beta+\gamma x)\). Their ratio gives the conditional odds. Similarly, conditioning on \(Y=y\), the odds for \(X=1\) versus \(X=0\) are \(\exp(\alpha+\gamma y)\). Consequently:

Here \(\operatorname{logit}(q)=\log\{q/(1-q)\}\). The intercepts \(\beta\) and \(\alpha\) differ because the two conditionals have different outcomes. The coefficient \(\gamma\), however, is shared: it describes the association contribution in both directions. This equality is a consequence of deriving both conditionals from one joint distribution, not a coincidence of two unrelated regressions.

To verify the result, the conditional odds of \(Y=1\) at \(X=1\) are \(p_{11}/p_{10}\), and at \(X=0\) they are \(p_{01}/p_{00}\). Their ratio is \((p_{11}p_{00})/(p_{10}p_{01})\). The conditional logit difference is the logarithm of that ratio. Reversing the roles of \(X\) and \(Y\) gives the same cross-product ratio, so the coefficient in the other conditional model is the same quantity.

Recovering a Joint Law From One Conditional

The factorization also shows what information is needed to reconstruct a joint distribution. A marginal distribution for one variable and a conditional distribution of the other given it suffice: multiply them cell by cell. The resulting cells must be nonnegative and sum to one. The reverse conditional can then be calculated from the reconstructed joint law whenever its conditioning marginal is positive.

Checks, Limitations, and Relevance to MICE

When a proposed pair of conditional models is intended to come from a common joint law, shared-parameter reasoning gives practical checks. For a binary pair with positive cells, calculate the log odds change in each conditional direction; both must equal the log odds ratio of the same joint table. More generally, reconstruct joint probabilities where possible and check that each conditional probability is recovered by dividing by its marginal.

The derivation has limits. The binary log-linear form is saturated for a binary pair; its simple shared-coefficient result does not automatically transfer to arbitrary outcome types, predictor sets, or model restrictions. Also, zero-probability cells can make odds ratios undefined, while zero marginal probabilities leave some conditional probabilities undetermined. Any calculation must respect the support of the joint law.

In MICE, the factorization is a useful reference when conditional imputation models are meant to represent a coherent joint distribution. If they are derived from one joint model, they inherit its parameter constraints. Fitting a set of conditional regressions separately does not, by itself, demonstrate that those constraints hold. The derivation clarifies what it means for parameters to be shared, without implying that every practical set of conditional regressions has this form.

Common Mistake

• Wrong reasoning: “Both conditional models use the same two variables, so they must describe a common joint distribution.”
• Correction: Sharing variables is not the same as sharing a joint law. A common joint distribution implies specific relationships between conditional parameters. For the positive-cell binary case, the two conditional log-odds slopes must agree and equal the joint log odds ratio.
• Another pitfall: A conditional probability is not a joint cell probability. To recover the cell, multiply by the conditioning marginal; for example, \(P(X=0,Y=1)=P(X=0)P(Y=1\mid X=0)\).