Sequential Sampling from Conditional Distributions

From a Joint Law to a Sampling Sequence

A joint distribution describes both the behavior of each variable and the dependence among them. A useful way to work with it is to build the variables in sequence: first draw one variable from its marginal distribution, then draw the next from its conditional distribution given what has already been drawn. The central point is that the conditional step carries the dependence. Replacing it with an unrelated draw from the second marginal generally changes the joint law.

The construction has two ingredients. First, a joint law factors into a marginal law and a conditional law. Second, a uniform random variable can be transformed into a draw from a specified distribution by applying the distribution’s generalized inverse. Together, these facts give a direct sampling method for discrete and continuous models.

Definition: For a CDF \(F\) on the real line and \(0<u<1\), its generalized inverse (or quantile function) is \(F^{-1}(u)=\inf\{x\in\mathbb{R}:F(x)\ge u\}\). If \(U\sim\operatorname{Uniform}(0,1)\), then \(F^{-1}(U)\) has CDF \(F\). For a conditional CDF \(F_{Y|X}(y|x)\), define \(F^{-1}_{Y|X}(u|x)=\inf\{y:F_{Y|X}(y|x)\ge u\}\) wherever the conditional distribution is specified.

The generalized inverse works whether the target distribution is discrete, continuous, or has both point masses and a continuous component. In a discrete distribution it selects the first value whose cumulative probability reaches \(u\). In a continuous distribution it solves the CDF equation when an ordinary inverse is available; otherwise, the infimum definition still applies.

Why the Inverse Transform Produces the Desired Distribution

The basic inverse-transform fact follows directly from the CDF definition. For any real \(t\), the event that the generalized inverse is at most \(t\) is, up to endpoints of probability zero, the event that \(U\le F(t)\). Thus the probability of that event is \(F(t)\).

Theorem: If \(U\sim\operatorname{Uniform}(0,1)\) and \(F\) is a CDF, then \(Q=F^{-1}(U)\) has CDF \(F\).

Proof. For \(0<u<1\), right-continuity and monotonicity of \(F\) imply \(F^{-1}(u)\le t\) if and only if \(u\le F(t)\). Therefore

P(Q\le t) =P\bigl(F^{-1}(U)\le t\bigr) =P\bigl(U\le F(t)\bigr) =F(t),

because a uniform random variable on \((0,1)\) has probability \(a\) of being at most \(a\), for \(0\le a\le1\). This holds for every real \(t\), so \(Q\) has CDF \(F\). \(\square\)

Sequential Sampling for Two Variables

For a discrete pair, write \(p_X(x)=P(X=x)\) and \(p_{Y|X}(y|x)=P(Y=y\mid X=x)\) when \(p_X(x)>0\). The joint probability factors as \(p_{X,Y}(x,y)=p_X(x)p_{Y|X}(y|x)\). For a pair with joint density, the corresponding factorization is \(f_{X,Y}(x,y)=f_X(x)f_{Y|X}(y|x)\) for almost every \((x,y)\), where the conditional density is defined for \(x\) with \(f_X(x)>0\), up to the usual almost-everywhere qualifications.

Theorem: Let \(X\) and \(Y\) be real-valued random variables, and let \(K(x,\cdot)\) be a conditional distribution of \(Y\) given \(X=x\). Let \(F_X^{-1}\) be the generalized inverse of the marginal CDF of \(X\), and let \(F^{-1}_{Y|X}(\cdot|x)\) be the generalized inverse of the conditional CDF. If \(U_1,U_2\) are independent \(\operatorname{Uniform}(0,1)\) random variables, define \(X^*=F_X^{-1}(U_1)\) and \(Y^*=F^{-1}_{Y|X}(U_2|X^*)\). Then \((X^*,Y^*)\) has the same joint distribution as \((X,Y)\). The conditional distribution must be chosen measurably in \(x\); changing it on a set of \(X\)-probability zero does not change the generated joint law.

Proof. By the inverse-transform result, \(X^*\) has CDF \(F_X\). Given \(X^*=x\), the independent variable \(U_2\) remains uniform, and applying the conditional generalized inverse gives \(Y^*\) the conditional distribution \(K(x,\cdot)\). More explicitly, for Borel sets \(A,B\subseteq\mathbb{R}\), the construction gives

P(X^*\in A,\;Y^*\in B) =\int_A K(x,B)\,P_X(dx).

The defining property of the conditional distribution of \(Y\) given \(X\) gives the same expression for \(P(X\in A,Y\in B)\). Since the probabilities agree for all Borel rectangles \(A\times B\), the joint distributions agree. The conditional distribution is only determined \(P_X\)-almost everywhere; \(X^*\) lands in any \(P_X\)-null exceptional set with probability zero, so the choice there has no effect. \(\square\)

For more than two variables, the same idea is applied repeatedly. A joint law can be factored into the marginal law of \(X_1\) and conditional laws of each later variable given the variables already drawn. Generate \(X_1\) first, then use a fresh independent uniform variable at each conditional step. The conditional distributions must match the chosen factorization order.

F_{X_1,\ldots,X_k}(dx_1,\ldots,dx_k) =F_{X_1}(dx_1) \prod_{j=2}^{k}F_{X_j|X_1,\ldots,X_{j-1}} (dx_j|x_1,\ldots,x_{j-1}).

Worked Examples

Worked Example: Generating a Dependent Binary Pair

Suppose \(P(X=1)=0.4\), so \(P(X=0)=0.6\), and suppose \(P(Y=1\mid X=0)=0.2\) and \(P(Y=1\mid X=1)=0.75\). Construct a sampler and verify its joint probabilities.

Let \(U_1,U_2\) be independent uniforms. Set \(X=1\) if \(U_1\le0.4\), and \(X=0\) otherwise. If \(X=0\), set \(Y=1\) if \(U_2\le0.2\); if \(X=1\), set \(Y=1\) if \(U_2\le0.75\). In each case set \(Y=0\) otherwise.

The resulting probabilities are \(P(X=0,Y=0)=0.6(1-0.2)=0.48\) and \(P(X=0,Y=1)=0.6(0.2)=0.12\). For \(X=1\), they are \(P(X=1,Y=0)=0.4(1-0.75)=0.10\) and \(P(X=1,Y=1)=0.4(0.75)=0.30\).

These four probabilities sum to \(0.48+0.12+0.10+0.30=1\), and their marginal probabilities for \(X=1\) and \(Y=1\) are \(0.10+0.30=0.40\) and \(0.12+0.30=0.42\), respectively. The conditional threshold used for \(Y\) changes with \(X\), which is what preserves the dependence.

Worked Example: A Continuous Pair on a Triangular Support

Let \(X\sim\operatorname{Uniform}(0,1)\), and, given \(X=x\) with \(0<x<1\), let \(Y\mid X=x\sim\operatorname{Uniform}(0,x)\). Find a sequential sampler and check the resulting joint density.

The marginal inverse is \(F_X^{-1}(u)=u\). For the conditional distribution, \(F_{Y|X}(y|x)=y/x\) when \(0\le y\le x\), so its inverse is \(F^{-1}_{Y|X}(u|x)=xu\). With independent uniforms \(U_1,U_2\), the sampler is therefore \(X=U_1,\;Y=U_1U_2\).

For \(0<y<x<1\), the marginal density of \(X\) is \(1\) and the conditional density of \(Y\) given \(X=x\) is \(1/x\). Thus the generated joint density is

f_{X,Y}(x,y)=\frac{1}{x},\qquad 0<y<x<1.

It integrates to one: \(\int_0^1\int_0^x (1/x)\,dy\,dx =\int_0^1 1\,dx=1\). As a further check, \(Y<X/2\) is equivalent under the sampler to \(U_2<1/2\), since \(U_1>0\) with probability one. Hence \(P(Y<X/2)=1/2\), matching the conditional uniform distribution on \((0,X)\).

Worked Example: Sequential Sampling from Normal Conditionals

Let \(X\sim N(0,1)\), and specify \(Y\mid X=x\sim N(0.6x,0.64)\). Use inverse conditional CDFs to generate the pair, and verify its marginal variance and covariance.

Write \(\Phi\) for the standard normal CDF. Its inverse gives \(X=\Phi^{-1}(U_1)\). The conditional standard deviation is \(\sqrt{0.64}=0.8\), so the second step is \(Y=0.6X+0.8\Phi^{-1}(U_2)\), with \(U_1,U_2\) independent uniforms.

The conditional mean is \(E[Y\mid X]=0.6X\), and the conditional variance is \(0.64\). Thus \(E[Y]=E[E[Y\mid X]]=0\). Using the variance decomposition, \(\operatorname{Var}(Y)=E[\operatorname{Var}(Y\mid X)]+\operatorname{Var}(E[Y\mid X]) =0.64+0.36=1\). Also, \(\operatorname{Cov}(X,Y)=\operatorname{Cov}(X,0.6X+0.8\Phi^{-1}(U_2))=0.6\), because \(X\) and \(\Phi^{-1}(U_2)\) are independent and both have mean zero. The conditional step therefore gives \(Y\) unit variance while retaining covariance \(0.6\) with \(X\).

Practical Choices and Common Pitfalls

In practice, conditional CDFs need not have closed-form inverses. For a discrete conditional law, accumulate its probabilities until the cumulative sum first reaches the uniform draw. For a continuous conditional law, solve the conditional CDF equation numerically when needed. At each stage, the conditional distribution is evaluated using the values generated in the earlier stages.

Sampling from the marginals separately. Drawing \(X\) and \(Y\) independently from their marginal distributions produces the product joint law. It is correct only when \(X\) and \(Y\) are independent. In the binary example, independent marginal draws would give \(P(X=1,Y=1)=0.4(0.42)=0.168\), not the required \(0.30\).
Using the wrong conditional distribution. The distribution needed at the second step is \(Y\mid X=x\), evaluated at the generated value of \(X\). A conditional law at a fixed, unrelated value does not in general preserve the target joint law.
Ignoring zero-probability conditioning values. In discrete models, \(P(Y\mid X=x)\) is only required by the usual ratio definition when \(P(X=x)>0\). In continuous models, conditional laws can be specified only almost everywhere with respect to the marginal law of \(X\). Values chosen on null sets do not affect the sampling distribution.
Treating every conditional draw as an independent marginal draw. Fresh uniforms are independent, but the generated variables need not be: later variables depend on earlier generated values through their conditional quantiles.

Key takeaway: To sample from a joint law, factor it into a marginal distribution and successive conditional distributions. Apply each conditional generalized inverse to a fresh independent uniform, using the already generated values as the conditioning information.

Check Your Understanding

Try these questions using the factorization and inverse-transform construction.

For a discrete pair, explain why the joint probability at \((x,y)\) is the product of the marginal probability of \(x\) and the conditional probability of \(y\) given \(x\).
If \(P(X=1)=0.3\), and \(P(Y=1\mid X=0)=0.1\), \(P(Y=1\mid X=1)=0.8\), describe the two uniform-threshold steps for generating \((X,Y)\).
For \(X\sim\operatorname{Uniform}(0,1)\) and \(Y\mid X=x\sim\operatorname{Uniform}(0,2x)\), write the sequential inverse-transform sampler.
Why does changing a conditional distribution at values of \(x\) in a set with \(P(X\in A)=0\) leave the generated joint law unchanged?
Under what condition can separate draws from the marginal distributions of \(X\) and \(Y\) replace a conditional sampling step without changing the joint law?