Conditional Densities and Bayes’ Rule

From Joint Densities to Conditional Densities

When two continuous random variables are observed together, their joint density describes their probabilities over regions in the plane. A conditional density answers a more focused question: after fixing the value of one variable, how is probability distributed over the other? This is useful when a measurement is known and we want to describe uncertainty about an associated quantity. The key calculation is a density ratio, but its denominator must be the correct marginal density and must be positive.

Let \(X\) and \(Y\) be real-valued random variables on a probability space \((\Omega,\mathcal{F},\mathbb{P})\), with joint density \(f_{X,Y}(x,y)\) with respect to area. The marginal densities are obtained by integrating out the other variable: \[ f_X(x)=\int_{\mathbb{R}} f_{X,Y}(x,y)\,dy,\qquad f_Y(y)=\int_{\mathbb{R}} f_{X,Y}(x,y)\,dx. \] These identities hold for almost every \(x\) or \(y\), respectively. A conditional density is a density in the variable not being conditioned on.

The defining feature is that, for a measurable set \(A\subseteq\mathbb{R}\), the conditional probability is found by integrating the conditional density: \[ \mathbb{P}(X\in A\mid Y=y)=\int_A f_{X\mid Y}(x\mid y)\,dx, \] for almost every \(y\) under the distribution of \(Y\). In particular, for an interval \((a,b)\), this is the integral from \(a\) to \(b\). Including or excluding either endpoint does not change the answer when the conditional distribution has a density, since a single point has integral zero.

Although the notation conditions on the event \(Y=y\), a continuous random variable usually has \(\mathbb{P}(Y=y)=0\). Thus the ordinary event ratio \(\mathbb{P}(X\in A,Y=y)/\mathbb{P}(Y=y)\) is not the definition. The density ratio supplies a version of the conditional distribution, and versions can differ at exceptional \(y\)-values of probability zero.

Why the Density Ratio Works

Proof. Since a joint density is nonnegative and integrates to one, Tonelli’s theorem gives \(\int_{\mathbb{R}}f_Y(y)\,dy=1\). In particular, \(f_Y\) is finite almost everywhere. For a \(y\) where \(0<f_Y(y)<\infty\), nonnegativity and the definition of \(f_Y\) give \[ \int_{\mathbb{R}} f_{X\mid Y}(x\mid y)\,dx =\frac{1}{f_Y(y)}\int_{\mathbb{R}}f_{X,Y}(x,y)\,dx=1. \] The ratio is nonnegative, so it is a probability density. For the identity, integrate the joint density over \(A\times B\) and use Tonelli’s theorem: \[ \mathbb{P}(X\in A,Y\in B) =\int_B\int_A f_{X,Y}(x,y)\,dx\,dy. \] For almost every \(y\in B\) with positive finite \(f_Y(y)\), the inner integral equals \(f_Y(y)\int_A f_{X\mid Y}(x\mid y)\,dx\). The remaining \(y\)-values contribute zero to the integral weighted by \(f_Y(y)\). Substitution yields the stated identity. \(\square\)

Integrating the conditional density over all \(x\) is an essential check: the result must be \(1\) at every conditioning value where the formula is being used. A ratio that fails this check signals an incorrect marginal, an incorrect support, or an unnormalized joint density.

Bayes’ Rule for Densities

Proof. From the definitions, \(f_{X\mid Y}(x\mid y)=f_{X,Y}(x,y)/f_Y(y)\) and \(f_{Y\mid X}(y\mid x)=f_{X,Y}(x,y)/f_X(x)\), wherever the denominators are positive and finite. Multiplying the second equality by \(f_X(x)\) gives \(f_{Y\mid X}(y\mid x)f_X(x)=f_{X,Y}(x,y)\). Dividing by \(f_Y(y)\) gives the claimed formula. To obtain the denominator identity, integrate the joint factorization \(f_{X,Y}(u,y)=f_{Y\mid X}(y\mid u)f_X(u)\) with respect to \(u\); the integral is the definition of \(f_Y(y)\). All identities involving density versions hold almost everywhere, as densities themselves are defined up to null sets. \(\square\)

Bayes’ rule converts a model for the observation given an unknown quantity into a conditional density for that unknown quantity after observing a value. The numerator combines the likelihood \(f_{Y\mid X}(y\mid x)\) and the prior density \(f_X(x)\); the denominator normalizes the result across all possible \(x\).

Common Mistakes and a Practical Check

A frequent error is to start with a function that looks like a joint density but does not integrate to one. For example, setting \(f_{X,Y}(x,y)=3\) on \(0<x<y<1\) gives \[ \int_0^1\int_0^y3\,dx\,dy=\int_0^1 3y\,dy=\frac32, \] so this function is not a probability density. Dividing it by \(3/2\) gives the normalized density \(2\) on that triangle, the density used in the first example. The corresponding marginal is \(2y\), not \(3y\), and the conditional density is \(1/y\) on \(0<x<y\).

Another error is to divide the joint density by the wrong marginal. To find \(X\) given \(Y=y\), integrate the joint density over \(x\) to obtain \(f_Y(y)\), then divide by that quantity. Dividing by \(f_X(x)\) instead produces the conditional density of \(Y\) given \(X=x\). In either direction, check that the resulting conditional density is nonnegative and integrates to one over its support. For an interval probability, integrate only over the intersection of the interval with that support.

Finally, a conditional density value is not itself a probability. Probabilities come from integrating it over a set. For example, in the first worked example the value of the density at \(y=3/4\) is \(4/3\), which is greater than one; this is valid because a density value is not bounded above by one. The probability of the interval is its area under the density, here \(2/3\).