Tutorials › Statistical Theory I - Probability and Distribution Theory › Bayes Rule for Continuous Latent Variables

Home  /  Tutorials  /  Conditional Distribution Theory

Conditional Distribution Theory · Tutorial 258 of 994

Bayes Rule for Continuous Latent Variables

Learn how to turn a prior density and an observation’s conditional density into a normalized posterior density for a continuous latent variable.

Intermediate 10 min read

What You'll Learn

  • State the conditions under which Bayes’ rule gives a posterior density.
  • Distinguish the prior density, likelihood, marginal density, and posterior density.
  • Derive a posterior from a normal prior and a normal observation model.
  • Update a gamma prior using an exponential observation.
  • Account for observation-dependent support when finding a posterior density.
  • Recognize why conditioning on an exact continuous observation needs careful interpretation.

From Prior Beliefs to a Posterior Density

Suppose an unknown quantity is itself modeled as a random variable \(\Theta\), with prior density \(\pi(\theta)\). An observation \(Y\) provides information about \(\Theta\) through a conditional density \(f_{Y\mid\Theta}(y\mid\theta)\). Bayes’ rule combines these two ingredients: values of \(\theta\) receive more posterior weight when they had substantial prior density and make the observed \(y\) relatively likely.

For a continuous observation, it is important to be precise about what “given the observation” means. Any single exact value of a continuously distributed \(Y\) has probability zero. The posterior is therefore defined through a conditional distribution (a regular conditional law), rather than by dividing probabilities of two positive-probability events. Under suitable density conditions, its density is given by a ratio.

Definition: Let \(\Theta\) have prior density \(\pi(\theta)\), and suppose \(Y\mid\Theta=\theta\) has conditional density \(f_{Y\mid\Theta}(y\mid\theta)\). Define the marginal density of \(Y\) by \(m(y)=\int f_{Y\mid\Theta}(y\mid u)\pi(u)\,du\). At any \(y\) for which \(0<m(y)<\infty\), the Bayes posterior density is \(f_{\Theta\mid Y}(\theta\mid y)=f_{Y\mid\Theta}(y\mid\theta)\pi(\theta)/m(y)\). The numerator is the likelihood at \(y\), multiplied by the prior density; the denominator normalizes it.

Here \(\theta\) and \(u\) range over the latent variable’s parameter space, and integration is with respect to its underlying measure (Lebesgue measure when that space is a subset of the real line). The conditional density must be jointly measurable and nonnegative, and it must integrate to one in \(y\) for prior-almost every \(\theta\). These assumptions make the joint density \(f_{\Theta,Y}(\theta,y)=\pi(\theta)f_{Y\mid\Theta}(y\mid\theta)\) valid. The marginal density \(m(y)\) is finite for almost every \(y\); the ratio formula applies wherever it is positive and finite. As with any conditional distribution, its values on a set of \(y\)-values having marginal probability zero can be specified differently without changing the joint model.

Why the Ratio Is a Posterior Density

The denominator is not an optional correction. Without it, prior-times-likelihood is generally not a density in \(\theta\). For each eligible observation \(y\), it makes the updated density integrate to one. The same formula also gives conditional probabilities for sets of latent values.

Theorem: Under the density conditions above, for every measurable set \(A\) of latent values and every \(y\) with \(0<m(y)<\infty\), \(P(\Theta\in A\mid Y=y)=\int_A f_{\Theta\mid Y}(\theta\mid y)\,d\theta\) is a version of the conditional probability for marginal-almost every \(y\). In particular, the posterior density integrates to one.

To verify normalization, integrate the proposed ratio over the latent space:

$$ \int \frac{f_{Y\mid\Theta}(y\mid\theta)\pi(\theta)}{m(y)}\,d\theta =\frac{1}{m(y)}\int f_{Y\mid\Theta}(y\mid\theta)\pi(\theta)\,d\theta =1. $$

For the conditional-probability claim, take any measurable set \(B\) of observation values. The joint density gives

$$ P(\Theta\in A,\ Y\in B) =\int_B\int_A f_{Y\mid\Theta}(y\mid\theta)\pi(\theta)\,d\theta\,dy =\int_B\left[\int_A f_{\Theta\mid Y}(\theta\mid y)\,d\theta\right]m(y)\,dy. $$

The last expression is the defining integral identity for a conditional probability given \(Y\): it integrates the proposed conditional probability against the marginal law of \(Y\) and reproduces the joint probability. Thus the ratio supplies a version of the conditional distribution. The identity is unaffected by changing the conditional probabilities on a marginal-null set of observations. \(\square\)

A useful consequence is the posterior expectation formula. If \(g\) is measurable and the relevant integrals are finite, then

$$ E[g(\Theta)\mid Y=y] =\frac{\int g(\theta)f_{Y\mid\Theta}(y\mid\theta)\pi(\theta)\,d\theta}{m(y)}. $$

This follows by integrating \(g\) against the posterior density just established. For a signed \(g\), it is enough to require the posterior expectation of \(|g(\Theta)|\) to be finite; for nonnegative \(g\), the identity holds with possibly infinite integrals.

Worked Posterior Calculations

Worked Example: Normal Prior and Normal Observation

Let \(\Theta\sim N(0,4)\), and suppose \(Y\mid\Theta=\theta\sim N(\theta,1)\). Find the posterior density after observing \(y=1\).

The prior density and conditional observation density are

$$ \pi(\theta)=\frac{1}{2\sqrt{2\pi}}e^{-\theta^2/8}, \qquad f_{Y\mid\Theta}(1\mid\theta)=\frac{1}{\sqrt{2\pi}}e^{-(1-\theta)^2/2}. $$

Their product is proportional, as a function of \(\theta\), to

$$ e^{-\theta^2/8-(1-\theta)^2/2} =e^{-1/10}e^{-(5/8)(\theta-4/5)^2}. $$

The exponent is a normal-kernel exponent with mean \(4/5\) and variance \(4/5\), because \(1/(2(4/5))=5/8\). The marginal density at the observation is \(m(1)=\frac{1}{\sqrt{10\pi}}e^{-1/10}\), as can also be checked by integrating the product of the two displayed densities. Dividing by this normalizing constant gives

$$ f_{\Theta\mid Y}(\theta\mid 1) =\sqrt{\frac{5}{8\pi}}\, e^{-(5/8)(\theta-4/5)^2}. $$

Therefore \(\Theta\mid Y=1\sim N(4/5,4/5)\). The observation shifts the posterior mean from the prior mean \(0\) toward the observed value \(1\), while the posterior variance is smaller than the prior variance \(4\).

Worked Example: Exponential Observation with a Gamma Prior

Let \(\Theta\) have a Gamma prior with shape \(2\) and rate \(3\), so that \(\pi(\theta)=9\theta e^{-3\theta}\) for \(\theta>0\). Given \(\Theta=\theta\), let \(Y\) have an exponential density with rate \(\theta\): \(f_{Y\mid\Theta}(y\mid\theta)=\theta e^{-\theta y}\) for \(y>0\). Find the posterior after observing \(y=1\).

For \(\theta>0\), the prior-times-likelihood is

$$ \pi(\theta)f_{Y\mid\Theta}(1\mid\theta) =9\theta^2e^{-4\theta}. $$

Its integral over the latent support is the marginal density at \(1\):

$$ m(1)=9\int_0^\infty \theta^2e^{-4\theta}\,d\theta =9\frac{\Gamma(3)}{4^3} =9\frac{2}{64} =\frac{9}{32}. $$

Consequently,

$$ f_{\Theta\mid Y}(\theta\mid 1) =\frac{9\theta^2e^{-4\theta}}{9/32} =32\theta^2e^{-4\theta},\qquad \theta>0. $$

This is a Gamma density with shape \(3\) and rate \(4\), since its normalizing constant is \(4^3/\Gamma(3)=32\). The posterior retains the prior’s positive support, while the observed waiting time changes both its shape and rate.

Worked Example: Observation-Dependent Posterior Support

Let \(\Theta\) be uniform on \((0,1)\). Given \(\Theta=\theta\), let \(Y\) be uniform on \((0,\theta)\). Find the posterior density after observing \(y=1/4\), and calculate \(P(\Theta>1/2\mid Y=1/4)\).

The prior density equals \(1\) for \(0<\theta<1\). The observation density is \(1/\theta\) when \(0<y<\theta\), and is zero otherwise. At \(y=1/4\), only values \(\theta>1/4\) are possible. Thus

$$ m(1/4)=\int_{1/4}^1\frac{1}{\theta}\,d\theta =\log 4. $$

The posterior density is therefore

$$ f_{\Theta\mid Y}(\theta\mid 1/4) =\frac{1}{\theta\log 4},\qquad \frac14<\theta<1, $$

and is zero elsewhere. Its integral is \(\int_{1/4}^1(\theta\log4)^{-1}\,d\theta=\log4/\log4=1\). For the requested probability,

$$ P(\Theta>1/2\mid Y=1/4) =\int_{1/2}^1\frac{1}{\theta\log4}\,d\theta =\frac{\log2}{\log4} =\frac12. $$

The lower posterior support bound comes from the observation model, not from a change to the prior: observing \(Y=1/4\) rules out every \(\theta\leq1/4\).

Interpretation and a Common Pitfall

The likelihood \(f_{Y\mid\Theta}(y\mid\theta)\), viewed as a function of \(\theta\), measures how compatible each latent value is with the observation. But it need not integrate to one with respect to \(\theta\). It becomes a posterior density only after multiplication by the prior density and division by \(m(y)\). In particular, the likelihood is not itself a probability distribution for \(\Theta\).

For an exact continuous observation, the phrase “posterior given \(Y=y\)” refers to the regular conditional distribution represented by the density ratio. It does not mean that \(P(Y=y)\) is positive. If the observation model supplies a particular pointwise density, the ratio can be evaluated at a \(y\) with positive, finite \(m(y)\); as a conditional-law statement, it is determined only up to marginal-almost-everywhere equality.

Common mistake: Treating prior-times-likelihood as the posterior density without checking its integral. For example, in the uniform-prior example, \(1/\theta\) on \((1/4,1)\) does not integrate to one: its integral is \(\log4\). The correction is to divide by the marginal density \(\log4\), giving \(1/(\theta\log4)\). Always identify the posterior support and verify that the proposed density integrates to one.

The normalizing constant also has a practical interpretation: \(m(y)\) is the density assigned to the observation before seeing it, under the full prior-and-observation model. It is sometimes called the marginal likelihood or prior predictive density. The ratio makes clear how posterior conclusions depend jointly on the observation model and the prior, rather than on the likelihood alone.

Check Your Understanding

Use the density-ratio rule and verify the support and normalization in each case.

  1. What conditions must hold for the posterior-density ratio at an observed value \(y\) to be defined?
  2. Why does conditioning on an exact value of a continuously distributed observation not use the elementary ratio \(P(A\cap B)/P(B)\) for the event \(B=\{Y=y\}\)?
  3. If a likelihood as a function of \(\theta\) integrates to \(7\) after multiplication by the prior, what adjustment is needed to obtain a posterior density?
  4. In the uniform-prior example, which latent values are excluded by observing \(Y=1/4\), and why?
  5. Explain why a likelihood function is not automatically a density for the latent variable.

Previous

Next

Back to Tutorials