Bayes’ Formula for Probability Kernels

From a Prior and Likelihood to a Posterior

Bayes’ formula is often introduced as a ratio of probabilities or densities. At the level of probability kernels, its central role is broader: it constructs a conditional law of an unknown random element given an observed one.

Let $E$ be the state space of an unknown quantity and $F$ the observation space. Suppose both are standard Borel spaces, $\mu$ is a probability measure on $E$, and $L$ is a probability kernel from $E$ to $F$. The measure $\mu$ describes the prior law, while $L(x,\cdot)$ describes the observation law when the unknown value is $x$.

Together they determine a joint law: first draw a value according to $\mu$, then draw an observation according to $L$ at that value. The observation’s marginal law is

\nu(B)=\int_E L(x,B)\,\mu(dx), \qquad B\in\mathcal{B}(F).

A posterior kernel $\Pi$ is a conditional law of the unknown value given the observation. It is a probability kernel from $F$ to $E$, and it satisfies

\int_B \Pi(y,A)\,\nu(dy) = \int_A L(x,B)\,\mu(dx)

for every measurable $A\subseteq E$ and $B\subseteq F$. This identity says that the joint probability of the unknown lying in $A$ and the observation lying in $B$ can be computed by either conditioning on the observation or integrating the likelihood over prior values in $A$.

Definition: Given a prior probability measure $\mu$ on $E$ and a likelihood kernel $L$ from $E$ to $F$, the induced observation law is $\nu(B)=\int_E L(x,B)\,\mu(dx)$. A posterior kernel is a probability kernel $\Pi$ from $F$ to $E$ satisfying $\int_B\Pi(y,A)\nu(dy)=\int_A L(x,B)\mu(dx)$ for all measurable $A$ and $B$.

The Kernel Form of Bayes’ Formula

The general formula can be expressed using Radon–Nikodym derivatives. For each measurable set $A\subseteq E$, define a finite measure on $F$ by

\nu_A(B)=\int_A L(x,B)\,\mu(dx).

Since $0\leq \nu_A(B)\leq \nu(B)$, the measure $\nu_A$ is absolutely continuous with respect to $\nu$. The posterior probability of $A$, as a function of the observation, is a version of the derivative $d\nu_A/d\nu$.

Theorem: Let $E$ and $F$ be standard Borel spaces, let $\mu$ be a probability measure on $E$, and let $L$ be a probability kernel from $E$ to $F$. Let $\nu(B)=\int_E L(x,B)\mu(dx)$. A posterior kernel $\Pi$ exists, and for every measurable $A\subseteq E$, $$ \Pi(y,A)=\frac{d\nu_A}{d\nu}(y) \quad\text{for }\nu\text{-almost every }y, \qquad \nu_A(B)=\int_A L(x,B)\,\mu(dx). $$ The exceptional $\nu$-null set may depend on $A$.

Proof. Define a probability measure $\gamma$ on $E\times F$ by $$ \gamma(C)=\int_E\int_F \mathbf{1}_C(x,y)\,L(x,dy)\,\mu(dx). $$ Its $F$-marginal is $\nu$. Since $E$ and $F$ are standard Borel, disintegration gives a probability kernel $\Pi$ from $F$ to $E$ such that $$ \gamma(A\times B)=\int_B\Pi(y,A)\,\nu(dy) $$ for all measurable $A\subseteq E$ and $B\subseteq F$. The definition of $\gamma$ also gives $$ \gamma(A\times B)=\int_A L(x,B)\,\mu(dx)=\nu_A(B). $$ Thus, for each $A$, the measures $\nu_A$ and $B\mapsto\int_B\Pi(y,A)\nu(dy)$ agree. By the Radon–Nikodym theorem, $\Pi(\cdot,A)$ is a version of $d\nu_A/d\nu$. The existence of the kernel supplies one coherent choice of posterior probabilities across measurable sets; individual derivatives, considered separately, are specified only up to null sets. $\square$

This is the kernel form of Bayes’ formula. It requires no density for the observation law and no pointwise likelihood function. In particular, it applies when the observation distribution is discrete, continuous, or neither.

Worked Example: A Finite Discrete Model

Let $E=\{a,b,c\}$ and $F=\{0,1\}$. Assign prior probabilities $\mu(\{a\})=0.5$, $\mu(\{b\})=0.3$, and $\mu(\{c\})=0.2$. Suppose the likelihood of observing $1$ at the three possible values is $0.8$, $0.4$, and $0.1$, respectively. The likelihood of observing $0$ is the complement in each case, so each likelihood is a probability distribution.

The probability of observing $1$ is

\nu(\{1\}) =0.5(0.8)+0.3(0.4)+0.2(0.1) =0.40+0.12+0.02 =0.54.

Because this observation has positive probability, the posterior probabilities are the prior-weighted likelihoods divided by $0.54$:

\begin{aligned} \Pi(1,\{a\})&=\frac{0.40}{0.54}=\frac{20}{27},\\ \Pi(1,\{b\})&=\frac{0.12}{0.54}=\frac{2}{9},\\ \Pi(1,\{c\})&=\frac{0.02}{0.54}=\frac{1}{27}. \end{aligned}

These numbers sum to one, since $20/27+6/27+1/27=1$. The posterior favors $a$, whose likelihood for the observed outcome is greatest, while still reflecting the prior weights.

Density-Based Bayes’ Formula

A familiar ratio formula follows when the likelihood kernel has a density with respect to a common reference measure. Let $\lambda$ be a $\sigma$-finite measure on $F$. Suppose there is a jointly measurable function $\ell:E\times F\to[0,\infty]$ such that, for every $x\in E$ and measurable $B\subseteq F$,

L(x,B)=\int_B\ell(x,y)\,\lambda(dy).

The function $\ell(x,y)$ is the likelihood density at observation $y$ for parameter value $x$. Integrating it against the prior gives the marginal density, also called the evidence:

m(y)=\int_E\ell(x,y)\,\mu(dx).

Theorem: Under the preceding assumptions, $\nu(dy)=m(y)\lambda(dy)$. At every $y$ for which $0<m(y)<\infty$, a posterior kernel is given by $$ \Pi(y,A)= \frac{\int_A\ell(x,y)\,\mu(dx)} {\int_E\ell(x,y)\,\mu(dx)}. $$ At observations where $m(y)=0$ or $m(y)=\infty$, the formula can be assigned any fixed probability measure on $E$; these observations form a $\nu$-null set.

Proof. For measurable $B\subseteq F$, Tonelli’s theorem gives $$ \nu(B) =\int_E\int_B\ell(x,y)\,\lambda(dy)\,\mu(dx) =\int_B m(y)\,\lambda(dy). $$ Since $\nu$ is a probability measure, $m$ is finite $\lambda$-almost everywhere. The set where $m=0$ has $\nu$-measure zero, as does the set where $m=\infty$. On the measurable set $0<m(y)<\infty$, the displayed ratio is a probability measure in $A$, because its numerator is nonnegative and its value at $A=E$ is one. It is measurable in $y$ for each $A$, by measurability of parameter integrals. Define it there and use any fixed probability measure on the exceptional set; this produces a probability kernel.

To verify the posterior identity, let $A\subseteq E$ and $B\subseteq F$ be measurable. The exceptional observations do not affect integration with respect to $\nu$. On the remaining set, the denominator $m(y)$ cancels the marginal density:

\begin{aligned} \int_B\Pi(y,A)\,\nu(dy) &=\int_B \frac{\int_A\ell(x,y)\,\mu(dx)}{m(y)} m(y)\,\lambda(dy)\\ &=\int_A\int_B\ell(x,y)\,\lambda(dy)\,\mu(dx)\\ &=\int_A L(x,B)\,\mu(dx). \end{aligned}

This is Bayes’ formula in density form. The numerator combines the prior and the likelihood restricted to $A$; the denominator normalizes the result to a probability measure.

Worked Example: Gaussian Prior and Gaussian Observation

Let the unknown value $X$ have prior law $N(0,\tau^2)$, where $\tau^2>0$. Given $X=x$, suppose $Y$ has law $N(x,\sigma^2)$, where $\sigma^2>0$. With Lebesgue measure as reference, the prior density is proportional to $\exp(-x^2/(2\tau^2))$, and the likelihood density at $y$ is proportional to $\exp(-(y-x)^2/(2\sigma^2))$. As a function of $x$, their product has exponent

-\frac12\left(\frac{x^2}{\tau^2}+\frac{(y-x)^2}{\sigma^2}\right).

Collecting the terms involving $x$ and completing the square yields

\frac{x^2}{\tau^2}+\frac{(y-x)^2}{\sigma^2} = \left(\frac{1}{\tau^2}+\frac{1}{\sigma^2}\right) \left(x-\frac{\tau^2}{\tau^2+\sigma^2}y\right)^2 +\frac{y^2}{\tau^2+\sigma^2}.

The last term does not depend on $x$, so it contributes only to the normalizing constant. The coefficient of the squared term is the reciprocal of $\tau^2\sigma^2/(\tau^2+\sigma^2)$. Therefore the posterior is

X\mid Y=y\ \sim\ N\left(\frac{\tau^2}{\tau^2+\sigma^2}y,\, \frac{\tau^2\sigma^2}{\tau^2+\sigma^2}\right).

The posterior mean is a weighted version of the observation: it is closer to zero when prior uncertainty is small, and closer to $y$ when observation noise is small.

Worked Example: An Observation with an Atom and a Density

Let $E=[0,1]$ and let the prior be uniform on this interval. Let the observation space be the disjoint union of a single symbol $c$ and the positive real line. Given $x$, the observation equals $c$ with probability $x$; otherwise it has density $(1-x)e^{-y}$ for $y>0$. This is a valid likelihood kernel because

x+\int_0^\infty(1-x)e^{-y}\,dy=x+(1-x)=1.

Use the reference measure that assigns mass one to $c$ and agrees with Lebesgue measure on $(0,\infty)$. The likelihood density relative to this measure is $x$ at $c$ and $(1-x)e^{-y}$ at $y>0$.

At the atom, the evidence is $\int_0^1x\,dx=1/2$. Thus the posterior density of $x$ given $Y=c$ is

\Pi(c,dx)=2x\,dx,\qquad 0\leq x\leq 1.

For a continuous observation $y>0$, the evidence density is

m(y)=\int_0^1(1-x)e^{-y}\,dx=\frac{e^{-y}}{2}.

Consequently, for every $y>0$, the posterior density is

\Pi(y,dx)=2(1-x)\,dx,\qquad 0\leq x\leq 1.

Indeed, $2x$ and $2(1-x)$ each integrate to one on $[0,1]$. The atom and the continuous part have different likelihoods, so their posterior laws differ. Treating the entire observation as if it had an ordinary Lebesgue density would miss the mass at $c$.

Versions, Null Observations, and Interpretation

A conditional law is determined only almost surely under the observation law $\nu$. If a particular observation has probability zero, the posterior value assigned at that observation is not fixed by the defining integral identity. In the density formula, observations with zero evidence are exactly of this kind: they have zero probability under the marginal law. The formula should therefore be read as an almost-everywhere statement, not as a claim that the data determine a unique posterior at every conceivable observation.

A second important distinction is between the general kernel formula and the density ratio. The Radon–Nikodym statement applies to arbitrary likelihood kernels. The ratio involving $\ell(x,y)$ requires a common reference measure and a jointly measurable likelihood density. When such a density exists, the ratio is a convenient explicit version; when it does not, the kernel formulation remains valid.

Bayes’ formula is useful because it keeps the modeling steps separate. The prior describes uncertainty before observing data; the likelihood specifies how data arise at each possible state; and the posterior is the conditional law induced by those two ingredients. The observation law also matters: it identifies which observations occur with positive probability and hence where the posterior is determined up to null sets.

Check Your Understanding

Use the definitions and calculations above to answer these questions.

How is the observation marginal $\nu$ obtained from a prior $\mu$ and likelihood kernel $L$?
Why is the measure $\nu_A$ absolutely continuous with respect to $\nu$, and what does its Radon–Nikodym derivative represent?
In the density formula, why can observations with zero evidence be assigned an arbitrary posterior probability measure?
For the finite example, what is the posterior probability of $b$ after observing $1$, and how is it calculated?
Why does the mixed observation example require a reference measure that includes both an atom and Lebesgue measure?

Bayes’ Formula for Probability Kernels

What You'll Learn

From a Prior and Likelihood to a Posterior

The Kernel Form of Bayes’ Formula

Worked Example: A Finite Discrete Model

Worked Example: A Finite Discrete Model

Density-Based Bayes’ Formula

Worked Example: Gaussian Prior and Gaussian Observation

Worked Example: Gaussian Prior and Gaussian Observation

Worked Example: An Observation with an Atom and a Density

Worked Example: An Observation with an Atom and a Density

Versions, Null Observations, and Interpretation

Check Your Understanding

Previous

Next