Divergences for Comparing Statistical Experiments

Why Compare Probability Distributions?

In a statistical experiment, different parameter values can produce different probability distributions for the data. Comparing those distributions helps describe how distinguishable the corresponding cases are. Several measures of separation are useful, but they do not all behave alike: Kullback–Leibler divergence is directional, whereas Hellinger distance is symmetric. Their definitions also make support conditions important.

We will compare two probability measures $P$ and $Q$ on the same measurable space. Choose a measure $\mu$ that dominates both, meaning that $P$ and $Q$ have densities $p$ and $q$ with respect to $\mu$. For discrete distributions, $\mu$ can be counting measure; for distributions with densities on the real line, it can be Lebesgue measure. The definitions below do not depend on which common dominating measure is chosen.

Definitions and Support Conditions

Definition: The Kullback–Leibler divergence from $P$ to $Q$ is $$ D_{\mathrm{KL}}(P\|Q)=\int p\log\!\left(\frac{p}{q}\right)\,d\mu. $$ If $P$ is not absolutely continuous with respect to $Q$, set $D_{\mathrm{KL}}(P\|Q)=+\infty$. In the integral, a point with $p=0$ contributes zero; if $p>0$ where $q=0$, the divergence is infinite. The logarithm is natural.

The notation $P\|Q$ records the direction: the distribution in the first position supplies the averaging weights. Equivalently, when $P$ is absolutely continuous with respect to $Q$, the divergence is $E_P[\log(dP/dQ)]$, interpreted as an extended nonnegative value. It is generally not a distance: reversing the two distributions can change the value, and it need not satisfy the triangle inequality.

Definition: The Hellinger affinity of $P$ and $Q$ is $$ \rho(P,Q)=\int \sqrt{pq}\,d\mu. $$ Using the convention in this tutorial, the squared Hellinger distance and Hellinger distance are $$ H^2(P,Q)=1-\rho(P,Q), \qquad H(P,Q)=\sqrt{1-\rho(P,Q)}. $$

Some references use a different constant factor in the definition of Hellinger distance. Here the stated convention fixes that factor: $H$ lies between zero and one. The affinity lies between zero and one as well. Unlike KL divergence, both affinity and Hellinger distance are symmetric in $P$ and $Q$.

Basic Properties

Theorem: For probability distributions $P$ and $Q$, $D_{\mathrm{KL}}(P\|Q)\geq 0$, with equality exactly when $P=Q$. Also, $$ H^2(P,Q)=\frac{1}{2}\int(\sqrt p-\sqrt q)^2\,d\mu, \qquad 0\leq \rho(P,Q)\leq 1. $$ Under the stated convention, $H$ is a metric on probability distributions.

Proof of the KL assertion. If $P$ is not absolutely continuous with respect to $Q$, the divergence is infinite and is therefore nonnegative. Otherwise, $q>0$ wherever $p>0$, up to null sets. For $t>0$, $-\log t\geq 1-t$, with equality only at $t=1$. Applying this with $t=q/p$, at points where $p>0$, gives $$ p\log(p/q)\geq p-q. $$ Integrating over the set where $p>0$ yields $$ D_{\mathrm{KL}}(P\|Q)\geq 1-\int_{\{p>0\}}q\,d\mu\geq 0, $$ because $\int_{\{p>0\}}p\,d\mu=1$ and $\int q\,d\mu=1$. Equality requires both $q$ to put no mass outside $\{p>0\}$ and equality in the logarithm inequality wherever $p>0$. Thus $p=q$ almost everywhere. Conversely, if $p=q$ almost everywhere, the divergence is zero.

Proof of the Hellinger assertions. Expanding the square and using $\int p\,d\mu=\int q\,d\mu=1$ gives $$ \frac{1}{2}\int(\sqrt p-\sqrt q)^2\,d\mu =\frac{1}{2}(1+1-2\rho(P,Q))=1-\rho(P,Q). $$ Cauchy–Schwarz gives $0\leq\rho(P,Q)\leq \sqrt{\int p\,d\mu\int q\,d\mu}=1$. Finally, $H(P,Q)=\|\sqrt p-\sqrt q\|_2/\sqrt{2}$, so nonnegativity, symmetry, and the triangle inequality follow from the corresponding properties of the $L^2(\mu)$ norm. If $H(P,Q)=0$, then $\sqrt p=\sqrt q$ almost everywhere, hence $P=Q$. These properties establish that $H$ is a metric. $\square$

Worked Calculations

Worked Example: Comparing Two Bernoulli Distributions

Let $P$ be Bernoulli with success probability $1/4$, and let $Q$ be Bernoulli with success probability $3/4$. On the two outcomes success and failure, their probability vectors are $p=(1/4,3/4)$ and $q=(3/4,1/4)$.

The KL divergence is $$ D_{\mathrm{KL}}(P\|Q) =\frac14\log\!\left(\frac{1/4}{3/4}\right) +\frac34\log\!\left(\frac{3/4}{1/4}\right) =-\frac14\log 3+\frac34\log 3 =\frac12\log 3. $$

The reverse divergence has the same value here because swapping the two outcomes swaps the two distributions: $$ D_{\mathrm{KL}}(Q\|P)=\frac12\log 3. $$ This equality is specific to this pair, not a general symmetry property of KL divergence. The affinity and squared Hellinger distance are $$ \rho(P,Q)=\sqrt{\frac14\frac34}+\sqrt{\frac34\frac14} =\frac{\sqrt{3}}{2}, \qquad H^2(P,Q)=1-\frac{\sqrt{3}}{2}. $$

Worked Example: A Three-Category Comparison

Take $P=(1/2,1/3,1/6)$ and $Q=(1/3,1/3,1/3)$. Their affinity is the sum of the square roots of the coordinatewise products: $$ \rho(P,Q) =\sqrt{\frac16}+\sqrt{\frac19}+\sqrt{\frac1{18}} =\frac{1}{\sqrt6}+\frac13+\frac{1}{3\sqrt2} \approx 0.9773. $$ Therefore $H^2(P,Q)=1-\rho(P,Q)\approx 0.0227$, a nonnegative value as required.

For the directed KL divergence, $$ D_{\mathrm{KL}}(P\|Q) =\frac12\log\!\left(\frac32\right) +\frac13\log(1) +\frac16\log\!\left(\frac12\right) =\frac12\log\!\left(\frac32\right)-\frac16\log 2 \approx 0.0872. $$ In the reverse direction, $$ D_{\mathrm{KL}}(Q\|P) =\frac13\log\!\left(\frac23\right) +\frac13\log(1) +\frac13\log 2 =\frac13\log\!\left(\frac43\right) \approx 0.0959. $$ The different values demonstrate that the direction of KL divergence matters even when both distributions have the same support.

Worked Example: Gaussian Distributions With a Shared Variance

Let $P=N(\mu,\sigma^2)$ and $Q=N(\nu,\sigma^2)$, where $\sigma>0$. Their densities are $$ p(x)=\frac{1}{\sqrt{2\pi}\sigma} \exp\!\left(-\frac{(x-\mu)^2}{2\sigma^2}\right), \qquad q(x)=\frac{1}{\sqrt{2\pi}\sigma} \exp\!\left(-\frac{(x-\nu)^2}{2\sigma^2}\right). $$ Taking the log density ratio and then expectation under $P$ gives $$ \log\!\left(\frac{p(x)}{q(x)}\right) =\frac{(x-\nu)^2-(x-\mu)^2}{2\sigma^2}. $$ Under $P$, $E_P[X]=\mu$ and $E_P[(X-\mu)^2]=\sigma^2$. Writing $X-\nu=(X-\mu)+(\mu-\nu)$, the cross term has expectation zero, so $E_P[(X-\nu)^2]=\sigma^2+(\mu-\nu)^2$. Hence $$ D_{\mathrm{KL}}(P\|Q)=\frac{(\mu-\nu)^2}{2\sigma^2}. $$

For affinity, the product of the square roots of the densities is $$ \sqrt{p(x)q(x)} =\frac{1}{\sqrt{2\pi}\sigma} \exp\!\left(-\frac{(x-\mu)^2+(x-\nu)^2}{4\sigma^2}\right). $$ The identity $$ (x-\mu)^2+(x-\nu)^2 =2\left(x-\frac{\mu+\nu}{2}\right)^2+\frac{(\mu-\nu)^2}{2} $$ separates the expression into a normal density centered at $(\mu+\nu)/2$, multiplied by a constant. Integrating that density over the real line gives one, leaving $$ \rho(P,Q)=\exp\!\left(-\frac{(\mu-\nu)^2}{8\sigma^2}\right), \qquad H^2(P,Q)=1-\exp\!\left(-\frac{(\mu-\nu)^2}{8\sigma^2}\right). $$ When $\mu=\nu$, both divergences are zero and the affinity is one, as they should be for identical distributions.

Reading the Measures Correctly

The examples show complementary features. KL divergence compares log density ratios averaged under its first distribution. Hellinger distance instead compares square roots of densities symmetrically. The affinity is often convenient in calculations because it is a single integral or sum; the distance is obtained directly from it by $H^2=1-\rho$. All three quantities are determined by the distributions, not by a particular choice of coordinates or a particular representation of their densities.

Support deserves special attention. A finite Hellinger distance does not imply finite KL divergence. For example, if $P$ puts probability one on outcome 1 and $Q$ assigns probability $1/2$ to each of outcomes 1 and 2, then $D_{\mathrm{KL}}(P\|Q)=\log 2$, but $D_{\mathrm{KL}}(Q\|P)=+\infty$, because $Q$ assigns positive probability to an outcome that has zero probability under $P$. Their affinity is $1/\sqrt{2}$, so their squared Hellinger distance is $1-1/\sqrt{2}$, which is finite.

Common Pitfall

Common mistake: Treating $D_{\mathrm{KL}}(P\|Q)$ as a symmetric distance, or assuming it is finite just because both distributions are valid probability models. The correction is to check the order and the support: KL is averaged under its first argument, and it is infinite if that argument assigns positive probability to a set that has probability zero under the second. Hellinger distance is symmetric and, with the convention used here, always lies in $[0,1]$.

These comparisons matter when assessing how much two candidate data-generating distributions differ. A small divergence indicates a form of closeness suited to that particular measure; it does not mean that every comparison or every direction gives the same numerical answer. Keeping the definition and support condition visible is especially useful when selecting distributions for lower-bound arguments.

Check Your Understanding

Answer each question using the definitions and calculations above.

For which condition on $P$ and $Q$ is $D_{\mathrm{KL}}(P\|Q)$ infinite?
Why does the definition of KL divergence usually give different values after swapping $P$ and $Q$?
If $\rho(P,Q)=0.8$, what are $H^2(P,Q)$ and $H(P,Q)$ under the convention used here?
For the Gaussian example, what happens to KL divergence and affinity as $|\mu-\nu|$ increases while $\sigma$ stays fixed?
Can two distributions have finite Hellinger distance but infinite KL divergence in one direction? Explain using the support condition.