Choosing Metrics for Conditional Distribution Error

Why the Error Metric Matters

A conditional distribution estimate can miss the truth in several distinct ways. It may assign too little probability to an event, put mass at the wrong locations, or make a small error in the body of the distribution while missing a rare but distant tail. No single discrepancy treats all these differences as equally important. Choosing a metric is therefore part of specifying the target of a conditional distribution analysis, not just a choice of computational convenience.

Let \(X\) be a covariate and \(Y\) an outcome. Write \(P_x\) for the true conditional law of \(Y\) given \(X=x\), and \(Q_x\) for an estimated or proposed conditional law. For a discrepancy \(d\) between probability laws, a common population summary is the expected conditional error \(E[d(P_X,Q_X)]\), where the expectation uses the target population distribution of \(X\). This summary assesses the conditional laws at the same covariate values; it is not a comparison of the marginal distributions of \(Y\).

Four Discrepancies

Definition: For probability laws \(P\) and \(Q\), the Kullback–Leibler divergence is \(D_{\mathrm{KL}}(P\|Q)=\int \log(dP/dQ)\,dP\) if \(P\) is absolutely continuous with respect to \(Q\), and \(+\infty\) otherwise. The total variation distance is \(d_{\mathrm{TV}}(P,Q)=\sup_A |P(A)-Q(A)|\), with the supremum over measurable sets. If both laws have densities \(p,q\) with respect to a common measure, then \(d_{\mathrm{TV}}(P,Q)=\frac12\int |p-q|\).

KL divergence measures the expected log density ratio under its first argument. Its direction matters: in general \(D_{\mathrm{KL}}(P\|Q)\ne D_{\mathrm{KL}}(Q\|P)\). If the true law is \(P_x\) and the proposed law is \(Q_x\), the convention \(D_{\mathrm{KL}}(P_x\|Q_x)\) penalizes assigning very little probability under \(Q_x\) to outcomes that occur under \(P_x\). A support mismatch can make this divergence infinite. Total variation instead records the largest difference in probability assigned to any single event. It lies between zero and one and does not measure how far apart two locations are.

Definition: Let \((\mathcal Y,\rho)\) be a metric space. For \(p\geq1\), the \(p\)-Wasserstein distance is \(W_p(P,Q)=\left(\inf_{\pi\in\Pi(P,Q)}\int \rho(y,z)^p\,d\pi(y,z)\right)^{1/p}\), provided \(P\) and \(Q\) have finite \(p\)th moments. Here \(\Pi(P,Q)\) is the set of joint laws, or couplings, with marginals \(P\) and \(Q\). The common choice \(W_1\) measures the least average transport distance needed to move one law into the other.

Unlike total variation, Wasserstein distance uses the geometry specified by \(\rho\). Moving a small amount of probability a long distance can therefore incur a large \(W_1\) cost. A Wasserstein distance is not determined by the laws alone without a choice of ground metric, and its finiteness requires an appropriate moment condition.

Definition: Let \(k\) be a positive definite kernel with reproducing kernel Hilbert space \(\mathcal H_k\). The maximum mean discrepancy (MMD) is \(\operatorname{MMD}_k(P,Q)=\sup_{\|g\|_{\mathcal H_k}\leq1}|E_P[g(Y)]-E_Q[g(Y)]|\), when the expectations are defined. If \(k\) is bounded, this is well-defined for all probability laws. Equivalently, its square is \(E[k(Y,Y')]+E[k(Z,Z')]-2E[k(Y,Z)]\), where \(Y,Y'\) are independent with law \(P\), and \(Z,Z'\) are independent with law \(Q\), also independent of \(Y,Y'\).

MMD depends on the chosen kernel, which determines which differences its function class can detect and how it weights them. For a characteristic kernel, the kernel mean embedding identifies the law, so MMD is zero only when \(P=Q\). For other kernels, distinct laws can have zero MMD. A bounded kernel keeps MMD bounded even when observations are arbitrarily far apart; this can be helpful for stable comparisons, but it also limits sensitivity to extreme distances.

Worked Comparisons

Worked Example: A Small Amount of Mass Far in the Tail

On the real line, let \(P=\delta_0\) and \(Q=(1-\varepsilon)\delta_0+\varepsilon\delta_R\), where \(\delta_a\) is a point mass at \(a\), \(0<\varepsilon<1\), and \(R>0\). The laws agree on most of their mass, but \(Q\) places a small amount far from zero.

KL divergence. The law \(P\) is absolutely continuous with respect to \(Q\), and \(dP/dQ\) at zero is \(1/(1-\varepsilon)\). Thus \(D_{\mathrm{KL}}(P\|Q)=\log(1/(1-\varepsilon))\). In the reverse direction, \(Q\) assigns positive mass to \(R\), where \(P\) assigns zero mass, so \(D_{\mathrm{KL}}(Q\|P)=+\infty\).

Total variation. The probability of the event \(\{R\}\) differs by \(\varepsilon\), and no event can have a larger difference: the only mass moved is \(\varepsilon\). Hence \(d_{\mathrm{TV}}(P,Q)=\varepsilon\).

Wasserstein distance. A coupling leaves mass \(1-\varepsilon\) at zero and moves mass \(\varepsilon\) from zero to \(R\), for cost \(\varepsilon R\). Since the only possible source of the mass at \(R\) is zero, this cost is necessary. Therefore \(W_1(P,Q)=\varepsilon R\).

Gaussian-kernel MMD. For \(k(u,v)=\exp(-(u-v)^2/2)\), the squared MMD is \(2\varepsilon^2(1-\exp(-R^2/2))\). With \(\varepsilon=0.01\) and \(R=1000\), KL in the \(P\|Q\) direction is \(-\log(0.99)\approx0.01005\), total variation is \(0.01\), and \(W_1=10\). The MMD is approximately \(\sqrt{0.0002}=0.01414\). Thus the transport cost grows with the tail distance, while this bounded-kernel MMD remains bounded. The reverse KL is infinite because of the support change.

Worked Example: A Location Shift of Normal Laws

Suppose at a fixed covariate value the true law is \(P=N(0,1)\) and the proposed law is \(Q=N(1,1)\). Both laws have the same support and variance; they differ by a location shift.

KL divergence. The log density ratio is \(\log(p(y)/q(y))=((y-1)^2-y^2)/2=1/2-y\). Since \(E_P[Y]=0\), \(D_{\mathrm{KL}}(P\|Q)=1/2\). By symmetry the reverse divergence is also \(1/2\).

Total variation. The two densities cross at \(y=1/2\). The probability difference on the region \(y<1/2\) is \(\Phi(1/2)-\Phi(-1/2)=2\Phi(1/2)-1\approx0.383\), where \(\Phi\) is the standard normal distribution function. This equals the total variation distance.

Wasserstein distance. Coupling \(Z\sim N(0,1)\) with \(Z+1\sim N(1,1)\) gives cost \(E|Z-(Z+1)|=1\), so \(W_1\leq1\). For any coupling \((U,V)\) of \(P,Q\), \(E|U-V|\geq|E[U]-E[V]|=1\), so \(W_1=1\).

Gaussian-kernel MMD. For the kernel \(k(u,v)=\exp(-(u-v)^2/2)\), differences of independent \(N(0,1)\) variables have law \(N(0,2)\), giving \(E[k(Y,Y')]=1/\sqrt{3}\); the same holds for \(Q\). The cross-difference has law \(N(-1,2)\), giving \(E[k(Y,Z)]=e^{-1/6}/\sqrt{3}\). Therefore \(\operatorname{MMD}_k^2(P,Q)=\frac{2}{\sqrt{3}}(1-e^{-1/6})\approx0.177\), and the MMD is about \(0.421\). The values differ because the metrics use different scales and definitions; they are not directly comparable numerically.

Worked Example: Weighting Conditional Error Across Covariates

Let \(X\) take values \(0\) and \(1\), with \(P(X=0)=0.9\) and \(P(X=1)=0.1\). Suppose \(P_0=Q_0=N(0,1)\), while \(P_1=N(0,1)\) and \(Q_1=N(10,1)\). The conditional laws are accurate at the common covariate value and shifted at the rare one.

At \(x=0\), all four discrepancies are zero. At \(x=1\), the calculations for a unit-variance normal location shift give \(D_{\mathrm{KL}}(P_1\|Q_1)=10^2/2=50\), \(d_{\mathrm{TV}}(P_1,Q_1)=2\Phi(5)-1\approx1\), and \(W_1(P_1,Q_1)=10\).

For the Gaussian kernel \(k(u,v)=\exp(-(u-v)^2/2)\), the same calculation as in the preceding example gives \(\operatorname{MMD}_k^2(P_1,Q_1)=\frac{2}{\sqrt{3}}(1-e^{-100/6})\), so the MMD is approximately \(1.075\). Averaging each conditional discrepancy over \(X\) multiplies its value at \(x=1\) by \(0.1\); for example, the expected \(W_1\) error is \(1\), and the expected KL error is \(5\). A different target population with a different probability for \(X=1\) would give different averages. The covariate distribution is therefore part of the definition of an expected conditional error.

Two Useful Relations

Theorem (Pinsker’s inequality): For probability laws \(P,Q\), \(d_{\mathrm{TV}}(P,Q)\leq\sqrt{D_{\mathrm{KL}}(P\|Q)/2}\). The inequality is interpreted as automatic if the KL divergence is infinite.

Proof. The infinite case is immediate, so suppose \(D_{\mathrm{KL}}(P\|Q)<\infty\). Let \(\mu=P+Q\), and let \(p=dP/d\mu\), \(q=dQ/d\mu\). Define \(A=\{y:p(y)\geq q(y)\}\), \(a=P(A)\), and \(b=Q(A)\). The positive and negative parts of \(p-q\) have equal integrals, so \(d_{\mathrm{TV}}(P,Q)=\int_A(p-q)\,d\mu=a-b\).

The log-sum inequality, applied on \(A\) and its complement, gives \(D_{\mathrm{KL}}(P\|Q)\geq a\log(a/b)+(1-a)\log((1-a)/(1-b))\), with boundary cases understood by limits. For fixed \(b\in(0,1)\), let the right side minus \(2(a-b)^2\) be \(h(a)\). Direct differentiation gives \(h(b)=h'(b)=0\) and \(h''(a)=1/[a(1-a)]-4\geq0\), since \(a(1-a)\leq1/4\). Thus \(h(a)\geq0\). The inequality extends to boundary values by continuity or as an infinite divergence. Consequently \(D_{\mathrm{KL}}(P\|Q)\geq2(a-b)^2=2d_{\mathrm{TV}}(P,Q)^2\), proving the claim. \(\square\)

Corollary (expected conditional Pinsker bound): If \(P_X,Q_X\) are measurable conditional probability laws and \(E[D_{\mathrm{KL}}(P_X\|Q_X)]<\infty\), then \(E[d_{\mathrm{TV}}(P_X,Q_X)]\leq\sqrt{E[D_{\mathrm{KL}}(P_X\|Q_X)]/2}\).

Proof. Apply Pinsker’s inequality at each covariate value where the conditional laws are defined. Taking expectations gives \(E[d_{\mathrm{TV}}(P_X,Q_X)]\leq E[\sqrt{D_{\mathrm{KL}}(P_X\|Q_X)/2}]\). The square-root function is concave on the nonnegative reals, so Jensen’s inequality bounds the last expression by \(\sqrt{E[D_{\mathrm{KL}}(P_X\|Q_X)]/2}\). The assumed finite expectation makes this bound finite. \(\square\)

This implication is one-way: a small expected KL error guarantees a small expected total variation error, but small total variation alone need not control KL. The tail example shows why: total variation can be small while reverse KL is infinite.

Matching a Metric to the Target

Metric	Most directly reflects	Important limitation
KL divergence	Log-probability or likelihood regret; under-assigning probability to outcomes generated by the truth	Directional; can be infinite under support mismatch and sensitive to low proposed density
Total variation	Largest discrepancy in event probabilities	Does not account for how far apart differing locations are
Wasserstein distance	Amount of probability moved and the distance it travels	Requires a meaningful ground metric and suitable moments
MMD	Differences visible to a chosen kernel function class	Depends on kernel choice; bounded kernels can downweight extreme distances

If the practical target is predictive log score, KL regret is closely aligned with that target and its direction must be stated. If the target is controlling the probability error of every event, total variation is natural. If the distance between outcomes has substantive meaning, Wasserstein distance can distinguish nearby from distant errors. If a kernel captures the features or scales relevant to the application, MMD provides a corresponding comparison, often with bounded sensitivity under a bounded kernel.

For conditional laws, also state the covariate distribution used in an average error. A score averaged over the observed population emphasizes common covariate regions; a score under another distribution answers a different question. In addition, report the direction of KL, the ground metric for Wasserstein distance, or the kernel for MMD. Without these choices, a numerical error summary may not specify a reproducible target.

Common Mistake

Wrong reasoning: “The laws differ by only one percent of probability, so every reasonable discrepancy must be small.” This ignores where the differing probability goes and which discrepancy is being used.
Correction: In the tail example, \(d_{\mathrm{TV}}=0.01\), but \(W_1=10\) when the moved mass travels a distance of 1000, and reverse KL is infinite. State the target discrepancy and inspect its assumptions before interpreting its size.

Key takeaway: KL emphasizes log-density mismatch, total variation emphasizes event-probability mismatch, Wasserstein distance incorporates outcome geometry, and MMD measures discrepancies through a selected kernel. For conditional distribution error, these choices must be paired with a specified covariate weighting.

Check Your Understanding

For each question, identify the relevant assumptions as well as the metric comparison.

Why can \(D_{\mathrm{KL}}(Q\|P)\) be infinite in the small-tail example even though \(d_{\mathrm{TV}}(P,Q)\) is only \(\varepsilon\)?
If a small amount of probability is moved farther from its original location, which of the four discrepancies is directly scaled by that transport distance in the example?
What additional choices are needed to interpret a Wasserstein distance and an MMD value?
State the expected conditional Pinsker bound and explain why it does not imply that small expected total variation forces small expected KL.
How would the expected conditional error change if the target population placed more probability on covariate values where the conditional estimate is inaccurate?