Common Pitfalls in Conditional Distribution Learning

Why These Pitfalls Matter

A conditional distribution forecast is a probability distribution that can change with the predictors. Its quality can be assessed in several different ways: by checking calibration, by evaluating a scoring rule, or by measuring distance between distributions. These assessments answer related but distinct questions. A forecast can match the overall outcome distribution while being wrong at each predictor value; a theoretically proper score does not ensure that a fitted model learns the truth; and convergence under one distribution metric need not imply convergence under another.

The central discipline is to identify exactly what is being claimed. Is the claim about the distribution averaged over predictors, about the forecast at each predictor value, about the population risk of a score, or about distance in a particular metric? The examples below show why these distinctions matter.

Definition: Let $X$ be a predictor and $Y$ an outcome, and suppose a regular conditional distribution $P_x=\mathcal{L}(Y\mid X=x)$ exists. A conditional distribution forecast is a measurable probability kernel $Q_x$. It is conditionally correct if $Q_X=P_X$ almost surely. It is marginally correct if its mixture distribution equals the outcome distribution: for every Borel set $A$, $E[Q_X(A)]=P(Y\in A)$. A forecast is calibrated with respect to its reported distribution if $E[\mathbf{1}\{Y\in A\}\mid Q_X]=Q_X(A)$ almost surely for every Borel set $A$. These are different properties; conditional correctness implies the latter two, but marginal correctness alone does not.

Pitfall 1: Mistaking Marginal Calibration for Conditional Accuracy

Marginal correctness averages over the predictor distribution. Errors at different predictor values can cancel in that average. Conditional correctness instead asks whether the forecast is right at almost every predictor value. In applications, the second question is usually essential: decisions made for a particular patient or setting depend on that setting's conditional distribution, not only on the population mixture.

Worked Example: Correct Marginal Distribution, Reversed Conditional Distributions

Let $X$ take values 0 and 1 with probability $1/2$ each, and let $Y$ be binary. Suppose the true conditional probabilities are $P(Y=1\mid X=0)=0.2$ and $P(Y=1\mid X=1)=0.8$. Now forecast $Q_0=\operatorname{Bernoulli}(0.8)$ and $Q_1=\operatorname{Bernoulli}(0.2)$, reversing the two conditional laws.

The true marginal probability of $Y=1$ is $(0.2+0.8)/2=0.5$. The forecast mixture also assigns probability $(0.8+0.2)/2=0.5$ to $Y=1$, so the forecast is marginally correct. But at $X=0$, it reports 0.8 instead of 0.2; at $X=1$, it reports 0.2 instead of 0.8. Thus it is not conditionally correct.

This example also shows that a correct marginal distribution does not rescue a predictor-specific analysis. For instance, a decision based on the forecast for $X=0$ would use the wrong event probability by 0.6.

Calibration diagnostics must therefore be interpreted in light of what they condition on. A check that only compares the overall forecast mixture with the observed outcome distribution cannot establish that forecasts are reliable within predictor-defined subgroups. Conversely, calibration with respect to the forecast $Q_X$ groups cases by the forecast itself, not necessarily by all information in $X$. It is weaker than requiring $Q_X=P_X$ almost surely.

Pitfall 2: Treating Propriety as a Convergence Guarantee

A scoring rule assigns a numerical loss to a forecast distribution and an observed outcome. Lower scores are better. A score is proper if, when the true distribution is $P$, the expected score is minimized by reporting $P$; it is strictly proper if this minimizer is unique. This is a population statement about the objective. By itself, it says nothing about whether a particular estimation procedure finds the population minimizer, whether the model class contains it, or whether empirical risk approaches population risk.

Definition: A scoring rule $S(Q,y)$ is proper on a class of distributions if, for every true distribution $P$ in the class, $E_{Y\sim P}[S(P,Y)]\leq E_{Y\sim P}[S(Q,Y)]$ for every forecast $Q$ for which the expectations are defined. It is strictly proper if equality is possible only when $Q=P$. For conditional forecasting, the analogous population comparison is made conditionally at each $X=x$, and then averaged over $X$, subject to existence of the expectations.

The binary Brier score makes the population-risk distinction explicit. For $Y\in\{0,1\}$, a forecast $q\in[0,1]$ assigns probability $q$ to $Y=1$, and its Brier loss is $(Y-q)^2$.

Result: Write $p(X)=P(Y=1\mid X)$, and let $q(X)$ be any measurable forecast in $[0,1]$. Then $$ E[(Y-q(X))^2]-E[p(X)(1-p(X))]=E[(q(X)-p(X))^2]. $$ Consequently, if a sequence of forecasts $q_n$ has Brier risk converging to the Bayes risk $E[p(X)(1-p(X))]$, then $E[(q_n(X)-p(X))^2]\to 0$.

Proof. Conditional on $X$, the outcome is Bernoulli with success probability $p=p(X)$. Therefore $$ E[(Y-q)^2\mid X] =p(1-q)^2+(1-p)q^2. $$ Expanding the right-hand side gives $$ p-2pq+pq^2+q^2-pq^2 =p-2pq+q^2. $$ Also $p(1-p)+(q-p)^2=p-p^2+q^2-2pq+p^2=p-2pq+q^2$. Hence the conditional risk equals $p(1-p)+(q-p)^2$. Taking expectations proves the identity. The convergence statement follows because the excess risk is exactly a nonnegative squared-error expectation. $\square$

Worked Example: A Proper Score With a Misspecified Model Class

Suppose $X$ has the two equally likely values 0 and 1, with true probabilities $p(0)=0.2$ and $p(1)=0.8$. Consider a model class that can only issue a constant forecast $q(X)=c$. The Brier score is strictly proper, but this class cannot represent the true conditional probabilities.

The mean squared difference from the truth is $$ E[(c-p(X))^2] =\frac{(c-0.2)^2+(c-0.8)^2}{2}. $$ Expanding gives $c^2-c+0.34=(c-0.5)^2+0.09$, so the best constant is $c=0.5$, with excess risk $0.09$. The forecast is the best member of the class but remains conditionally wrong by 0.3 at each value of $X$.

Thus strict propriety does not remove approximation error. Even if the score is ideal for comparing distributions, a consistency claim also needs suitable conditions on the model class and the estimation procedure, such as risk convergence to the relevant population minimum and a way to relate score excess to the desired error measure.

The Brier identity is a useful case where the score excess directly equals a familiar conditional probability error. It should not be silently generalized to every proper score and every notion of distributional convergence. Different scores can distinguish distributions in different ways, and their risk differences need not control a chosen metric without additional assumptions.

Pitfall 3: Ignoring Support in Log-Score Arguments

For a density forecast $q$, the logarithmic loss is $-\log q(Y)$, with loss $+\infty$ when $q(Y)=0$. Support is therefore not a technical afterthought. If the true distribution gives positive probability to outcomes that the forecast assigns zero density, the expected log loss is infinite. Expressions involving Kullback–Leibler divergence likewise require absolute continuity: the true distribution must assign no mass to sets on which the forecast distribution is zero.

Worked Example: A Forecast With Too-Narrow Support

Let the true outcome have the uniform distribution on $[0,1]$, with density $p(y)=1$ there. Forecast the uniform distribution on $[0,0.5]$, whose density is $q(y)=2$ for $0\leq y\leq0.5$ and zero otherwise.

With probability $1/2$, the outcome lies in $(0.5,1]$. On that event $q(Y)=0$, so the log loss is $+\infty$. Since this event has positive probability, $E[-\log q(Y)]=+\infty$. The forecast may produce finite losses for outcomes in its own support, but that does not make its expected log score finite under the true distribution.

The relevant check is not whether the two supports overlap, but whether the true distribution is absolutely continuous with respect to the forecast distribution. Here it is not: the true distribution assigns probability $1/2$ to $(0.5,1]$, while the forecast assigns that set probability zero.

In conditional distribution learning, this check is predictor-specific. A forecast can have adequate support for common predictor values but fail under conditional extrapolation. Any finite-risk or divergence argument must state the needed support and integrability conditions for the conditional distributions being compared.

Pitfall 4: Treating Distribution Metrics as Interchangeable

A metric, a divergence, and weak convergence describe differences in different senses. For example, total variation detects changes in the probability assigned to sets, while Wasserstein distance also reflects the geometry of the outcome space. Weak convergence tests integrals of bounded continuous functions. There is no universal rule that convergence in one of these senses implies convergence in all the others.

Definition: For probability measures $P,Q$, total variation distance is $\sup_A|P(A)-Q(A)|$, where the supremum is over measurable sets. On the real line, the 1-Wasserstein distance $W_1(P,Q)$ is the infimum of $E|U-V|$ over all pairs $(U,V)$ with marginal distributions $P$ and $Q$; it is finite for measures with finite first moments. A sequence $P_n$ converges weakly to $P$ if $\int g\,dP_n\to\int g\,dP$ for every bounded continuous function $g$.

Worked Example: Weak and Wasserstein Convergence Without Total Variation Convergence

Compare $P_n=\delta_{1/n}$ with $P=\delta_0$, point masses at $1/n$ and 0. The only possible coupling places its two coordinates at these two points, so $W_1(P_n,P)=1/n\to0$. But for the set $A=\{1/n\}$, $P_n(A)=1$ and $P(A)=0$. Thus the total variation distance is 1 for every $n$.

The sequence does converge weakly: for any bounded continuous $g$, continuity at 0 gives $g(1/n)\to g(0)$, so $\int g\,dP_n\to\int g\,dP$. This example shows that even Wasserstein convergence to a point mass need not imply total variation convergence.

Worked Example: Weak and Total Variation Convergence Without Wasserstein Convergence

For integers $n\geq2$, let $P_n=(1-1/n)\delta_0+(1/n)\delta_n$ and $P=\delta_0$. For any bounded continuous $g$, $$ \left|\int g\,dP_n-\int g\,dP\right| =\frac{|g(n)-g(0)|}{n} \leq \frac{2\|g\|_\infty}{n}\to0. $$ So $P_n$ converges weakly to $P$. The total variation distance is $1/n$, since the only excess mass is $1/n$ at $n$.

In any coupling with $V\sim\delta_0$, $V=0$ almost surely, so $E|U-V|=E|U|=(1/n)n=1$ for $U\sim P_n$. Hence $W_1(P_n,P)=1$ for every $n$. The small probability of a distant outcome disappears for weak convergence and total variation, but its distance-weighted contribution remains in $W_1$. The first moments are finite for each $n$, but they are not uniformly controlled.

The practical lesson is to name the metric and the conditions behind any implication. A result in one metric cannot be substituted for a result in another just because both are called distributional error. For conditional distributions, the same caution applies to metrics evaluated pointwise in $X$, averaged over $X$, or uniformly over a predictor space.

A Diagnostic Checklist

Before interpreting a conditional distribution result, check the target of the claim and the assumptions connecting it to the reported evidence. The following questions help prevent the most common overstatements.

Calibration: Is the claim about the overall mixture, calibration grouped by the forecast, or correctness conditional on the predictors?
Score: Is the score proper for the target distribution, and does its excess risk control the specific error measure being claimed?
Learning procedure: Is the model class rich enough, and is there evidence that estimation risk approaches the population optimum?
Support: Does the forecast assign positive probability or density wherever the true conditional distribution does, when the argument requires it?
Metric: Which distance or convergence notion is used, and what moment, compactness, or regularity assumptions are needed for the desired implication?

A careful conclusion separates what has actually been established from what would require another argument. Marginal agreement is not conditional accuracy; a proper score is not a learning theorem; finite log loss requires support compatibility; and distribution metrics are not interchangeable without additional structure.

Check Your Understanding

Use the distinctions in this tutorial to answer each question.

What does marginal correctness establish about a conditional forecast, and what does it fail to establish?
Why does a strictly proper score not, by itself, show that a fitted conditional distribution estimator converges to the truth?
For the binary Brier score, what quantity equals the excess population risk over the Bayes risk?
What absolute-continuity condition is relevant when a true distribution is evaluated using a log-score forecast?
Give one example from the tutorial showing that convergence in one distribution metric does not imply convergence in another.
Why does the sequence $(1-1/n)\delta_0+(1/n)\delta_n$ converge weakly to $\delta_0$ while its 1-Wasserstein distance from $\delta_0$ stays equal to 1?