Proper Scoring Rules for Conditional Distributions

Why Score a Predictive Distribution?

A distributional forecast is useful when uncertainty matters, not just a single predicted value. For example, a clinical model might predict a patient's full distribution of recovery time given observed characteristics. A scoring rule evaluates that forecast after the outcome is observed. The central question is whether the rule gives the best expected score to the distribution that actually generates the outcome.

Throughout, a score is treated as a loss: smaller values are better. This convention matters because some references call a quantity to be maximized a score. Here, propriety means minimizing expected loss. We first define the idea for an unconditional distribution, then apply it to conditional laws.

Definition: Let $\mathcal P$ be a class of probability distributions on an outcome space $\mathcal Y$. A scoring rule is a measurable loss $S(Q,y)$ for a forecast distribution $Q\in\mathcal P$ and realized outcome $y\in\mathcal Y$. Its risk when the true distribution is $P$ is $$ R(P,Q)=\int_{\mathcal Y} S(Q,y)\,P(dy), $$ when this expectation is well-defined. The scoring rule is proper on $\mathcal P$ if $R(P,P)\le R(P,Q)$ for every $P,Q\in\mathcal P$. It is strictly proper if equality is possible only when $Q=P$.

The definition compares a forecast with the distribution that generates the observation. It does not say that the realized loss $S(P,y)$ must be smaller than $S(Q,y)$ for every individual outcome. Rather, it is the expected loss over repeated outcomes from $P$ that must be minimized by forecasting $P$. Scores may be unbounded, so claims about risks require that the expectations and comparisons be meaningful; the examples below state useful conditions.

Conditional Propriety

Let $X$ be a predictor taking values in a standard Borel space, and let $Y$ be the outcome. Write $P_x$ for a regular conditional distribution of $Y$ given $X=x$. A distributional prediction based on $X$ is a probability kernel $x\mapsto Q_x$: for each $x$, $Q_x$ is a distribution, and $x\mapsto Q_x(A)$ is measurable for each measurable outcome set $A$. The true conditional law is itself such a kernel, up to choices on sets of predictor values having probability zero.

Definition: The conditional risk of a forecast kernel $Q$ at predictor value $x$ is $$ R_x(P_x,Q_x)=\int S(Q_x,y)\,P_x(dy). $$ A scoring rule is conditionally proper if $R_x(P_x,Q_x)\ge R_x(P_x,P_x)$ for $P_X$-almost every $x$, for every admissible measurable forecast kernel $Q$. It is conditionally strictly proper if equality at $x$ implies $Q_x=P_x$, subject to the score's distributional identifiability, for $P_X$-almost every $x$.

The conditional statement is pointwise in the covariate value, apart from null sets. It does not require that the forecast be selected independently of $X$; indeed, its purpose is to evaluate forecasts that depend on $X$. The basic implication for an overall risk follows by averaging conditional risks, with care about integrability.

Theorem: Suppose the score is conditionally proper and the conditional risks are measurable and integrable, so that the overall risks are finite real numbers. Then $$ E[S(Q_X,Y)]\ge E[S(P_X,Y)]. $$ If the score is conditionally strictly proper, equality of these finite overall risks implies $Q_X=P_X$ for $P_X$-almost every $X$.

Proof. By the defining property of conditional expectation, the overall risk of $Q$ is the expectation of its conditional risk: $$ E[S(Q_X,Y)]=E\!\left[R_X(P_X,Q_X)\right]. $$ Likewise, $E[S(P_X,Y)]=E[R_X(P_X,P_X)]$. Conditional propriety gives a nonnegative measurable difference $$ \Delta(X)=R_X(P_X,Q_X)-R_X(P_X,P_X)\ge 0 $$ almost surely. Integrability and finiteness allow subtraction of the two overall risks, giving $$ E[S(Q_X,Y)]-E[S(P_X,Y)]=E[\Delta(X)]\ge 0. $$ This proves the risk inequality. If the finite risks are equal, then $E[\Delta(X)]=0$. A nonnegative random variable with expectation zero is zero almost surely: for each positive integer $k$, $P(\Delta(X)>1/k)\le kE[\Delta(X)]=0$, and $\{\Delta(X)>0\}$ is the union of these events. Thus $\Delta(X)=0$ almost surely. Conditional strict propriety now gives $Q_X=P_X$ almost surely. $\square$

The finiteness condition is important. Equality of two extended overall risks that are both $+\infty$ does not allow their difference to be treated as an expectation of zero. In some problems, one can instead establish directly that the expected conditional regret $E[\Delta(X)]$ is well-defined and equals zero; that condition also supports the strictness conclusion.

The Log Score

For distributions with densities $p$ and $q$ relative to a common dominating measure, the log score is the negative log density assigned to the observation:

S(q,y)=-\log q(y).

For discrete distributions, densities in this definition are probability mass functions. For continuous distributions they are densities, and the density's numerical value can exceed one; the score remains defined as long as the relevant log expectations exist. Strict propriety follows from the logarithm inequality, under conditions that make the risk comparison valid.

Result: Suppose $P$ and $Q$ have densities $p$ and $q$, and $E_P[-\log p(Y)]$ is finite. If the cross-entropy $E_P[-\log q(Y)]$ is well-defined, then $$ E_P[-\log q(Y)]-E_P[-\log p(Y)] =\int p(y)\log\!\left(\frac{p(y)}{q(y)}\right)dy\ge 0. $$ Equality holds exactly when $p=q$ almost everywhere. If $q=0$ on a set of positive $P$-probability, the forecast risk is $+\infty$ and cannot improve on the true forecast.

Proof. First suppose $q>0$ wherever $p>0$, and the displayed expectations are finite. The elementary inequality $\log t\le t-1$ for $t>0$ follows by differentiating $t-1-\log t$, whose minimum is zero at $t=1$. Apply it with $t=q(y)/p(y)$, at points where $p(y)>0$: $$ p(y)\log\!\left(\frac{p(y)}{q(y)}\right) \ge p(y)-q(y). $$ Integrating over the support of $p$ gives a lower bound $1-\int_{\{p>0\}}q(y)\,dy\ge 0$. If equality holds, the logarithm inequality must be equality $P$-almost everywhere, so $q(y)/p(y)=1$ $P$-almost everywhere. Since both densities integrate to one, this gives $p=q$ almost everywhere. If $q=0$ on a set where $p>0$, the log loss is infinite there; the risk comparison follows by the usual extended-risk interpretation. $\square$

Worked Example: Log Loss for a Binary Outcome

Let $Y$ be Bernoulli with $P(Y=1)=0.8$. Compare the true forecast $p=0.8$ with the forecast $q=0.6$. For a Bernoulli forecast $q$, the expected log loss is $$ R(p,q)=-p\log q-(1-p)\log(1-q). $$

For the true forecast, $$ R(0.8,0.8)=-0.8\log(0.8)-0.2\log(0.2) =0.1785148+0.3218876 =0.5004024. $$ For $q=0.6$, $$ R(0.8,0.6)=-0.8\log(0.6)-0.2\log(0.4) =0.4086605+0.1832581 =0.5919186. $$ The incorrect forecast has the larger expected loss, by $0.0915162$. The strict comparison is about repeated outcomes drawn from the Bernoulli law, not a guarantee that the true forecast wins on each single trial.

Worked Example: Conditional Log Score Across Two Groups

Suppose $X$ takes two values, each with probability $1/2$, and the true conditional probabilities are $P(Y=1\mid X=x_1)=0.8$ and $P(Y=1\mid X=x_2)=0.3$. A forecast that uses the correct conditional probabilities has conditional risks $$ h(0.8)=-0.8\log(0.8)-0.2\log(0.2)=0.5004024 $$ and $$ h(0.3)=-0.3\log(0.3)-0.7\log(0.7)=0.6108643. $$ Thus its overall expected loss is $$ \frac{h(0.8)+h(0.3)}{2}\approx 0.55563336\approx 0.5556. $$

Now forecast $q=0.5$ in both groups. Its conditional risk is $-\log(0.5)=0.6931472$ in either group, so the overall risk is also $0.6931472$. The conditional forecast has lower expected loss by approximately $0.1375138$. Averaging over $X$ does not erase the importance of getting the conditional law right.

The Continuous Ranked Probability Score

For a real-valued outcome, the continuous ranked probability score (CRPS) compares the forecast cumulative distribution function with the step function determined by the observed value. Write $F$ for the true CDF and $G$ for the forecast CDF. Define

Definition: The CRPS of forecast CDF $G$ at observation $y$ is $$ \operatorname{CRPS}(G,y)=\int_{-\infty}^{\infty}\left(G(z)-\mathbf{1}\{y\le z\}\right)^2\,dz. $$

This integral is finite for distributions with finite first absolute moment. For such a true law and forecast, its expected score has a simple risk difference, which proves both propriety and strict propriety.

Theorem: Let $P$ and $Q$ be distributions on the real line with finite first absolute moments, and let their CDFs be $F$ and $G$. Then $$ E_{Y\sim P}[\operatorname{CRPS}(G,Y)] - E_{Y\sim P}[\operatorname{CRPS}(F,Y)] = \int_{-\infty}^{\infty}(G(z)-F(z))^2\,dz. $$ Consequently CRPS is proper, and it is strictly proper: equality of risks implies $P=Q$.

Proof. Fix $z$, and let $B_z=\mathbf{1}\{Y\le z\}$. Under $P$, $E[B_z]=F(z)$. Expanding the square and taking expectation gives $$ E[(G(z)-B_z)^2]=G(z)^2-2G(z)F(z)+F(z). $$ For the true forecast, the same expression with $G(z)=F(z)$ is $F(z)-F(z)^2$. Subtracting yields $$ E[(G(z)-B_z)^2]-E[(F(z)-B_z)^2]=(G(z)-F(z))^2. $$ Integrating this pointwise identity gives the stated risk difference. The moment assumptions ensure the CRPS risks are finite, so the expectations and integrals can be taken and subtracted; equivalently, Tonelli's theorem applies to each nonnegative score integral. The resulting difference is nonnegative, proving propriety. If it is zero, $G(z)=F(z)$ for Lebesgue-almost every $z$. CDFs are right-continuous: if they differed at any point, right-continuity would make them differ on an interval immediately to the right, contradicting equality almost everywhere. Hence $G=F$ everywhere, so $Q=P$. $\square$

Worked Example: CRPS for a Binary Outcome on the Real Line

Let $Y$ equal 1 with probability $p$ and 0 with probability $1-p$, and let a forecast assign probability $q$ to 1. For $0\le z<1$, the forecast CDF is $G(z)=1-q$. The indicator $\mathbf{1}\{Y\le z\}$ equals 1 when $Y=0$ and 0 when $Y=1$. Outside the interval from 0 to 1, the CDF and indicator agree, apart from endpoints that do not affect the integral. Therefore, $$ \operatorname{CRPS}(G,0)=\int_0^1((1-q)-1)^2\,dz=q^2, $$ and $$ \operatorname{CRPS}(G,1)=\int_0^1((1-q)-0)^2\,dz=(1-q)^2. $$

The expected score under the true probability $p$ is $$ (1-p)q^2+p(1-q)^2 =q^2-2pq+p =(q-p)^2+p(1-p). $$ At $q=p$, the risk is $p(1-p)$; the excess risk for any other $q$ is exactly $(q-p)^2$. Thus the true probability uniquely minimizes expected CRPS in this example.

A Common Mistake: Equal Infinite Risks

A tempting but invalid argument is: “The overall risks are equal, so their nonnegative conditional difference has expectation zero.” That argument requires finite overall risks, or a separately established, well-defined expected conditional regret equal to zero. If both risks are infinite, their equality as extended real numbers does not imply anything about the difference.

Worked Example: Why Infinite Overall Log Risks Do Not Identify a Forecast

Let $X$ take positive integer values with $P(X=n)=c/n^2$, where $c$ normalizes the probabilities. Given $X=n$, let the true law $P_n$ be uniform on the first $M_n=\lceil e^{n^2}\rceil$ positive integers. Let the forecast $Q_n$ be uniform on the first $2M_n$ integers. Both conditional log risks are finite: $$ R_n(P_n,P_n)=\log M_n,\qquad R_n(P_n,Q_n)=\log(2M_n). $$ Their difference is $\log 2$ for every $n$, and $P_n\ne Q_n$ for every $n$.

Nevertheless, each overall risk is infinite. Since $\log M_n\ge n^2$, the true overall risk is at least $\sum_n(c/n^2)n^2=\infty$. The forecast risk is larger conditionally by $\log 2$, but is also infinite. Writing $+\infty=+\infty$ therefore gives no equality-of-risk conclusion. In fact, the expected conditional regret is $\sum_n(c/n^2)\log 2=\log 2>0$; it is the subtraction of the two infinite overall risks that is invalid.

When Propriety Matters

Propriety gives a principled target for learning an entire conditional distribution: in expectation, reporting the true conditional law cannot be improved on by another forecast. Strict propriety adds identification, so that attaining the same finite risk requires the same conditional distribution almost surely. These properties make scores useful objectives for fitting and comparing distributional predictions, while leaving separate questions about estimation error and finite-sample performance.

Before using a scoring rule, check the loss convention, the distribution class, and the conditions under which its expected risk exists. For conditional comparisons, distinguish pointwise conditional risk from an overall average, and do not infer strict identification from equality of infinite risks. Those checks are part of the mathematical content of a score, not merely technical qualifications.

Check Your Understanding

Use the definitions and risk comparisons above to answer the following.

In the loss convention used here, what inequality defines propriety, and what additional condition defines strict propriety?
For the log score, why does $\log t\le t-1$ imply that the true distribution minimizes expected loss?
In the conditional theorem, why does a nonnegative conditional risk difference with expectation zero vanish almost surely?
For CRPS, what is the expected risk difference between CDFs $F$ and $G$, and why does a zero difference identify the distribution?
Why does equality of two infinite overall log risks not imply equality of the conditional forecasts?