Proper Scoring Rules and Conditional Regret

Why Score Probability Forecasts?

A probability forecast contains more information than a single predicted class. For example, reporting probabilities \((0.6,0.4)\) distinguishes a forecast that favors the first outcome slightly from one that is nearly certain about it. A scoring rule evaluates such forecasts after the outcome is observed. The central question is whether its expected score encourages a forecaster to report the probabilities they actually believe.

Throughout, take a finite outcome space \(\{1,\ldots,K\}\), and represent a probability distribution by a vector in the simplex \(\Delta_K=\{p\in[0,1]^K:\sum_{k=1}^K p_k=1\}\). We use the loss convention: a smaller score is better. Thus a scoring rule is a function \(S(q,k)\) giving the loss when forecast \(q\in\Delta_K\) is issued and outcome \(k\) occurs.

Definition: For a true distribution \(p\in\Delta_K\), the expected score of reporting \(q\in\Delta_K\) is \(L(p,q)=\sum_{k=1}^K p_k S(q,k)\). The scoring rule is proper if \(L(p,p)\leq L(p,q)\) for every \(p,q\in\Delta_K\). It is strictly proper if this inequality is strict whenever \(q\neq p\).

Propriety is an expected-score condition, not a claim about the score on every individual outcome. A truthful forecast can receive a larger realized loss than a mistaken forecast on a particular case. The guarantee is that, before the outcome is known and under distribution \(p\), reporting \(p\) minimizes expected loss.

The Divergence Associated With a Score

The extra expected loss from reporting \(q\) when the truth is \(p\) is the natural measure of the forecast's regret. It is determined by the scoring rule itself.

Definition: The divergence associated with \(S\) is \(D_S(p,q)=L(p,q)-L(p,p)=\sum_{k=1}^K p_k\{S(q,k)-S(p,k)\}\). Its first argument is the true distribution and its second is the reported distribution. If \(S\) is proper, \(D_S(p,q)\geq 0\); if \(S\) is strictly proper, \(D_S(p,q)>0\) whenever \(p\neq q\).

This divergence is not necessarily a metric. It can be asymmetric, and it need not satisfy a triangle inequality. Its defining role is decision-theoretic: it is exactly the excess expected score incurred by the incorrect report.

Worked Example: Multiclass Brier Score

For \(K\) outcomes, the Brier loss is \(S(q,y)=\sum_{j=1}^K(q_j-\mathbf{1}\{y=j\})^2\). Let the true probabilities be \(p=(0.5,0.3,0.2)\), and consider the report \(q=(0.4,0.4,0.2)\). First derive the score's regret.

For a random outcome with distribution \(p\), expanding the square and taking expectations gives \(L(p,q)=\sum_j q_j^2-2\sum_j p_jq_j+1\). At the truthful report, this is \(L(p,p)=1-\sum_jp_j^2\). Subtracting and completing the square, \(D_S(p,q)=\sum_j(q_j-p_j)^2\).

Here \(\sum_jp_j^2=0.25+0.09+0.04=0.38\), so the truthful expected score is \(1-0.38=0.62\). For the reported forecast, \(\sum_jq_j^2=0.16+0.16+0.04=0.36\) and \(\sum_jp_jq_j=0.20+0.12+0.04=0.36\). Therefore its expected score is \(0.36-2(0.36)+1=0.64\). The regret is \(0.64-0.62=0.02\), also equal to \((0.4-0.5)^2+(0.4-0.3)^2+(0.2-0.2)^2=0.02\).

Worked Example: Log Loss and Kullback–Leibler Regret

For positive forecast probabilities, log loss is \(S(q,y)=-\log q_y\). Let \(p=(0.6,0.4)\) and report \(q=(0.8,0.2)\). The truthful expected score is \(-0.6\log(0.6)-0.4\log(0.4)\), approximately \(0.673012\). The reported expected score is \(-0.6\log(0.8)-0.4\log(0.2)\), approximately \(0.777661\).

The difference is \(0.6\log(0.6/0.8)+0.4\log(0.4/0.2)=0.6\log(0.75)+0.4\log(2)\). Numerically, this is \(0.6(-0.287682)+0.4(0.693147)\), or approximately \(-0.172609+0.277259=0.104650\). Thus the regret is the Kullback–Leibler divergence from \(p\) to \(q\), and is positive for these unequal distributions.

Propriety and Regret: A Direct Equivalence

The divergence gives a compact way to state exactly what propriety means. The following result applies to any real-valued scoring rule on the finite outcome space.

Theorem: A scoring rule \(S\) is proper if and only if \(D_S(p,q)\geq0\) for every \(p,q\in\Delta_K\). It is strictly proper if and only if \(D_S(p,q)>0\) for every pair \(p\neq q\).

Proof. By definition, \(D_S(p,q)=L(p,q)-L(p,p)\). Therefore \(D_S(p,q)\geq0\) for all \(p,q\) is precisely the inequality \(L(p,p)\leq L(p,q)\) required for propriety. Requiring strict positivity whenever \(q\neq p\) is precisely the strict version of the same inequality. \(\square\)

This equivalence is simple but useful: instead of separately comparing expected scores for each report, one can calculate the associated divergence and inspect its sign. It also makes clear why nonnegative regret alone does not establish strict propriety; the regret must be positive for every incorrect distribution.

Conditional Regret Given Predictors

In prediction problems, the distribution of the outcome can depend on observed predictors \(X\). For a finite outcome \(Y\in\{1,\ldots,K\}\), define the posterior probability vector \(\eta(X)\) by \(\eta_k(X)=P(Y=k\mid X)\). A forecast \(q(X)\) is a measurable, predictor-dependent probability vector. Conditional on \(X\), the expected score is computed under \(\eta(X)\).

Result: For an integrable scoring rule and forecast, the conditional regret of reporting \(q(X)\) instead of \(\eta(X)\) is \(E[S(q(X),Y)\mid X]-E[S(\eta(X),Y)\mid X]=D_S(\eta(X),q(X))\), almost surely. If the two expected scores are finite, their population risk difference is \(E[D_S(\eta(X),q(X))]\).

Proof. Since \(Y\) has finitely many possible values, conditioning on \(X\) gives \(E[S(q(X),Y)\mid X]=\sum_{k=1}^K P(Y=k\mid X)S(q(X),k)=\sum_{k=1}^K\eta_k(X)S(q(X),k)\). Applying the same calculation to the truthful forecast gives \(E[S(\eta(X),Y)\mid X]=\sum_{k=1}^K\eta_k(X)S(\eta(X),k)\). Subtracting these finite sums yields \(D_S(\eta(X),q(X))\) by definition. Taking expectations of the conditional identity gives the population risk identity when the risks are finite. \(\square\)

For a proper score, this conditional regret is nonnegative at each predictor value, so the posterior forecast minimizes conditional expected score. For a strictly proper score, it is zero exactly when the report agrees with the posterior. This is the scoring-rule version of pointwise Bayes-risk minimization: the divergence quantifies the conditional cost of deviating from the Bayes forecast.

Worked Example: Conditional Brier Regret Across Two Groups

Suppose \(X\) takes two values, \(a\) and \(b\), each with probability \(1/2\), and the outcome is binary. Let \(P(Y=1\mid X=a)=0.7\) and \(P(Y=1\mid X=b)=0.2\). A forecast reports \(0.5\) in both groups. Under binary Brier loss, the regret for a true probability \(r\) and report \(s\) is \(2(s-r)^2\): the two coordinates of the probability vector differ by \(s-r\) and \(r-s\).

In group \(a\), the regret is \(2(0.5-0.7)^2=2(0.04)=0.08\). In group \(b\), it is \(2(0.5-0.2)^2=2(0.09)=0.18\). The mean conditional regret is therefore \(\tfrac12(0.08)+\tfrac12(0.18)=0.13\). By the conditional-regret identity, the forecast's population risk exceeds that of the posterior forecast by \(0.13\).

Proper Does Not Always Mean Strict

A score may reward getting one feature of a distribution right while ignoring the rest. Such a score can be proper for the feature it evaluates without identifying the entire probability vector. The distinction matters whenever a forecast is meant to communicate a full distribution.

Worked Example: A Proper but Non-Strict Score

Consider three outcomes and the score \(S(q,y)=(q_1-\mathbf{1}\{y=1\})^2\), which evaluates only the probability assigned to outcome 1. Let \(p=(0.4,0.36,0.24)\). For any report \(q\), the expected score is \(0.4(1-q_1)^2+0.6q_1^2\).

Expanding gives \(0.4-0.8q_1+q_1^2=(q_1-0.4)^2+0.24\). It is minimized when \(q_1=0.4\), so reporting \(p\) is optimal. But \(q=(0.4,0.1,0.5)\) is a different distribution with the same first coordinate. Its expected score is also \(0.24\), exactly the truthful score. Thus the rule is proper but not strictly proper, and its divergence \((q_1-p_1)^2\) can be zero even when \(q\neq p\).

Common Pitfall and Practical Meaning

Confusing realized score with regret. A single outcome's score does not generally show whether the forecast was optimal. Regret compares expected scores under the true distribution.
Assuming every proper score identifies the full distribution. Only strict propriety guarantees positive regret for every distinct report. A coarsened score can ignore distinctions among distributions.
Treating a divergence as a metric. The score-induced quantity measures excess expected loss; symmetry and the triangle inequality are not part of its definition.

The practical value of this framework is that a scoring rule does more than rank forecasts: it specifies what kinds of errors are costly. Brier regret penalizes squared probability differences, whereas log-loss regret is Kullback–Leibler divergence. Conditional on predictors, the same quantity measures the extra risk from a forecast that departs from the posterior probabilities.

Key takeaway: A proper scoring rule makes truthful probabilities minimize expected loss. Its associated divergence is exactly the excess expected score of a report, and conditional on predictors it becomes the conditional regret of departing from the posterior distribution.

Check Your Understanding

Use the definitions and derivations above to answer these questions.

State the difference between a proper and a strictly proper scoring rule.
For a proper score, what does \(D_S(p,q)=0\) tell you, and why does the answer depend on strict propriety?
Explain why the log-loss regret in the worked example is ordered as divergence from \(p\) to \(q\), rather than from \(q\) to \(p\).
If a forecast equals the posterior for every predictor value, what is its conditional regret under a proper score?
How can a score be proper while failing to distinguish two different probability distributions?