Proper Scoring Rules for Probability Forecasts

Why Score Probability Forecasts?

A probability forecast says more than which outcome seems most likely. If a weather service reports a 70% chance of rain, that number makes a claim about the outcome distribution. A useful scoring loss should reward forecasts that represent this distribution accurately, rather than merely reward a particular outcome after it occurs. The key comparison is therefore between reports in expected loss under the distribution that actually generates the outcome.

This idea is captured by propriety. It provides a criterion for evaluating a loss designed for probability forecasts: when the true distribution is $p$, does reporting $p$ minimize expected loss? This tutorial develops the definition and illustrates it for binary outcomes. A probability forecast can be wrong on a particular realization and still be well chosen in this expected-loss sense.

Reports, Outcomes, and Expected Loss

Suppose the possible outcomes are $1,\ldots,K$, where $K$ is a positive integer. A probability distribution over those outcomes is a vector in the probability simplex $\Delta_K=\{p\in\mathbb{R}^K:p_y\geq 0,\ \sum_{y=1}^K p_y=1\}$. A forecaster reports a distribution $q\in\Delta_K$. We use the loss convention: smaller values are better.

Definition: A (finite-valued) scoring loss is a function $S(q,y)\in\mathbb{R}$ that assigns a loss to report $q\in\Delta_K$ when outcome $y\in\{1,\ldots,K\}$ occurs. If the true outcome distribution is $p$, the expected loss of report $q$ is $$ L(p,q)=\sum_{y=1}^K p_y S(q,y). $$ The scoring loss is proper if, for every $p\in\Delta_K$, $L(p,p)\leq L(p,q)$ for every $q\in\Delta_K$. It is strictly proper if $L(p,p)<L(p,q)$ for every $p\in\Delta_K$ and every $q\in\Delta_K$ with $q\ne p$.

The true distribution $p$ appears as the first argument of $L$, because it determines how likely each outcome is. The report $q$ appears as the second argument, because it is the choice being evaluated. Propriety is a comparison of expected losses under one fixed $p$: it does not require the truthful report to have the lowest loss for every possible realized outcome.

If $K\geq 2$, the simplex contains distinct reports. Strict propriety then requires every report different from $p$ to have strictly greater expected loss. For $K=1$, the simplex has only one report, so the condition involving $q\ne p$ is vacuous. Under the definition above, any scoring loss on that one-point simplex is strictly proper, including a loss that does not depend on the report.

Binary Probability Forecasts

For a binary outcome, write $Y\in\{0,1\}$. Any distribution for $Y$ is described by a single probability $p=P(Y=1)$, and any report is described by $q\in[0,1]$, the reported probability of outcome 1. The probability of outcome 0 is $1-p$ under the truth and $1-q$ under the report. If the losses for outcomes 1 and 0 are $S(q,1)$ and $S(q,0)$, respectively, then

$$ L(p,q)=pS(q,1)+(1-p)S(q,0). $$

This expression makes the central test explicit: hold the true probability $p$ fixed, calculate the expected loss as a function of the report $q$, and check whether a minimum occurs at $q=p$. The following examples show why the choice of scoring loss matters.

Worked Example: The Binary Brier Loss

For a binary outcome, define the Brier loss by $S(q,y)=(q-y)^2$. Suppose the true probability of outcome 1 is $p=0.7$, and the forecast reports $q=0.4$. If outcome 1 occurs, the loss is $(0.4-1)^2=0.36$. If outcome 0 occurs, the loss is $(0.4-0)^2=0.16$. Thus the expected loss of this report is

$$ L(0.7,0.4)=0.7(0.36)+0.3(0.16)=0.252+0.048=0.300. $$

For a general true probability $p$, the expected loss at report $q$ is

\begin{aligned} L(p,q) &=p(1-q)^2+(1-p)q^2\\ &=p-2pq+pq^2+q^2-pq^2\\ &=p-2pq+q^2\\ &=p(1-p)+(q-p)^2. \end{aligned}

At the truthful report $q=p=0.7$, this is $0.7(0.3)+0^2=0.21$, which is less than $0.300$. The difference is $0.300-0.210=0.090=(0.4-0.7)^2$. Since $(q-p)^2\geq 0$ for every report, this calculation verifies that the binary Brier loss is proper. It also shows that, for this loss, any different report has larger expected loss.

Worked Example: An Absolute-Error Loss That Is Not Proper

Consider instead $S(q,y)=|q-y|$, the absolute error between the reported probability and the binary outcome. Let the true probability be $p=0.7$. For any report $q\in[0,1]$, its expected loss is

\begin{aligned} L(0.7,q) &=0.7|q-1|+0.3|q-0|\\ &=0.7(1-q)+0.3q\\ &=0.7-0.4q. \end{aligned}

The second line uses $q\leq 1$ and $q\geq 0$. This expected loss decreases as $q$ increases, so it is minimized at $q=1$, not at the truthful report $q=0.7$. In particular, $L(0.7,1)=0.3$, while $L(0.7,0.7)=0.7(0.3)+0.3(0.7)=0.42$. The loss is not proper: its expected value encourages this forecaster to report 1 even though the true probability is 0.7.

Two Useful Consequences of the Definition

The defining inequality can be read directly as a nonnegative expected penalty for departing from the truth. For a proper loss, define the expected-loss difference between report $q$ and truthful report $p$ by

L(p,q)-L(p,p) =\sum_{y=1}^K p_y\bigl(S(q,y)-S(p,y)\bigr).

Propriety says this difference is nonnegative. Strict propriety says it is positive whenever the reports differ. This difference is an expected quantity: an individual term $S(q,y)-S(p,y)$ can be negative for a particular outcome even when the weighted sum is nonnegative.

A simple special case helps distinguish weak from strict propriety. Suppose the loss depends on the outcome but not on the report: $S(q,y)=c_y$ for fixed finite numbers $c_1,\ldots,c_K$. Then every report has the same expected loss. For $K\geq2$, this loss is proper but not strictly proper, because a distinct report ties the truthful one. For $K=1$, it is also strictly proper according to the definition: there is no distinct report that could violate the strict inequality.

Worked Example: A Report-Independent Loss

Let the binary losses be $S(q,1)=2$ and $S(q,0)=5$ for every $q\in[0,1]$. If the true probability is $p=0.3$, then every report has expected loss

$$ L(0.3,q)=0.3(2)+0.7(5)=0.6+3.5=4.1. $$

In particular, $q=0.3$ and $q=0.8$ both have expected loss 4.1. The truthful report is a minimizer, so the loss is proper, but the different report $q=0.8$ ties it, so the loss is not strictly proper. This example shows that propriety alone does not guarantee that the expected loss can distinguish truthful reports from untruthful ones.

The binary case also illustrates why a loss for probability forecasts should be judged through its expected value, not by whether it seems intuitive for one outcome. When outcome 1 occurs, a report near 1 can receive a smaller loss than a report near 0; that single comparison does not determine whether the loss is proper. Propriety averages the two outcome-specific losses with their probabilities under the true distribution.

Common Mistake: Judging a Forecast From One Realization

A common mistake is to say, “The forecast assigned high probability to the outcome that occurred, so the scoring loss must be proper.” This reasoning looks only at the observed outcome and does not compare expected losses under the true distribution. A loss can reward high reports when outcome 1 occurs and still favor reports that systematically exaggerate or understate probabilities.

The correction is to write $L(p,q)$ using the true outcome probabilities, then compare it with $L(p,p)$ for all allowable reports. For the absolute-error example, outcome 1 receives less loss from a high report, but outcome 0 receives less loss from a low report. Weighting both possibilities by $p=0.7$ and $1-p=0.3$ gives $L(0.7,q)=0.7-0.4q$, which is minimized by $q=1$. The expectation, rather than a single realized score, reveals the failure of propriety.

Key takeaway: A scoring loss is proper precisely when truthful reporting minimizes expected loss under every true outcome distribution. Strict propriety additionally rules out ties with different reports, except that this requirement is vacuous when there is only one possible report.

Check Your Understanding

Use the definition of expected loss to answer each question.

For a binary outcome with true probability $p$, write the expected loss of a report $q$ in terms of $S(q,1)$ and $S(q,0)$.
For the binary Brier loss, what is $L(p,q)-L(p,p)$? What does its sign imply about propriety?
For the absolute-error loss and true probability $p=0.3$, find the report that minimizes expected loss.
Why can a loss be proper without being strictly proper? Give the report-independent example’s reasoning in terms of expected loss.
Under the stated definition, why is strict propriety vacuous when $K=1$?