Strict Propriety of the Brier Score

Why the Brier Score Rewards Truthful Forecasts

A probability forecast is useful only if its numerical value communicates the forecaster’s actual uncertainty about an event. A scoring rule evaluates a forecast after the outcome is observed. The central question is whether, before observing that outcome, a forecaster minimizes expected loss by reporting the true probability. For the binary Brier score, the answer follows from an exact decomposition: expected loss is the sum of an uncertainty term and a squared error in the reported probability.

This result is a concrete instance of strict propriety for a probability scoring rule. It also clarifies why even a perfectly truthful forecast can have positive expected loss: when the event is uncertain, some loss is unavoidable.

Definition: Let $Y\in\{0,1\}$ indicate whether an event occurs, with $P(Y=1)=p$, where $p\in[0,1]$. A probability forecast is a number $q\in[0,1]$, interpreted as the reported probability that $Y=1$. The binary Brier loss under the convention used here is $$ \ell(q,Y)=(Y-q)^2. $$ Lower loss is better. Some conventions multiply this loss by a positive constant, but this tutorial uses the unscaled squared error.

The loss is evaluated after the outcome is known: it is $(1-q)^2$ if the event occurs and $q^2$ if it does not. To assess a forecast before the outcome is known, average these two possible losses using their probabilities. This expected loss is also called the risk of the fixed forecast $q$ under event probability $p$.

The Expected-Loss Decomposition

For a fixed event probability $p$, write the expected Brier loss as $R_p(q)=E[\ell(q,Y)]$. Because $Y$ takes only the values zero and one, this expectation is a two-term sum. The key algebraic step is to rewrite that sum as a quantity independent of $q$ plus a squared distance from $q$ to $p$.

Theorem: If $Y$ is Bernoulli with success probability $p\in[0,1]$ and $q\in[0,1]$ is a fixed forecast, then $$ R_p(q)=E[(Y-q)^2]=p(1-p)+(q-p)^2. $$ Thus the expected Brier loss consists of the binary outcome’s variance $p(1-p)$ and the squared reporting error $(q-p)^2$.

Proof. The event $Y=1$ has probability $p$, and the event $Y=0$ has probability $1-p$. Therefore

\begin{aligned} R_p(q) &=p(1-q)^2+(1-p)q^2\\ &=p(1-2q+q^2)+(1-p)q^2\\ &=p-2pq+pq^2+q^2-pq^2\\ &=p-2pq+q^2. \end{aligned}

On the other hand,

$$ p(1-p)+(q-p)^2 =p-p^2+q^2-2pq+p^2 =p-2pq+q^2. $$

The two expressions are equal. Since $Y$ is Bernoulli, $\operatorname{Var}(Y)=p(1-p)$, so the first term is the variance of the outcome. The calculation is valid also at $p=0$ and $p=1$; no division by $p$ or $1-p$ is used. $\square$

The decomposition separates two different sources of expected loss. The term $p(1-p)$ describes uncertainty in the outcome itself and does not depend on the forecast. The term $(q-p)^2$ is entirely due to the report’s distance from the true event probability. At $q=p$, this second term vanishes, but the uncertainty term generally remains.

Worked Example: Comparing Two Forecasts at Probability 0.7

Suppose an event has probability $p=0.7$. Compare a truthful forecast $q=0.7$ with a forecast $q=0.4$.

For the truthful forecast, the expected loss is $$ R_{0.7}(0.7)=0.7(1-0.7)+(0.7-0.7)^2=0.21+0=0.21. $$

For the forecast $q=0.4$, the decomposition gives $$ R_{0.7}(0.4)=0.7(0.3)+(0.4-0.7)^2=0.21+0.09=0.30. $$

Directly checking the two possible outcomes gives the same result: $0.7(1-0.4)^2+0.3(0.4)^2=0.7(0.36)+0.3(0.16)=0.252+0.048=0.30$. The excess expected loss of the second report is $0.30-0.21=0.09$, exactly $(0.4-0.7)^2$.

Strict Propriety

A scoring rule is proper when the expected loss is minimized by reporting the true probability. It is strictly proper when that minimizing report is unique. The decomposition proves both properties for the binary Brier score, with strictness coming directly from the squared reporting error.

Definition: A probability scoring rule is proper if, whenever the true event probability is $p$, the expected loss at report $q=p$ is no greater than the expected loss at any other allowed report $q$. It is strictly proper if equality is possible only when $q=p$.

Proof of strict propriety. For any $p,q\in[0,1]$, the expected-loss decomposition gives

R_p(q)-R_p(p) =\bigl[p(1-p)+(q-p)^2\bigr]-\bigl[p(1-p)+(p-p)^2\bigr] =(q-p)^2.

A square is nonnegative, so $R_p(q)\ge R_p(p)$ for every permitted report $q$. Moreover, $(q-p)^2=0$ if and only if $q=p$. Thus $q=p$ is the unique minimizing report, including when $p$ is at either endpoint of the probability interval. The Brier score is strictly proper. $\square$

The equality also gives the exact penalty for misreporting: moving the forecast a distance $d$ from the truth increases expected loss by $d^2$. This is not just a statement that truthful forecasts perform well on average; it quantifies the difference for every pair of probabilities. For example, a forecast error of $0.2$ adds $0.04$ expected loss, regardless of whether the true probability is $0.3$ or $0.8$, provided the forecast remains in $[0,1]$.

Worked Example: A Certain Event and an Endpoint Forecast

Suppose the event is certain, so $p=1$. The truthful report is $q=1$, and its expected loss is $$ R_1(1)=1(1-1)+(1-1)^2=0. $$

Now consider the report $q=0.8$. The decomposition gives $$ R_1(0.8)=1(1-1)+(0.8-1)^2=0+0.04=0.04. $$

Because $Y=1$ with probability one, direct evaluation confirms this: $(1-0.8)^2=0.2^2=0.04$. In general, at $p=1$, $R_1(q)=(1-q)^2$, which is zero only at $q=1$. The boundary case therefore satisfies strict propriety just as interior probabilities do.

Using the Decomposition on a Finite Sample

The same algebra applies to an empirical distribution of binary outcomes. If a data set contains a proportion $\hat p$ of ones and a proportion $1-\hat p$ of zeros, the average Brier loss of a fixed forecast $q$ is $\hat p(1-q)^2+(1-\hat p)q^2$. This is the Brier risk under the empirical distribution, and the decomposition holds with $\hat p$ in place of $p$. It describes the observed average for those outcomes; it does not by itself say that the empirical proportion equals the population probability.

Worked Example: Average Loss for Four Observed Outcomes

Suppose the observed outcomes are $1,0,1,1$. Their empirical event proportion is $\hat p=3/4$. For the report $q=1/2$, each of the four outcomes has squared error $1/4$, so the average loss is $$ \frac{1}{4}\left(\frac14+\frac14+\frac14+\frac14\right)=\frac14=0.25. $$

The empirical decomposition gives the same value: $$ \hat p(1-\hat p)+(q-\hat p)^2 =\frac34\frac14+\left(\frac12-\frac34\right)^2 =\frac{3}{16}+\frac{1}{16} =\frac14. $$

At the empirical proportion $q=3/4$, the average loss is $$ \frac{1}{4}\left[\left(1-\frac34\right)^2+\left(0-\frac34\right)^2+\left(1-\frac34\right)^2+\left(1-\frac34\right)^2\right] =\frac{1}{4}\left(\frac{1}{16}+\frac{9}{16}+\frac{1}{16}+\frac{1}{16}\right) =\frac{3}{16}=0.1875. $$

The difference is $1/4-3/16=1/16$, equal to $(1/2-3/4)^2$. This calculation minimizes average loss for the observed outcomes by matching their empirical proportion.

Interpretation and a Common Pitfall

The uncertainty term $p(1-p)$ is largest at $p=1/2$, where it equals $1/4$, and is zero at $p=0$ and $p=1$. This reflects the variability of the binary outcome, not the quality of the report. In particular, comparing raw expected Brier losses across settings with different event probabilities can mix forecast accuracy with differences in outcome uncertainty. The decomposition makes those contributions visible.

Common pitfall: treating one realized score as evidence that one forecast was better. If an event with probability $0.7$ does not occur, a forecast of $0.4$ receives loss $0.16$, while a forecast of $0.7$ receives loss $0.49$. It would be wrong to conclude from this single outcome that $0.4$ was the better probability forecast. The strict-propriety result compares expected losses before the outcome is known: across the two possible outcomes with probabilities $0.7$ and $0.3$, the truthful forecast has lower expected loss. A single outcome can favor the less accurate report by chance.

The key distinction is between the score realized after one outcome and the expected score under the event’s probability distribution. The Brier score does not guarantee that the truthful forecast wins on every individual event. It guarantees that no different fixed probability report has lower expected loss when the stated probability is the true one.

Key takeaway: For a binary event, the Brier expected loss equals $p(1-p)+(q-p)^2$. The first term is uncertainty that remains even with a truthful forecast; the second is the exact excess loss from reporting $q$ instead of $p$. Because that excess is zero only at $q=p$, the Brier score is strictly proper.

Check Your Understanding

Use the expected-loss decomposition to answer the following questions.

For $p=0.4$ and $q=0.6$, calculate the uncertainty term, the squared reporting error, and the expected Brier loss.
Why does a truthful forecast generally have positive expected Brier loss when $0<p<1$?
State the exact increase in expected loss when the forecast changes from $p$ to $q$.
Does strict propriety still hold when the true probability is $p=0$ or $p=1$? Explain using the decomposition.
Why can a forecast different from $p$ receive a smaller realized loss than the truthful forecast for one observed outcome, even though the Brier score is strictly proper?