Brier Scores for Multiclass Probability Estimation

Why Score Probability Estimates?

A classifier that reports only a class label hides how certain it is. A probability estimate gives more information: for example, it can distinguish a prediction of \((0.51,0.29,0.20)\) from one of \((0.95,0.03,0.02)\), even if both select the same most likely class. A scoring rule evaluates such probability forecasts against the observed label. The multiclass Brier score is one standard choice, with a particularly simple conditional risk.

Throughout, there are \(K\) possible classes. A probability vector belongs to the simplex \(\Delta_K=\{q\in\mathbb{R}^K:q_k\geq 0,\ \sum_{k=1}^K q_k=1\}\). Write \(Y\in\{1,\ldots,K\}\) for the observed class, \(p_k=P(Y=k)\) for its true probability, and \(q_k\) for the probability forecast. The calculations are conditional on the information used to make the forecast, so \(p\) and \(q\) are fixed vectors in \(\Delta_K\).

Definition: The multiclass Brier score of a forecast \(q\in\Delta_K\) when the observed class is \(Y=y\) is \(S_B(q,y)=\sum_{k=1}^K(q_k-\mathbf{1}\{y=k\})^2\). Equivalently, if \(e_y\) is the one-hot vector with a 1 in coordinate \(y\) and 0 elsewhere, then \(S_B(q,y)=\|q-e_y\|_2^2\). Smaller scores are better. This tutorial uses the unscaled sum; a convention that multiplies it by a positive constant changes the numerical scale but not its minimizing forecast.

The score compares every reported class probability with the corresponding entry in the one-hot encoding of the outcome. It is not just the squared error for the probability assigned to the class that occurred: the other class probabilities also contribute.

Worked Example: Computing a Score for One Outcome

Suppose there are three classes, the forecast is \(q=(0.50,0.30,0.20)\), and class 2 occurs. The one-hot vector is \(e_2=(0,1,0)\). Thus

$$ S_B(q,2)=(0.50-0)^2+(0.30-1)^2+(0.20-0)^2 =0.25+0.49+0.04=0.78. $$

The result includes the squared errors for classes 1 and 3 as well as class 2. The score would be zero only if the forecast assigned probability 1 to the class that occurred.

Conditional Risk and Strict Propriety

A single score depends on the realized class and can vary from one observation to another. To compare forecasts for a given true distribution \(p\), take its expectation over the possible outcomes. This expected score is the conditional risk.

Definition: Given a true class-probability vector \(p\in\Delta_K\), the conditional Brier risk of forecasting \(q\in\Delta_K\) is \(R_B(p,q)=E[S_B(q,Y)]=\sum_{y=1}^K p_y S_B(q,y)\). A scoring rule is proper if \(q=p\) minimizes this risk over \(q\in\Delta_K\), and strictly proper if this minimizer is unique.

Theorem: For every \(p,q\in\Delta_K\), the multiclass Brier risk satisfies \(R_B(p,q)=1-\sum_{k=1}^Kp_k^2+\sum_{k=1}^K(q_k-p_k)^2\). Consequently, the Brier score is strictly proper: its unique risk-minimizing forecast is \(q=p\).

Proof. Expand the squared terms in the definition and interchange the finite sums:

\begin{aligned} R_B(p,q) &=\sum_{y=1}^K p_y\sum_{k=1}^K(q_k-\mathbf{1}\{y=k\})^2\\ &=\sum_{k=1}^K\left(q_k^2-2q_k\sum_{y=1}^Kp_y\mathbf{1}\{y=k\} +\sum_{y=1}^Kp_y\mathbf{1}\{y=k\}\right)\\ &=\sum_{k=1}^K(q_k^2-2q_kp_k+p_k)\\ &=\sum_{k=1}^K(q_k-p_k)^2+1-\sum_{k=1}^Kp_k^2. \end{aligned}

In the last equality, \(\sum_k p_k=1\) was used. The term \(1-\sum_kp_k^2\) does not depend on \(q\), while \(\sum_k(q_k-p_k)^2\geq 0\). This squared distance is zero exactly when every \(q_k=p_k\). Since \(p\) itself is in \(\Delta_K\), it is an available forecast, and it is the unique minimizer. \(\square\)

The decomposition separates the risk into an irreducible term determined by uncertainty in the outcome and an excess term measuring the squared Euclidean distance from the true probabilities. Even a perfectly reported probability vector can therefore have positive expected score when the outcome is uncertain.

Worked Example: Comparing Two Forecasts

Let the true probabilities be \(p=(0.50,0.30,0.20)\), and forecast \(q=(0.40,0.40,0.20)\). First compute the risk at the correct forecast:

$$ R_B(p,p)=1-(0.50^2+0.30^2+0.20^2) =1-(0.25+0.09+0.04)=0.62. $$

The squared distance between the forecast and truth is \((0.40-0.50)^2+(0.40-0.30)^2+(0.20-0.20)^2=0.01+0.01+0=0.02\). Therefore

$$ R_B(p,q)=0.62+0.02=0.64. $$

This can also be checked by averaging the outcome-specific scores. If class 1 or 2 occurs, the score is \(0.56\); if class 3 occurs, it is \(0.96\). The expected score is \(0.50(0.56)+0.30(0.56)+0.20(0.96)=0.64\), as above.

Comparison with Logarithmic Loss

The logarithmic loss for forecast \(q\) and observed class \(y\) is \(-\log q_y\). Its conditional risk is the expected negative log probability assigned to the outcome. We use the convention that \(0\log 0=0\) in entropy expressions; if \(p_y>0\) but \(q_y=0\), the log loss for that possible outcome, and hence the conditional risk, is infinite.

Definition: The logarithmic loss is \(S_{\log}(q,y)=-\log q_y\). Its conditional risk is \(R_{\log}(p,q)=-\sum_{k=1}^Kp_k\log q_k\), with the zero-probability convention above. The entropy of \(p\) is \(H(p)=-\sum_{k=1}^Kp_k\log p_k\), and the Kullback–Leibler divergence is \(D_{\mathrm{KL}}(p\|q)=\sum_{k:p_k>0}p_k\log(p_k/q_k)\), taken to be infinite if \(q_k=0\) for any \(k\) with \(p_k>0\).

Theorem: For any \(p,q\in\Delta_K\), \(R_{\log}(p,q)=H(p)+D_{\mathrm{KL}}(p\|q)\). The logarithmic loss is strictly proper: its unique risk-minimizing forecast is \(q=p\).

Proof. If \(q_k=0\) for some \(k\) with \(p_k>0\), the risk and the divergence are both infinite, so the identity holds in the extended-real sense. Otherwise, all terms for which \(p_k>0\) have \(q_k>0\), and separating the logarithm gives

-\sum_{k:p_k>0}p_k\log q_k =-\sum_{k:p_k>0}p_k\log p_k +\sum_{k:p_k>0}p_k\log\frac{p_k}{q_k} =H(p)+D_{\mathrm{KL}}(p\|q).

It remains to check that the divergence is nonnegative and equals zero only at \(p=q\). For \(p_k>0\), the inequality \(\log t\leq t-1\) for \(t>0\) implies

D_{\mathrm{KL}}(p\|q) =-\sum_{k:p_k>0}p_k\log\frac{q_k}{p_k} \geq \sum_{k:p_k>0}(p_k-q_k) =1-\sum_{k:p_k>0}q_k \geq 0.

Equality requires both equality in \(\log t\leq t-1\), which holds only at \(t=1\), for every \(k\) with \(p_k>0\), and no probability mass of \(q\) outside that support. These conditions give \(q_k=p_k\) for all \(k\). Thus \(D_{\mathrm{KL}}(p\|q)\) is minimized uniquely at \(q=p\), proving strict propriety. \(\square\)

The two conditional-risk decompositions have parallel forms but different excess-risk quantities: squared Euclidean distance for the Brier score and Kullback–Leibler divergence for logarithmic loss. Their baseline risks also differ: \(1-\sum_kp_k^2\) for Brier and \(H(p)\) for log loss. These are not interchangeable numerical scales.

Worked Example: The Correct Forecast Still Has Positive Risk

Let \(p=q=(0.70,0.20,0.10)\). Because the forecast is correct, the squared-distance term in Brier risk is zero:

$$ R_B(p,p)=1-(0.70^2+0.20^2+0.10^2) =1-(0.49+0.04+0.01)=0.46. $$

The logarithmic risk is the entropy of \(p\):

R_{\log}(p,p) =-0.70\log(0.70)-0.20\log(0.20)-0.10\log(0.10) \approx 0.249672+0.321888+0.230259=0.801819.

Neither risk is zero because the realized class is not known in advance. The values \(0.46\) and approximately \(0.801819\) describe the baseline risks under different scoring rules; their numerical sizes do not show that one forecast is better than the other, since here the forecasts are identical and correct.

Worked Example: A Zero Probability for a Possible Class

Suppose \(p=(0.80,0.20,0)\), but the forecast is \(q=(1,0,0)\). For the Brier score, the squared-distance term is \(0.20^2+(-0.20)^2+0^2=0.08\), and the baseline is \(1-(0.80^2+0.20^2)=1-(0.64+0.04)=0.32\). Hence \(R_B(p,q)=0.32+0.08=0.40\). Directly, class 1 gives score 0 and class 2 gives score 2, so the expected score is \(0.80(0)+0.20(2)=0.40\).

For logarithmic loss, the forecast assigns probability zero to class 2 even though \(p_2=0.20\). That outcome has positive probability and incurs infinite loss, so \(R_{\log}(p,q)=\infty\). This illustrates a key difference: Brier risk remains finite for every forecast in the simplex, while logarithmic risk can be infinite.

What the Comparison Tells Us

Both scores reward reporting the true class probabilities in the precise sense of strict propriety. Their different risk decompositions nevertheless matter in practice. The Brier excess risk is a squared distance, and the individual score is bounded between 0 and 2 for probability forecasts in \(\Delta_K\). Logarithmic loss is sensitive to probabilities near zero and has no finite upper bound: assigning a very small probability to an event that occurs can incur a very large penalty.

Neither score is itself a class decision rule. If a downstream task requires one class, a separate decision rule is needed; the probability score instead assesses the quality of the reported distribution. Similarly, strict propriety is a population statement: it identifies the risk-minimizing probability vector when the true conditional distribution is known. It does not say that a finite-sample fitted forecast will equal that distribution.

Common Mistake

A frequent error is to say that the Brier risk is simply the squared error in the probability assigned to the observed class. That omits the other \(K-1\) coordinates. For example, with \(q=(0.50,0.30,0.20)\) and outcome \(y=2\), using only \((q_2-1)^2=0.49\) misses \(q_1^2+q_3^2=0.29\), giving the incorrect score \(0.49\) rather than \(0.78\). The correction is to compare the whole forecast vector with the one-hot outcome vector.

A related mistake is to compare the numerical Brier and log risks as if they were measured on the same scale. Their baselines and excess-risk terms are different. Compare forecasts under a specified score, and interpret each risk using its own decomposition.

Key takeaway: The multiclass Brier risk equals a true-distribution baseline plus squared distance from the truth, so it is uniquely minimized by the true probability vector. Logarithmic risk has the same unique minimizer but adds Kullback–Leibler divergence instead; its severe penalty for assigning zero probability to a possible class distinguishes it from Brier risk.

Check Your Understanding

Use the definitions and risk decompositions to answer these questions.

For a three-class outcome \(y=1\) and forecast \(q=(0.60,0.25,0.15)\), compute the multiclass Brier score.
For \(p=(0.50,0.30,0.20)\), what is the minimum possible Brier risk, and which forecast attains it?
State the excess-risk term for Brier risk and the corresponding excess-risk term for logarithmic risk.
Why can logarithmic risk be infinite when Brier risk is finite?
Why does a correct probability forecast generally have positive expected score when the outcome remains uncertain?