Proper Scores for Discriminant Probabilities

Why Score the Probabilities?

A classifier may report a probability for every class, then assign a class by choosing the largest probability. That final assignment is useful when the decision is simply whether the predicted class is correct. But it discards information: forecasts of \(0.51\) and \(0.99\) for the same winning class lead to the same assignment, even though they express very different confidence. Proper scores evaluate the probabilities themselves, while zero-one loss evaluates the resulting class assignment.

This distinction matters when a model is intended to communicate uncertainty or when decisions depend on the probabilities, not only on which class is most likely. A proper score is not a replacement for classification risk; it answers a different question. Lower scores are better for both log loss and the Brier score.

Definitions for Multiclass Probability Forecasts

Let the response \(Y\) take one of \(K\) class labels, \(1,\ldots,K\). A probability forecast is a vector \(q=(q_1,\ldots,q_K)\) in the probability simplex: \(q_k\geq 0\) for every \(k\), and \(\sum_{k=1}^K q_k=1\). Write \(e_y\) for the vector that is 1 in position \(y\) and 0 in all other positions. The score is observed after the outcome \(Y=y\), so it can depend on both \(q\) and \(y\).

Log loss penalizes a forecast according to the probability it assigned to the class that occurred; assigning that class a probability near zero incurs a large penalty. The Brier score is the squared Euclidean distance between the forecast vector and the one-hot vector for the observed class. For \(K\) classes, it lies between 0 and 2: the squared distance is nonnegative, and its maximum over the simplex and class outcomes is 2.

For a covariate value \(X=x\), let \(\eta_k=P(Y=k\mid X=x)\), and write \(\eta=(\eta_1,\ldots,\eta_K)\) for the true conditional probability vector. To compare forecasts as probabilistic predictions, consider their expected score given \(X=x\), where the expectation averages over the possible class outcomes with probabilities \(\eta_1,\ldots,\eta_K\).

Strict Propriety of Log Loss

A scoring rule is called proper when the true probability vector minimizes expected score, and strictly proper when it is the unique minimizer. The next result states this property precisely for log loss. It allows true probabilities to be zero; a forecast that assigns zero probability to an outcome with positive true probability has infinite expected log loss.

Proof. If \(q_k=0\) for an index with \(\eta_k>0\), the expected score is infinite, whereas \(q=\eta\) has finite expected score. Such a \(q\) cannot minimize it. Now suppose \(q_k>0\) whenever \(\eta_k>0\). Subtract the expected score at \(\eta\) from that at \(q\). The difference is \(D(\eta\Vert q)=\sum_{k:\eta_k>0}\eta_k\log(\eta_k/q_k)\). For each index with \(\eta_k>0\), apply \(\log t\leq t-1\) to \(t=q_k/\eta_k\). Multiplication by \(-\eta_k\) gives \(\eta_k\log(\eta_k/q_k)\geq \eta_k-q_k\). Summing over the support \(S=\{k:\eta_k>0\}\) gives \(D(\eta\Vert q)\geq\sum_{k\in S}(\eta_k-q_k)=1-\sum_{k\in S}q_k\geq0\). Thus \(q=\eta\) minimizes expected log loss. Equality requires both \(\log t=t-1\) at every index in \(S\), which holds only when \(q_k/\eta_k=1\) there, and \(\sum_{k\in S}q_k=1\). These conditions imply \(q_k=\eta_k\) on \(S\) and \(q_k=0\) off \(S\), so \(q=\eta\). The minimizer is unique. \(\square\)

The excess \(D(\eta\Vert q)\) is the Kullback–Leibler divergence from \(\eta\) to \(q\), with the usual infinite value when \(q\) puts zero mass on a class that has positive \(\eta\). The proof shows why the direction matters: the expected score uses the true probabilities as weights and the forecast probabilities inside the logarithm.

Strict Propriety of the Brier Score

For the Brier score, the expected loss can be expanded directly. Since \(\mathbf{1}\{Y=k\}\) has expectation \(\eta_k\), and \(\sum_k\mathbf{1}\{Y=k\}^2=1\), we obtain \(\begin{aligned} E_{\eta}[S_{\mathrm{Br}}(q,Y)] &=\sum_{k=1}^K E_{\eta}[(q_k-\mathbf{1}\{Y=k\})^2]\\ &=\sum_{k=1}^K(q_k^2-2q_k\eta_k+\eta_k)\\ &=\sum_{k=1}^K(q_k-\eta_k)^2+1-\sum_{k=1}^K\eta_k^2. \end{aligned}\) In the second line, the constants contribute \(\sum_k\eta_k-1=0\); equivalently, the expanded expression is \(\sum_k(q_k^2-2q_k\eta_k+\eta_k)\), which equals the final line because \(\sum_k\eta_k=1\). This identity gives the minimizer and the regret without any approximation.

Proof. For each outcome \(y\), expand the squared differences and then average over \(Y\): \(\begin{aligned} E_{\eta}[S_{\mathrm{Br}}(q,Y)] &=\sum_{k=1}^K\{q_k^2-2q_kE_{\eta}[\mathbf{1}\{Y=k\}]+E_{\eta}[\mathbf{1}\{Y=k\}]\}\\ &=\sum_{k=1}^K(q_k^2-2q_k\eta_k+\eta_k)\\ &=\sum_{k=1}^K(q_k-\eta_k)^2+1-\sum_{k=1}^K\eta_k^2. \end{aligned}\) Setting \(q=\eta\) makes the squared-distance term zero. Subtracting that minimum leaves exactly \(\sum_k(q_k-\eta_k)^2\), which is nonnegative and equals zero only when every coordinate of \(q\) equals the corresponding coordinate of \(\eta\). Hence the Brier score is strictly proper. \(\square\)

Probability Scores and Zero-One Classification Risk

Given \(q\), a usual zero-one classification rule predicts a class in \(\arg\max_k q_k\), with a specified tie-breaking rule if needed. Its zero-one loss is 0 when that predicted class equals \(Y\) and 1 otherwise. Conditional on \(X=x\), the risk of predicting a fixed class \(a\) is \(1-\eta_a\); hence a class with largest \(\eta_k\) minimizes conditional zero-one risk. This criterion depends on the selected class, not on the full vector of forecast probabilities.

For example, two forecasts can have the same maximizing class but quite different log and Brier scores. A score comparison can therefore reveal differences that zero-one risk cannot. Conversely, a probability score is not itself the fraction misclassified: a model with a smaller average log loss need not have a smaller zero-one risk on a particular evaluation sample. The choice of metric should follow the purpose of the evaluation.

Using Proper Scores in Evaluation

For a test sample of \(m\) observations with forecasts \(q_i\) and outcomes \(y_i\), the average log loss is \(m^{-1}\sum_{i=1}^m S_{\log}(q_i,y_i)\); the average Brier score is defined analogously. When comparing probabilistic classifiers, both forecasts should be evaluated on the same observations. The averages summarize realized performance, while strict propriety is a statement about expected score under the data-generating probabilities. In a finite sample, the model with the smaller observed score is not guaranteed to be the model with the smaller population expected score.

A common mistake is to call the classifier with the lowest zero-one error the best probability forecaster. That conclusion does not follow: zero-one error only records whether each assigned class was correct, whereas log loss and the Brier score use the full probability vector. Another mistake is to interpret strict propriety as a guarantee that any finite collection of scores will rank models correctly. Propriety describes the expected score when outcomes follow the stated probabilities; sampling variation remains in observed averages.

These scores are useful because they make probability forecasts evaluable with a clear target: in expectation, reporting the true conditional class probabilities minimizes each score. They do not make the scores interchangeable, nor do they remove the need to state which performance objective matters. Use zero-one risk when the target is the correctness rate of class assignments; use a proper score when the quality of the reported probabilities is part of the target.