Brier Score and Quadratic Probability Regret

Why Score a Probability Forecast?

A classifier may be asked not only to name the most likely class, but also to report how likely each class is. These are different tasks: the forecast \((0.51,0.49)\) expresses substantial uncertainty, while \((0.99,0.01)\) expresses near certainty, even though both choose the first class. A useful probability score should therefore evaluate the whole forecast, not merely whether its most likely class was correct.

The Brier score does this by comparing the forecast with a one-hot encoding of the observed class. Its conditional risk has a particularly transparent form: the true probability vector minimizes that risk, and the penalty for any other forecast is exactly its squared Euclidean distance from the truth.

Definition of the Multiclass Brier Score

Suppose the response \(Y\) takes one of \(K\) classes, indexed by \(1,\ldots,K\). A probability forecast is a vector \(q=(q_1,\ldots,q_K)\) in the probability simplex \(\Delta_K=\{q\in\mathbb{R}^K:q_k\geq 0,\ \sum_{k=1}^K q_k=1\}\). For an observed class \(y\), represent the outcome by the vector whose \(y\)-th entry is one and whose other entries are zero.

Expanding the score for a particular outcome gives \(S(q,y)=(1-q_y)^2+\sum_{k\ne y}q_k^2=1-2q_y+\sum_{k=1}^Kq_k^2\). Thus the score depends on the probability assigned to the class that occurred and on the squared size of the whole forecast vector.

Conditional Risk and Its Minimizer

Let \(X\) denote the information available when a forecast is made. For a fixed value \(X=x\), write \(p_k=P(Y=k\mid X=x)\) and \(p=(p_1,\ldots,p_K)\). The vector \(p\) is the true conditional class-probability vector. For a proposed forecast \(q\), its conditional Brier risk is the expected score over the possible outcomes, holding \(X=x\) fixed.

Proof. By the definition of conditional risk and the expanded score,

The second line uses \(\sum_y p_y=1\); the final line follows by expanding the squared differences. The first two terms in the last line do not depend on \(q\), and the remaining term is nonnegative. It equals zero exactly when \(q_k=p_k\) for every \(k\). Since \(p\) is itself in \(\Delta_K\), it is an available forecast and is the unique minimizer. Subtracting its risk, \(1-\sum_kp_k^2\), leaves precisely \(\sum_k(q_k-p_k)^2\), as claimed. \(\square\)

From Conditional Regret to Population Excess Risk

In a prediction problem, the forecast can depend on \(X\). Let \(p_k(X)=P(Y=k\mid X)\) be versions of the conditional probabilities, and let \(q(X)\in\Delta_K\) be a measurable forecast rule. Conditional on \(X\), the theorem applies with \(p=p(X)\) and \(q=q(X)\). Integrating that identity gives an exact population comparison.

Derivation. The conditional identity holds almost surely in \(X\). Taking expectations on both sides and using iterated expectation yields \(\mathbb{E}[S(q(X),Y)]=\mathbb{E}[1-\sum_kp_k(X)^2]+\mathbb{E}[\sum_k(q_k(X)-p_k(X))^2]\). Setting \(q(X)=p(X)\) shows that the first expectation on the right is \(\mathbb{E}[S(p(X),Y)]\), which gives the displayed result. All terms are integrable: for any probability vectors, the Brier score lies between 0 and 2, and the squared distance is bounded. The squared-distance expectation is nonnegative, so the population risk cannot be lower than that of \(p(X)\). It is zero exactly when the nonnegative squared distance is zero almost surely, which means \(q(X)=p(X)\) almost surely. \(\square\)

What the Distance Formula Tells Us

The regret is measured in the ordinary squared Euclidean distance between probability vectors. For example, shifting probability mass between two classes changes two coordinates, and both squared changes contribute. The formula is exact, not an approximation: a forecast twice as far from \(p\) in Euclidean distance has four times the conditional regret.

The result also clarifies what is being optimized. If the goal is to forecast probabilities, the minimizer is the full vector \(p\), not a vector that puts all probability on the most likely class. Choosing a single class is a different decision problem. For example, with \(p=(0.6,0.4)\), the probability forecast \(p\) has minimum conditional Brier risk \(1-(0.6^2+0.4^2)=0.48\). The one-hot forecast \((1,0)\) has regret \((1-0.6)^2+(0-0.4)^2=0.32\), and hence risk \(0.80\). It chooses the most likely class, but it is not the best probability forecast.

Common Mistake

A common mistake is to replace the probability vector by its most likely class before evaluating the Brier score. That discards the uncertainty information the score is designed to assess. Another is to compare only the probability assigned to the class that occurred. The Brier score includes squared errors for every class, including classes that did not occur. For the outcome in the first worked example, omitting the non-occurring classes would miss the \(0.6^2\) and \(0.1^2\) contributions and would not give the defined score.

The correction is to retain the full forecast vector and compare it coordinate by coordinate with the one-hot outcome. For conditional risk, average that full score over the true class probabilities. The resulting squared-distance identity then gives both the minimizer and the size of the penalty for a misspecified forecast.