Brier-Score Aggregation Of Probability Predictors

Why the Brier Score Has Useful Risk Geometry

A probability forecast is judged not only by whether its most likely outcome is correct, but also by whether its probabilities are appropriate. The Brier score measures the squared distance between a forecast probability vector and the indicator vector of the outcome that occurs. Its squared-error form gives probability forecasts a particularly transparent risk geometry: expected score is minimized by the true conditional probabilities, and averaging forecasts has an exact diversity decomposition.

This tutorial uses the sum-of-squares convention for the multiclass Brier score, with no division by the number of classes. Multiplying every score by the same positive constant would change numerical values but not which forecast has lower risk.

A predictor uses available information \(X\) to produce its forecast. For multiclass outcomes, write \(p(X)\in\Delta_K\). Its Brier risk is \(R(p)=E[S(p(X),Y)]\), provided the expectation exists. It always exists here because the forecast and outcome vectors are bounded. Define the conditional class-probability vector by \(\eta(X)=E[e_Y\mid X]\); its \(k\)th component is \(P(Y=k\mid X)\).

A candidate dictionary consists of a finite collection of measurable probability predictors \(p_1,\ldots,p_M\). Given fixed weights \(\alpha_1,\ldots,\alpha_M\) satisfying \(\alpha_j\geq0\) and \(\sum_{j=1}^M\alpha_j=1\), their aggregate is \(\bar p(X)=\sum_{j=1}^M\alpha_jp_j(X)\). Because the simplex is convex, \(\bar p(X)\) is itself a valid probability forecast.

Conditional Risk and the True Probability Vector

The first identity explains what Brier risk measures. Conditional on the predictors \(X\), the expected score is the squared distance from the forecast to the true conditional probabilities, plus a term determined by the outcome distribution that no forecast can remove.

Consequently, \(R(p)\) is minimized by \(p(X)=\eta(X)\) almost surely, and for any two probability predictors \(p\) and \(q\),

To prove the conditional identity, expand the squared distance after writing \(p-e_Y=(p-\eta)+(\eta-e_Y)\). The cross term has conditional expectation zero because \(E[e_Y-\eta\mid X]=0\). Also, since \(e_Y\) has one component equal to one and the rest zero, \(\|e_Y\|_2^2=1\). Therefore

Taking expectations gives the risk decomposition. The second term does not depend on \(p\), so the conditional probabilities minimize risk. Subtracting the decomposition for \(q\) from the one for \(p\) gives the stated risk difference. \(\square\)

For binary outcomes, let \(\eta(X)=P(Y=1\mid X)\). The same result becomes \(E[(p(X)-Y)^2\mid X]=(p(X)-\eta(X))^2+\eta(X)(1-\eta(X))\). The final term is conditional outcome uncertainty; the first is the squared probability error. Thus a lower Brier risk corresponds exactly to a lower expected squared distance from the true conditional probability, after averaging over \(X\).

The Exact Gain from Averaging

For squared probability loss, the benefit of averaging can be quantified exactly. At each value of \(X\), the aggregate’s score is the weighted average of the candidates’ scores minus their weighted dispersion around the aggregate. This identity holds for every realized outcome, not just after taking expectations.

Taking expectations yields the Brier-risk identity

For the proof, write \(p_j-e_Y=(p_j-\bar p)+(\bar p-e_Y)\), square, and sum using the weights. The cross term vanishes because

The remaining terms are \(\sum_j\alpha_j\|p_j-\bar p\|_2^2\) and \(\|\bar p-e_Y\|_2^2\sum_j\alpha_j=\|\bar p-e_Y\|_2^2\), proving the samplewise identity. Taking expectations proves the risk identity. \(\square\)

The final expectation is the weighted diversity of the candidates. It is nonnegative, so the aggregate risk cannot exceed the weighted average candidate risk. It is strictly smaller exactly when candidates with positive weight disagree on a set of \(X\)-values having positive probability. If all positive-weight forecasts agree almost surely, the diversity term is zero and averaging gives no risk reduction.

Conditions, Interpretation, and a Common Pitfall

The identities require valid probability forecasts and weights that are nonnegative and sum to one. The risk conclusions concern predictions evaluated on the same population distribution. In practice, weights may be chosen using a training sample; the fixed-weight result applies to an independent test pair after conditioning on that training sample, provided the test pair follows the target distribution. If weights are instead chosen using the outcome being scored, the fixed-weight population-risk interpretation does not automatically apply.

The central pitfall is to replace “no worse than the weighted average risk” with “no worse than the best candidate risk.” The diversity identity gives

It does not say that \(R(\bar p)\leq\min_jR(p_j)\). The correction is to identify the benchmark before interpreting the inequality: averaging improves on the weighted mean risk by exactly \(D\), while beating a particular candidate requires the additional condition \(\sum_j\alpha_jR(p_j)-D\leq R(p_k)\). Probability forecasts can also be diverse while the average remains farther from the truth than the best forecast, as the binary example demonstrated.

This geometry is useful when probability forecasts are combined: it separates candidate accuracy from candidate disagreement, and shows how disagreement contributes to the exact reduction relative to the weighted risk benchmark. The same calculation works for each observed case in a data set, so it applies to empirical Brier scores as well as population risks.