Decomposing The Brier Score For Competing Transitions

Why Decompose a Competing-Destination Score?

When a prediction concerns one of several competing destinations, an overall score can summarize how close the forecast was to the observed outcome. But the total alone can hide where the error occurred. A forecast may assign too little probability to the destination that occurred, too much to one rival, or spread probability across several destinations. Decomposing the score makes those contributions visible without treating the destination predictions as unrelated outcomes.

The setup is a fixed prediction origin and horizon. For a person in a specified state at the origin, the outcome by the horizon is one category from a mutually exclusive, exhaustive set. Depending on the prediction target, categories might be the possible outgoing transitions, with an additional category such as “no transition by the horizon” when needed. The categories must match the target used to generate the predictions.

For \(n\) validation subjects with fully observed outcomes, the average score is \(\overline{BS}=n^{-1}\sum_{i=1}^n BS_i\). Smaller values indicate closer probability forecasts for this target and horizon. Under this convention the individual score ranges from 0 to 2: zero is attained by assigning probability one to the observed category, while the maximum of 2 occurs when probability one is assigned to a different category. Some sources rescale the score, for example by dividing by two. Always check the convention before comparing reported values.

Destination Contributions and Their Dependence

Define the contribution for destination \(k\) on subject \(i\) as \(BS_{ik}=(p_{ik}-Y_{ik})^2\). Then \(BS_i=\sum_k BS_{ik}\), and the average contribution for destination \(k\) is \(\overline{BS}_k=n^{-1}\sum_i BS_{ik}\). Consequently,

Each contribution is exactly the binary Brier loss for the indicator of that destination. Thus, summing the destination-specific binary losses gives the multicategory score. This is a descriptive decomposition of squared error, not a claim that the \(K\) forecasts are statistically independent.

In fact, the probability constraint links the errors. Write \(e_{ik}=p_{ik}-Y_{ik}\). Because both the predicted probabilities and the outcome indicators sum to one, \(\sum_k e_{ik}=0\). If category \(r\) occurred, then \(e_{ir}=p_{ir}-1\), while for every other category \(k\), \(e_{ik}=p_{ik}\). The total is therefore

The first term measures the squared shortfall in the probability assigned to the realized category. The remaining terms measure probability assigned to the non-realized alternatives. They are distinct pieces of the total, but cannot be varied freely: increasing probability for one category requires reducing probability elsewhere. A destination-specific contribution should therefore be interpreted alongside the other contributions, not as if it were an isolated forecast.

A Useful Expected-Score Derivation

The decomposition also clarifies why the score rewards well-targeted probabilities. Suppose the outcome for a particular prediction setting has true category probabilities \(q_1,\ldots,q_K\), with \(\sum_k q_k=1\), and the forecast is the fixed vector \(p=(p_1,\ldots,p_K)\). For each category, \(Y_k\) is an indicator with \(E(Y_k)=q_k\) and \(Y_k^2=Y_k\). Expanding the score and taking expectation gives

The last line follows by expanding \(\sum_k(p_k-q_k)^2\) and adding \(1-\sum_kq_k^2\). The second term is the outcome uncertainty for this categorical distribution; it does not depend on the forecast. The first term is the squared distance between the forecast probabilities and the true probabilities. Thus, in expectation, the score is minimized when \(p_k=q_k\) for every category. This is the strictly proper scoring property of the multicategory Brier score.

Worked Example: Summing Contributions Across Destinations

Consider four hypothetical subjects with three possible destinations, \(A\), \(B\), and \(C\). The table shows each forecast, the observed destination, and the destination-specific contributions. Each forecast sums to one, and each outcome is exactly one category.

For example, subject 3 had outcome \(A\), so the three contributions are \((0.50-1)^2=0.25\), \((0.30-0)^2=0.09\), and \((0.20-0)^2=0.04\). Their sum is \(0.38\). Across all subjects, the destination contributions are

Their sum is \(0.0975+0.0825+0.0450=0.2250\). Calculating from the subject totals gives the same result: \((0.14+0.24+0.38+0.14)/4=0.2250\). In this small hypothetical validation set, the average squared error contribution is largest for destination \(A\), but that does not establish that its predictions are worse in a broader population. The empirical average depends on which outcomes and forecasts happened to occur in the validation sample.

Using the Decomposition Responsibly

The destination contributions help identify how an aggregate score is built, but they do not replace checks on the prediction target or the validation data. Before computing or interpreting the score, verify the following:

• Categories define one outcome. At the chosen origin and horizon, each subject must have exactly one observed category. If “no transition by the horizon” is part of the target, include it as a category rather than silently omitting it.
• Forecasts use the same categories. The ordering and meaning of predicted probabilities must match the outcome indicators. Check that probabilities lie between zero and one and sum to one, allowing only negligible numerical rounding differences.
• All destinations are included. Scoring just one destination produces a binary Brier score for that indicator, not the full multicategory score. If only some competing outcomes are included, the result no longer describes the complete categorical forecast.
• Outcomes are ascertained for the calculation. The simple empirical average here assumes the category is observed for every included subject by the horizon. Right-censored outcomes require additional methods; treating a censored subject as “no transition” can give a false score. The IPCW Brier Score addresses censoring separately.

A useful computational diagnostic is to calculate the subject totals in two ways: sum contributions across categories for each subject and average those totals; then average each category’s contributions and sum across categories. The results must agree apart from rounding. A discrepancy often indicates a coding error, a missing category, or inconsistent handling of subjects. Also inspect the contribution pattern: very large losses can arise from highly confident forecasts assigned to the wrong destination, while small losses can reflect probabilities reasonably close to the observed one-hot outcome. Neither pattern alone diagnoses calibration across the population.

Compared with reporting one binary Brier score for each destination, the multicategory total evaluates the whole coherent probability vector in one number. Reporting the class-wise contributions alongside it preserves information about where squared error accumulates. Compared with a classification error rate, the Brier score uses the full probabilities rather than only the most likely category: two forecasts that choose the same destination can receive different scores if one is much more confident. Conversely, the score is not a direct measure of discrimination or a calibration curve. It combines aspects of forecast accuracy, and its value depends on the outcome mix and the target horizon.

A common mistake is to add destination probabilities as if they were separate risks, or to interpret a small contribution for one destination as evidence that the entire forecast is good. The probabilities are constrained to sum to one, and the total includes every destination. Another mistake is to call the class-wise terms independent Brier scores. Each term is indeed a binary squared-error contribution, but all terms for a subject share the same categorical outcome and constrained forecast vector. Report the total for the complete prediction task, and use the decomposition to explain its components rather than to make unsupported class-by-class claims.