What a Transition-Specific Brier Score Measures
A transition prediction is a probability, but its validation outcome for one person is binary: the specified transition either occurs within the stated horizon or it does not. The Brier score measures the average squared difference between these predicted probabilities and their observed binary outcomes. It gives a single summary of prediction error for one transition target, at one prediction origin and one horizon.
The target must be defined before calculating the score. Here, consider people who are in origin state \(r\) at prediction time \(s\), and a particular allowed outgoing edge \(r\to k\). Let \(h>0\) be the prediction horizon. The event is that the person’s first transition out of \(r\) occurs by time \(s+h\) and is to state \(k\). A competing outgoing transition, or remaining in \(r\) through the horizon, is a non-event for this edge. This definition treats the edge as a first-exit target; a question about later transitions after leaving \(r\) requires a different outcome definition.
The target is not simply “is the person in state \(k\) at time \(s+h\)?” A person may enter \(k\) and then leave it before the horizon, or may arrive in \(k\) through another path. The binary outcome above records a specified first transition out of \(r\). Keeping the event definition, prediction origin, and horizon aligned is essential.
Calculation and Interpretation
Each case contributes a squared error. If the transition occurs, the contribution is \((1-\hat p_i)^2\); if it does not, the contribution is \(\hat p_i^2\). The score is the arithmetic mean of these contributions. It is nonnegative, has no units, and ranges from 0 to 1. A score of 0 means every predicted probability exactly matched its binary outcome; a score nearer 0 indicates smaller average squared prediction error on the cases scored.
The score is not the proportion of people classified incorrectly. It evaluates the probability values themselves: predicting 0.90 for a non-event is penalized more than predicting 0.55 for a non-event. Because the squared differences are averaged, a few predictions that are very far from their outcomes can noticeably increase the score.
Worked Example: Scoring Five Edge-Specific Predictions
Suppose five hypothetical people are all in state \(r\) at the prediction origin. The target is their first exit from \(r\) to \(k\) within 12 months. Follow-up determines the outcome for everyone through 12 months. Their predictions and outcomes are:
| Person | \(\hat p_i\) | \(Y_i\) | \((Y_i-\hat p_i)^2\) |
|---|---|---|---|
| 1 | 0.20 | 0 | 0.0400 |
| 2 | 0.40 | 1 | 0.3600 |
| 3 | 0.60 | 1 | 0.1600 |
| 4 | 0.70 | 0 | 0.4900 |
| 5 | 0.10 | 0 | 0.0100 |
For example, person 2 has an event, so the contribution is \((1-0.40)^2=0.36\). Person 4 does not have the target event, so the contribution is \((0-0.70)^2=0.49\). Thus
The Brier score is 0.212 for this hypothetical set of five 12-month predictions. This is an average squared error, not a statement that 21.2% of predictions were wrong. Five cases are also too few to support a stable general conclusion about how well the model predicts in a broader population.
Outcome Coding With Competing Exits
For a specified edge, an exit to a different destination is a non-event, even though it is a real transition. That outcome definition follows from the question being scored: did the specified edge occur as the first exit within the horizon? Treating a competing exit as an event for every outgoing edge would give one person multiple positive outcomes when only one first exit occurred. Treating it as censoring would also misrepresent this binary target when the competing exit is observed.
Worked Example: A Competing Transition Is a Non-Event
Consider four hypothetical people with a prediction origin in state \(r\), a 6-month horizon, and the target edge \(r\to k\). Their first exits are fully observed through month 6. The predictions are 0.25, 0.50, 0.75, and 0.40. Person 1 exits to a different state \(m\) at month 2; person 2 exits to \(k\) at month 5; person 3 has no exit by month 6; and person 4 exits to \(k\) at month 1.
For the edge \(r\to k\), the outcomes are \(0,1,0,1\), respectively. In particular, person 1’s exit to \(m\) is a non-event for the target edge, as is person 3’s lack of an exit by the horizon. The squared errors are
Therefore,
This score concerns only the 6-month probability of the specified \(r\to k\) edge. The competing transition is not an unobserved outcome: it establishes that the target edge was not the first exit within the horizon.
Why Squared Probability Error Is Useful
The Brier score is a proper scoring rule: in expectation, a person or model achieves the smallest score by reporting the true event probability. This gives a justification for scoring the numerical probabilities rather than only their classifications.
To see this, suppose that for a given information set the true probability of the binary event is \(\pi\), and a prediction is \(a\), where \(0\leq a\leq1\). The binary outcome \(Y\) equals 1 with probability \(\pi\) and 0 with probability \(1-\pi\). The expected squared error is
The first term, \(\pi(1-\pi)\), does not depend on the prediction. The second term is nonnegative and equals zero exactly when \(a=\pi\). Thus, if the true conditional probability is \(\pi\), the expected Brier loss is minimized by predicting \(a=\pi\). A single observed outcome can still make a well-calibrated probability look wrong; the result concerns expected loss over repeated comparable cases, not whether each individual prediction matches its outcome.
Worked Example: Comparing Two Prediction Sets
Suppose two hypothetical models predict the same 4-month \(r\to k\) event for the same four fully followed validation cases. The observed outcomes are \(1,0,0,1\). Model A predicts \(0.80,0.30,0.40,0.60\); Model B predicts \(0.60,0.20,0.20,0.80\). For Model A, the four contributions are
So Model A’s score is \((0.0400+0.0900+0.1600+0.1600)/4=0.1125\). For Model B, the contributions are \(0.1600, 0.0400, 0.0400,\) and \(0.0400\), giving a score of \(0.2800/4=0.0700\). Model B has the lower Brier score on these four cases, so its predicted probabilities have smaller average squared error here. This comparison does not prove that Model B will score better in a new sample; the validation set is small.
Assumptions, Diagnostics, and Alternatives
The arithmetic is simple, but a meaningful score depends on a well-defined and observable validation target. Check the following before interpreting a result:
- Same origin and horizon. Each probability must refer to the same prediction origin and horizon as the outcome. A probability for 12 months cannot be scored against a 6-month outcome.
- Same edge and event definition. Confirm that \(Y_i=1\) denotes the specified first exit \(r\to k\), not occupancy of \(k\) at the horizon or an event along a later path.
- Valid probabilities. Predictions must be between 0 and 1. Check for coding errors, missing predictions, or probabilities that were calculated using information unavailable at the prediction origin.
- Known outcomes. Each included case needs a determined event status through the horizon. If observation ends before the horizon without establishing whether the edge occurred, the outcome is unknown, not automatically 0.
- Relevant validation cases. The scored cases should represent the population and prediction setting to which the reported performance is meant to apply. For repeated observations per person or other dependence, uncertainty calculations should account for that structure.
A useful diagnostic is to inspect the prediction and outcome records alongside the score. Count target events and competing exits, and check that each competing exit is coded as 0 for this edge. A large score may reflect overconfident predictions, systematic prediction errors, a small or unusual validation sample, or incorrect outcome coding. The score alone cannot identify which explanation applies. Calibration summaries, including calibration-in-the-large and calibration curves, can help examine agreement between predicted and observed probabilities, but they answer a different question from the overall squared-error summary.
Accuracy after choosing a probability cutoff answers how often a chosen classification is correct; it discards how close the probabilities are to the outcomes and depends on the cutoff. The Brier score retains the probability scale. A log score is another proper probability score and penalizes confident errors especially strongly. Calibration summaries focus on agreement between predictions and observed event rates, while the Brier score combines aspects of probability error into one average. These summaries are complementary, not interchangeable.
A standard unweighted sample Brier score also requires observed outcomes for the cases it scores. Simply dropping people whose follow-up ends before the horizon can change which cases are represented and may distort the intended validation summary. Do not silently code such cases as non-events or treat a competing transition as ordinary censoring. Methods for handling right censoring require additional assumptions and weighting; they are not part of this unweighted calculation.
Check Your Understanding
Answer each question, then check the brief solutions.
- For a target edge \(r\to k\), a person exits \(r\) to \(m\) before the horizon. What is the binary outcome for this target edge, and why?
- A person has \(Y=1\) and predicted probability \(\hat p=0.30\). Calculate the squared-error contribution.
- Three fully observed cases have outcomes \(0,1,0\) and predictions \(0.20,0.60,0.50\). Calculate their Brier score.
- Why is a person whose follow-up stops before the horizon without a known transition not automatically coded \(Y=0\)?
- In one sentence, distinguish what the Brier score summarizes from what classification accuracy summarizes.
Solutions. 1. The outcome is 0 because the first exit was to a competing destination, not along \(r\to k\). 2. \((1-0.30)^2=0.49\). 3. The contributions are \(0.20^2=0.04\), \((1-0.60)^2=0.16\), and \(0.50^2=0.25\), so the score is \((0.04+0.16+0.25)/3=0.15\). 4. The target event status through the horizon is unknown; lack of observed follow-up is not evidence that the edge did not occur. 5. The Brier score averages squared errors in predicted probabilities, whereas accuracy counts correct classifications after applying a chosen cutoff.