Multicategory Brier Scores for Dynamic State Predictions

Why score dynamic state predictions?

A model can produce a probability for every possible state at a future time, but a prediction needs to be evaluated against what happens at that time. The multicategory Brier score does this by comparing the full predicted probability vector with the state that is actually observed. It evaluates the predictions at a specified horizon; it is not a summary of prediction performance over all times.

For a fair comparison, the models should be evaluated for the same target population, landmark time, prediction horizon, and state definitions. The predictions must use only information available at the landmark. This is especially important when the prediction is updated at successive landmarks: a score at one landmark concerns that landmark’s eligible population, not automatically the population used at another.

Definition and assumptions

Here \(I(\cdot)\) equals 1 when its condition is true and 0 otherwise. A valid prediction has \(\hat p_{ik}(s,h)\geq 0\) for every state and \(\sum_{k=1}^{K}\hat p_{ik}(s,h)=1\). The observed outcome is represented by a one-hot vector: exactly one state indicator is 1. The score is dimensionless, and lower values indicate smaller average squared probability errors. This tutorial uses the unnormalized sum over states shown above; dividing every score by 2 is another convention, but it changes the scale and must be applied consistently.

Before calculating the score, the prediction and outcome must refer to the same state system and horizon. The target individuals should be eligible at the stated landmark, and each prediction should be based only on information available by that landmark. For the simple average above, every horizon state must be observed. When follow-up ends before the horizon, the missing outcome cannot be treated as an observed state; censoring requires explicit handling.

Why the score rewards accurate probabilities

The score uses the whole probability vector, not just the state assigned the largest probability. To see why this is useful, suppose the true state probabilities for an individual with given prediction-time information are \(q_1,\ldots,q_K\), while the model reports \(p_1,\ldots,p_K\). The state indicators are random, with probability \(q_k\) of being 1 for category \(k\). The expected Brier loss, averaging over the possible horizon states, is

If the prediction equals the true probability vector, \(p_k=q_k\), its expected loss is \(1-\sum_k q_k^2\). Subtracting that value from the expected loss for any other probability vector gives

Thus the expected score is minimized by reporting the true state probabilities. This is the proper-scoring property of the Brier score. For a single observation, the loss can range from 0 to 2: it is 0 for a probability of 1 on the observed state, and approaches 2 when probability 1 is assigned to a different state. A dataset average need not reach either extreme.

Calculating a score and comparing models

For complete horizon outcomes, calculate each individual’s loss against the observed state and then average those losses. The model comparison is made on the resulting average scores, with the same individuals and outcome definition used for each model. The difference is a descriptive comparison on that evaluation set; it does not by itself establish that one model will perform better in every population or at every horizon.

The Brier score reflects several aspects of probabilistic prediction together. It rewards assigning high probability to the observed state and penalizes misplaced probability across all states. A lower score is useful for comparing overall probabilistic accuracy, but it does not identify which aspect of performance produced the difference. In particular, it is not a direct measure of calibration alone, nor is it the proportion of individuals whose most likely predicted state was correct.

When the horizon state is censored

If a participant’s follow-up ends before \(s+h\), the state at the horizon may be unknown. Averaging only the losses for participants whose outcomes happen to be observed can give a biased score: those participants may differ systematically from those lost to follow-up. One approach is inverse-probability-of-censoring weighting (IPCW).

This form estimates the average loss in the original landmark target set, rather than only among those observed at the horizon. It requires that censoring be independent of the horizon outcome, conditional on the information used to estimate \(G_i(h)\), and that \(\hat G_i(h)>0\) for target individuals. The latter is a positivity condition: relevant individuals must have a nonzero chance of remaining observable through the horizon. The censoring model should use suitable prediction-time information and be estimated appropriately for the evaluation design.

The construction is justified by inverse weighting. If observation is independent of the horizon loss conditional on the censoring-model information, then the conditional expected value of \(R_i/G_i\) is \(G_i/G_i=1\). Consequently, weighting the observed loss by \(1/G_i\) recovers its expected contribution in the target set. If conditional independence fails, or observation probabilities are misspecified, the weighted score can still be biased.

Diagnostics, alternatives, and limitations

Before interpreting a score, check the prediction and evaluation process, not just the arithmetic. Useful checks include:

• Verify that every probability vector is nonnegative and sums to 1, allowing only minor rounding discrepancies.
• Confirm that the state labels and horizon are identical across the models being compared, and that predictions were formed without future information.
• Tabulate how many horizon outcomes are observed. With IPCW, inspect the estimated observation probabilities and weights; very small \(\hat G_i(h)\) creates large contributions and high variability.
• Check that the censoring assumptions are plausible for the information used in the censoring model. A weighted score does not automatically correct informative censoring.
• Keep the evaluation target fixed. A score difference across distinct landmark populations or horizons can reflect a change in the people or prediction task, not just a change in model quality.

For the IPCW formula above, weights are not normalized by their sum: the denominator is the original target-set size \(n\). Some analyses use a normalized, weighted average instead. That alternative can be useful descriptively, but it is a different finite-sample estimator and should be identified rather than silently substituted. In either case, extreme weights deserve attention; any weight truncation changes the estimator and should be reported.

Other measures answer different questions. Classification accuracy uses only the most likely predicted state and discards the rest of the probability vector. The logarithmic score is another proper score but penalizes assigning very small probability to the observed state particularly strongly. Calibration assessments examine agreement between predicted probabilities and observed frequencies. These approaches complement rather than replace a Brier score: none should be interpreted as covering every aspect of model performance.

Finally, a score is an average over a particular evaluation set. It may conceal poor performance in a subgroup or at a different horizon. A lower observed score is not, by itself, evidence of a statistically meaningful difference; quantifying uncertainty in a model comparison requires an appropriate inferential procedure. When observations are dependent, the uncertainty procedure must also respect that dependence.