The Brier Score for Survival Predictions

Why Score Event-Risk Predictions?

A survival prediction may report a person’s probability of experiencing the event by a specified time. To evaluate such predictions, we need a rule that compares each probability with what happened to that person. The horizon-specific Brier score does this by averaging squared differences between predicted event risks and observed event indicators. It rewards probabilities close to the observed outcomes, while penalizing confident predictions that are wrong.

The score is useful because it responds to more than one aspect of prediction quality. It reflects whether predicted risks are numerically appropriate and whether predictions distinguish people whose outcomes differ. It is not, however, a pure measure of either calibration or discrimination. The horizon, the event rate, and the validation population all affect its value.

Definition and Conditions for the Ordinary Score

Suppose predictions are made at a common landmark time, and let \(\tau\) denote the prediction horizon after that landmark. For a person in the prediction population, let \(Y(\tau)=1\) if the event occurs within the specified horizon and \(Y(\tau)=0\) otherwise. Let \(\hat p(\tau)\) be the predicted probability of that event, conditional on the information available when the prediction is made. In a joint-model setting, that information may include a person’s longitudinal history up to the landmark.

For this ordinary calculation, each person’s event status at the horizon must be known. This holds, for example, when everyone’s event status is fully observed through the horizon or when there is no censoring before it. If someone is censored before the horizon, their event indicator may be unknown; treating that person as a non-event would generally misclassify an unknown outcome. The ordinary formula cannot simply be applied to such records as though their statuses were known. Censoring-adjusted versions address that problem separately.

The same score can be written using predicted survival probabilities. If \(\hat S_i(\tau)=1-\hat p_i(\tau)\) and the observed survival indicator is \(1-Y_i(\tau)\), then \((1-Y_i(\tau))-\hat S_i(\tau)=-(Y_i(\tau)-\hat p_i(\tau))\); squaring gives the same contribution. The choice between event-risk and survival-probability notation does not change the score.

Why the Score Reflects Calibration and Discrimination

Calibration concerns whether the probabilities assigned by a model agree with event frequencies. Discrimination concerns whether predictions tend to be higher for people who experience the event than for people who do not. A squared-error score responds to both: it penalizes probabilities that are too high or too low, and it can reward useful separation between people with different outcomes.

A short population-level decomposition makes the joint sensitivity precise. Let \(X\) denote the information used for prediction, let \(q(X)=P\{Y(\tau)=1\mid X\}\) be the true conditional event risk, and let \(p(X)\) be the model’s predicted risk. Write \(\pi=P\{Y(\tau)=1\}=E\{q(X)\}\). Assume the outcome is binary and observed, and that the expectations refer to the target prediction population. Conditional on \(X\), the outcome has mean \(q(X)\) and variance \(q(X)\{1-q(X)\}\). Thus, $$ E\left[\{Y(\tau)-p(X)\}^{2}\mid X\right] =q(X)\{1-q(X)\}+\{p(X)-q(X)\}^{2}. $$ Taking expectations gives the first identity below. Since \(E\{q(X)^2\}=\pi^2+\operatorname{Var}\{q(X)\}\), rearrangement gives the second: $$ \begin{aligned} \operatorname{BS}(\tau) &=E\left[q(X)\{1-q(X)\}\right]+E\left[\{p(X)-q(X)\}^{2}\right]\\ &=\pi(1-\pi)-\operatorname{Var}\{q(X)\} +E\left[\{p(X)-q(X)\}^{2}\right]. \end{aligned} $$

The first term, \(\pi(1-\pi)\), is the uncertainty associated with the event prevalence at the horizon. The variance term represents the reduction in uncertainty available from knowing \(X\): if people have meaningfully different true risks, the outcome is more predictable from those characteristics. The final term is the average squared difference between the model’s predictions and the true conditional risks. It penalizes probability error. A constant prediction equal to \(\pi\) has no separation based on \(X\), while a useful risk model can reduce expected score by identifying people with different risks.

This decomposition also explains why a model with stronger ranking ability need not have a lower Brier score: improved separation can be offset by inaccurate probability levels. Conversely, a model can be well calibrated on average but offer little separation if it assigns nearly everyone the same risk. The Brier score combines these features in one proper scoring rule rather than reporting them separately.

Interpreting and Comparing Scores

A Brier score has no universal cutoff that labels a prediction model acceptable. Its size depends partly on how common the event is at the chosen horizon. For a population with event prevalence \(\pi\), a model that predicts the same risk \(\pi\) for everyone has expected Brier score \(\pi(1-\pi)\). This constant-risk benchmark is useful context, but it does not replace evaluation against the intended prediction task.

A lower score indicates smaller average squared errors for the evaluated outcomes and predictions. It does not mean that every individual risk is accurate, nor does it identify whether any advantage arose from better calibration, better separation, or both. For a focused diagnosis, examine calibration and discrimination separately. Calibration summaries assess agreement between predicted risks and observed event frequencies; time-dependent AUC measures ranking discrimination at a horizon. Neither is interchangeable with the Brier score.

Comparisons are most interpretable when models are evaluated on the same people, at the same landmark and horizon, for the same event definition, and in the same validation population. Scores from different horizons can differ because both the event prevalence and the prediction task change over time. Scores from populations with different event rates are also not automatically comparable. Report the horizon, the population, and how outcomes were observed along with the score.

Diagnostics, Limitations, and a Common Pitfall

Before interpreting a score, check that each prediction corresponds to the correct person and was formed using only information available at the prediction time. Verify the landmark, horizon, event definition, and coding of the zero-one outcome. Confirm that the predictions are probabilities on the zero-to-one scale and that event status is known at the horizon. If censoring prevents outcome ascertainment, do not silently code censored people as event-free or discard them without considering the consequences. The ordinary score then does not directly evaluate the full target population.

The score is an average, so a small number of confidently incorrect predictions can contribute substantially. It can also be imprecise in small validation samples, and an empirical difference between models may reflect sampling variation. A score alone does not provide an uncertainty interval or a test of whether a difference is meaningful. It is also not a causal measure: it evaluates predictions against observed outcomes, not the effect of changing a predictor.

The Brier score is especially useful when the target is a probability prediction and average squared probability error is relevant. It is less informative on its own when the goal is specifically to understand calibration, ranking, or the source of a performance difference. For survival predictions, censoring creates an additional limitation: a person censored before the horizon may not have a known binary outcome there. The basic score is therefore a starting definition under fully observed horizon outcomes, not a solution to incomplete follow-up.