Logarithmic Scores for Event-Risk Predictions

Why Score Probabilistic Event-Risk Predictions?

A survival model may assign each person a probability of experiencing the event by a chosen horizon. Evaluating those probabilities requires more than checking whether the model ranks people correctly: a prediction of 0.8 and a prediction of 0.55 are different statements about risk, even if both rank the person above someone else. A logarithmic score evaluates the probabilities themselves and rewards predictions that assign high probability to what actually happens.

Censoring complicates this evaluation. At a horizon of two years, for example, a person censored after one year has unknown two-year event status. Treating that person as event-free would be incorrect. The censoring-aware score uses inverse-probability weights to account for the fact that some outcomes are observed only when censoring has not already occurred.

An event by the horizon incurs loss \(-\log p_i(t)\); remaining event-free through the horizon incurs loss \(-\log\{1-p_i(t)\}\). Thus, the score is sensitive both to whether predictions are accurate and to how much probability they assigned to the outcome that occurred. It is a score for probabilistic accuracy, not a measure of discrimination alone.

Correcting the Score for Right Censoring

Suppose \(T_i\) is the event time and \(C_i\) is the censoring time. We observe \(U_i=\min(T_i,C_i)\) and \(\Delta_i=I(T_i\leq C_i)\). An event by \(t\) is known if \(\Delta_i=1\) and \(U_i\leq t\). Event-free status through \(t\) is known if \(U_i>t\). Someone censored at or before \(t\) without an observed event has unknown status at the horizon and does not receive a direct loss contribution.

Let \(Z_i\) denote the information used to account for censoring, including relevant predictors available at prediction time. Write \(G(s\mid Z_i)=P(C_i>s\mid Z_i)\). For simplicity, assume censoring times are continuous; in settings with ties, the survival-function limits and event-time convention must be defined consistently. The estimated censoring-adjusted mean logarithmic loss is

The first term scores observed events by the horizon, weighting each by the inverse estimated probability of remaining uncensored until the event time. The second scores people observed event-free beyond the horizon, weighting each by the inverse probability of remaining uncensored through \(t\). A censored person with no observed event by \(t\) contributes zero directly. The divisor is \(n\), not the sum of the observed weights: this inverse-probability form targets average loss per person in the evaluation population.

Why the Weighting Works

The score is based on the target loss that would be available if event status at \(t\) were known for everyone. The weights correct the selection created when only people who remain uncensored long enough have known status. Here is the justification, conditional on \(Z_i\), under the assumptions above.

This argument explains both the method and its vulnerability. If censoring is informative even after conditioning on \(Z_i\), the observation probabilities used in the weights do not correctly describe which outcomes are observed. If the probabilities are near zero, a few observed people receive very large weights and can dominate the estimate.

Why Confident Errors Are Penalized Strongly

For a known binary outcome with true event probability \(q\), the expected logarithmic loss from predicting \(p\) is

Differentiating with respect to \(p\) gives

For \(0<q<1\), this derivative is zero at \(p=q\), negative when \(p<q\), and positive when \(p>q\). The second derivative is \(q/p^2+(1-q)/(1-p)^2>0\), so the expected loss is minimized at the true probability. This establishes the strict propriety of the logarithmic score: in expectation, truthful probabilities give the lowest loss.

Interpretation, Diagnostics, and Alternatives

A lower censoring-adjusted logarithmic score indicates better average probabilistic accuracy at the specified horizon, for the population and prediction setting being evaluated. Scores can be compared only when they use the same horizon, outcome definition, evaluation population, censoring adjustment, and logarithm convention. Unlike a risk ratio or a calibration slope, the score is not itself an effect size with a natural clinical unit. It also does not reveal whether any improvement came from calibration, discrimination, or both.

The Integrated Brier Score in Joint Models is a useful alternative overall prediction metric. Its squared-error penalty is less sensitive than logarithmic loss to assigning an extremely low probability to an event that occurs, or an extremely high probability to an event-free outcome. The logarithmic score’s stronger penalty is valuable when overconfident errors matter, but it can make results sensitive to a small number of extreme predictions or large censoring weights.

Before interpreting a score, check the censoring model and the weights as well as the predictions. Useful diagnostics include examining the range and distribution of \(\widehat G\), identifying very large inverse weights, and checking whether censoring is plausibly independent of event time after conditioning on the information used. A lack of positivity—such as no one remaining under observation to a late horizon in a predictor subgroup—means the target loss there is not reliably recoverable from these data. A misspecified censoring model can bias the score even when the event-risk model is accurate.

A further pitfall is to assign censored people the event-free loss simply because no event was observed before censoring. For someone censored before \(t\), the outcome by \(t\) is unknown; using \(-\log\{1-p_i(t)\}\) treats missing status as a known negative outcome. The correction is to omit that direct contribution and use the censoring-adjusted terms for observed cases and observed event-free people.