Calibration of Conditional Survival Predictions

What Calibration Asks

A conditional survival prediction is made for a subject who has remained event-free to a specified prediction time. The model uses information available at that time, such as the subject’s longitudinal marker history, to predict the chance of an event over a future horizon. Calibration asks whether those predicted probabilities match event frequencies in the relevant validation population.

The distinction between a prediction and the probability it aims to predict is essential. A model can assign a risk of 0.20 to a subject even when that subject’s true conditional event probability is 0.35. Calibration is a property of the model’s predictions across subjects, not a definition that makes each prediction correct by construction.

The model’s prediction \(\hat q_i(s,h)\) may be obtained from its conditional survival prediction as one minus the predicted probability of surviving from \(s\) through \(s+h\), given survival to \(s\) and the available history. The event probability and prediction refer to the same event, landmark, horizon, eligibility condition, and target population. If any of these differ, the comparison does not assess the stated calibration target.

For example, a conditional risk of 0.20 means a 20% predicted probability of the target event during the next \(h\) time units, among subjects still event-free at \(s\) with the relevant histories. It does not mean that every subject assigned 0.20 has an individual outcome that can be checked against a probability. Calibration compares probabilities with event frequencies across subjects.

Why the Definition Uses a Conditional Mean

Suppose the model’s prediction is determined by information available at \(s\). For subjects at risk at \(s\), the event indicator has conditional expectation \(p_i(s,h)\) given that history. Averaging over subjects assigned the same predicted value gives the central calibration relationship.

This follows by conditioning first on \(H_i(s)\): because \(D_i(s,h)\) is an indicator of the event in the stated interval, its conditional expectation given survival to \(s\) and the history is \(p_i(s,h)\). Averaging that conditional expectation among subjects with \(\hat q_i(s,h)=q\) yields the displayed equality. Thus calibration requires the average true risk among subjects assigned \(q\) to equal \(q\). It does not require every individual’s true risk to equal the model’s prediction.

In a validation sample, the conditional expectation is not directly observed. It is estimated from outcomes among subjects with similar predictions, while accounting for incomplete follow-up when necessary. With continuously varying predictions, a finite sample will rarely contain many subjects with exactly the same value. Any practical comparison therefore uses a specified estimation approach and has sampling uncertainty.

Calibration Is Not Discrimination

Discrimination concerns whether predictions rank subjects appropriately: subjects who experience the event sooner or within a horizon should tend to receive higher predicted risks than subjects who do not. Calibration concerns whether the numerical probabilities are accurate on average. A model can rank subjects well but systematically overpredict risk; it can also predict the overall event frequency accurately while assigning risks that do not distinguish subjects well.

Assessing Calibration With Censored Follow-Up

At a fixed horizon, some validation subjects may be censored before their event status for that horizon is known. A subject censored before \(s+h\) without a prior event is not known to have remained event-free through the horizon. Counting such a subject as a non-event generally biases the observed event frequency downward. Simply discarding all subjects without complete follow-up can also bias the comparison if censoring is related to event risk.

A censoring-adjusted comparison can use inverse probability of censoring weighting (IPCW). In broad terms, an observed event is weighted by the inverse probability of remaining uncensored up to its event time, conditional on the variables in the censoring model. This compensates for similar events that may have been hidden by earlier censoring. The adjustment requires appropriate assumptions: conditional independence of censoring and the event process given the variables used, a correctly specified censoring distribution (or a sufficiently flexible estimate), and positivity, meaning that the probability of remaining uncensored is not zero over the times and covariate patterns being evaluated.

The censoring model and the prediction model have different roles. The prediction model supplies \(\hat q_i(s,h)\); the censoring model supports estimation of observed risk when outcomes are incompletely observed. IPCW does not repair a misspecified prediction model, and it cannot identify horizon-specific event probabilities in regions where essentially no one remains observable.

Practical Checks, Alternatives, and Limitations

Before interpreting a calibration comparison, verify that the predictions and validation outcomes refer to the same target. In particular, check that the prediction time, horizon, event definition, eligibility rule, and conditioning information agree. Confirm that subjects classified as at risk at \(s\) were actually event-free then, and that outcome follow-up is sufficient or censoring is handled under defensible assumptions.

A calibration assessment is also limited by sample size and the distribution of predictions. Few subjects at a particular risk level mean that the conditional event frequency is estimated imprecisely. Predictions outside the range well represented in the validation cohort cannot be assessed reliably from that cohort. In addition, calibration in one validation population does not guarantee calibration in a population with a different case mix, event incidence, follow-up process, or clinical setting.

A common practical summary is to compare observed and predicted risks within prediction groups. This can be useful descriptively, but it only assesses agreement at the chosen grouping resolution; it may hide differences within groups, as the second worked example demonstrates. A calibration plot provides a broader visual comparison across risk values, while later summaries such as calibration-in-the-large and a calibration slope focus on particular systematic patterns. No one summary replaces the need to define the prediction target and assess uncertainty.

A frequent mistake is to define the model prediction itself as \(P(s<T_i\le s+h\mid T_i>s,H_i(s))\), then conclude that calibration must hold. That expression is the true conditional event probability, not an arbitrary model’s prediction. The correction is to keep the model output \(\hat q_i(s,h)\) separate from the true probability \(p_i(s,h)\), and assess whether the mean outcome among subjects assigned \(\hat q_i(s,h)=q\) equals \(q\).