Why Absolute-Risk Calibration Matters
A survival model can rank patients correctly without giving accurate probabilities. For example, it may assign higher predicted risk to patients who experience an event earlier, yet systematically predict a 30% risk for a group whose event risk is closer to 15%. Ranking performance and the accuracy of predicted probabilities are different properties. Calibration assesses the latter.
Fix a prediction time origin and a horizon \(t\). For patient \(i\), let \(T_i\) be the event time and let \(\hat{p}_i(t)\) be the model’s predicted probability of experiencing the event by \(t\). In a survival model that predicts survival \(\hat{S}_i(t)\), the corresponding event-risk prediction is \(\hat{p}_i(t)=1-\hat{S}_i(t)\). The observed event indicator at the horizon is \(I(T_i\leq t)\), where \(I(\cdot)\) equals 1 when its condition holds and 0 otherwise.
With complete follow-up to \(t\), a calibration plot compares predicted risks with observed event proportions across groups of similar predictions. Its reference line is the 45-degree line: predicted risk equals observed risk. With right-censored data, however, a person censored before \(t\) has an unknown event status at \(t\). Counting such a person as event-free would bias the observed risk downward. The observed-risk estimate and the prediction-error measure must account for this incomplete follow-up.
Calibration Plots With Censored Follow-Up
One practical plot divides patients into groups based on their predicted risks. For each group, the horizontal coordinate is the group’s mean predicted risk at \(t\), and the vertical coordinate is an estimate of the group’s event risk by \(t\). If censoring is independent of event time within each group, the Kaplan–Meier estimate of survival in that group gives an observed-risk estimate of \(1-\hat{S}_{KM}(t)\). This condition can fail if people with different predicted risks or other relevant covariates have different censoring patterns within a broad group.
When censoring depends on measured variables, one option is to estimate observed group risks using inverse-probability-of-censoring weights. For each patient whose event status at \(t\) is observed, weight an observed event by the inverse estimated probability of remaining uncensored until just before that event time. Weight an observed event-free status at \(t\) by the inverse probability of remaining uncensored through \(t\). A ratio of the weighted number of events to the total weight of observed statuses estimates the event risk in the group. Patients censored before \(t\) do not supply an observed status at \(t\).
Worked Example: Reading a Group-Based Calibration Point
Suppose a hypothetical validation set contains 12 patients in a predicted-risk group, with mean predicted event risk of 0.20 by two years. Their follow-up yields an event at 0.5 years, a censoring time at 0.8 years, and events at 1.2 and 1.8 years; the other patients are followed beyond two years without an event. Assume censoring is independent of event time within this group.
The Kaplan–Meier survival estimate at two years multiplies the conditional survival proportions at the three event times. The risk sets are 12 at 0.5 years, 10 at 1.2 years after one event and one censoring, and 9 at 1.8 years.
Thus the estimated observed event risk is \(1-0.7333=0.2667\), or about 26.7%. The calibration point is approximately (0.20, 0.267). It lies above the equality line, suggesting that this group’s predictions are lower than its estimated event risk. With only 12 patients, this estimate is uncertain; the point alone does not establish systematic miscalibration.
Grouped plots are easy to interpret, but grouping discards detail and the result can depend on the number and boundaries of the groups. A smooth calibration curve can show departures within the range of predictions, but it too has estimation uncertainty and can be unstable when data are sparse. Confidence intervals or bands should reflect estimation of observed risks; resampling the validation data is one way to assess uncertainty, provided the resampling scheme respects the study design.
Time-Dependent Brier Scores
Calibration focuses on whether predicted probabilities match observed frequencies. The Brier score summarizes squared prediction error at a particular horizon, combining probability accuracy across patients. At time \(t\), the complete-data score for patient \(i\) is \(\{I(T_i\leq t)-\hat{p}_i(t)\}^2\). The population Brier score is its expectation, and the sample score is the mean across patients. Smaller values indicate better probabilistic predictions in the evaluated population and at that horizon.
For censored data, define \(Y_i=\min(T_i,C_i)\) and \(\Delta_i=I(T_i\leq C_i)\), where \(C_i\) is the censoring time and \(\Delta_i=1\) indicates an observed event. Let \(Z_i\) denote variables used to model censoring. The censoring survival probability is \(G(u\mid Z_i)=P(C_i>u\mid Z_i)\); thus \(G(u-\mid Z_i)=P(C_i\geq u\mid Z_i)\) for event-time weighting, including ties. An inverse-probability-of-censoring-weighted estimate is:
An observed event by \(t\) contributes the squared error \(\{1-\hat{p}_i(t)\}^2\), upweighted for the chance that the event would have been observed rather than censored first. A patient known to be event-free past \(t\) contributes \(\hat{p}_i(t)^2\), upweighted for the chance of remaining uncensored through \(t\). A patient censored at or before \(t\), without an observed event, contributes zero because their status at \(t\) is unknown; the inverse weights of the other observed patients correct for this missing status under the assumptions below. The sum is divided by the full sample size \(n\), not by the number with observed status at \(t\).
Assumptions and Justification of the Weighting
The censoring-adjusted score requires more than a censoring model that ignores relevant predictors. In particular, the covariates used for censoring adjustment must account for the dependence of censoring on the prediction-dependent loss. A clear sufficient setup is that the prediction \(\hat{p}_i(t)\) is a function of variables included in \(Z_i\), and that event time and censoring time are conditionally independent given \(Z_i\). If other measured variables affect censoring, they should also be included when needed to make this conditional independence credible.
The reason for conditioning on prediction-related variables can be seen from the event contribution. Given \(T_i\) and \(Z_i\), conditional independence gives \(P(C_i\geq T_i\mid T_i,Z_i)=G(T_i-\mid Z_i)\). Therefore, the probability of observing an event at \(T_i\) cancels its inverse weight in expectation. Similarly, for a patient with \(T_i>t\), the probability of being observed event-free through \(t\) is \(G(t\mid Z_i)\), which cancels the corresponding weight.
The same argument for the event-free term gives \(E[I(T_i>t,\ C_i>t)\hat{p}_i(t)^2/G(t\mid Z_i)\mid T_i,Z_i]=I(T_i>t)\hat{p}_i(t)^2\). Adding the two terms recovers the complete-data squared error in expectation. This argument depends on using the censoring probability conditional on variables that make the censoring assumption valid and account for prediction-related differences. A marginal censoring curve is not generally adequate when censoring and predicted risks vary together.
Worked Example: Calculating an IPCW Brier Score
Consider four hypothetical validation patients at horizon \(t=2\) years. Their predictions and observed follow-up are: patient 1 has \(\hat{p}=0.20\), an event at 0.5 years, and \(\hat{G}(0.5-\mid Z)=0.80\); patient 2 has \(\hat{p}=0.60\), is event-free beyond two years, and \(\hat{G}(2\mid Z)=0.80\); patient 3 has \(\hat{p}=0.40\), is censored at one year without an event; patient 4 has \(\hat{p}=0.70\), an event at 1.5 years, and \(\hat{G}(1.5-\mid Z)=0.50\). Assume the stated censoring probabilities are appropriate conditional on \(Z\), which includes the variables determining these predictions.
Patient 1 contributes \((1-0.20)^2/0.80=0.64/0.80=0.80\). Patient 2 contributes \(0.60^2/0.80=0.36/0.80=0.45\). Patient 3 contributes zero because their event status at two years is unknown. Patient 4 contributes \((1-0.70)^2/0.50=0.09/0.50=0.18\). Hence
This is a hypothetical estimate of the two-year Brier score, not a percentage of patients misclassified. With only four patients it is extremely imprecise; the example illustrates the arithmetic, not the sample size needed for reliable evaluation.
Worked Example: Comparing Brier Scores for Two Risk Models
Suppose four hypothetical patients have fully observed two-year outcomes \(1,0,1,0\), where 1 means an event by two years. Model A predicts risks \(0.80,0.20,0.70,0.30\), and Model B predicts \(0.60,0.40,0.60,0.40\). There is no censoring before the horizon, so the ordinary mean squared errors apply.
For Model A, the squared errors are \((1-0.80)^2=0.04\), \((0-0.20)^2=0.04\), \((1-0.70)^2=0.09\), and \((0-0.30)^2=0.09\). Its Brier score is \((0.04+0.04+0.09+0.09)/4=0.065\).
For Model B, each prediction is 0.20 away from its observed outcome, giving squared error \(0.20^2=0.04\) for all four patients. Its Brier score is \((0.04+0.04+0.04+0.04)/4=0.04\). Model B has the lower score in this small example, so its predicted probabilities are closer to the observed binary outcomes under squared-error loss. This comparison does not by itself establish that Model A is better in another population or at another horizon.
Interpretation, Diagnostics, and Alternatives
A Brier score is a proper scoring rule: in expectation, predicting the true event probability minimizes expected squared error. It reflects both calibration and the ability to distinguish individuals with different outcomes, so a lower score does not identify which property caused the improvement. Scores also depend on the event frequency and time horizon; compare models on the same validation patients, at the same horizon, using the same censoring adjustment. A time-integrated Brier score averages or integrates the time-specific score over a prespecified interval and therefore summarizes performance over that interval rather than at one time point.
Calibration plots and Brier scores answer complementary questions. A plot helps locate overprediction or underprediction across the risk range. The Brier score gives a single measure of overall probability error. The time-dependent ROC curve and concordance summarize discrimination or ranking, not whether a predicted risk of, for example, 20% corresponds to an observed risk near 20%. Strong discrimination cannot substitute for calibration.
Useful diagnostics include inspecting predicted-risk distributions, comparing censoring patterns across risk ranges, checking that estimated censoring probabilities do not approach zero, and examining the stability of weighted scores. Very small \(\hat{G}\) values create large weights and can make estimates dominated by a few observations. Calibration estimates should also be checked for sparse groups and dependence on grouping choices. If censoring depends on measured covariates, a censoring model that omits those covariates can leave the observed patients unrepresentative even when event and censoring times are independent conditional on the omitted model’s variables.
No single evaluation measure resolves every concern. Group-based calibration is interpretable but coarse; smooth calibration estimates can be variable; IPCW scores depend on censoring assumptions and estimation; and discrimination measures do not assess absolute-risk accuracy. Predictions should be evaluated on data not used to fit or tune the model, at clinically meaningful horizons, with uncertainty and limitations reported. If the censoring model is poorly supported or positivity is weak, the resulting calibration and Brier estimates may not be trustworthy.
Check Your Understanding
Answer each question, then compare with the solutions below.
- A model predicts a 10% event risk by three years for a group whose estimated observed risk is 18%. Is the group overpredicted or underpredicted?
- At a fixed horizon, a patient has predicted risk 0.30 and an observed event by that time. What is the unweighted squared error?
- Why does a patient censored before the horizon contribute no directly observed binary outcome to the IPCW Brier-score sum?
- What should be done if a variable affects both censoring and the predicted risk?
Solutions. (1) The model underpredicts risk in that group: observed risk exceeds mean predicted risk by about 8 percentage points. (2) The squared error is \((1-0.30)^2=0.49\). (3) Their event status at the horizon is unknown, so treating them as event-free would be incorrect; the observed statuses of other patients receive inverse-censoring weights. (4) Include that variable, or an adequate representation such as the predicted risk when appropriate, in the conditional censoring model, and assume event time is independent of censoring conditional on the resulting adjustment variables. Check that censoring probabilities remain positive over the evaluation period.