Why Diagnose Calibration?
A competing-risk prediction can rank people well and still give risks that are systematically too high or too low. Calibration asks whether predicted probabilities agree with event risks observed in a population. This matters when predictions inform decisions at a particular time horizon: a predicted 20% risk by two years should correspond to an observed risk near 20% among people with similar predictions, not merely identify people who tend to experience the event sooner.
Calibration is specific to both the event cause and the time horizon. A model may be well calibrated for cause 1 by one year but poorly calibrated for cause 1 by five years, or calibrated for cause 1 but not cause 2. The assessment should therefore state the cause and horizon explicitly.
The cumulative incidence is the probability of experiencing the specified cause by the horizon before experiencing a competing cause. A competing event is therefore part of the outcome process, not ordinary loss to follow-up. The prediction being checked must refer to the same cause, horizon, population, and predictor information used to define the validation groups.
Constructing a Grouped Calibration Plot
Start with a validation sample that was not used to fit the prediction model, or use predictions generated by a suitable resampling scheme. For each validation person, obtain the predicted cumulative incidence for the chosen cause at \(\tau\). Sort people by this predicted risk and divide them into a modest number of groups, such as five or ten. For each group, calculate the mean predicted risk and estimate the observed cumulative incidence by \(\tau\), accounting for censoring and competing events.
Plot the group’s mean prediction on the horizontal axis and its censoring-adjusted observed risk on the vertical axis. The 45-degree line represents perfect agreement. Points above that line indicate that observed risks exceed predictions in those groups; points below it indicate that predictions exceed observed risks. Plotting uncertainty intervals for observed risks helps show how much precision the data provide.
The grouped prediction is the arithmetic mean of the individual predicted risks in that group. It should not generally be replaced by a prediction for an “average” covariate profile: with nonlinear prediction functions, that prediction need not equal the average of the individual risks. Grouping is a descriptive summary, not a replacement for individual predictions.
Worked Example: Calculating an Observed Risk With Competing Events
Suppose a hypothetical validation group contains 100 people. At day 30, 8 experience cause 1, 5 experience cause 2, and 7 are censored. At day 60, among those still at risk, 4 experience cause 1, 6 experience cause 2, and 10 are censored. We want the cause-1 cumulative incidence by day 60. Assume censoring is independent of event times and causes within this group, and that the recorded competing events are correctly classified.
At day 30, \(Y_g(30)=100\) and \(d_g(30)=8+5=13\). Thus the estimated probability of remaining free of either event just after day 30 is \(1-13/100=0.87\). Censoring does not enter this event-free survival decrement.
At day 60, \(Y_g(60)=100-8-5-7=80\). The cause-1 cumulative incidence has a day-30 contribution of \(8/100=0.08\), and a day-60 contribution of \(0.87(4/80)=0.0435\). Therefore
The observed group risk is 12.35% by day 60. If the mean predicted cause-1 risk in this group is 10.0%, the group lies above the identity line: observed risk exceeds predicted risk by 2.35 percentage points. That difference describes this group; by itself it does not establish that the model is miscalibrated in the wider target population.
Assumptions and a Short Justification
A grouped Aalen–Johansen estimate is useful only if its assumptions are plausible. The validation sample should represent the population to which calibration is being assessed, and predicted risks should be available for the people whose outcomes are evaluated. The horizon must lie within a period with adequate follow-up. Most importantly, censoring must be independent of the event process within the group, or conditionally independent given information appropriately handled in the analysis. If censoring depends on predictors that vary substantially within a group, the pooled group estimate may be biased.
The estimator’s form follows from a sequence of conditional probabilities. At an event time \(t\), among people still event-free and at risk, the estimated chance of cause \(j\) occurring at that time is \(d_{jg}(t)/Y_g(t)\). The estimated chance of remaining free of all event types immediately before that time is \(\widehat S_g(t-)\). Their product estimates the probability of reaching \(t\) event-free and then experiencing cause \(j\) at \(t\). Adding these mutually exclusive event-time contributions through \(\tau\) gives the cumulative incidence.
This construction also explains why treating competing events as censoring is wrong for this purpose. Doing so would estimate a hypothetical cause-specific risk in a setting where competing events are removed, rather than the actual probability of experiencing cause \(j\) in the presence of competing causes. The Aalen–Johansen estimate updates event-free survival using events of every cause, while adding only cause-\(j\) events to the cause-\(j\) cumulative incidence.
Worked Example: Comparing Predictions Across Risk Groups
Consider a hypothetical validation set divided into three groups of equal size. At a selected two-year horizon, a cause-1 prediction procedure gives these group summaries. The observed values are Aalen–Johansen cumulative-incidence estimates, not raw event proportions.
| Risk group | Mean predicted risk | Observed cause-1 risk | Observed minus predicted |
|---|---|---|---|
| Low | 0.04 | 0.05 | +0.01 |
| Middle | 0.12 | 0.15 | +0.03 |
| High | 0.30 | 0.36 | +0.06 |
For the middle group, the calibration difference is \(0.15-0.12=0.03\), or 3 percentage points. In the high group it is \(0.36-0.30=0.06\), or 6 percentage points. All three differences are positive, and the gap grows across groups. The plot would place all three points above the identity line, with the largest discrepancy in the high-risk group. This pattern is consistent with increasing underprediction across the displayed groups.
The conclusion is limited to the evidence shown. Without group sizes, confidence intervals, or more detailed risk information, one cannot tell whether the differences are statistically distinguishable from sampling variation or whether the pattern holds within each group. A useful plot reports group sizes and uncertainty intervals, and checks that the observed estimates are not being driven by very few people still under observation near two years.
Practical Workflow and Diagnostics
Choose the event cause, prediction horizon, and validation population. Confirm that the predicted quantity is a cumulative incidence for that cause at that horizon.
For each person, extract the model’s predicted risk at the chosen horizon (for example, using a prediction function such as riskRegression::predictRisk with the cause and time specified). Check that predictions are finite and between 0 and 1.
Sort by predicted risk, define groups, and calculate each group’s mean prediction and Aalen–Johansen estimate for the same cause and horizon. Retain the group size and numbers at risk as diagnostics.
Plot observed against predicted risk with the identity line and uncertainty intervals. Look for consistent displacement from the line, curvature, and isolated groups with sparse follow-up before describing the calibration pattern.
A clear vertical displacement from the identity line suggests systematic overprediction or underprediction over the corresponding range of risks. Curvature suggests that the calibration error changes with predicted risk. A single point far from the line may instead reflect imprecision, especially if its group is small or has few people at risk near the horizon. Confidence intervals for observed cumulative incidence can be shown group by group; resampling the validation people can also quantify uncertainty in the whole plotted pattern. If uncertainty from model development is part of the question, the model fitting process must be included in the resampling.
A useful diagnostic is to repeat the plot with a different reasonable number of groups. If the apparent pattern changes sharply, the data may not support a stable fine-grained diagnosis. Also check prediction distributions, group sizes, follow-up near the horizon, and censoring patterns across groups. Empty or nearly empty risk sets at the horizon make observed risk estimates unstable, regardless of how smooth the plot looks.
Worked Example: Why a Raw Proportion Can Mislead
Suppose a hypothetical group has 100 people. Twenty are censored before the one-year horizon, and before that censoring time there are no observed events. Of the remaining 80 people, 12 experience cause 1 and 8 experience a competing event by one year; the others remain event-free. Assume censoring is independent of event risk in this group and the event times occur after the censoring time.
The naive proportion \(12/100=0.12\) treats all 20 censored people as if they were known not to have cause 1 by one year. Their outcomes by the horizon are actually unknown. Since censoring occurs before any events in this example, the Aalen–Johansen cause-1 estimate uses 80 people at risk at the event time:
The censoring-adjusted estimate is 15%, not 12%. If the mean predicted risk were 15%, the adjusted estimate would agree with the prediction in this group, whereas the naive proportion would misleadingly suggest overprediction. In other data patterns the bias need not have this direction; the key point is that censoring before the horizon does not reveal the person’s event status at the horizon.
What Calibration Plots Can and Cannot Show
Grouped plots are interpretable but coarse. Averages can hide opposing errors within a group: predictions may be too high for one subset and too low for another, producing an apparently accurate group average. Group boundaries are also somewhat arbitrary, and estimates become unstable when groups are too small. A smooth calibration curve can show finer patterns, but smoothing does not remove the need for censoring-aware estimation and can create a false impression of precision.
Calibration is distinct from discrimination. Discrimination concerns how well predictions order people by risk; calibration concerns agreement between predicted and observed risks. A model can rank people well while systematically overpredicting every group. Conversely, a model can predict the same overall risk for everyone and be calibrated on average while offering little ability to distinguish individuals. A Brier score combines aspects of prediction error into a summary, but it does not show where, across the prediction range, risks are miscalibrated.
Calibration plots are also not causal analyses. Agreement between predictions and observed risks does not establish that changing a predictor would change risk. Nor does a plot alone identify why a discrepancy occurs. Differences in population, follow-up, outcome coding, censoring, or model specification may all contribute. If the validation population differs from the development population, report that context rather than interpreting the plot as a universal property of the model.
Check Your Understanding
Answer each question before reading the brief solutions.
- A group has mean predicted cause-2 risk of 0.18 at three years and an Aalen–Johansen estimate of 0.13. What does the difference indicate?
- Why should a competing event not be counted as ordinary censoring when estimating the observed cumulative incidence of cause 1?
- In a group, 6 cause-1 events and 4 competing events occur at the same event time when 50 people are at risk, with no earlier events. What is the cause-1 contribution to the cumulative incidence at that time?
- Name two reasons a grouped calibration plot might appear close to the identity line even when individual predictions are not well calibrated.
- What should be checked if the selected horizon is near the end of follow-up?
Solutions. (1) Observed risk is 5 percentage points below predicted risk, so the predictions are higher than the observed group estimate; this is evidence of overprediction in that group, subject to uncertainty. (2) A competing event prevents the target event from occurring first and is part of the risk process; censoring it would estimate a different quantity. (3) The contribution is \(1\times 6/50=0.12\), because no earlier events have reduced event-free survival. (4) Group averaging can conceal offsetting errors within a group, and a small number of broad groups can conceal curvature or local discrepancies. (5) Check the number at risk and amount of follow-up near the horizon, group sizes, censoring patterns, and uncertainty of the observed cumulative-incidence estimates.