Evaluating Cumulative Incidence Predictions

Why Cumulative Incidence Predictions Need Special Evaluation

A predicted risk for one event type is not the same as a predicted risk in a setting where that event is the only possible failure. If a competing event occurs first, the event of interest can no longer happen. For example, when evaluating a prediction of relapse, death before relapse is a competing event. Treating such deaths as if they were ordinary censoring can overstate the observed probability of relapse.

Evaluation begins by specifying the population, prediction time, cause, and horizon. For a dynamic prediction, consider people who are alive and event-free at a landmark time \(t_0\), with recorded history \(H_i(t_0)\). Let \(T_i\) be the time from that landmark to the first event, and let \(J_i\) identify its cause. For a cause \(k\) and a future horizon \(\tau\), the prediction is the conditional probability of cause \(k\) occurring by \(\tau\), before any competing event. All predictions being evaluated must refer to the same target, landmark, and horizon.

Calibration Against Observed Cumulative Incidence

Calibration asks whether predicted risks agree with observed event probabilities. A calibration plot places predicted risk on the horizontal axis and an estimate of observed cause-specific cumulative incidence on the vertical axis. Agreement is indicated by proximity to the 45-degree line. The observed quantity must account for competing events and censoring; a raw fraction of observed cause-\(k\) events is generally not an adequate estimate when follow-up is incomplete.

A simple summary divides people into risk groups and compares the mean prediction in each group with the group’s estimated cumulative incidence. For a group with event times \(u_1,\ldots,u_m\), the Aalen–Johansen estimate for cause \(k\) is

Here \(Y(u_j)\) is the number still at risk immediately before \(u_j\), \(d_{kj}\) is the number of cause-\(k\) events at that time, and \(\widehat S(u_j-)\) estimates the probability of remaining free of every event just before that time. In this calculation, competing events reduce the event-free survival for later times, but they are not counted as cause-\(k\) events. Right censoring removes a person from the risk set after censoring.

The group estimate is then compared with the group’s average predicted risk, not with an individual outcome for each person. Grouping can reveal systematic overprediction or underprediction, but the result depends on group boundaries and may conceal departures within a group. Smooth calibration curves can show more detail, but require enough data and careful smoothing. Uncertainty should be shown where feasible, since a noisy curve is not evidence of precise calibration.

Prediction Error at a Fixed Horizon

Calibration is one part of prediction assessment. A horizon-specific Brier score measures squared error between predicted risk and the corresponding binary outcome. For cause \(k\), define \(Y_{ik}(\tau)=1\) if person \(i\) experiences cause \(k\) by \(\tau\), and \(Y_{ik}(\tau)=0\) otherwise. In complete data, the Brier score is the average of \(\{Y_{ik}(\tau)-p_{ik}(\tau)\}^2\). A lower score indicates smaller average squared prediction error in the evaluated sample. It reflects both calibration and the ability to distinguish people with different outcomes; it is not a pure measure of calibration.

With right censoring before \(\tau\), the outcome is unknown for some people. Inverse-probability-of-censoring weighting (IPCW) uses the probability of remaining uncensored to correct the contributions from observed outcomes. Let \(X_i=\min(T_i,C_i)\), where \(C_i\) is the censoring time, and let \(\Delta_i=1\) indicate that an event was observed. Write \(G(t\mid Z_i)=P(C_i\geq t\mid Z_i)\) for the conditional censoring survival probability given appropriate observed predictors \(Z_i\). One horizon-specific estimator is

An observed event by \(\tau\), whether cause \(k\) or a competing cause, contributes an outcome that is known: one for cause \(k\), zero for a competing cause. A person event-free and uncensored past \(\tau\) also has outcome zero for cause \(k\) by that horizon. Someone censored before \(\tau\) without an event contributes neither term because the horizon outcome is unknown. The denominator remains the full sample size \(n\).

The assumptions are substantive. Conditional on \(Z_i\), censoring must be independent of the event process relevant to the outcome; the censoring model must be adequately estimated; and the probability of remaining uncensored must be positive over the times and covariate patterns being evaluated. The prediction and censoring model may use information available by the landmark, but not information that would only become available later. Very small estimated values of \(G\) create large weights and unstable scores, so the censoring distribution and weight distribution should be checked.

Why IPCW Targets the Complete-Outcome Score

The weighting argument can be seen conditionally. For an observed event at time \(t\), its contribution is available only if the person remains uncensored through \(t\). Under conditional independent censoring, the probability of that observation is \(G(t-\mid Z)\). Dividing its squared error by this probability restores its expected contribution in the complete-data target. For a person with no event by \(\tau\), the outcome is observed only if they remain uncensored through \(\tau\), so the corresponding inverse probability is \(1/G(\tau\mid Z)\).

Here \(a\) and \(b\) are the relevant squared errors, fixed given the event outcome and prediction. The equalities follow because the conditional expectation of each censoring indicator is its corresponding censoring survival probability. Thus, with a correct censoring model and the stated independence and positivity conditions, the weighted average estimates the complete-outcome Brier score. Estimating \(G\) introduces additional sampling uncertainty, which should be reflected in uncertainty intervals when they are reported.

Summarizing Error Across a Time Interval

A Brier score at one horizon answers a horizon-specific question. If predictions are evaluated at several horizons, an integrated Brier score (IBS) can summarize the scores over a prespecified interval. For an interval from 0 to \(\tau_{\max}\), the normalized form is

In practice, the integral is approximated from a grid of times. The interval and time grid should be stated, since changing them changes the summary and the prediction targets being averaged.

Assumptions, Diagnostics, and Alternatives

Before interpreting a calibration plot or prediction-error score, verify that the evaluated cohort and each prediction match the intended landmark and horizon. Check that competing events are classified separately from censoring, that the censoring model uses appropriate predictors, and that estimated censoring survival probabilities are not near zero in important parts of the data. If weights are highly variable, the estimate may be dominated by a few people. Sparse risk groups, few cause-\(k\) events, and heavy censoring also produce imprecise calibration estimates.

A practical comparison of approaches is:

These summaries answer different questions. A calibration plot is directly about agreement between predicted and observed probabilities; the Brier score is an overall error measure. Neither by itself establishes that predictions are useful for a particular decision. Detailed discrimination measures and decision-based evaluation address different aspects of performance and should not be substituted for calibration or prediction error.