Calibration-In-The-Large For A Transition

Why compare average predicted and observed risk?

A model can give each person a different predicted probability of a transition, while validation asks how well those predictions agree with what happened in the group. A useful first summary is the difference between the average predicted risk and the observed transition probability at a specified horizon. This checks the model’s overall level of prediction: does it predict too much, too little, or approximately the right total amount of transition?

This comparison is meaningful only after fixing the prediction origin, target transition, horizon, and validation population. For example, a prediction for transition \(r\to s\) within one year of entering state \(r\) is not interchangeable with a prediction for a different destination or a different time origin. The target event must also be defined consistently for every subject.

For a direct transition \(r\to s\), one possible target is that the subject’s first exit from origin state \(r\), by \(h\), is to state \(s\). Under that definition, a first exit to another destination is a competing event, not a target event; remaining in \(r\) through \(h\) is also not a target event. Predictions and observations must use this same definition. Other validation questions can define a different event, but changing the event changes the quantity being calibrated.

Assumptions and calculation

When every subject’s event status through \(h\) is known, the observed probability for a binary target is the event proportion. Thus, if \(Y_i=1\) indicates that subject \(i\) experienced the specified transition by \(h\), and \(Y_i=0\) otherwise, then \(\widehat{O}(h)=\sum_iY_i/n\). This simple proportion requires complete ascertainment of the target status at the horizon. It is not justified when some subjects are lost to follow-up or otherwise censored before \(h\).

With right censoring, the observed probability needs an estimator that handles censoring and competing transitions. For a first transition from a state with competing outgoing transitions, the cumulative incidence for the target destination can be estimated using the competing-risks methods introduced in the earlier material on observed transition probabilities. Its interpretation relies on suitable independent-censoring conditions (possibly conditional on measured covariates), adequate follow-up support through the horizon, and correct handling of the competing destinations. Simply counting observed target events and dividing by the full sample size generally underestimates risk when some subjects are censored before \(h\).

The predictions must refer to the same horizon and target event as the observed estimate, and the validation sample must represent the population for which the comparison is intended. If predictions were developed using the same subjects, apparent agreement may be optimistic; validation in independent data, or a design that accounts for model development, is needed to assess performance beyond the development sample. The calculation itself is descriptive: it does not establish that the model is clinically useful or that individual predictions are accurate.

Why the gap has this interpretation

The sign follows directly from comparing two averages on the same probability scale. With complete outcome ascertainment, the observed probability is \(\bar{Y}=n^{-1}\sum_iY_i\). Hence $$ G(h)=\frac{1}{n}\sum_{i=1}^n(q_i-Y_i). $$ Each subject contributes their predicted risk minus their binary outcome. A positive average means predicted probabilities collectively exceed the observed event frequency; a negative average means they fall short. This identity explains the summary, but it does not make \(q_i-Y_i\) a reliable individual-level error measure: one binary outcome is highly variable relative to a probability prediction.

A related population justification is also useful. Let \(\pi_i\) denote the true event probability for subject \(i\), conditional on the information used for prediction. If \(E(Y_i\mid\text{prediction information})=\pi_i\), then $$ E\{G(h)\mid q_1,\ldots,q_n\} =\frac{1}{n}\sum_{i=1}^n(q_i-\pi_i). $$ Thus, the expected gap compares the average prediction with the average true risk in the validation population. A gap near zero indicates agreement of these averages, not that \(q_i=\pi_i\) for each subject.

What an overall match can hide

Calibration-in-the-large is a single population-level summary. It can be zero even when predictions systematically miss in different parts of the population. A smoothed calibration curve, covered in the earlier material on smoothed calibration curves for multi-state predictions, can reveal departures across the range of predicted risks. A calibration slope addresses another feature: whether predictions are too extreme or insufficiently spread. Neither should be replaced by the overall gap; each answers a different question.

Diagnostics, uncertainty, and limitations

Before interpreting a calculated gap, check that the prediction origin, horizon, transition definition, and target population match. Then check how the observed probability was estimated. In particular, look for early censoring, sparse risk sets near the horizon, and competing destinations handled as if they were censoring. Also inspect the distribution of predictions: a few extreme predictions can influence their average, while the single gap gives no indication of where those predictions occur.

For a complete sample with independent subjects, the observed event proportion has sampling uncertainty; the predicted average may also vary if the predictions are estimated rather than fixed. With censoring, uncertainty in the estimated observed probability must be included as well. A subject-level bootstrap that repeats the relevant estimation steps can provide an interval for the gap when the resampling scheme reflects the validation design. An interval describes uncertainty about the population gap under that design and assumptions; it is not a probability statement about a fixed parameter. A narrow interval around zero supports close average agreement, but does not rule out subgroup miscalibration.

The calibration gap should also not be confused with a calibration intercept from a regression model, which uses a model-based parameterization and may use a different sign convention. Here the target is specifically the arithmetic difference between mean predicted risk and observed transition probability. Reporting both the two component estimates and their signed difference makes the result interpretable and prevents ambiguity.

Common pitfall

• Wrong reasoning: “The average predicted risk equals the observed transition probability, so the predictions are calibrated for every subject.”
• Correction: Equality establishes only aggregate calibration at the stated horizon and in the stated population. Opposing errors across risk groups can cancel, and no individual’s true risk is observed directly.
• Wrong reasoning: “A competing transition is just a censored target event.”
• Correction: For a first-exit target, a competing transition prevents that target event from occurring as the first exit. It must be represented as a competing event in estimating the target probability.