Calibration of Landmark Risk at Fixed Horizons

What Calibration Means at a Landmark

A landmark prediction is made for a defined group: patients who are alive and event-free at a chosen landmark time and whose required predictor information is available then. Calibration asks whether those predictions agree with subsequent event experience in that group. At a specified horizon, the question is direct: among eligible patients assigned a predicted risk near 20%, is the observed risk by that horizon also near 20%?

Let \(s\) denote the landmark time and \(h>0\) the follow-up horizon after it. For patient \(i\), let \(T_i\) be the event time measured from the study time origin, \(C_i\) the censoring time, and \(X_i(s)\) the predictor information available at the landmark. The patient is landmark-eligible if they are event-free at \(s\) and the information required by the prediction rule is available. A fitted landmark model supplies a predicted probability \(p_i(h)\) of the event during \((s,s+h]\), conditional on eligibility and the information used by that model.

Calibration is specific to the landmark population, the time origin, the prediction horizon, and the event definition. A model can be calibrated among patients who reach one landmark but not among all patients at baseline. It can also be calibrated at one horizon and miscalibrated at another. The introductory material on landmark risk establishes why the conditioning set is the landmark-eligible group; calibration evaluates predictions within that set, not in a different population.

Observed Outcomes and the Endpoint Convention

If every eligible patient’s event status through \(s+h\) is known, define \(Y_i(h)=1\) if the event occurs during \((s,s+h]\), and \(Y_i(h)=0\) otherwise. The observed event proportion in a group of \(m\) patients is \(\sum_i Y_i(h)/m\). Compare it with the group’s mean prediction, \(\sum_i p_i(h)/m\). Their difference is an easy-to-interpret calibration summary, but grouping can conceal miscalibration within groups and does not by itself describe the entire calibration curve.

With censoring, some patients’ horizon outcomes are unknown. Write \(U_i=\min(T_i-s,C_i-s)\) for observed follow-up after the landmark, and let \(\Delta_i=1\) if an event is observed and \(\Delta_i=0\) if censoring occurs first. Use the following endpoint convention throughout: an observed event at exactly \(h\) counts as an event by the horizon; a patient censored exactly at \(h\), with no observed event by then, is known to be event-free through \(h\). A patient censored before \(h\) without an event has unknown horizon status.

This convention resolves an important boundary case. Patients with an observed event at \(U_i\leq h\) have a known event outcome. Patients observed event-free through at least \(h\) also have a known outcome. In particular, a patient censored at \(h\) belongs in the latter group, not among patients with unknown status.

Calibration Summaries and Plots

A calibration plot displays observed event risk against predicted risk. For grouped summaries, divide predictions into a small number of prespecified or clearly described groups. Plot each group’s mean prediction on the horizontal axis and its observed event proportion on the vertical axis. The 45-degree line represents ideal agreement. A smooth calibration curve can show patterns that grouping hides; its shape depends on the smoothing method and sample size, so it should not be read as exact at sparsely populated risk levels.

Two compact summaries are often useful. The calibration-in-the-large difference is the overall observed risk minus the overall mean predicted risk. A positive value indicates that observed risk exceeds average predicted risk; a negative value indicates that predictions are higher than observed risk on average. A related recalibration model uses the predicted log odds as an offset:

The ideal offset intercept is \(a=0\). A positive fitted \(a\) indicates that the observed odds tend to be higher than the predicted odds after preserving the predictions’ relative ordering and spacing on the log-odds scale. This offset model adjusts overall level; it does not test whether the spread of predictions is appropriate. A distinct recalibration model estimates both an intercept \(a\) and slope \(b\), replacing the right side with \(a+b\operatorname{logit}\{p_i(h)\}\). Ideal values are \(a=0\) and \(b=1\); a slope below 1 commonly indicates predictions that are too extreme, while a slope above 1 commonly indicates predictions that are not varied enough. These estimates require enough outcome information and variation in predicted risks to be estimable.

Accounting for Censoring With IPCW

Inverse-probability-of-censoring weighting (IPCW) uses patients with known horizon outcomes to represent patients whose outcomes are unobserved because of earlier censoring. Let \(G_i(t)=P(C_i-s\geq t\mid Z_i)\), where \(Z_i\) is a set of variables sufficient to make censoring independent of event time, conditional on those variables. Under independent censoring given \(Z_i\), correct estimation of \(G_i(t)\), and positivity \(G_i(t)>0\) over the time range used, an observed outcome with probability \(G_i(t)\) of remaining observable can be weighted by \(1/G_i(t)\).

For a marginal observed risk in the landmark cohort, an illustrative IPCW estimator is:

Here \(\widehat{G}_i\) estimates the probability of remaining uncensored through the stated time, using the same “greater than or equal to” convention in the definition of \(G_i\). The event term includes observed events at \(h\). The event-free term includes patients followed beyond \(h\) without an event by \(h\), and patients censored exactly at \(h\) without an event. Patients censored before \(h\) without an event contribute to neither term because their horizon status is unknown. If the censoring-time distribution is continuous, ties at \(h\) are usually absent; with recorded-time ties, the endpoint convention must still be explicit.

The justification is inverse-probability weighting. If an event by \(h\) is observed at time \(u\), it is observed with probability \(G_i(u)\) under the censoring assumption; multiplying its observation indicator by \(1/G_i(u)\) removes that observation probability in expectation. Likewise, a known event-free outcome through \(h\) is observed with probability \(G_i(h)\), so its weighted contribution uses \(1/G_i(h)\). Summing the weighted event and event-free contributions estimates the corresponding outcome totals; their ratio estimates the event proportion. Estimation of \(G_i\) may use a censoring survival estimate, or a conditional model when censoring depends on measured predictors. The form of the estimator must match the censoring assumptions and the target population.

Uncertainty, Diagnostics, and Alternatives

Calibration summaries are estimates, not exact properties established by a plot. Confidence intervals for grouped observed risks or for a calibration parameter communicate sampling uncertainty. When model fitting, prediction, and evaluation involve resampling or data splitting, uncertainty procedures should respect the subject as the sampling unit. In IPCW analyses, uncertainty should also reflect estimation of the censoring distribution; a subject-level bootstrap that repeats the relevant estimation steps is one practical approach. Large or highly variable weights signal that few patients remain observable at some times. Truncating extreme weights may stabilize estimates, but changes the procedure and can introduce bias; report the choice and examine sensitivity.

Useful checks include confirming that every prediction and outcome refers to the same landmark and horizon, verifying event and censoring time conventions, inspecting numbers at risk near \(h\), and checking whether \(\widehat{G}(t)\) approaches zero. Compare calibration across clinically relevant risk ranges, but avoid creating many tiny groups. If censoring differs by measured patient characteristics, an unconditional censoring Kaplan–Meier estimate may be inadequate; use a defensible conditional censoring model or an appropriate stratified approach. If censoring is informative even after conditioning on measured variables, IPCW does not automatically solve the problem.

Grouped observed-versus-predicted summaries are transparent and easy to communicate, but depend on grouping and are noisy in small samples. A smooth curve uses information across prediction values, but can obscure sparse regions and depends on smoothing choices. Calibration-in-the-large compresses overall agreement into one number and can miss offsetting errors across risk levels. The two-parameter intercept-and-slope assessment adds information about scale but still cannot describe every nonlinear departure. These approaches complement one another; none replaces inspection of the underlying data and uncertainty.

Calibration is not discrimination. A model can rank patients well while systematically overpredicting risk, or be calibrated on average while failing to separate higher- from lower-risk patients. Discrimination is a separate evaluation target. Calibration also does not establish that a predictor causes an event or that acting on a prediction improves outcomes. The estimated agreement pertains to the landmark-eligible population and setting represented in the validation data. Changes in eligibility, predictor measurement, clinical practice, or event incidence can alter calibration.

Common Pitfalls

• Dropping patients censored exactly at the horizon. That discards a known event-free outcome under the endpoint convention used here. Include such patients in the event-free IPCW term, with the horizon censoring-survival weight.
• Counting a censored-before-horizon patient as event-free. Without event observation or follow-up through \(h\), the patient’s horizon outcome is unknown; simply coding them as a non-event biases the observed risk downward.
• Using the wrong denominator. For a calibration plot, each group’s observed risk must represent the same landmark-eligible group and horizon as the predictions, with censoring addressed when needed.
• Equating a small calibration difference with good calibration everywhere. Overall errors can cancel. Inspect calibration across the prediction range and report uncertainty.
• Calling a prediction causal. Calibration measures agreement between predicted and observed risks; it does not show that changing a predictor would change a patient’s outcome.