Calibration-In-The-Large

What Calibration-in-the-Large Measures

A survival prediction made at a specified prediction origin often concerns the probability of an event occurring within a fixed horizon. For example, a joint model may predict a patient's risk of an event during the next year, conditional on the patient's longitudinal history available at that origin. When these predictions are evaluated in a validation cohort, a basic question is whether their average agrees with the cohort's average event risk.

This comparison is calibration-in-the-large. It is a marginal check: it compares average predicted risk with average observed risk, rather than asking whether every prediction is accurate or whether the relationship between predicted and observed risks has a particular shape. The introductory material on horizon-specific calibration plots describes visual calibration checks; here the focus is the signed average difference, including how to estimate observed risk when outcomes are censored.

The sign convention matters. This tutorial uses predicted minus observed, so positive values indicate overprediction. Some reports use the reverse subtraction; whenever a result is presented, state the convention explicitly. A value near zero indicates agreement of the two marginal means, not that each person's prediction is accurate, nor that risks are calibrated at every prediction level.

Observed Event Risk With Censoring

If every subject's event status at \(\tau\) is known, estimate the observed event risk by the sample proportion with an event by that horizon. With right censoring, a subject censored before \(\tau\) has unknown event status at \(\tau\). Treating that subject as event-free, or simply dropping all such subjects, can distort the estimate.

Write \(T_i\) for the event time, \(C_i\) for the censoring time, \(Y_i=\min(T_i,C_i)\) for the observed follow-up time, and \(\delta_i=I(T_i\leq C_i)\) for the event indicator. Let \(Z_i\) denote the observed variables used to model censoring. A status at \(\tau\) is ascertained if the event is observed by \(\tau\), or if the subject is known to be event-free through \(\tau\). Define \(R_i(\tau)=1\) in either case and \(R_i(\tau)=0\) when censoring occurs before \(\tau\) without an observed event. Thus, when \(\delta_i=1\) and \(Y_i\leq\tau\), the event status is observed; when \(Y_i\geq\tau\) with no event by \(\tau\), the subject is known to be event-free through the horizon.

Let \(G(t\mid Z_i)=P(C_i\geq t\mid Z_i)\) be the conditional censoring survival function. Under conditional independent censoring, the probability that a status is ascertained depends on its event information: for an event by \(\tau\), it is \(G(T_i-\mid Z_i)\); for a subject event-free through \(\tau\), it is \(G(\tau\mid Z_i)\). Denote this probability by \(\pi_i\) for the relevant case. The left limit \(G(T_i-\mid Z_i)\) accommodates the event being observed at its event time; the handling of tied event and censoring times must be specified consistently in an application.

Under these assumptions, an inverse-probability-of-censoring weighted (IPCW) estimator of observed event risk is

Only observed events by \(\tau\) contribute to the numerator. A subject censored before the horizon contributes zero because \(R_i(\tau)=0\); an observed event contributes its event indicator divided by the probability that it would have been ascertained. The cohort size \(n\), rather than the number of uncensored subjects, is the denominator. The corresponding estimate of calibration-in-the-large is

Why the IPCW Estimate Targets Event Risk

The weighting justification requires conditioning on the complete event information and \(Z_i\), not just on the observed follow-up time. Conditional on \(T_i\) and \(Z_i\), the probability of ascertainment is \(\pi_i\): it is \(G(T_i-\mid Z_i)\) for an event by \(\tau\), and \(G(\tau\mid Z_i)\) for someone who remains event-free through \(\tau\). Conditional independent censoring therefore gives \(E\{R_i(\tau)\mid T_i,Z_i\}=\pi_i\).

Since \(B_i(\tau)\) is determined by \(T_i\), it is fixed when conditioning on \(T_i,Z_i\). Consequently,

Averaging over \(T_i,Z_i\) shows that the weighted event contribution has expectation \(E\{B_i(\tau)\}\), the target event risk. Similarly, \(E\{R_i(\tau)/\pi_i\mid T_i,Z_i\}=1\), which explains why a ratio version with a weighted denominator can also be consistent under suitable conditions. The estimator displayed above uses the fixed cohort denominator \(n\). With estimated censoring probabilities, its performance additionally depends on the censoring model being adequately estimated; the finite-sample estimate need not equal the target exactly.

Interpretation, Diagnostics, and Alternatives

Report the sign convention, horizon, units, and the two averages being compared. For example, a difference of \(0.04\) under the predicted-minus-observed convention means that average predicted risk exceeds estimated observed risk by 4 percentage points at the specified horizon. It does not mean that every subject's predicted risk is 4 percentage points too high. Nor does it establish why the difference arose: it may reflect prediction-model error, a difference between the validation and development populations, or uncertainty in estimating observed risk.

Several checks are especially important when censoring is present:

• Check horizon and prediction origin. Predictions and outcomes must refer to the same origin and follow-up interval. A prediction made using information recorded after that origin does not represent the intended forecast.
• Check ascertainment and censoring probabilities. Very small \(\widehat\pi_i\) values create large weights and can make the estimate sensitive to a few subjects. Inspect the distribution of weights and whether important censoring predictors are represented in \(Z_i\).
• Check who remains observable. If censoring depends on event risk after conditioning on the variables in the censoring model, IPCW does not generally recover the target risk. A plausible model and adequate follow-up are substantive requirements, not just computational details.
• Do not infer a pattern from one average. Equal mean predicted and observed risk can hide offsetting overprediction and underprediction. A calibration plot can reveal patterns that a single summary cannot.

The simplest alternative when all outcomes are known by \(\tau\) is the unweighted event proportion, as in the first example. When censoring before \(\tau\) is present, the proportion among only those with known status is generally not a valid substitute: ascertainment may select subjects with different event risks. IPCW instead adjusts observed contributions using estimated ascertainment probabilities, under its assumptions. It can become unstable when those probabilities are small, and misspecification of the censoring model can bias the result.

Calibration-in-the-large is also narrower than a calibration curve: it summarizes only the difference in marginal means. A curve can show how observed risk varies across predicted risks, while a zero average difference alone cannot. Neither summary by itself establishes good discrimination. The mean difference is therefore useful as a compact diagnostic, but should be interpreted alongside other validation evidence rather than as a complete assessment of prediction quality.

Common Mistakes and Limitations

A second mistake is to describe a negative predicted-minus-observed difference as overprediction. Under the convention defined here, a negative difference means average predicted risk is lower than observed risk: underprediction. Another is to interpret a result near zero as proof of individual-level calibration. The statistic checks only equality of marginal averages; substantial errors in opposite directions can cancel. Finally, a weighted estimate is not automatically reliable merely because a formula was applied. If censoring probabilities are poorly estimated or close to zero, the estimate may be biased or highly variable.

The target population also matters. The predictions should be evaluated in a cohort relevant to the intended use, and their mean should be computed on the same set of validation subjects used to estimate observed risk. Differences in eligibility, prediction time, or follow-up definitions can turn an apparent calibration problem into a mismatch of estimands. Calibration-in-the-large describes average agreement for the specified population and horizon; it does not guarantee transportability to another population or another prediction horizon.