Why Smooth a Calibration Plot?
A horizon-specific calibration plot compares predicted event risks with observed event outcomes at a chosen time \(\tau\). Grouping predictions into a few risk categories is easy to display, but it hides variation within each group and makes the picture depend on arbitrary cut points. A smooth curve instead estimates how the observed event risk changes across the continuous range of predicted risks.
Censoring complicates this task. For a person censored before \(\tau\) without an observed event, the event status at \(\tau\) is unknown. Treating that person as event-free is incorrect, while simply dropping all such people can distort the curve. Inverse-probability-of-censoring weighting (IPCW), introduced in the earlier material on censoring-adjusted evaluation, allows assessable subjects to represent similar subjects whose horizon outcomes are unobserved, under specified assumptions.
Here \(T_i\) is the event time, \(C_i\) the censoring time, \(Y_i=\min(T_i,C_i)\) the observed follow-up time, and \(\Delta_i=I(T_i\leq C_i)\) the event indicator. Define the assessable set at \(\tau\) to contain subjects whose event by \(\tau\) is observed or whose event-free status through \(\tau\) is known. A subject censored before \(\tau\) without an observed event is not assessable. For an assessable subject, \(D_i(\tau)\) is observed; for a non-assessable subject it is not.
IPCW Kernel Smoothing
Let \(G(t\mid Z_i)=P(C_i\geq t\mid Z_i)\) be the censoring survival probability conditional on variables \(Z_i\) sufficient to make censoring independent of the event time. Write \(\hat G\) for an estimate of this probability. An observed event by \(\tau\) is weighted by the inverse probability of remaining uncensored to its event time; a subject known to be event-free through \(\tau\) is weighted by the inverse probability of remaining uncensored through \(\tau\). Thus, for assessable subjects, define
The left limit \(\hat G(Y_i-\mid Z_i)\) is used for an event at time \(Y_i\), representing being uncensored just before that event time. The second case covers subjects known to be event-free through \(\tau\). We do not define or use \(W_i(\tau)\) for non-assessable subjects. In particular, no calculation below divides by a censoring probability for someone whose horizon status is unknown.
For a kernel function \(K\), bandwidth \(h>0\), and prediction risk \(r_0\) at which the curve is to be estimated, define \(K_h(u)=K(u/h)/h\). A local-constant, or kernel-ratio, estimate is
where \(\mathcal A_\tau\) is the assessable set. Both sums run only over subjects for whom \(D_i(\tau)\) and \(W_i(\tau)\) are defined. The denominator must be positive. This ratio estimates a weighted local event proportion; it is not an unweighted average over all validation subjects. For display, it is evaluated over a grid of \(r_0\) values and plotted against the identity line \(m_\tau(r)=r\).
These conditions explain what the curve can and cannot show. If censoring depends on event risk after conditioning on \(Z_i\), or if \(\hat G\) is near zero, weighting may not remove bias and may greatly amplify noise. If prediction values are concentrated in a narrow range, the curve outside that range is mainly extrapolation, not empirical evidence.
Worked Examples
Worked Example: Calculating a Censoring-Adjusted Smooth Estimate
Suppose four hypothetical validation subjects have predictions near \(r_0=0.30\) at horizon \(\tau=2\) years. Three are assessable: subject A had an observed event at year 1 with \(\hat G(1-\mid Z_A)=0.80\) and \(r_A=0.20\); subject B was known event-free through year 2 with \(\hat G(2\mid Z_B)=0.50\) and \(r_B=0.30\); subject C had an observed event at year 1.5 with \(\hat G(1.5-\mid Z_C)=0.75\) and \(r_C=0.40\). Subject D was censored at year 1 without an event and is not assessable at year 2. Use the triangular kernel \(K(u)=(1-|u|)I(|u|\leq1)\), bandwidth \(h=0.20\), and only the assessable subjects.
The respective censoring weights are \(W_A=1/0.80=1.25\), \(W_B=1/0.50=2.00\), and \(W_C=1/0.75=1.333\). The kernel weights, with the common \(1/h\) factor omitted because it cancels in the ratio, are \(K((0.20-0.30)/0.20)=0.50\), \(K(0)=1\), and \(K((0.40-0.30)/0.20)=0.50\). Their observed horizon outcomes are \(1,0,1\).
The estimated event risk by two years among subjects with predictions locally near 0.30 is about 0.392, after censoring adjustment. This local estimate lies above the predicted risk 0.30, so this small neighborhood suggests underprediction. It is not a precise conclusion from three assessable subjects. Subject D contributes neither a zero outcome nor a term with an undefined weight: that subject is excluded from both sums because the horizon outcome is unknown.
Worked Example: Why Boundary Smoothing Needs Attention
Consider three hypothetical local outcome estimates at risks \(r=0.00,0.05,0.10\), equal weights, and a true linear relationship \(m(r)=0.10+2r\) over this small range. At the lower boundary \(r_0=0\), use a uniform kernel that gives equal weight to observations within \(h=0.10\). The corresponding mean outcomes are 0.10, 0.20, and 0.30. The local-constant estimate is
It exceeds the true boundary value \(m(0)=0.10\). The reason is geometric: at the boundary, all included risks are at or above zero, so a symmetric neighborhood around zero is unavailable. The average risk in this window is \(0.05\), and the linear trend increases the average outcome by \(2(0.05)=0.10\).
A local-linear fit regresses the three outcomes on \(r\) with an intercept and slope, using the local weights. Here the observations lie exactly on \(0.10+2r\), so the fitted line has intercept 0.10 and slope 2. The local-linear estimate at the boundary is therefore 0.10, the correct value in this constructed example. In real data, local-linear estimates can be less stable and may fall outside \([0,1]\); boundary correction reduces a source of bias but does not create information or guarantee a valid risk-valued estimate.
Worked Example: Checking Bandwidth Sensitivity
Suppose five hypothetical assessable subjects have predicted risks \(0.10,0.20,0.30,0.40,0.50\), outcomes \(0,0,1,1,1\), and censoring weights all equal to 1. At \(r_0=0.30\), use the triangular kernel. With \(h=0.20\), the kernel weights are \(0.5,1,0.5\) for risks \(0.20,0.30,0.40\), and zero for the other two. Thus
With the larger bandwidth \(h=0.40\), the weights at risks \(0.10,0.20,0.30,0.40,0.50\) are \(0.5,0.75,1,0.75,0.5\). The estimate becomes
The larger bandwidth uses more distant subjects and produces a smoother but markedly different estimate. The example is intentionally small; it shows why bandwidth must be reported and examined rather than selected to make the curve look appealing. It does not establish which estimate is closer to the unknown calibration function.
Bandwidth, Boundaries, and Diagnostics
Bandwidth controls the bias-variance trade-off. A small \(h\) uses a narrow neighborhood: it can reveal local departures from calibration, but the curve may be jagged because few subjects contribute. A large \(h\) averages over a wider range: the curve is smoother, but local patterns may be blurred and the result may conceal meaningful miscalibration. A practical display should make the bandwidth explicit and inspect whether the substantive interpretation changes over a reasonable range of values. Selecting a bandwidth is not a substitute for adequate validation data.
The prediction domain is \([0,1]\), so estimates close to 0 and 1 have one-sided neighborhoods. A local-constant smoother can have appreciable boundary bias, as the preceding example demonstrates. A local-linear smoother fits a weighted line near each target risk and uses its fitted value at \(r_0\); it often reduces first-order boundary bias when the underlying calibration function is locally smooth. However, its estimate can overshoot, becoming less than 0 or greater than 1, especially with sparse data, large weights, or a narrow bandwidth. Clipping such values to \([0,1]\) changes the estimator and should be disclosed rather than done silently. Reflection methods impose an artificial mirrored pattern beyond the boundary and are not automatically appropriate for risk predictions.
Diagnostics should focus on where the displayed curve is supported and whether weighting is stable. At each part of the risk range, inspect the number of assessable subjects, the distribution of predictions, the weighted denominator, and the size of the largest IPCW weights. A curve that changes sharply when one subject is removed, or that is driven by a few very large weights, is fragile. A smooth line can look convincing even where the denominator is tiny; plotting the prediction distribution or marking regions with sparse support helps reveal that problem.
| Choice or approach | Strength | Limitation to check |
|---|---|---|
| Local-constant kernel smoothing | Simple, transparent weighted local average | Boundary bias and sensitivity to bandwidth |
| Local-linear smoothing | Often reduces bias near a boundary and in locally linear regions | Can be unstable or produce estimates outside the risk range |
| Grouped horizon-specific plot | Easy to summarize and communicate | Cut points discard within-group detail and can hide local patterns |
A grouped plot is a useful descriptive alternative, not a replacement for the smooth curve: its summary depends on the chosen groups. Conversely, smoothing does not remove the assumptions of IPCW, eliminate sampling uncertainty, or establish that a model is well calibrated. The curve is a descriptive estimate of a relationship in the validation population at one horizon. Comparisons across horizons require their own outcome definitions and censoring adjustments.
Common Mistakes and Practical Limits
- Assigning censored subjects an outcome of zero. A person censored before \(\tau\) is not known to be event-free through \(\tau\). Exclude that person from the assessable sums and use censoring adjustment for the assessable observations.
- Using weights outside their defined set. The IPCW weight and horizon outcome above are defined only for assessable subjects. Sum over \(\mathcal A_\tau\); do not rely on multiplying an undefined weight by zero.
- Ignoring positivity. If the probability of remaining uncensored is very small, inverse weights become large. A smooth curve may then be dominated by a few subjects, and the available data may not support reliable estimation.
- Choosing bandwidth by appearance alone. A visually smooth curve is not necessarily accurate. Report the bandwidth and assess sensitivity and local data support.
- Interpreting a boundary estimate as strongly evidenced. Near zero or one, there may be few predictions and only one-sided support. The estimate can be sensitive to the smoother and to a small number of observations.
The central distinction is between the target relationship \(m_\tau(r)\), the censoring adjustment used to estimate it, and the smoothing rule used to display it. Correct weighting addresses censoring only under its assumptions. The kernel and bandwidth determine how neighboring predictions are combined, not whether the predictions are valid or whether the fitted joint model is causally interpretable.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions below.
- At horizon \(\tau\), a subject is censored before \(\tau\) with no observed event. Should the subject's outcome be coded zero in the smoothed calibration estimate? Why?
- Why does a local-constant smoother tend to be biased near the lower prediction boundary when the calibration function is increasing?
- In the bandwidth example, what were the estimates at \(r_0=0.30\) for \(h=0.20\) and \(h=0.40\)?
- State two conditions needed for the IPCW calibration estimate to be reliable.
Solutions. (1) No. The subject's event status by \(\tau\) is unknown, so the subject is not in the assessable set and contributes to neither sum. (2) At the boundary, the local neighborhood contains risks at or above the target, not a symmetric set around it. For an increasing function, their average outcome tends to exceed the value at the boundary. (3) The estimates were \(0.25\) and approximately \(0.6429\), respectively, illustrating substantial bandwidth sensitivity in the small hypothetical data set. (4) Examples include conditional independence of censoring and event time given the censoring-model variables, positivity of the censoring survival probability over the relevant times, adequate estimation of that probability, and sufficient assessable observations near the target prediction risk.