Smoothed Calibration Curves For Multi-State Predictions

Why Smooth Calibration Curves?

A calibration assessment asks whether predicted probabilities agree with observed outcomes. For multi-state predictions, this comparison must refer to a particular origin state, prediction horizon, and transition target. A single overall summary can conceal an important pattern: predictions may be too high at one part of the prediction range and too low at another. A smoothed calibration curve makes that pattern visible by comparing predicted probabilities with estimated observed frequencies across the range of predictions.

The curve is descriptive, not a new prediction model. Its purpose is to assess the agreement of predictions with outcomes among individuals receiving similar predictions. Because outcomes may be censored before the horizon, the observed frequencies must account for incomplete follow-up. The resulting curve is only as credible as its endpoint definition, censoring adjustment, and smoothing choices.

This definition treats a competing outgoing transition as a non-target outcome, not as censoring. It concerns the specified transition from the origin state; it does not automatically describe later paths through other states or the probability of occupying a state at the horizon. Individuals whose status at \(\tau\) cannot be determined because they were censored earlier have an unknown \(Y_i\), not a zero.

Estimating the Observed Frequency Under Censoring

Suppose individual \(i\) enters the prediction set in state \(j\), receives prediction \(\hat p_i\), and has follow-up until either the first exit from \(j\) or censoring. Let \(U_i\) be the time of the first outgoing transition, \(J_i\) its destination, and \(C_i\) the censoring time, all measured from the prediction origin. The endpoint is known if an outgoing transition is observed by \(\tau\), or if the individual is known to remain in \(j\) through \(\tau\). Write \(R_i=1\) when the endpoint is known and \(R_i=0\) otherwise. For \(R_i=1\), \(Y_i=1\) only when \(U_i\leq\tau\) and \(J_i=k\).

Let \(G_i(u)=P(C_i\geq u\mid X_i)\) be the censoring survival probability at time \(u\), conditional on the variables \(X_i\) used to model censoring. Under a clearly stated tie convention, define \(q_i=G_i(U_i)\) for an observed outgoing transition by \(\tau\), and \(q_i=G_i(\tau)\) when the individual is known to remain in the origin state through \(\tau\). Thus \(q_i\) is the probability, under the censoring model, of observing the information that establishes the endpoint. In practice \(G_i\) is estimated; the estimate must be positive for the times and covariate patterns contributing to the curve.

A simple smooth estimate uses a kernel weight to give most influence to individuals whose predictions are near a chosen value \(p\). With bandwidth \(h>0\), kernel function \(K\), and estimated observation probabilities \(\hat q_i\), a censoring-adjusted local mean is

Only individuals with known endpoints contribute. Each is weighted both by proximity to \(p\) and by the inverse estimated probability of endpoint ascertainment. The numerator accumulates weighted target-transition outcomes, while the denominator accumulates the corresponding weighted observations. The ratio estimates the local observed frequency. A common compact kernel is \(K(u)=1-|u|\) for \(|u|\leq1\), and zero otherwise. This local-constant ratio remains between zero and one when its denominator is positive.

Why the Weighting Works

The basic justification is an inverse-probability argument. Consider an individual’s full endpoint and event history, and suppose the probability that the endpoint is observed for that history is \(q_i\). Under the censoring assumptions, the conditional expectation of the weighted observed outcome, given the full history, is

The same argument gives \(E(R_i/\!q_i\mid\text{full history})=1\). Therefore, locally around a prediction value \(p\), the weighted numerator represents the target outcomes and the weighted denominator represents the corresponding population count. Their ratio estimates the conditional mean \(E(Y_i\mid\hat p_i\approx p)\). Smoothing replaces an unstable estimate at one exact prediction value with a local estimate using nearby predictions. This argument does not rescue a misspecified censoring model, near-zero observation probabilities, or inadequate data near \(p\).

Reading the Curve and Choosing a Smoother

Plot \(\hat m_h(p)\) vertically against \(p\) horizontally, and include the diagonal reference line \(m(p)=p\). Points or curve segments above the line indicate that the observed target-transition frequency is higher than predicted locally; segments below indicate that it is lower. The vertical difference \(\hat m_h(p)-p\) is a local calibration discrepancy on the probability scale. Interpret it only over the prediction range with adequate data and censoring support.

The bandwidth \(h\) controls the balance between detail and stability. A small bandwidth emphasizes nearby predictions but can produce a jagged curve with high variance. A large bandwidth averages over a wider range and may obscure local departures or bend the curve toward a broad average pattern. A bandwidth should be selected before interpreting small features, or its sensitivity should be shown with more than one reasonable choice. Kernel choice matters less than bandwidth in many applications, but should still be reported.

A grouped calibration plot is an alternative: divide predictions into bins and plot each bin’s mean prediction against a censoring-adjusted observed frequency. Binning is easy to explain, but results depend on cut points and can hide within-bin patterns. A smooth curve avoids abrupt bin boundaries, but depends on the smoothing choice and can imply more precision than the data support. A regression-based smoother, such as a logistic model for the binary endpoint with a flexible function of prediction, is another option. Its shape is constrained by the chosen model and may extrapolate beyond well-supported prediction values. None of these approaches makes censoring assumptions unnecessary.

Diagnostics, Uncertainty, and Common Pitfalls

A credible curve requires more than a smooth line. Inspect the distribution of predictions and the number of individuals contributing across the horizontal axis. Also inspect the estimated censoring probabilities and the resulting weights. Very large inverse weights can make a few observations dominate a local estimate; near-zero estimated \(G_i(u)\) signals a positivity problem rather than a reason to trust an unstable correction. Consider reporting weight summaries, effective sample size, and the range over which the curve is supported. If censoring varies by covariates, assess whether the censoring model includes the variables needed to make censoring conditionally independent.

Quantify uncertainty, for example, with a bootstrap that resamples independent individuals and repeats estimation of the censoring distribution and the curve. If predictions come from a fitted model in the same data, refit that prediction model within each bootstrap sample when the goal is to include model-fitting uncertainty. A pointwise confidence band addresses uncertainty at individual prediction values; it is not automatically a simultaneous band for the whole curve. Sparse regions and extreme weights should be made visible rather than concealed by a smooth line.

A common mistake is to code everyone censored before \(\tau\) as \(Y_i=0\), then smooth those zeros together with observed outcomes. That treats unknown endpoint status as a known absence of the target transition and generally biases observed frequencies downward. The correction is to identify which endpoints are ascertainable, treat competing outgoing transitions as known non-target outcomes when observed, and adjust for censoring under a defensible censoring model. Another mistake is to interpret the curve as a state-occupation probability: the target here is a specified outgoing transition by the horizon, not necessarily the state occupied at the horizon.

The curve is limited by the chosen prediction origin, horizon, endpoint definition, and validation population. It assesses calibration conditional on the predicted value, not discrimination or overall predictive accuracy. A visually close curve can coexist with poor ranking or substantial error, and a curve with wide uncertainty may not distinguish meaningful departures from sampling variation. Report the smoothing method, bandwidth, censoring model, endpoint and horizon, and uncertainty method so that the assessment can be interpreted and reproduced.