Residual Diagnostics For Transition Predictions

Why Examine Transition-Prediction Residuals?

A transition prediction is a probability for a specified destination, starting state, and time horizon. A residual summary compares those predictions with outcomes observed in a validation sample. Its central question is direct: on average, are predicted probabilities higher or lower than the observed transition probability? The answer can reveal systematic overprediction or underprediction, but it is not a complete assessment of a model.

Censoring complicates this comparison. If a person’s follow-up ends before the horizon, their state at that horizon may be unknown. Counting that person as having no transition, or simply omitting them without checking the censoring mechanism, can distort the observed outcome summary. This tutorial defines a fixed-horizon residual and shows how inverse-probability-of-censoring weighting can account for incomplete observation under explicit assumptions.

Define the Outcome Before the Residual

Consider a validation sample of \(n\) people who are in state \(h\) at a specified landmark. Let \(\tau\) be the horizon measured from that landmark. For person \(i\), let \(Y_i(\tau)\) equal 1 if that person is in destination state \(k\) at the horizon, and 0 otherwise. This definition concerns state occupancy at the horizon: a person may reach \(k\) through intermediate states, if the model’s prediction is defined that way.

Let \(\hat{p}_i(\tau)\) be the model’s predicted probability that person \(i\) will be in state \(k\) at the horizon, conditional on the information used to make the prediction. The corresponding sample mean prediction is \(\bar{p}(\tau)=n^{-1}\sum_{i=1}^n\hat{p}_i(\tau)\). If every horizon outcome is observed, the simplest residual summary is the observed proportion minus the mean predicted probability.

This is an average calibration diagnostic, not an individual-level error measure. A residual near zero says that the two averages are close; it does not show that predictions are accurate for every person, that they are well calibrated throughout the prediction range, or that the model discriminates well between people with different outcomes.

Accounting for Censoring at the Horizon

Let \(C_i\) denote person \(i\)’s censoring time after the landmark, and define \(\Delta_i(\tau)=I(C_i\geq\tau)\), where \(I(\cdot)\) is 1 when its condition holds and 0 otherwise. The horizon outcome \(Y_i(\tau)\) is observed when \(\Delta_i(\tau)=1\). If \(\Delta_i(\tau)=0\), the person’s state at \(\tau\) is unknown unless the event history itself determines it; the methods here use the general case in which it is unknown.

Write \(G_i(\tau)=P(C_i\geq\tau\mid Z_i)\), where \(Z_i\) is the observed information used to model censoring. A practical estimate is \(\hat{G}_i(\tau)\). A weighted estimate of the observed destination-state proportion is $$ \hat{O}_{w}(\tau)=\frac{1}{n}\sum_{i=1}^{n}\frac{\Delta_i(\tau)Y_i(\tau)}{\hat{G}_i(\tau)}. $$ For a person censored before the horizon, the numerator is zero; their unobserved outcome is not coded as a known zero. The residual summary is \(\hat{R}_{w}(\tau)=\hat{O}_{w}(\tau)-\bar{p}(\tau)\).

The weighting has a simple rationale. A person observed through \(\tau\) represents similar people who would have been censored earlier. If, conditional on \(Z_i\), being uncensored through \(\tau\) is independent of the horizon outcome, then the inverse of the probability of remaining uncensored corrects for the reduced observation probability. Small \(\hat{G}_i(\tau)\) values produce large weights, making results unstable and sensitive to a few observations.

Why the Weighting Works

The basic justification is an expectation calculation. For clarity, first suppose the true \(G_i(\tau)\) is known, and write \(Y_i=Y_i(\tau)\), \(\Delta_i=\Delta_i(\tau)\), and \(G_i=G_i(\tau)\). Conditional independent censoring gives \(E[\Delta_i\mid Y_i,Z_i]=G_i\). Consequently, $$ E\left[\frac{\Delta_iY_i}{G_i}\,\middle|\,Y_i,Z_i\right] =\frac{Y_i}{G_i}E[\Delta_i\mid Y_i,Z_i] =Y_i. $$ Taking expectations again gives \(E[\Delta_iY_i/G_i]=E[Y_i]\). Thus the weighted outcome contribution has the same expectation as the fully observed horizon outcome. Subtracting the average prediction targets the difference between the mean outcome probability and the mean prediction.

This argument requires positivity: if \(G_i(\tau)=0\) for a group, no one in that group can be observed through the horizon, so the outcome probability there cannot be recovered by weighting. It also assumes the censoring model describes the relevant observation probabilities. In practice, replacing \(G_i\) with an estimated value adds uncertainty and possible model error; the expectation calculation alone does not guarantee an unbiased finite-sample estimate.

Heterogeneous Censoring and Diagnostics

Censoring probabilities often differ across people. A single censoring probability is then inadequate unless it correctly represents the observation process for everyone. Inverse-probability weighting can accommodate measured differences, but it cannot repair unmeasured dependence between censoring and the horizon outcome. Before interpreting the residual, inspect the estimated censoring probabilities, the resulting weights, and how many people remain observable at the horizon.

Interpreting Residuals and Choosing an Approach

The complete-outcome residual and the weighted residual answer the same broad question: whether the observed destination-state probability differs from the mean predicted probability at a specified horizon. Their difference is how they handle incomplete outcomes. If everyone’s horizon state is observed, use the direct observed proportion. If some outcomes are unknown because of censoring, weighting can use the observed outcomes under its assumptions. A complete-case comparison instead describes the people observed through the horizon and can be misleading if those people differ in outcome risk from those censored earlier.

Pseudo-observations, introduced in the earlier material on pseudo-observations for transition-probability calibration, provide another route to calibration summaries under censoring. They are constructed from an estimator of the transition probability and can be used in calibration analyses. The weighted residual here is a direct fixed-horizon observed-minus-predicted summary. These approaches are not interchangeable in every setting: their construction, modeling choices, and uncertainty calculations differ. State which estimator was used and which population and horizon it targets.

A near-zero average residual does not establish calibration at every prediction level. Overprediction for some people can offset underprediction for others. Nor does it establish discrimination, which concerns how well predictions distinguish people with different outcomes. The residual is most informative when reported with uncertainty, its horizon and transition definition, the number observed through the horizon, and a description of censoring adjustment. Inference should account for estimation of the censoring distribution; resampling subjects and re-estimating that distribution is one practical approach.

Limitations and Practical Checks

A weighted residual can be unstable when few people are observed through the horizon or when some estimated censoring probabilities are close to zero. Large weights can make the estimate depend heavily on a small number of outcomes. Report the range of weights and the number of observed horizon outcomes; investigate whether apparent discrepancies are driven by sparse data. Weight truncation may reduce variance but changes the estimator and can introduce bias, so it should not be applied silently or treated as a universal fix.

The diagnostic also depends on matching the prediction to the observed outcome. The starting state, landmark, destination, horizon, and conditioning information must agree. A prediction for being in state \(k\) by a horizon is not automatically comparable with an outcome defined as the first transition to \(k\); those are different targets. Finally, a residual estimates a population-level discrepancy with sampling error. Small samples, rare destinations, and uncertain censoring estimates can all make its sign and magnitude unstable.