Worked Validation Of An Illness-Death Prediction Model

What Is Being Validated?

An illness-death model predicts how a person’s state may evolve over time: initially healthy people may become ill or die, and initially ill people may die. Validation asks whether predictions agree with outcomes in a cohort not used to develop the model. It is important to define the target precisely: a predicted transition probability is evaluated for a specified starting state, prediction time, and horizon.

Consider states \(0\) (healthy), \(1\) (ill), and \(2\) (dead), where death is absorbing and illness persists until death. At horizon \(t\), a person starting healthy can be in any of the three states; a person starting ill can be ill or dead. The healthy-to-ill probability at \(t\) is therefore the probability of being in state 1 at \(t\), not the probability of ever having become ill by \(t\). Someone who became ill and subsequently died is in state 2 at the horizon.

The three assessments in this tutorial are consequently distinct: among healthy starters, assess \(p_{01}(t)\) and \(p_{02}(t)\); among ill starters, assess \(p_{12}(t)\). These are state-occupancy probabilities at the chosen horizon. Calibration for one probability does not guarantee calibration for the others.

Assumptions and Evaluation Plan

Before calculating performance, specify the origin state, time origin, horizon, target population, and outcome definition. Predictions must be calculated for that same horizon and population. Validation subjects should be independent of model development for an external or held-out validation; when predictions are estimated during resampling, the resampling scheme must also respect any clustering or repeated-subject structure. The model’s predicted probabilities should be valid probabilities: each must lie between 0 and 1, and the probabilities across possible states must sum to 1, up to rounding.

With complete follow-up to \(t\), the state outcome for every subject is observed. With right censoring before \(t\), some horizon states are unknown. A simple complete-case analysis can be biased if being observed through \(t\) is associated with prognosis. An inverse-probability-of-censoring-weighted (IPCW) analysis instead requires censoring to be conditionally independent of the future state, given variables in the censoring model, and requires a positive probability of remaining observed long enough to determine the outcome for each relevant covariate pattern. The censoring model must be adequate; weights cannot repair unmeasured informative censoring.

For a death outcome, death before \(t\) determines the horizon state because death is absorbing. A person known to be alive through \(t\) also has a known death indicator. A person censored while alive before \(t\), however, has an unknown death indicator at \(t\). For state 1, a person who is alive through \(t\) has a known state, but a person censored before \(t\) may later become ill or die. In practice, the censoring weights and observed-history rules must match the particular state outcome; do not treat a censored person as event-free at the horizon.

Horizon-Specific Calibration

A direct calibration check groups subjects with similar predicted probabilities and compares the group’s mean prediction with its observed event proportion. For \(p_{01}(t)\), for example, form groups among healthy starters and compare mean predicted probability of being ill at \(t\) with the observed fraction in state 1. Repeat separately for \(p_{02}(t)\) among healthy starters and \(p_{12}(t)\) among ill starters. A calibration plot can show these comparisons across the probability range; a smooth curve can supplement bins when the sample is large enough.

With complete follow-up, the observed fraction in a group is the usual sample proportion. Under censoring, estimate the observed state probability using an appropriate censoring-adjusted method, such as IPCW or pseudo-observations. The introductory material on pseudo-observations provides one route for estimating transition-probability calibration with incomplete follow-up. Whatever estimator is used, state its assumptions and uncertainty method.

Brier Scores and Why They Work

The Brier score measures squared prediction error for a binary outcome. For a specified transition \(r\) to \(s\), use the horizon indicator \(Y_{i,s}(t)\) and predicted probability \(p_{rs,i}(t)\):

Here \(n_r\) is the number of validation subjects starting in state \(r\). The score is between 0 and 1; lower values indicate smaller squared errors. A transition-specific Brier score treats being in the named destination state as 1 and being in any other state as 0. For an overall assessment of the state distribution, the multicategory Brier score sums the squared errors across all possible states for each subject, then averages:

where \(\mathcal S_0=\{0,1,2\}\) for healthy starters and \(\mathcal S_1=\{1,2\}\) for ill starters. This convention does not divide the multicategory score by two. The convention matters when comparing reported scores.

The squared-error score rewards accurate probabilities, not just correct classifications. To see why, suppose the true conditional probability of the outcome is \(q\), and the forecast is \(p\). Since \(Y\) is binary,

The first term does not depend on the forecast; the second is minimized at \(p=q\). Thus the expected Brier score is minimized by reporting the true conditional probability. The score also reflects outcome uncertainty: even a perfectly calibrated prediction can have a nonzero expected score when outcomes are inherently uncertain.

Scoring When Follow-Up Is Censored

For a binary horizon outcome \(Y_i(t)\), let \(\tilde T_i\) be the last time at which that outcome is known: \(t\) if observed through the horizon, or the observed death time if death before \(t\) determines the outcome. Let \(G_i(u)\) denote the probability of remaining uncensored through time \(u\), conditional on the variables in the censoring model. In the simple setting where outcome status is known after death or when observed through \(t\), an IPCW Brier estimator has contributions weighted by \(1/G_i(\tilde T_i)\); a subject censored alive before \(t\) contributes zero because the horizon outcome is unknown. Divide the total weighted loss by the original number of eligible subjects, not merely the number with observed outcomes. More complex state outcomes may require history-specific weights or other censoring-adjusted estimators.

Uncertainty, Diagnostics, and Alternatives

Calibration estimates and Brier scores vary across validation samples. A practical approach is to bootstrap subjects, repeating the complete calculation in each resample. For clustered data, resample clusters rather than individuals. In an external validation where the prediction model is fixed, retain each sampled subject’s prediction and outcome together; if the target includes model-fitting uncertainty, model development must also be repeated within each resample. For IPCW analyses, re-estimate the censoring model in each resample. Percentile or other justified bootstrap intervals can then summarize sampling uncertainty. A confidence interval quantifies uncertainty under the sampling and modeling assumptions; it is not the probability that a fixed parameter lies in the observed interval.

Inspect calibration at the chosen horizon separately for each starting state and destination. Useful warning signs include very sparse groups, large gaps between predicted and observed probabilities, unstable weighted estimates, predictions near 0 or 1 unsupported by enough outcomes, and estimated probabilities that fail to sum to 1. Compare a binned plot with a smooth calibration curve when data support smoothing. A single average calibration difference can hide underprediction at low risks and overprediction at high risks.

The Brier score summarizes probability error across the full cohort, while calibration plots localize disagreement between prediction and observation. Discrimination measures, such as those based on ranking, answer a different question: they do not establish that the probabilities themselves are accurate. Comparing Brier scores across horizons or populations also requires caution because outcome frequency, follow-up, censoring, and case mix can differ. No single summary replaces checking calibration, uncertainty, and the data conditions that support the calculation.

A second frequent error is to score healthy-to-ill as “ever ill by \(t\)” while the model predicts the probability of occupying state 1 at \(t\). These are different estimands when death can occur after illness. Define the horizon state first, then make the observed indicator and predicted probability refer to that same state.