What Does It Mean to Validate a Multi-Step Prediction?
A multi-state model can predict the probability that a person will follow a specified sequence of transitions by a given time. For example, a prediction might concern moving from an initial state to an intermediate state and then to a final state within two years. Validation asks whether predictions of that kind agree with what is observed in a separate set of individuals whose outcomes can be followed to the same horizon.
The key is to validate the same event that the model predicted. A probability of completing a route is not automatically the probability of occupying its final state at the horizon: the individual might complete the route and then move again. Nor is it necessarily the probability of making just one transition. Define the route event and horizon before calculating an observed frequency.
For a route defined as “the sequence has occurred by \(t\),” two route events can overlap: someone who follows \(s_0\to s_1\to s_2\to s_3\) has also completed the shorter sequence \(s_0\to s_1\to s_2\). Their observed frequencies should not be added as though they were mutually exclusive categories. If the purpose is to compare categories that sum to one, define each category as a distinct, complete history through the horizon.
Validation also requires a cohort and follow-up scheme that allow the route outcome to be determined. The examples below first use outcomes fully ascertained at the horizon. If individuals are censored before their route status is known, a raw frequency among only those with known outcomes can be misleading; the censoring mechanism and an appropriate method must be considered.
Aggregate Agreement: Predicted Probability and Observed Frequency
For \(n\) individuals with known route status at \(t\), the average predicted probability and observed route frequency are
The difference \(\bar Y^\pi(t)-\bar p^\pi(t)\) describes aggregate calibration for this route and horizon. A positive value means the route occurred more often than the predictions indicated; a negative value means it occurred less often. The observed-to-expected ratio, \(\bar Y^\pi(t)/\bar p^\pi(t)\), is another summary when the average prediction is nonzero. A ratio above one indicates more observed route completions than expected from the predictions.
These are summaries, not proofs that a model is calibrated. Predictions can be too high for some individuals and too low for others while their average is close to the observed frequency. Therefore, also compare observed and predicted frequencies across groups of individuals with similar predicted probabilities. Such grouping is a diagnostic summary, not a replacement for examining the underlying event definition and follow-up.
Worked Example: Checking a Two-Transition Route
Suppose a hypothetical validation cohort contains 200 individuals, all followed for two years. The route is defined as moving from state 0 to state 1 during the first year, then from state 1 to state 2 during the second year, and remaining in state 2 through the two-year horizon. Assume state 2 is absorbing. A model assigns the same predicted probability, 0.20, to this route for every individual. Forty-three individuals meet the route definition.
The mean prediction is \(\bar p^\pi(2)=200(0.20)/200=0.20\). The observed frequency is \(\bar Y^\pi(2)=43/200=0.215\). Thus the observed-minus-predicted difference is \(0.215-0.20=0.015\), or 1.5 percentage points. The observed-to-expected ratio is \(0.215/0.20=1.075\). In this hypothetical cohort, the route occurred slightly more often than the model's average prediction, but this aggregate comparison alone does not establish whether individual predictions are well calibrated.
The route probability could be formed as a product in this deliberately specified two-interval setup if the second probability is conditional on the first transition and the model's conditional probabilities refer to the stated route event. For instance, if the first transition probability is 0.40 and the conditional probability of the second transition and required final condition is 0.50, then \(0.40\times0.50=0.20\). Multiplying probabilities without the appropriate conditioning or event definitions is not justified in general.
Why the Aggregate Comparison Works—and What It Does Not Show
If the predictions are calibrated for the specified route and horizon, then, conditional on the information used to make each prediction, the expected route indicator equals the predicted probability. Writing this condition as \(E\{Y_i^\pi(t)\mid \hat p_i^\pi(t)\}=\hat p_i^\pi(t)\), taking expectations gives
This is the justification for comparing the average prediction with the observed frequency. It also explains the limitation: an aggregate difference near zero is compatible with errors that cancel. For example, overprediction among low-risk individuals can offset underprediction among high-risk individuals. Calibration within prediction groups helps reveal this pattern.
The equation concerns expected agreement under the stated calibration condition; in a finite validation sample, the observed frequency varies. A small difference is not proof of calibration, and a nonzero difference can reflect sampling variation as well as systematic miscalibration. This tutorial's summaries describe agreement; they do not, by themselves, quantify uncertainty or test a hypothesis.
Worked Example: Grouped Calibration for One Route
A hypothetical validation set of 100 individuals is divided into three groups by predicted probability of completing the same route by the same horizon. Outcomes are known for all individuals.
| Prediction group | Number of individuals | Mean predicted probability | Observed route completions | Observed frequency |
|---|---|---|---|---|
| Low | 40 | 0.10 | 5 | 0.125 |
| Medium | 35 | 0.25 | 10 | 0.286 |
| High | 25 | 0.50 | 14 | 0.560 |
In the low group, the predicted number of route completions is \(40(0.10)=4\), compared with 5 observed; the observed frequency is \(5/40=0.125\). In the medium group, the expected count is \(35(0.25)=8.75\), compared with 10 observed; the frequency is \(10/35=0.2857\). In the high group, the expected count is \(25(0.50)=12.5\), compared with 14 observed; the frequency is \(14/25=0.56\).
Overall, the mean prediction is \(\{4+8.75+12.5\}/100=0.2525\), while the observed frequency is \((5+10+14)/100=0.29\). The observed-minus-predicted difference is \(0.0375\), and the observed-to-expected ratio is \(0.29/0.2525=1.149\). In this sample, the route was more frequent than predicted overall and in each displayed group. The grouped results suggest the discrepancy is not limited to just one prediction range, though the groups are coarse and the sample is finite.
Horizon, Censoring, and Event Ascertainment
The horizon is part of the outcome definition, not a detail to choose after looking at the results. A prediction for route completion by one year must be compared with status by one year; a two-year observed frequency answers a different question. Similarly, a prediction for completing a sequence and being in its final state at the horizon cannot be checked using a frequency that counts completion followed by any later state.
Right censoring creates a further issue. If an individual is censored before \(t\) without having completed the route, whether that person would complete it by \(t\) is unknown. Coding that individual as a non-completer biases the frequency downward when some would have completed the route. Simply discarding all individuals with unknown status is justified only under appropriate assumptions about censoring; for example, censoring must not selectively remove individuals with different route outcomes in a way that distorts the observed subset. When those assumptions are inadequate, censoring-adjusted validation methods are needed.
Worked Example: Incomplete Status Before the Horizon
In a hypothetical cohort of 50 individuals, the prediction horizon is two years. Route status is known at two years for 42 individuals: 9 completed the route and 33 did not. Eight were censored before two years without having completed it, so their status by the horizon is unknown. Suppose the mean predicted route probability for all 50 individuals is 0.22.
The observed frequency among individuals with known status is \(9/42=0.2143\). It is not correct to call \(9/50=0.18\) the observed two-year route frequency: that calculation treats all eight unknown outcomes as route non-completions. Nor should \(9/42\) automatically be treated as an unbiased estimate for the full cohort. It can represent the cohort frequency under suitable independent-censoring and representativeness assumptions, but if censoring relates to route outcome or its predictors, adjustment is needed.
If those assumptions were credible, the complete-case observed-minus-predicted comparison would be \(0.2143-0.22=-0.0057\), about -0.57 percentage points. Without justification for using the observed subset to represent the cohort, that comparison is not a sound validation conclusion. The appropriate response is to address censoring with a method suited to the design and assumptions, rather than to label unknown outcomes as failures.
Diagnostics, Alternatives, and Common Mistakes
A useful route-validation check begins with the data structure, not the arithmetic. Verify that each person's observed history is reconstructed in time order; the required transitions occur in the specified order; the same start state, horizon, and terminal condition are used for predictions and outcomes; and censoring has not been silently coded as a negative outcome. Then inspect both aggregate agreement and grouped agreement. Sparse groups or very few observed route completions make frequency estimates unstable, so apparent departures should be interpreted cautiously.
This approach complements other validation summaries rather than replacing them. Transition-specific calibration evaluates a single transition, whereas route-specific validation asks about a sequence of transitions as a whole. State-occupation probabilities concern the state occupied at a horizon, which may combine many histories. A Brier score summarizes squared prediction error, and a time-dependent AUC assesses ranking or discrimination; neither directly answers whether a predicted route probability agrees with the route's observed frequency at each prediction level.
- Wrong: “The route happened more often than predicted, so every individual's risk was underestimated.” Correction: Aggregate excess does not show how errors are distributed across individuals; compare prediction groups and retain the finite-sample context.
- Wrong: “The person reached the final state, so the predicted route occurred.” Correction: The person may have reached that state by a different sequence. Check the ordered history, not only the endpoint.
- Wrong: “The average route probability is close to the observed frequency, so the model is calibrated.” Correction: Errors can cancel in the aggregate. Examine calibration across ranges of predicted probability.
- Wrong: “Anyone censored before the horizon did not complete the route.” Correction: Unless the route was already completed, the horizon status may be unknown and requires appropriate handling.
Route probabilities also have limits as validation targets. A rare sequence can yield very few observed completions even in a sizeable cohort, making frequencies volatile. Grouping predictions can conceal important variation or become uninformative when too few individuals are available. Finally, a validation result applies to the population, follow-up, horizon, and event definition studied; it does not guarantee the same performance elsewhere or at another time.
Check Your Understanding
Answer each question, then compare your reasoning with the brief solutions.
- For a route defined as completion of \(0\to1\to2\) by three years, what must be fixed before comparing predicted probabilities with observed frequencies?
- In a validation set, the mean predicted probability is 0.18 and the observed route frequency is 0.21. Compute the observed-minus-predicted difference and interpret its sign.
- Why can an overall observed frequency close to the mean prediction fail to show good calibration?
- At a fixed horizon, 6 of 30 people with known route status completed the route; 5 more had unknown status because of earlier censoring. Why is \(6/35\) not automatically the observed route frequency?
- Can the frequencies for routes \(0\to1\to2\) and \(0\to1\to2\to3\), each defined as completed by the horizon, be added as disjoint outcome probabilities? Explain.
Solutions. (1) Specify the horizon, the exact transition order, the start state, and whether completion alone or completion plus a particular state at the horizon defines the event. (2) \(0.21-0.18=0.03\), so the route occurred 3 percentage points more often than predicted in that validation set. (3) Overprediction for some individuals can cancel underprediction for others; compare observed and predicted frequencies across prediction groups. (4) It treats all five unknown outcomes as non-completions; the horizon route status is not known for them. (5) No. Completion of the longer route implies completion of the shorter route, so the route events overlap under those definitions and their frequencies are not disjoint probabilities.