Why the Target Must Be Named Precisely
A prediction that someone will move from state A to state B answers a different question from a prediction that the person will be in state B at a specified future time. A person can enter B and then leave before the horizon; another can reach B by a route that did not begin with the A-to-B transition. These distinctions affect the event label used for calibration, Brier scores, and discrimination measures.
The phrase “transition probability” can itself be ambiguous. In multi-state settings, it is often used for the probability of occupying a state at a later time, conditional on the state at an earlier time. In transition-specific validation, it may instead refer to the probability of a specified edge event. The reliable approach is to state the event, origin, and horizon explicitly rather than infer the target from its label.
All three probabilities can be conditioned on information available at the prediction origin, such as baseline covariates. For clarity, those covariates are omitted from the notation here. A prediction and its validation outcome must use the same origin state, time horizon, and event definition. The state process and edge definitions must also be specified consistently, and the event history must be observed well enough to determine the target. If follow-up ends before the horizon, the outcome is unknown rather than automatically negative; appropriate handling of censoring is needed.
How the Events Differ
The first-exit event asks which state is reached on the first departure from the origin state, and whether that departure occurs by \(h\). Other outgoing edges compete with the edge of interest: if a subject first exits \(i\) to \(k\ne j\), the subject is not a positive outcome for the first-exit \(i\to j\) target, even if a later path reaches \(j\). A subject who does not leave \(i\) by \(h\) is also negative for that target.
The occupation event asks only which state the subject occupies at the horizon. A subject who entered \(j\) and then left it is negative for occupation of \(j\) at \(t_0+h\). A subject who reaches \(j\) by some other route can be positive for occupation even without a first exit from \(i\) to \(j\). The event “ever traverses \(i\to j\)” has a different rule again: it is positive if that edge is traversed at least once, whether or not the subject remains in \(j\).
For any two events \(F\) and \(O\), the difference between their probabilities is
This follows by partitioning each event into its overlap and its non-overlap: \(P(F)=P(F\cap O)+P(F\cap O^c)\), while \(P(O)=P(F\cap O)+P(F^c\cap O)\). For a first-exit target \(F=\{T_i\le h,J_i=j\}\) and an occupation target \(O=\{X(t_0+h)=j\}\), the first term counts people whose first exit was to \(j\) but who are not in \(j\) at the horizon. The second counts people in \(j\) at the horizon whose first exit was not to \(j\), or who did not exit \(i\) by the horizon. The terms can balance, so the two probabilities can be numerically equal even when the events differ. Agreement of the events almost surely guarantees equal probabilities, but it is not necessary for numerical equality.
Worked Example: First Exit and Occupation in a Hypothetical Cohort
Suppose 100 hypothetical subjects are all in state A at the prediction origin. By horizon \(h\), 50 have stayed in A throughout; 24 followed A→B and remained in B; 6 followed A→B→A; 16 followed A→C and remained in C; and 4 followed A→C→A. Assume these paths describe everyone’s history through the horizon.
First exit from A to B. The subjects with first path A→B are the 24 who remain in B and the 6 who return to A. Thus 30 of 100 have the first-exit event: $$ \widehat r_{AB}(h)=\frac{24+6}{100}=0.300. $$
Occupation of B at the horizon. Only the 24 subjects who remain in B are in B at \(h\): $$ \widehat q_{AB}(h)=\frac{24}{100}=0.240. $$
The six who returned to A are positive for the first-exit A→B event but negative for occupation of B at \(h\). The 50 who never left A are negative for both. This example shows why the observed edge-event proportion cannot be substituted for the observed state-occupation proportion.
The transition and occupation events may coincide under strong path restrictions. For example, suppose B is absorbing and every path starting at A that reaches B must first exit A directly to B. Then first exit A→B by \(h\) implies occupation of B at \(h\), because B is absorbing. Conversely, occupation of B at \(h\) means B was reached by \(h\); the path restriction makes its first exit from A a direct transition to B. Under these conditions the two events coincide. Absorption of B alone is not enough: a path A→D→A→B reaches absorbing B even though its first exit from A was to D.
Worked Example: A Later Visit Does Not Change the First Exit
Consider 10 hypothetical subjects starting in A. Four follow A→B→C by the horizon, two follow A→B and remain in B, two follow A→C→B, and two remain in A. These paths are mutually exclusive and account for all 10 subjects.
The first exit from A is to B for the four subjects on A→B→C and the two on A→B who remain in B. Therefore, the first-exit A→B proportion is $$ \frac{4+2}{10}=0.600. $$ The two subjects on A→C→B merely visit B later; their first exit from A was to C, so they are not positive for the first-exit A→B target.
At the horizon, B contains the two subjects who remained there after A→B and the two who arrived by A→C→B. Thus the occupation proportion is $$ \frac{2+2}{10}=0.400. $$ The four on A→B→C are positive for the first-exit A→B event but not for occupation of B. The two on A→C→B show the reverse mismatch: they occupy B at the horizon without a first exit from A to B.
Choose the Validation Target Before the Metric
For a first-exit prediction, define the binary validation outcome as one if the first exit from \(i\) is to \(j\) by \(h\), and zero if a different edge is the first exit or no exit occurs by \(h\). For an occupation prediction, define the outcome as one if \(X(t_0+h)=j\), and zero otherwise. For an “ever traversed” prediction, use whether the specified edge was traversed at least once during the interval. Each outcome is a different random variable, even though all may be recorded for the same subject.
Calibration compares predicted probabilities with observed frequencies of the defined outcome. A Brier score squares the difference between each predicted probability and its corresponding binary outcome. Time-dependent AUC likewise depends on which observations count as cases and controls. Consequently, a metric is interpretable only when its predictions and labels target the same event. A transition-specific score calculated with horizon-occupation labels is not a noisy version of the intended edge score; it evaluates another estimand.
Worked Example: Matching Calibration and Brier Outcomes
Imagine 100 hypothetical subjects starting in A, divided into two groups of 50. In Group 1, 20 have first exit A→B and 12 occupy B at the horizon; in Group 2, 12 have first exit A→B and 18 occupy B at the horizon. Suppose a model predicts first-exit probabilities of 0.50 and 0.24 for Groups 1 and 2, respectively, and occupation probabilities of 0.30 for each group.
For the first-exit target, the observed group proportions are \(20/50=0.400\) and \(12/50=0.240\). The first group’s prediction of 0.50 is higher than its observed first-exit frequency; the second group’s prediction of 0.24 matches its observed frequency. Overall, the observed first-exit proportion is $$ \frac{20+12}{100}=0.320, $$ and the average predicted first-exit probability is $$ \frac{50(0.50)+50(0.24)}{100}=0.370. $$
For the occupation target, the observed group proportions are \(12/50=0.240\) and \(18/50=0.360\). The model predicts 0.30 in both groups, so it overpredicts occupation in Group 1 and underpredicts it in Group 2. Overall, 30 of 100 subjects occupy B at the horizon, giving an observed occupation proportion of 0.300, equal to the average predicted occupation probability.
The overall agreement for occupation does not establish groupwise calibration, and it says nothing about calibration of the first-exit predictions. The event-specific group comparisons are essential: even the overall means differ for the first-exit target, while the occupation means agree.
Assumptions, Diagnostics, and Common Failure Modes
A defensible validation analysis needs a clearly defined state process, edge set, prediction origin, and horizon. It also needs predictions that were formed using information available at the origin and observed histories sufficient to classify the stated event. If there is right censoring before the horizon, an unresolved subject cannot simply be coded as event-free. Use an appropriate censoring adjustment under its assumptions, such as the inverse-probability-of-censoring approach covered in the earlier material on IPCW Brier scores; inspect whether the adjustment is supported by the follow-up data.
A practical diagnostic is to tabulate histories that separate the targets. Among subjects whose first exit from \(i\) is to \(j\), count how many have left \(j\) by the horizon: they remain positive for that first-exit target but are negative for occupation of \(j\) at the horizon. Separately, identify subjects occupying \(j\) at the horizon whose first exit from \(i\) was to another state, or who did not exit \(i\) by the horizon. These subjects are positive for occupation but not for the first-exit target. People who merely visit \(j\) at an intermediate time need not be positive for a first-exit \(i\to j\) event; they are positive only if that specified first exit occurred.
Other warning signs include a model that reports an edge-event probability while the validation code extracts the state at the horizon; a horizon defined from a different origin in the two calculations; or inconsistent treatment of competing exits. In settings where states are observed only at scheduled visits, the exact transition time or order may be uncertain. The analysis must then state how ambiguous histories are classified rather than silently treating visit-to-visit changes as fully observed paths.
| Prediction target | Positive validation outcome | Key diagnostic |
|---|---|---|
| First exit \(i\to j\) by \(h\) | First exit from \(i\) occurs by \(h\) and goes to \(j\) | Check competing first exits and subsequent departures from \(j\) |
| Any traversal of \(i\to j\) by \(h\) | At least one traversal of the edge occurs in the interval | Check returns to \(i\) and repeated edge traversals |
| Occupation of \(j\) at \(h\) | The subject’s state at the horizon is \(j\) | Check departures from \(j\) and alternative paths into \(j\) |
A common mistake is to say, “The model predicts A→B, so anyone who is in B at the horizon is a case.” That reasoning substitutes an occupation event for a first-exit edge event. The correction is to write the event before constructing the label: for first exit, record the destination of the first departure from A; for occupation, record the state at the horizon. A second mistake is to infer identical events from equal observed proportions. Equal probabilities can occur by coincidence; only event agreement almost surely guarantees equality as a consequence of the events themselves.
Check Your Understanding
Answer each question using the event definition in the question. Full solutions follow the questions.
- A subject starts in A, follows A→B→A, and is in A at horizon \(h\). Is the subject positive for first exit A→B by \(h\), occupation of B at \(h\), or both?
- A subject follows A→C→B and is in B at \(h\). Classify the subject for first exit A→B and occupation of B.
- Give a sufficient path restriction, in addition to B being absorbing, under which first exit A→B by \(h\) and occupation of B at \(h\) coincide.
- In the two-group example, what is the overall observed first-exit A→B proportion, and what is the overall observed occupation proportion?
- Can two different event definitions have equal probabilities? What does equality of their probabilities establish about whether the events are identical?
Solutions. (1) The subject is positive for first exit A→B because the first departure from A is to B and occurs by \(h\). The subject is negative for occupation of B because the state at \(h\) is A. (2) The subject is negative for first exit A→B because the first exit from A is to C. The subject is positive for occupation of B because the state at \(h\) is B. (3) Require that every path starting in A that reaches B must first exit A directly to B. With B absorbing, first exit A→B ensures occupation of B at \(h\); the path restriction ensures that occupation of B at \(h\) implies that same first exit. (4) The first-exit proportion is \((20+12)/100=0.320\). The occupation proportion is \((12+18)/100=0.300\). (5) Yes. Distinct events can have equal probabilities. Equal probabilities alone do not establish that the events are identical; event agreement almost surely is sufficient for equal probabilities, but not necessary.