Landmark Validation Of Updated Transition Predictions

Why a Landmark Changes the Validation Question

A prediction made at baseline answers a different question from an updated prediction made later. At a landmark time $s$, a subject may have entered a new state, accumulated a different duration in that state, or acquired new clinical information. The updated prediction should therefore be checked against what happens after that landmark, among the subjects who are still eligible to receive it.

The central comparison is specific to a transition and a horizon. For example, one might ask whether updated predictions at month 6 agree with the probability of moving from state $a$ to state $b$ by month 12, among subjects in state $a$ at month 6. Subjects who already left $a$ before month 6 do not belong to that landmark risk set. A competing transition out of $a$ during the next six months prevents the specified $a$-to-$b$ transition during that interval and is counted as a non-target outcome.

The landmark time, eligible state, transition, prediction horizon, and information available for updating must be specified before examining validation outcomes. This makes the prediction and its observed counterpart refer to the same target population and time interval.

Definition: Fix a landmark $s$, a horizon $h>0$, and a transition $a\to b$. Let $E_s$ denote eligibility at the landmark; for a transition out of state $a$, this commonly means being in state $a$ at time $s$, with the required history observed. Let $H_s$ be the information available for updating the prediction at $s$. For an eligible subject $i$, $p_i(s,h)$ is the updated predicted probability of experiencing the specified $a\to b$ transition during $(s,s+h]$, under the stated transition definition. Let $Z_i(s,h)=1$ if that transition occurs in the interval and $Z_i(s,h)=0$ otherwise. The landmark validation target is the distribution of $Z_i(s,h)$ among subjects with $E_s=1$.

For a direct transition out of $a$, the event definition should say whether $a\to b$ must be the first exit from $a$ after $s$. In the examples below, it is: leaving $a$ for a different destination before reaching $b$ makes $Z=0$. If the model predicts a multi-step path or a different event definition, the observed outcome must use that same definition rather than silently substituting a direct transition.

In a semi-Markov model, a transition prediction can depend on time already spent in the current state. The prediction and validation outcome must use the same sojourn-time origin and elapsed duration. Aligning those origins is necessary even when the landmark itself is correctly chosen.

Aggregate Calibration at a Landmark

At a fixed landmark and horizon, calibration in the large compares the average predicted probability with the average target outcome in the eligible validation population. Write $E(\,\cdot\mid E_s=1)$ for an expectation over that population. The aggregate calibration condition is

E\{p_i(s,h)\mid E_s=1\} = E\{Z_i(s,h)\mid E_s=1\}.

This is an average statement. It does not say that each subject’s prediction equals that subject’s true conditional event probability given the full history. Nor does equality of these two averages guarantee that predictions are well calibrated at every prediction level or in every subgroup. The earlier material on calibration-in-the-large and smoothed calibration curves distinguishes these questions.

If all eligible outcomes are observed through $s+h$, the finite-sample comparison is direct. For $N_s$ eligible subjects, the mean prediction is $N_s^{-1}\sum_i p_i(s,h)$, and the observed transition proportion is $N_s^{-1}\sum_i Z_i(s,h)$. Their difference is a descriptive validation summary; a zero difference in one finite sample is not proof of perfect calibration.

Justification: The aggregate calibration condition is a statement about the marginal expectation in the eligible validation population. By the definition of the event probability, $$ E\{Z_i(s,h)\mid E_s=1\} = P\{Z_i(s,h)=1\mid E_s=1\}. $$ Thus aggregate calibration requires the mean prediction to equal this marginal event probability. It does not require $p_i(s,h)=E\{Z_i(s,h)\mid E_s=1,H_s\}$ for every subject. That stronger equality is a subject-level conditional probability assumption, not a consequence of calibration in the large.

A separate, stronger calibration condition can be stated conditional on the predicted value: $E\{Z_i(s,h)\mid p_i(s,h),E_s=1\}=p_i(s,h)$. This asks for agreement within prediction levels. Conditioning instead on all of $H_s$ asks a still more detailed question and must not be inferred from aggregate agreement.

Worked Example: Aggregate Calibration With Complete Outcomes

Suppose a hypothetical validation sample has eight subjects eligible at month 6 for a specified transition from state $a$ to state $b$. The horizon is the following four months, and all eight outcomes are observed. Their updated predictions sum to 2.40. Two subjects make the target transition by month 10; the others do not.

The mean prediction is $$ \bar p=\frac{2.40}{8}=0.300. $$ The observed transition proportion is $$ \hat q=\frac{2}{8}=0.250. $$ The observed-minus-predicted difference is $0.250-0.300=-0.050$, or $-5.0$ percentage points. In this hypothetical sample, the average prediction exceeds the observed proportion by 5.0 percentage points. This is a landmark-specific aggregate comparison, not evidence that every individual prediction is too high.

The calculation is appropriate only because the same eight subjects define both averages, all were eligible at month 6, and their outcomes were ascertained through the same month-10 horizon. It would not answer the stated question if the prediction mean included subjects who had already left state $a$ before month 6.

Updated Predictions and Competing Destinations

An updated prediction uses information available at the landmark, not information that becomes known afterward. Depending on the model and target, that information may include the current state, covariates measured by $s$, and elapsed time in the current state. The prediction is evaluated against a later outcome defined from $s$, not from baseline. For a semi-Markov prediction, using the correct elapsed sojourn time is part of matching the prediction to its target.

For a direct transition $a\to b$, a competing exit from $a$ is not a target transition. It is coded $Z=0$, as is remaining in $a$ through the horizon. This coding does not claim that competing destinations are unimportant; it defines the binary outcome for this particular transition-specific probability. A state-occupation probability or a probability of reaching $b$ by any path is a different target and requires a correspondingly different outcome definition.

Worked Example: Updated Predictions by Landmark History

Consider eight hypothetical subjects eligible in state $a$ at a prespecified month-3 landmark. A newly measured marker is used to update their predictions for the direct transition $a\to b$ over the next three months. Outcomes are completely observed. Four subjects are in a higher-marker group and four in a lower-marker group.

In the higher-marker group, predictions are $0.40, 0.50, 0.60, 0.50$, with outcomes $1,0,1,1$. The mean prediction is $2.00/4=0.500$, and the observed proportion is $3/4=0.750$.

In the lower-marker group, predictions are $0.10, 0.20, 0.20, 0.30$, with outcomes $0,0,0,1$. The mean prediction is $0.80/4=0.200$, and the observed proportion is $1/4=0.250$.

Across all eight eligible subjects, the mean prediction is $(2.00+0.80)/8=0.350$, while the observed proportion is $(3+1)/8=0.500$. The overall observed-minus-predicted difference is $0.150$. The group summaries also show that the discrepancy is not uniform: it is $0.250$ in the higher-marker group and $0.050$ in the lower-marker group. These small groups do not establish a general calibration pattern, but they illustrate why a matching overall average can conceal subgroup discrepancies.

The marker must have been available by month 3 for these to be legitimate updated predictions. If the groups had instead been defined using a measurement obtained after month 3, the validation would use future information and would no longer assess the specified landmark predictions.

When Outcomes Are Censored Before the Horizon

If a subject’s follow-up ends before $s+h$ without a target or competing transition, $Z_i(s,h)$ is unknown: that subject cannot simply be coded as a non-event. A complete-case proportion can be biased if being observed through the horizon is related to the transition outcome. Right-censoring methods, such as inverse-probability-of-censoring weighting (IPCW), can be used under appropriate assumptions; the earlier material on IPCW Brier scores and censoring-weight diagnostics gives the broader framework.

For an IPCW approach, censoring must be conditionally independent of the target outcome given the variables used in the censoring model, and the probability of remaining observed must be positive for the relevant histories through the needed times. The censoring weights must be estimated appropriately for the outcome and event-time observation process. If these conditions fail, weighting does not by itself repair the validation estimate. The weighted estimator also describes a target population only insofar as its weights and conditioning set correspond to that population.

Worked Example: A Simple Censoring-Weighted Landmark Proportion

In a hypothetical landmark risk set of six eligible subjects, four have outcome status observed through the horizon: two make the target transition, one makes a competing transition, and one remains in state $a$. Two are censored before the horizon with no transition observed. Suppose, for illustration, that each of the four subjects with known horizon status has the same estimated probability $G=0.80$ of having an observable status through the horizon, and that censoring is independent of the target outcome in this eligible group.

Using inverse-probability weights $1/G=1/0.80=1.25$ for the four known outcomes, the weighted target count is $2(1.25)=2.50$. The weighted total is $4(1.25)=5.00$. The weighted proportion is therefore $$ \hat q_{\mathrm{IPCW}}=\frac{2(1.25)}{4(1.25)} =\frac{2.50}{5.00} =0.500. $$

The competing transition and the subject remaining in $a$ contribute to the weighted denominator but not the target numerator. The two censored subjects are not assigned $Z=0$. Because the weights are identical here, they cancel in the ratio; with varying probabilities of observation, their values would affect the result. This deliberately simple calculation illustrates the weighting principle, not a general prescription for estimating censoring weights.

Landmark Comparisons, Diagnostics, and Limitations

A landmark analysis is a sequence of target-specific validation questions, one for each prespecified landmark and horizon. At each one, eligibility is reassessed and the predictions are updated using the information available then. The eligible population can change across landmarks because subjects enter, leave, or die; therefore, differences in observed calibration across landmarks may reflect both model performance and changes in the populations being evaluated.

For a landmark comparison, check the following before interpreting a metric:

Eligibility: confirm that every evaluated subject met the stated landmark rule and that subjects who had already experienced an excluding transition are not included.
Time origin and horizon: confirm that prediction and outcome start at the same landmark and cover the same interval. For semi-Markov predictions, align elapsed sojourn time as well.
Transition definition: verify that target, competing, and remaining-state outcomes are coded consistently with the prediction.
Information timing: use only information available by the landmark to update predictions.
Outcome ascertainment: identify censoring before the horizon and assess whether the chosen adjustment and its assumptions are plausible.

A common mistake is to compare an updated month-6 prediction with an outcome measured from baseline, or to include everyone who was eligible at baseline even if some had left the origin state before month 6. Either choice changes the target. The correction is to define $E_s$ first, then calculate both the prediction summary and the later outcome summary over that same landmark risk set and interval.

Another mistake is to argue that matching average predictions and outcomes means every subject’s probability was correctly predicted. That conclusion is too strong: predictions can average to the event proportion even when some subjects’ conditional risks are lower and others’ are higher. Aggregate calibration is useful for detecting overall under- or overprediction, but it does not replace assessment of calibration across prediction levels, discrimination, or prediction error. The earlier material on transition-specific Brier scores and time-dependent AUC addresses different aspects of predictive performance.

Landmark-specific summaries also do not automatically provide valid uncertainty estimates, especially when the same subject contributes at several landmarks. Nor does good calibration in one eligible validation population guarantee calibration in a different population. These questions require methods and assumptions beyond the point comparisons developed here.

Prespecify the target.
Name the landmark, eligible state or rule, transition, horizon, and outcome definition.

Form the landmark risk set.
Include only subjects eligible at the landmark, using information observed by that time.

Match predictions to outcomes.
Use the updated predictions and later transition statuses for the same eligible subjects and interval; handle censoring rather than coding unknown outcomes as non-events.

Interpret at the right level.
Describe aggregate calibration as an average comparison, and keep it distinct from subject-level or subgroup calibration.

Key takeaway: Landmark validation concerns updated predictions among subjects eligible at a prespecified time, for a clearly defined later transition outcome and horizon. Aggregate agreement compares average predictions with the marginal event probability in that landmark population; it does not imply that each subject’s prediction is their true conditional risk.

Check Your Understanding

Answer each question using the landmark, eligibility, transition, and horizon definitions given.

At month 4, a subject is no longer in the origin state for a proposed transition. Should that subject contribute to validation of predictions for that transition among subjects in the origin state at month 4?
In a validation set of 20 eligible subjects with fully observed outcomes, the mean prediction is $0.30$ and 4 subjects make the target transition. Calculate the observed proportion and observed-minus-predicted difference.
Why does equality of the average prediction and event proportion not establish that every subject’s prediction equals their conditional event probability?
For a direct transition out of state $a$, a subject instead makes a competing transition out of $a$ during the horizon. How is the binary target outcome coded?
What is wrong with coding a subject censored before the horizon, without an observed transition, as a non-event?

Solutions. (1) No. The subject is outside the specified month-4 eligible risk set. (2) The observed proportion is $4/20=0.200$; the observed-minus-predicted difference is $0.200-0.300=-0.100$, or $-10.0$ percentage points, so predictions exceed the observed proportion in this sample. (3) The equality is only between marginal averages. It does not impose equality conditional on each subject’s history; different individual risks and predictions can offset in the aggregate. (4) Code $Z=0$, because the specified direct transition did not occur as defined. (5) The outcome is unknown, not known to be zero. Treating it as a non-event can bias the observed proportion; use a suitable censoring adjustment under its assumptions or acknowledge that the outcome is not fully observed.