Why Landmark Analyses Fail in Practice
A landmark analysis makes a prediction for people who have reached a specified time without the event of interest. Its validity therefore depends on aligning four things: who is eligible at the landmark, what information was available then, how the future outcome is known, and what unit is kept together during validation. An error in any one can produce apparently strong predictions that answer the wrong question.
The examples below use a fixed landmark time \(s\) and prediction horizon \(h\). At time \(s\), the target is the risk of an event during the next \(h\) time units, conditional on being event-free at \(s\). These are conditional predictions for landmark survivors, not predictions for the original cohort at baseline.
The distinction between \(T_i>s\) and \(C_i\ge s\) matters. A subject whose event occurred at or before \(s\) is not a landmark survivor. A subject censored before \(s\) has unknown event status at \(s\), so cannot be classified as an eligible survivor from the available record. A subject followed without an event through \(s\) can be eligible, even if censoring occurs later.
Failure 1: Incorrect Eligibility at the Landmark
A common error is to include everyone who entered the original study, or everyone who has a recorded biomarker, in a prediction set at \(s\). That changes the conditioning population. The landmark risk is defined among people known to have survived event-free through the landmark; it is not the risk among all enrolled people.
Worked Example: Auditing Eligibility at Six Months
Suppose the landmark is \(s=6\) months. The hypothetical records are: A has an event at month 5; B is event-free through month 6 and has an event at month 9; C is censored at month 4; and D has no event and is followed through month 6.
A is ineligible because \(T_A=5\le6\): the event preceded the prediction time. C is ineligible because \(C_C=4<6\): the record does not establish that C was event-free at month 6. B is eligible because the event occurs after the landmark, and D is eligible because follow-up establishes event-free status through the landmark. Thus the landmark cohort contains B and D, not all four subjects.
If the analysis instead includes A, it uses information from a person who could not have received a prediction at month 6. If it includes C as an event-free survivor, it silently assumes that no event occurred after month 4 and before month 6. Neither assumption is supported by the data.
The event-time convention should be explicit. Under the definition above, an event at exactly \(s\) is not included among survivors because eligibility requires \(T_i>s\). Consistent handling of event dates, visit dates, and administrative censoring at \(s\) avoids ambiguous boundary cases.
Failure 2: Predictors From the Wrong Time
A landmark prediction must use only information available at or before \(s\). A biomarker measured after \(s\), a treatment decision made in response to post-landmark deterioration, or a retrospectively calculated summary that uses future visits leaks outcome-period information into the predictor. Such leakage can make validation performance look better than performance achievable at the time the prediction is supposed to be made.
A useful audit is to write the measurement time beside every candidate predictor and check that it is no later than \(s\). “Latest measurement” is not a sufficient specification: it must mean the latest eligible measurement before or at the landmark, not the latest measurement in the subject’s complete record.
Worked Example: Enforcing Predictor Timing and a Look-Back Window
At a six-month landmark, suppose a hypothetical subject has biomarker values of 10 units at month 2, 12 units at month 5, and 18 units at month 7. The analysis prespecifies a three-month look-back window, \([s-L,s]=[3,6]\), where \(L=3\) months, and uses the most recent measurement in that window.
The month-7 value is excluded because \(7>6\), even though it is the most recent value in the full record. The month-2 value is excluded because \(2<3\), so it is too old under the prespecified window. The month-5 value is the valid predictor: it is inside \([3,6]\), and the resulting predictor is 12 units.
If there were no measurement in the window, the predictor should be treated according to a prespecified missing-data rule, such as marking it missing or using a justified imputation procedure. Quietly carrying the month-2 value forward would change the measurement rule and make some subjects’ “current” values substantially older than others.
The look-back window is part of the prediction definition, not just a data-cleaning convenience. A short window may leave more predictors missing; a long window may use stale measurements. Report the window and check the distribution of time since the selected measurement. If the most recent measurements cluster near the far end of a long window, predictions may be based on information that is no longer clinically current.
Failure 3: Treating Censored Follow-Up as a Known Outcome
For an \(h\)-unit prediction horizon, a subject’s binary outcome is known if an event is observed in \((s,s+h]\), or if the subject is observed event-free through \(s+h\). If follow-up ends before \(s+h\) without an event, the horizon outcome is unknown—not a non-event. Coding every such subject as event-free usually biases estimated risk downward.
Survival methods can use censored follow-up without pretending that the unobserved part of the horizon is known. Their interpretation requires appropriate assumptions about censoring; in particular, censoring must be independent of future event time conditional on the variables used by the method, or otherwise be handled by a suitable model or weighting approach. If censoring is related to prognosis even after conditioning, naïve complete-case estimates may also be biased.
Worked Example: Classifying Outcomes at a Fixed Horizon
Consider a hypothetical prediction at \(s=6\) months for an event by \(s+h=18\) months. Subject E has an observed event at month 14. Subject F has no recorded event and is censored at month 15. Subject G has no event and is followed through month 18.
For E, the event is in \((6,18]\), so the horizon outcome is known to be an event. For F, follow-up ends before month 18 with no observed event. F’s event status by month 18 is unknown, so F must not be labelled event-free. For G, event-free observation reaches month 18, so G is known not to have had the event by the horizon.
If these were the only three eligible subjects, the simple fraction \(1/3\) would not be a justified estimate of the event risk: one of the three outcomes is unknown. The observed event in E and known non-event in G do not reveal F’s unobserved status. A survival-based analysis uses the available event and censoring times under its censoring assumptions rather than replacing F’s unknown outcome with a zero.
Always tabulate, by landmark and horizon, the numbers with an observed event by the horizon, known event-free follow-up through it, and censoring before it. A large or prognosis-dependent group censored before the horizon is a warning that risk estimates and performance measures may be unstable or biased.
Failure 4: Repeated Rows Treated as Independent Subjects
Some analyses create a row for each subject at each eligible landmark. These rows are not independent: the same subject may contribute multiple predictions, and nearby landmark records can share much of their history and follow-up. Treating each row as a new person can overstate the effective sample size and produce standard errors that are too small.
A related error occurs during validation. If rows are randomly split, one subject’s landmark record may enter the training data while another record from the same subject enters the test data. The test then evaluates predictions on subjects whose repeated records have already helped fit the model. The earlier material on subject-level validation emphasizes the appropriate unit: split subjects, not rows.
Worked Example: Keeping a Subject’s Landmarks Together
Suppose four hypothetical subjects, A through D, each contribute a row at month 3 and a row at month 6. There are eight rows but only four independent subjects. A row-wise random split might put A’s month-3 row in training and A’s month-6 row in testing. The test result for A is then not an independent assessment of performance on a new subject.
Instead, assign subjects to training or testing first, then place all rows from each subject in that subject’s assigned group. For example, if A and B are assigned to training and C and D to testing, both rows for A and B remain in training and both rows for C and D remain in testing. The split respects the actual unit that generates the correlated records.
This example has only four subjects, so it cannot support a stable performance estimate; it illustrates the split rule, not a recommended sample size. With repeated landmarks, report how many distinct subjects and how many rows are in each partition. Uncertainty calculations should also account for within-subject dependence when multiple landmark records contribute.
A Practical Failure-Mode Audit
The following checks connect the target prediction to the data actually analyzed. They do not replace a model-specific assessment, but they often reveal errors before fitting or evaluation begins.
State the landmark \(s\), horizon \(h\), event definition, and time origin. Check that every included subject is known to be event-free at \(s\).
For every predictor, verify that its source information was available no later than \(s\). Flag summaries that use visits or decisions after \(s\).
Specify the allowable interval and the rule for selecting or summarizing measurements. Count missing predictors and examine time since the selected measurement.
Separate observed events by \(s+h\), known event-free follow-up through \(s+h\), and censoring before \(s+h\). Do not convert the last category into non-events.
Keep all landmark rows from one subject in the same partition. Check subject counts as well as row counts, and account for repeated records in uncertainty summaries.
Comparisons, Diagnostics, and Limits
A fixed-landmark analysis is relatively direct when the question concerns risk among survivors at one specified time. Repeating the analysis at several landmarks can describe predictions at several decision times, but it creates correlated records and requires careful subject-level validation. A joint model is an alternative for connecting longitudinal measurements and event risk through a shared model; it has different modeling assumptions and targets. These approaches should be compared using the same prediction time, horizon, eligible population, and validation subjects if the goal is a fair predictive comparison.
Landmarking does not remove selection induced by survival to \(s\): its target is explicitly conditional on reaching \(s\). Nor does choosing a landmark automatically solve informative visit timing, measurement error, missing predictors, or informative censoring. Diagnostics should therefore include eligibility counts, predictor measurement times, missingness by landmark, follow-up completeness by horizon, and the number of unique subjects in each validation set. Implausibly strong test performance should prompt a search for timing leakage and subject overlap before being interpreted as evidence of useful prediction.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions below.
- At \(s=10\) months, a subject has an event at month 10.5. Is the subject eligible for a prediction made at month 10? Explain.
- At a 12-month landmark, a biomarker was measured at month 11 and again at month 13. Which value can be used for a prediction made at month 12?
- For a horizon ending at month 24, a subject is event-free but censored at month 22. Is the horizon outcome a non-event, an event, or unknown?
- Why is a row-wise train-test split unsuitable when each subject contributes records at several landmarks?
Solutions. (1) Yes, if event-free status is established through month 10: the event at 10.5 is after the landmark, so it contributes a future event outcome. (2) The month-11 value is eligible because it was observed by month 12; the month-13 value is future information and is excluded. (3) Unknown: follow-up stops before month 24 without an event, so event-free status through the horizon is not established. (4) Rows from the same subject can enter both partitions, allowing subject-specific information to overlap; assign subjects first and keep all their landmark rows together.