Why Landmark Choice Is a Support Question
A landmark prediction describes future state occupation for a cohort defined at a particular time. A landmark that is too late may leave few eligible individuals; one that is too early may not match the clinical decision point. Even when the landmark is substantively appropriate, the data may not support predictions at the intended follow-up horizons. The practical question is therefore not simply how many people are in the cohort, but whether the available observations contain enough information about follow-up, censoring, and relevant state transitions to estimate the probabilities with useful precision.
This tutorial focuses on diagnosing that support. It does not supply a universal minimum cohort size: the required information depends on the states, transitions, censoring pattern, intended precision, and prediction method. The earlier material on landmark state-occupation prediction and bootstrap intervals provides the prediction and uncertainty framework used here.
The eligibility rule must be clear. For example, a cohort could include all individuals alive and under observation at \(s\), or only those in a specified transient state at \(s\). These define different target populations and may have very different sizes and state compositions. Record the number eligible, the numbers in each starting state, and any exclusions before assessing later follow-up.
What to Examine Before Choosing a Landmark
Assess support at the landmark and after it. At the landmark, inspect the total eligible sample size and the counts in each state from which predictions will be made. After the landmark, examine how much follow-up is observed through each intended horizon, when censoring occurs, and how many individuals remain at risk for the transitions that contribute to the predictions. A total cohort count can conceal weak support for a particular state or transition.
- Landmark cohort size and composition: Count eligible individuals overall and by starting state or prediction-relevant subgroup. Small groups can produce unstable subgroup predictions even if the full cohort is large.
- Follow-up coverage: For each intended horizon, summarize how many individuals have observation extending to that time, and when losses to follow-up or administrative censoring occur. A horizon beyond the data’s observed follow-up range is not empirically supported without additional model assumptions.
- Transition-specific information: For each important transition, examine the number of individuals at risk over time and the number of observed transitions. A large number of participants cannot compensate for almost no observed transitions from a rare state.
- Uncertainty: Examine interval widths or other suitable uncertainty summaries for the state probabilities at the intended horizons. Uncertainty should be evaluated for the predictions that matter, not just for an overall event count.
- Data quality and censoring: Check that state changes and censoring times are recorded consistently. Estimation procedures generally require censoring to be independent of future state evolution, at least conditional on the variables accounted for by the method.
The last condition is substantive. If individuals with worsening health are more likely to leave observation, and the prediction method does not account for relevant predictors of that censoring, observed follow-up may be unrepresentative. A large sample does not remove bias from informative censoring. Likewise, using an estimator that handles censoring does not make predictions reliable when only a few individuals remain under observation in the parts of follow-up that determine them.
Worked Example: Comparing Two Candidate Landmarks
Hypothetical data. Investigators want predictions two years after a landmark for people in either of two eligible states. They consider landmarks at 12 and 18 months after enrollment. At 12 months, 180 people are eligible: 120 are in state A and 60 in state B. At 18 months, 92 are eligible: 64 are in state A and 28 in state B. For the 12-month landmark, 147 people have observed follow-up through two years after that landmark; for the 18-month landmark, 53 do. The intended clinical decision is made at 12 months.
The cohort-size comparison is direct: \(180\) are eligible at 12 months versus \(92\) at 18 months. The 12-month landmark also has more people in each starting state. Follow-up coverage is \(147/180=0.817\), or 81.7%, at the 12-month landmark, compared with \(53/92=0.576\), or 57.6%, at the 18-month landmark.
These ratios are descriptive, not estimates of the prediction probabilities. They show that more of the 12-month cohort is observed through the target time, but do not by themselves establish adequate support: the transitions and censoring patterns still need inspection. Since the intended decision occurs at 12 months, moving the landmark to 18 months would also change the target population and decision time. On these data, the 12-month landmark is the more plausible candidate, subject to checking state-specific risk sets, transitions, and prediction uncertainty.
Why Sparse Support Inflates Uncertainty
A simple calculation shows why the number of independent observations matters. If a state indicator at a fixed horizon were observed for every person in a cohort of size \(n_s\), the estimated probability of occupying state \(k\) would be a sample proportion. Under independent sampling, its variance is \(p_k(1-p_k)/n_s\), where \(p_k\) is the true probability. The corresponding standard error decreases in proportion to \(1/\sqrt{n_s}\): with other features unchanged, quadrupling the sample size halves this standard error.
In multi-state data, state occupation at a horizon is usually estimated using event histories, not a fully observed binary outcome for everyone. The Aalen–Johansen estimator, for example, combines transition information over time. For a transition from state \(r\), the observed transition count at a time is compared with the number at risk in state \(r\). When that risk set is small, a few transitions can have substantial influence on the estimated transition probability. Uncertainty in those transitions can then affect downstream state-occupation probabilities.
This explains why the total number enrolled is an incomplete support measure. The relevant information depends on how many individuals occupy each origin state and remain observable while transitions from that state could occur. It also explains why a prediction may be stable for common states but unstable for a rare state, or stable at an early horizon but imprecise later.
Worked Example: A Precision Check at a Fixed Horizon
Hypothetical data. At a proposed landmark, 100 eligible individuals have complete observation at one target horizon. The estimated probability of occupying state B is \(0.20\). For a rough precision check, use the binomial standard error for a fully observed state indicator.
Substituting the estimate and sample size gives \(SE=\sqrt{0.20(1-0.20)/100}=\sqrt{0.0016}=0.0400\). A rough normal-approximation interval is \(0.20\pm1.96(0.0400)\), or \(0.20\pm0.0784\), giving endpoints 0.122 and 0.278 after rounding. This interval is about 15.7 percentage points wide.
Suppose the investigators had prespecified that a width no greater than 12 percentage points would be useful for this particular decision. The rough interval does not meet that goal, so the landmark lacks the desired precision under this simplified calculation. The calculation assumes independent observations and complete horizon outcomes; it is a diagnostic approximation, not a substitute for an appropriate multi-state uncertainty analysis when censoring and transition histories are involved.
Using Uncertainty and Diagnosing Failure
When candidate landmarks are substantively acceptable, uncertainty estimates help assess whether their support is adequate for the intended use. Bootstrap intervals can reflect sampling variation in the prediction procedure, provided the resampling scheme matches the sampling unit and design. Inspect interval widths for each important state and horizon, and look for estimates that change markedly under modest changes to the data or model. A very wide interval, a boundary estimate driven by only a few observations, or frequent bootstrap replicates with poorly supported transitions are warnings, even if a formal procedure returns a number.
Do not interpret a particular interval-width threshold as universal. A difference of five percentage points may matter greatly for one decision and little for another. Set a precision goal in the outcome’s units and in context, then check whether the data meet it. If the intended horizon falls near the end of follow-up, report how much information remains there and treat unstable tail estimates with caution.
Worked Example: A Large Cohort With a Rare Transition
Hypothetical data. A landmark cohort contains 240 individuals. During follow-up, 180 are observed in state A, 54 in state B, and only 6 in state C. The prediction requires estimating the probability of moving from state C to an absorbing event state within one year. Only 3 of the 6 individuals in state C have enough follow-up for a year, and 1 transition to the absorbing state is observed.
The total cohort size, 240, sounds substantial, but it is not the relevant denominator for the transition from C: only 6 individuals ever occupy C in the observed cohort, and only 3 have a full year of follow-up from there. The single observed transition provides little information about the transition probability. The resulting prediction may be highly sensitive to that one history, and uncertainty should be expected to be large.
The appropriate conclusion is not that the transition probability is zero or that the prediction is necessarily unusable in every context. Rather, this data set provides weak empirical support for a precise one-year prediction from state C. A different model might produce a numerical estimate, but any smoothing or extrapolation would rely on added assumptions that should be made explicit.
Alternatives, Limitations, and Common Pitfalls
A landmark with weak support should not be rescued by silently changing the analysis. Pooling adjacent landmarks may increase the number of observations, but it can combine people with different landmark times, histories, or target populations. It is appropriate only if the pooled target and the assumptions that make observations comparable are scientifically defensible. A transition-specific parametric model can estimate probabilities beyond sparse observed regions, but that is model-based extrapolation, not new empirical support. State-occupation predictions based on transition models also depend on their modeling assumptions.
Other limitations remain even after careful diagnostics. Support summaries cannot prove that censoring is independent, that recorded states are correct, or that the target cohort represents the population of interest. Small subgroups may remain uncertain even when the overall prediction is precise. Moreover, selecting a landmark after trying many candidates can make reported performance and uncertainty overly optimistic if the selection process is ignored. Prefer landmarks justified by the clinical question and specified before examining which one produces the most favorable predictions.
Common mistake: “There are hundreds of participants at this landmark, so predictions at every later time must be reliable.” This reasoning overlooks state composition, censoring, and sparse transition-specific risk sets. The correction is to evaluate support separately for each relevant state and intended horizon, then inspect uncertainty and the assumptions required by the estimator.
A second mistake is to treat the number still observed at the target time as the only measure of support. Individuals censored before that time may still contribute information before censoring, while many recorded observations at the target time do not guarantee enough information about a rare transition. Use follow-up coverage as one diagnostic alongside the full risk-set and transition history.
A Practical Assessment Sequence
Define eligibility at landmark \(s\), the starting-state groups, and the prediction horizons. Keep the landmark tied to the intended decision or scientific question.
Report the total eligible count and counts by state or subgroup. Flag groups too small for the intended level of precision.
Summarize follow-up through each horizon, censoring patterns, risk sets, and observed transitions relevant to the predictions.
Use uncertainty summaries appropriate to the estimator. Check censoring, state recording, and model assumptions, especially where support is weakest.
Choose a landmark that meets the scientific objective and has adequate support for the declared precision goal. If none does, narrow the target, collect more data, or state the limitations rather than presenting unsupported precision.
Check Your Understanding
Answer each question, then compare your reasoning with the brief solutions.
- Why can a cohort of 300 individuals still provide weak support for a transition-specific prediction?
- At a landmark, 80 of 100 eligible people are observed through the intended horizon. What does the 80% figure tell you, and what does it not establish?
- Why might moving the landmark later to gain a more relevant clinical time point reduce prediction support?
- What is the consequence of pooling adjacent landmarks merely to increase sample size?
- For a rare-state prediction, which is more informative: the total cohort count or the number at risk in that state and observed transitions from it? Explain.
Solutions. (1) The relevant origin state may contain few people, or censoring may leave few at risk when the transition can occur; the overall count does not measure transition-specific information. (2) It describes observed horizon coverage. It does not establish that censoring is non-informative, that risk sets are adequate, or that the prediction is precise. (3) Fewer people may remain eligible at a later landmark, and less post-landmark follow-up may be available. (4) Pooling can change the target population and requires assumptions that make the landmark cohorts comparable; larger \(n\) alone does not justify it. (5) The state-specific risk set and transition count are more directly informative because they determine the evidence for that transition; the total count can conceal sparse support.