Landmark Validation with Subject-Level Data Splits

Why the Split Unit Matters

A landmark data set may contain several records for one patient: one record for each eligible landmark time. Each record uses information available by that landmark to predict a later event outcome. If records from the same patient are placed in both the training and validation sets, the validation set is no longer a clean assessment on patients unseen during model fitting. The model may benefit from patient-specific patterns already represented in training, and the resulting performance estimate can be optimistic for deployment to new patients.

The essential design rule is simple: assign patients to partitions first, then carry all of each patient’s landmark records into that partition. The number of rows is not the number of independent patients. Keeping this distinction explicit is especially important when patients contribute different numbers of eligible landmarks.

For example, write a record as \((i,k)\), where \(i\) identifies the patient and \(k\) identifies that patient’s landmark occasion. A record can contain predictors measured by landmark time \(s_k\) and an outcome defined over a later prediction horizon. The split is based on \(i\), not on \((i,k)\). The landmark time, event status, and number of rows do not change which patient owns a record.

What the Validation Design Targets

A subject-level split is appropriate when the intended use is prediction for patients who were not used to fit the model. It estimates performance on held-out patients sampled from a population resembling the development patients, subject to the split design and the chosen performance measure. It does not automatically estimate performance at a different hospital, in a later calendar period, or under a changed clinical process.

It is also important to define what a “typical prediction” means. If one patient contributes five landmark rows and another contributes one, a summary that gives equal weight to every row gives the first patient five times as much influence. A summary that first averages within each patient and then across patients targets a different weighting. Subject-level splitting prevents cross-partition leakage, but it does not decide this weighting question. Choose and report the target that matches the intended use.

The split must also respect how predictions will be made. If deployment concerns new patients, a row-wise split is not a substitute for a patient-wise split. If the aim is to predict future records for patients already observed, a chronological split may instead be relevant; that answers a different question and must ensure that later information cannot enter earlier predictions.

Why Row-Wise Splitting Leaks Patient Information

A random split of rows treats repeated records as if they were separate units for allocation. But records from one patient are typically related: they may share baseline characteristics, biomarker trajectories, measurement patterns, and event history. A model trained on one landmark row can therefore have an advantage when predicting a different row from the same person. This is a dependence and information-overlap problem, even when the outcome for the validation row was not directly used during fitting.

A simple calculation shows how readily this can happen. Suppose each row is independently assigned to training with probability \(q\), and a patient contributes \(m\) rows. The probability all of that patient’s rows land in training is \(q^m\); the probability all land in validation is \((1-q)^m\). Therefore, the probability the patient appears in both partitions is

The expression follows because the only ways a patient avoids being split across partitions are for all \(m\) rows to be in training or for all \(m\) rows to be in validation. Those two events are disjoint, so subtract their probabilities from one. Under subject-level allocation, by contrast, the probability of a patient appearing in both partitions is zero by design.

Practical Design for a Holdout Split

A defensible holdout workflow begins with the patient identifiers, not with a randomization of the landmark table’s rows. Create a unique list of patients, allocate those identifiers to training and validation, and join the allocation back to every landmark record. Check that the sets of patient identifiers are disjoint and that all eligible records for each assigned patient have been handled consistently.

These steps address distinct leakage routes. A perfect patient split can still be undermined if a biomarker summary uses measurements taken after the landmark, if validation data help select predictors, or if imputation and scaling are estimated using both partitions. Conversely, fitting preprocessing within training folds does not repair a split that has already placed the same patients on both sides.

Diagnostics, Alternatives, and Limitations

After creating a split, audit it rather than relying on the code’s intended behavior. Confirm that no identifier occurs in both partitions; count unique patients separately from rows; inspect the number of landmark records per patient in each partition; and compare the distribution of landmark times and key patient-level characteristics. If relevant outcomes are sparse, inspect their patient-level counts as well. Large imbalances can make an estimate unstable, though balancing cannot guarantee representativeness.

Common alternatives serve different purposes. A random row split is easy but invalid for new-patient validation when patients repeat. Grouped cross-validation reuses patients across training and validation roles across folds while ensuring that, within each fold, a patient is held out as a whole. A temporal split evaluates a later period and can reflect changes over time, but requires a clear time ordering and protection against future information. External validation uses a distinct data source or setting and addresses transport to that setting; a random subject split from one cohort does not provide this evidence.

A holdout split sacrifices some data for model fitting, while repeated grouped cross-validation uses data more efficiently but can have substantial uncertainty and does not constitute independent external validation. In either design, censoring and incomplete follow-up still affect whether a landmark outcome is observed. Grouping prevents patient overlap; it does not by itself correct outcome-label uncertainty, informative missingness, or a mismatch between the validation cohort and intended users.

When patients contribute multiple eligible landmarks, specify whether the evaluation target is a randomly selected landmark record or a randomly selected patient. The first gives more weight to patients with more rows; the second requires patient-level aggregation or weighting. Also specify how a patient with several validation landmarks contributes predictions. Those choices affect the interpretation of the resulting summary and should not be left to the data layout alone.

Common Mistakes

• Splitting the landmark table by row. The mistaken reasoning is that a large number of rows guarantees a large validation sample. The correction is to count independent patients and assign each patient as a group.
• Assuming grouped splitting eliminates every form of leakage. It prevents a patient’s rows from crossing partitions, but future measurements, global preprocessing, or validation-guided model selection can still leak information.
• Balancing every row instead of patients. A patient with many rows can dominate row-based counts. Check event and covariate balance at the patient level, while preserving all records for each patient.
• Calling random holdout validation external validation. A held-out subset from the same source can assess performance on unseen patients from a similar source; it does not establish performance in a new health system or time period.
• Reporting the number of rows as the validation sample size. Report both the number of unique patients and the number of landmark records, because they describe different aspects of the evaluation data.