Why Leakage Matters for Dynamic Prediction
A validation estimate is useful only if it reflects how predictions will be made in practice. In a joint model, a prediction at time \(s\) may use a subject’s longitudinal biomarker history up to \(s\), together with survival information that the subject has remained event-free through \(s\). If a validation analysis quietly uses a biomarker measured after \(s\), or lets held-out data influence model fitting or selection, it evaluates a prediction task with more information than would actually be available.
This problem is called information leakage. It can make predictions appear more accurate, better calibrated, or more useful than they would be at deployment. The key safeguard is to define exactly what is known at prediction time, then ensure that every step producing a prediction uses only that information and the training data permitted by the validation design.
“Available by \(s\)” is operational, not merely a timestamp in a database. A sample collected at \(s\) but assayed and reported two days later cannot support a prediction made at \(s\), unless the intended prediction time is actually after the result becomes available. Likewise, a measurement recorded later but backdated, a future event indicator, or a summary calculated using follow-up after \(s\) is not part of the prediction-time information set.
Leakage From Future Measurements
A common error is to construct each subject’s prediction record using the complete longitudinal trajectory, then label the record as a prediction at an earlier landmark. For example, a marker measured at \(s+3\) may be strongly associated with an event in \((s,s+w]\), but it cannot be used to predict that event from time \(s\). This remains leakage even if the future value is used only to calculate a slope, average, maximum, or “last observed” value.
The subject must also be alive and event-free at the prediction time for the conditional target above. Event status after \(s\) is an outcome for validation, not a predictor available at \(s\). It is legitimate to use held-out follow-up to determine whether a prediction was accurate; it is not legitimate to use that follow-up to alter the prediction.
Worked Example: A Future Biomarker Changes the Risk Prediction
Suppose a hypothetical frozen prediction rule uses a standardized marker \(z\) available at the landmark and returns risk \(r(z)=1/\{1+\exp[-(-2+0.8z)]\}\) over a specified horizon. At \(s\), a hypothetical subject’s last available marker has \(z=1\). A later measurement, obtained after \(s\), has \(z=3\). The intended prediction uses only the value available at \(s\).
Using the valid value gives \(r(1)=1/\{1+\exp[-(-2+0.8(1))]\}=1/\{1+\exp(1.2)\}\approx 0.2315\). Thus the predicted event risk is about 23.2% for the stated horizon.
If the later value is substituted, the calculation becomes \(r(3)=1/\{1+\exp[-(-2+0.8(3))]\}=1/\{1+\exp(-0.4)\}\approx 0.5987\). That is about 59.9%, but it is not a valid prediction at \(s\): the value \(z=3\) was not yet known. This example checks the timing assumption explicitly—the first value is available by the landmark, and the second is not. The large difference illustrates how future information can distort apparent performance.
Preprocessing Is Part of Model Fitting
Leakage can occur before the joint model is fitted. Imputation, centering and scaling, variable screening, spline-knot selection, batch correction, and construction of data-driven cut points all learn something from data. In validation, these operations must be learned using the training portion of each split and then applied unchanged to its held-out portion. A held-out subject’s predictors must not influence the fitted preprocessing parameters in a standard inductive validation design.
The same principle applies within cross-validation: preprocessing is repeated inside each training fold, not once on the complete data before folds are created. Otherwise information from a held-out fold can affect the representation supplied to the model. There are settings in which unlabeled deployment covariates are intentionally available for transductive preprocessing; that is a different prediction scenario and must be specified and reproduced in validation.
Worked Example: Scaling With Training Data Only
Consider hypothetical training marker values \(0,2,4\), and a held-out value of \(10\). Suppose scaling uses the training mean and population standard deviation. The training mean is \((0+2+4)/3=2\), and the training standard deviation is \(\sqrt{[(0-2)^2+(2-2)^2+(4-2)^2]/3}=\sqrt{8/3}\approx1.633\).
The held-out value is therefore transformed using the training scaler as \(z=(10-2)/1.633\approx4.899\). This is the appropriate transformation for a new subject under this fitted preprocessing rule.
If the held-out value is included when estimating the scaler, the combined mean becomes \((0+2+4+10)/4=4\), and the combined population standard deviation is \(\sqrt{[(-4)^2+(-2)^2+0^2+6^2]/4}=\sqrt{14}\approx3.742\). The held-out value then becomes \(z=(10-4)/3.742\approx1.604\). The held-out observation has altered its own transformation. This example checks the intended assumption: the model is to be applied to new subjects using preprocessing learned from training data, so the training scaler—not the combined-data scaler—must be used.
Model Selection Must Be Separated From Evaluation
Leakage also arises when the validation results guide the choice of model and the same results are then reported as an unbiased assessment of that chosen model. Choices include association structure, marker functional form, covariate set, prediction horizon handling, tuning parameters, and thresholds. Selecting the best-looking candidate exploits random variation in the validation results; the selected estimate is generally optimistic.
A common remedy is nested validation. Within each outer training sample, use inner resampling to select the model and any tuning choices. Refit the selected procedure using the outer training sample, then predict the untouched outer test sample. Aggregate the outer test performance to evaluate the complete selection-and-fitting procedure. An independently reserved test set is another option, provided it is not used to revise the model before the reported evaluation.
Worked Example: Reusing a Selection Set Gives an Optimistic Result
Suppose three-fold inner validation on hypothetical training data gives mean Brier scores of \(0.190\) for candidate A and \(0.180\) for candidate B; smaller Brier scores indicate better probabilistic accuracy. Candidate B is selected because \(0.180<0.190\). Reporting \(0.180\) as the final validation performance reuses the data that selected B, so it does not provide an independent estimate of B’s performance.
In a nested analysis, the selected candidate is refitted on the outer training data and evaluated on an untouched outer test fold. Suppose its hypothetical outer test Brier score is \(0.220\). The arithmetic comparison is \(0.220-0.180=0.040\): performance on the outer test fold is worse by 0.040 Brier-score units than the selected inner estimate. The outer score, not the winning inner score, contributes to evaluation of the model-selection procedure. This example assumes the outer fold was not used to choose candidate B, tune it, or revise the analysis.
A Leakage-Resistant Validation Workflow
A reliable workflow makes the prediction time, unit of splitting, and boundary between fitting and evaluation explicit. In dynamic prediction, several landmark records may belong to one person. If records from the same person appear in both training and test data, the model may effectively learn that person’s characteristics from one record and be tested on another. For validation of predictions to new subjects, split by subject, keeping all of each subject’s landmark records in the same partition.
State the eligible population, landmark time, prediction horizon, available covariates and biomarker history, and the target outcome. Define whether the intended use is prediction for new subjects, new centers, or later calendar time.
For new-subject prediction, assign subjects—not individual landmark rows—to partitions. For transport to new centers or future periods, make the split reflect that target rather than relying only on random subject splits.
Estimate preprocessing parameters and select variables, model structure, and tuning choices using training data alone. Repeat these operations within inner folds when using nested validation.
For each test subject and landmark, use only measurements and other inputs available by that prediction time. Apply the training-fitted preprocessing and model without adapting them to that subject’s future information.
Use appropriate observed follow-up to assess predictions, accounting for censoring as required by the chosen performance measure. Do not use test outcomes to alter predictions, select the reported model, or choose a threshold whose performance is then reported on the same test data.
Diagnostics, Alternatives, and Limitations
Useful leakage checks follow the information flow. For every predictor, ask when it is measured, when it becomes available, and whether any summary uses observations after the landmark. Verify that every data-adaptive transformation is fitted separately within training partitions. Confirm that repeated records from one subject do not cross a subject-level split, and maintain a record of which data informed each model or decision. Suspiciously strong performance, particularly when it collapses under a temporal or center-based split, is a warning—not proof—of leakage or a mismatch between validation and deployment.
The main alternatives differ in the prediction setting they represent. A random subject split can estimate performance for new subjects from a similar source population. A center-held-out or temporal split may be more relevant when deployment is at new centers or later times, but can also reflect genuine population or practice changes. Nested resampling protects evaluation from model-selection reuse; a single untouched external test set can serve a similar role. Neither strategy corrects a wrongly defined prediction-time information set: a future marker remains unavailable regardless of how carefully subjects were split.
Leakage prevention does not remove all uncertainty. Small samples can make validation estimates imprecise, censoring can limit which outcomes are observed, and shifts between development and deployment populations can reduce performance. Also, the choice of split determines the question answered; a design suited to new subjects may not estimate performance after a major change in clinical practice. Evaluation methods may legitimately use held-out follow-up to determine outcomes or estimate censoring adjustments, but those operations must not feed back into prediction or model choice. The distinction is between using outcomes to score a prediction and using them to create or revise it.
Check Your Understanding
Answer each question, then compare with the brief solutions.
- A prediction is made at month 6 using a marker value measured at month 9. Is the prediction time-valid? Explain.
- In a subject-level cross-validation fold, marker scaling is calculated once using the full dataset before splitting. What is wrong, and what should be done instead?
- Five landmark records from one subject are divided between training and test data. Why can that invalidate validation for new-subject prediction?
- Candidate models are ranked using an outer test fold, and the winning model’s performance on that fold is reported. Is the outer-fold result an independent evaluation?
- May held-out event follow-up be used to assess a predicted risk without being information leakage?
Solutions. 1. No. The month-9 marker was not available at month 6; use only the history available by month 6. 2. The test observations influence the transformation. Estimate scaling within each training fold and apply those estimates unchanged to its test fold. 3. The training records can reveal subject-specific information that is then present in the test records, so the test set no longer represents entirely new subjects. Keep all records for a subject in one partition. 4. No. The test fold helped select the winner, so its score is reused for selection and evaluation; use inner selection within training data and an untouched outer fold for evaluation. 5. Yes. Follow-up is needed to determine the validation outcome, provided it is used to score the prediction and not to alter the prediction or select the reported model.