Avoiding Future-Information Leakage in Landmark Prediction

Why prediction time must govern the inputs

A landmark prediction answers a time-specific question: given what is known at a chosen time, what is the probability of occupying each state at a future horizon? The phrase what is known at the chosen time is essential. A prediction that uses a laboratory result obtained later, or information that a competing event has already occurred, is not a prediction that could have been made at the landmark. It answers a different, easier question using a larger information set.

This problem is called future-information leakage (also called look-ahead leakage). It can arise in a statistical model, in how the analysis data are assembled, or in the validation process. The calculation may be mathematically correct and still fail to represent a usable prediction. The remedy is to specify the prediction time and target first, then verify that every input was genuinely available by that time.

Definition: Let \(s\) be a landmark time, \(h>0\) a prediction horizon, and \(X(t)\) the state occupied at time \(t\). A landmark state-occupation prediction for state \(k\) has target \(P\{X(s+h)=k\mid \mathcal{H}(s)\}\), where \(\mathcal{H}(s)\) denotes the subject's history and information available for prediction at time \(s\). A predictor is eligible only if its value, or the information used to construct it, was available by \(s\). Information first observed or generated after \(s\) is future information for this prediction.

The conditioning information usually includes the current state \(X(s)\), relevant recorded history up to \(s\), and covariates measured and available by \(s\). The target is the state at \(s+h\), with the event and censoring process treated according to the analysis plan. Future follow-up is necessarily used to determine training outcomes and validation outcomes. That is not leakage by itself: future outcomes are labels against which predictions are learned or evaluated. Leakage occurs when information from that future is allowed to influence a prediction input, feature construction, model choice, or assessment in a way that would not be available at the prediction time.

A time-indexed audit

Before fitting or using a model, write down four times or time-specific objects: the landmark \(s\), the horizon \(h\), the prediction target, and the information set permitted as input. Then check each candidate variable according to when it was measured, when it became available, and what records were used to calculate it. A variable's name is not enough to establish eligibility. For example, “current treatment” could mean treatment at \(s\), which may be eligible, or treatment received at any point during the next six months, which is not.

The operational availability date matters. A blood sample taken at \(s\) but not processed or accessible until \(s+2\) days was not available to a prediction made at \(s\), unless the intended prediction is explicitly made after the result becomes available. Conversely, a value recorded in a database after \(s\) might document a measurement that was already available to the clinician at \(s\); using it is defensible only if that availability is established and matches the intended deployment setting. The audit concerns information actually available, not merely a timestamp in one data system.

Specify the prediction.
State who is being predicted for, the landmark, the horizon, and the state or event probabilities required.

Freeze the information set.
List histories and measurements available by the landmark. Include the current state and the operational availability of each proposed predictor.

Trace every input.
Inspect derived variables, missing-data procedures, feature selection, and data joins for any use of post-landmark measurements or event records.

Validate the same prediction task.
Ensure that prediction inputs are restricted to the landmark information set, while future outcomes are used only as outcomes for assessment.

Worked Example: A Laboratory Result Obtained After the Landmark

Suppose a hypothetical clinic wants, at day 90 after diagnosis, to predict each patient's state at day 180: alive without progression, alive after progression, or dead. At day 90, the patient is alive without progression. A proposed model uses age, current state, and a laboratory result measured on day 120. The day-120 result is strongly associated with progression by day 180.

Audit. The landmark is \(s=90\) days and the horizon is \(h=90\) days. The target is \(X(180)\), conditional on information available at day 90. Age and current state can be inputs if their values are available at day 90. The day-120 result cannot: \(120>90\). It may be useful for a prediction made at day 120, but that would be a different landmark prediction with a different information set.

Suppose the day-120 result is high for 18 of 20 patients who progress by day 180, and for 4 of 20 patients who do not progress. Those counts describe a post-landmark measurement's association with the future outcome; they do not justify using it in a day-90 prediction. The valid correction is to remove it and refit or reapply the model using day-90-available predictors. The resulting prediction may be less sharp, but it answers the question that could actually have been answered on day 90.

The example assumes the stated measurement dates reflect when results became available and that the intended clinical decision occurs on day 90. If results were already available then, despite being entered later, the assumption would need to be reconsidered and documented.

Why leakage changes the conditional prediction

The justification is direct. Let \(\mathcal{H}(s)\) be the information available at the landmark, and let \(Z\) be a measurement first available after \(s\). The intended probability conditions on \(\mathcal{H}(s)\). A model using \(Z\) instead conditions on the larger information set that includes \(Z\). In general, these are not the same conditional probability.

P\{X(s+h)=k\mid \mathcal{H}(s)\} \quad\ne\quad P\{X(s+h)=k\mid \mathcal{H}(s),Z\}.

The second probability may be useful for a later decision made after observing \(Z\), but it cannot be reported as the first. To see why, average the second probability over the possible values of \(Z\), conditional on the landmark information:

P\{X(s+h)=k\mid \mathcal{H}(s)\} = \sum_z P\{X(s+h)=k\mid \mathcal{H}(s),Z=z\} P(Z=z\mid\mathcal{H}(s)),

for discrete \(Z\), with the corresponding integral for continuous \(Z\). A prediction that uses the observed future value \(Z\) gives one conditional probability from this collection; it does not give the average that was available at \(s\). The equality can hold in special cases, such as when \(Z\) adds no information about the future state given \(\mathcal{H}(s)\), but that does not make \(Z\) an available predictor. Eligibility is a time-and-use requirement, not a test of whether the variable improves prediction.

Worked Example: An Event Indicator That Reveals the Outcome

Consider a hypothetical prediction at day 30 of whether a patient will have entered the death state by day 90. A data analyst joins each patient's day-90 status to the day-30 predictor table and adds “dead by day 90” as a predictor. In a sample of 50 patients, 7 are dead by day 90 and 43 are alive. The added predictor is therefore 1 for exactly the 7 patients with the outcome and 0 for the other 43.

If the analyst uses this indicator to predict death by day 90, the apparent classification is perfect: all 7 deaths are flagged and none of the 43 survivors are flagged. That result is tautological, not evidence of useful day-30 prediction. The indicator is the future outcome itself, not day-30 information.

The correction is to remove the day-90 status from the predictor set. It remains the outcome label used to determine whether a day-30 prediction was correct. The example assumes the status is known at day 90 and correctly determines the target; it does not imply that an outcome label should be discarded from model development or validation.

Leakage beyond obvious future measurements

Some sources of leakage are less visible than a future laboratory value. Review the entire pipeline, not just the final model formula. Common risks include:

Future event history: a record of progression, death, or a transition after \(s\) is joined to the landmark predictors or used to define a predictor.
Summaries that cross the landmark: “ever treated,” maximum laboratory value, or number of visits may include observations after \(s\), even if the summary has no explicit future date in its name.
Information from later visits: a value carried backward from a later record, or a change score calculated using a post-landmark value, is not available at the earlier landmark.
Preprocessing informed by held-out data: imputation, scaling, feature selection, or other data-adaptive steps can leak information if they use validation outcomes or information unavailable in the intended prediction setting.
Selection based on future survival or follow-up: restricting the prediction sample to people known to remain observed or event-free beyond \(s\) changes who is being predicted for and may use information unavailable at \(s\).

Some exclusions are legitimate because the prediction target is defined among people in a particular state at \(s\). For example, requiring a subject to be alive at \(s\) is part of defining the risk set for a prediction conditional on being alive then. The problem is selecting or classifying people using what happens after \(s\) while presenting results as if the prediction had been made using only information at \(s\). State the eligibility rule in terms of information available at the landmark.

Worked Example: Rebuilding a Prediction With Eligible Inputs

Suppose a hypothetical model at day 30 predicts day-90 state occupation for a patient currently in a transient illness state. Its proposed inputs are current state, age, a symptom score measured on day 30, and whether the patient received rescue treatment by day 60. The first three are available by day 30; rescue treatment by day 60 is not. After removing that input, the fitted day-30 model gives probabilities \(0.55\) for remaining in the transient state, \(0.30\) for entering a recovery state, and \(0.15\) for entering an absorbing event state.

Check the probability vector. The values are nonnegative and sum to one: \(0.55+0.30+0.15=1.00\). They therefore form a coherent set of state-occupation probabilities for this example. For this patient, the model's day-30 prediction is a 55% probability of the transient state, 30% of recovery, and 15% of the absorbing event state at day 90.

These numbers are hypothetical model outputs, not estimates from a real study. The audit establishes that the inputs are time-eligible; it does not establish that the model is well calibrated or otherwise accurate. Those properties require appropriate evaluation using outcomes observed after day 30, with the prediction inputs still restricted to day-30 information.

Diagnostics, alternatives, and limitations

A practical diagnostic is to create a variable-level audit table with each input's definition, measurement time, availability time, and source records. Flag any variable whose construction touches post-landmark data. Then rerun the complete pipeline with only eligible inputs. Compare the resulting predictions and validation summaries with the original as a sensitivity check. A dramatic deterioration after removing future information is a warning about how much the original analysis depended on look-ahead; it is not a reason to retain the leaked variables.

A second diagnostic is to inspect data joins and transformations. Check whether one row represents information as of the landmark or whether later rows have been merged into it. Confirm that any “last observed,” “ever,” or “change from baseline” variable uses only records dated on or before \(s\). Verify that model selection and preprocessing are performed without access to held-out outcome information. When deployment would involve repeated updates, each update should have its own clearly defined landmark and information cutoff. Using a measurement after one landmark may be valid for a prediction made at a later landmark, but it does not retroactively make that measurement available earlier.

The main alternative to a leaked prediction is not to ignore all future data. Future outcomes are needed as training or assessment labels. Instead, preserve the time ordering: construct predictors using information available at \(s\), and use later follow-up only to identify the state at \(s+h\) or otherwise assess the prediction target. In transition-based prediction, this means the history and covariates supplied for the landmark calculation must match the landmark information set. Other model forms do not remove the requirement; any prediction intended for time \(s\) must obey the same information boundary.

This audit cannot by itself fix every problem in dynamic prediction. It does not establish that the landmark sample represents the intended population, that outcomes are measured correctly, or that censoring and missingness have been handled appropriately. It also cannot make an unavailable predictor usable merely because it is strongly associated with outcome. A later landmark can support a legitimate updated prediction when its inputs are then available, but the question and prediction time have changed.

Key takeaway: For a landmark prediction at \(s\), every predictor and every step that constructs it must use only information available by \(s\). Later observations may define the outcome used for learning or evaluation, but must not leak into the prediction inputs.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions.

At day 60, a model predicts state at day 120 using a laboratory result measured and returned on day 75. Is this eligible as a day-60 predictor? What would make it eligible for a different prediction?
A future event indicator is used as a predictor, while the same event by the horizon is the outcome. Explain why this can produce misleadingly strong apparent performance.
A patient's current state and treatment at the landmark are used to predict a state at a later horizon. What timing condition must be checked for the treatment variable?
Why may future follow-up be used to assess a landmark prediction without constituting future-information leakage?
A data set includes only people who are known to be alive at the end of the prediction horizon. What is the concern if the stated prediction is intended for people eligible at the landmark?

Solutions. (1) No. The result was not available until day 75, after the day-60 prediction. It could be eligible for a prediction made at or after day 75 if it is then available and the target is defined from that later prediction time. (2) The predictor contains the outcome information itself, so the model can reproduce the label rather than forecast it from landmark information; apparent performance does not represent prediction at the landmark. (3) Confirm that the treatment value describes treatment known by the landmark, not treatment initiated or updated afterward. (4) Future follow-up supplies the outcome label needed to determine the state at the horizon; leakage arises only if that information influences the predictors or prediction process as though it were known at the landmark. (5) Survival to the horizon is future information. Restricting to those known to survive can select the sample using what happens after the prediction time, so the analysis no longer evaluates the intended landmark-eligible group.