Tutorials › Joint Modeling of Longitudinal and Time-to-Event Data › Defining the Prediction Target for Validation

Calibration and Prediction Validation · Tutorial 547 of 697

Defining the Prediction Target for Validation

Learn to specify a conditional survival prediction target precisely and distinguish it from baseline survival, a different landmark, or risk of a competing event.

Advanced 9 min read

What You'll Learn

  • Define who belongs to the target population at a prediction origin.
  • Write the conditional survival estimand using the landmark and a forward time horizon.
  • Distinguish survival probability from event risk and cause-specific cumulative incidence.
  • Check that the event definition, prediction information, and evaluation target align.
  • Identify how changing the population or prediction origin changes the target.

Why the Target Must Come First

A numerical prediction has no unique interpretation until its target is stated. For example, a predicted survival probability of 0.80 might refer to survival for one year from diagnosis, survival for one year from a later clinical visit, or survival in a particular eligible patient population. Those are different quantities, even if the model produces the same number.

In dynamic prediction, a joint model uses a person’s longitudinal marker history, together with other available information, to predict survival from a specified prediction origin. Before evaluating such predictions, define four things: the target population, the prediction origin, the event of interest, and the time horizon. These choices define what the model’s prediction is supposed to estimate; they are not details that can be repaired by choosing a different evaluation statistic.

Definition: Let \(T_i\) be the time of the specified event for person \(i\), and let \(\mathcal H_i(t_0)\) denote that person’s longitudinal history available by prediction origin \(t_0\). Let \(X_i\) denote other specified covariates, and let \(h>0\) be a forward time horizon. For a person who is event-free at \(t_0\), the conditional survival target is $$ S_i(t_0,h)=P\!\left(T_i>t_0+h\mid T_i>t_0,\mathcal H_i(t_0),X_i\right). $$ The corresponding conditional event risk over the horizon is \(1-S_i(t_0,h)\). The target population is the population of eligible people, and eligible histories and covariates, to which these conditional predictions are intended to apply.

The formula describes survival for h time units after \(t_0\), conditional on being event-free at \(t_0\) and on the information specified in the conditioning set. The origin and horizon therefore work together: \(t_0+h\) is the end of the prediction window on the study’s time scale. A prediction of 0.80 is not interpretable as a probability of surviving “one year” unless the origin, the one-year duration, and the event definition are all clear.

A conditional prediction is individual-specific when it depends on that person’s marker history and covariates. It is not, by itself, an average survival probability for the target population. If a population-level summary is also wanted, the population over which individual risks are averaged must be stated. For example, the mean predicted risk is an average over the eligible people in the specified target population, not over an unspecified collection of records.

Specify the Four Parts of the Target

Target population. State who the prediction is for, including relevant eligibility conditions. In a landmark setting, eligibility commonly requires being alive and free of the event at \(t_0\). Also state the setting the predictions are intended to represent. Predictions for patients meeting a specialist-clinic eligibility rule need not have the same target as predictions for all people diagnosed in a region. The target population includes the distribution of marker histories and covariates among eligible people, not only a demographic label.

Prediction origin. State the time at which the prediction is made and the time scale used to express it. The history \(\mathcal H_i(t_0)\) must contain only measurements and information available by that origin. A marker value measured after \(t_0\) cannot be used to represent information available when a prediction is made at \(t_0\). Earlier material on dynamic prediction explains how the longitudinal history enters the prediction; the target definition makes explicit the point in that history at which the prediction is formed.

Event of interest. Define the endpoint operationally. “Failure” or “progression” is not enough unless the event criteria, and whether the endpoint is a first event or a composite, are clear. If the endpoint is death from any cause, then any death is the event. If the endpoint is a particular cause of death and other causes can occur first, then those other causes are competing events. The probability of the target cause is not generally the complement of event-free survival.

Time horizon. State the duration \(h\) measured forward from \(t_0\), with units. A 12-month horizon beginning at month 6 ends at month 18 on the original time scale. It is not the same target as 12 months beginning at diagnosis. Where the prediction is reported as survival, the outcome is whether the specified event has not occurred by \(t_0+h\), among those event-free at \(t_0\).

Worked Example: Translating a Prediction Into Its Target

Suppose a hypothetical clinic evaluates predictions for adults who are alive and have not experienced the defined clinical event at a follow-up visit 6 months after diagnosis. The endpoint is first occurrence of that event, the horizon is 12 months from the visit, and a joint model predicts conditional survival of 0.82 for one eligible patient.

The target population is adults meeting the clinic’s eligibility criteria and event-free at the 6-month visit. The prediction origin is month 6 after diagnosis. The endpoint is the specified first clinical event. The horizon is 12 months after the visit, ending at month 18 after diagnosis.

The model’s predicted conditional event risk is $$ 1-S_i(6,12)=1-0.82=0.18. $$ Thus the prediction is an 18% conditional risk of the defined event by month 18, given that the patient was event-free at month 6 and given the information included in the prediction. It is not a 18% risk from diagnosis, nor an unconditional risk for every adult in the clinic.

Why the Conditional Form Matters

The conditioning on survival to the prediction origin is essential. Let \(A\) denote the information used to condition the model, and suppose the relevant conditional probabilities are defined on the same basis. Since surviving beyond \(t_0+h\) implies surviving beyond \(t_0\), the conditional probability rule gives

$$ P(T_i>t_0+h\mid T_i>t_0,A) = \frac{P(T_i>t_0+h\mid A)} {P(T_i>t_0\mid A)}. $$

This identity shows why survival to the landmark cannot be silently dropped: the conditional target is a probability within the group that has survived event-free to \(t_0\), not the probability calculated from the original time origin. In a joint model, the conditioning information may include the longitudinal history available at the landmark. The prediction must use the same origin and conditioning set as the target against which it is evaluated.

For an endpoint defined as the first occurrence of any event in a chosen event set, \(1-S_i(t_0,h)\) is the conditional probability of at least one such event during the window. If the goal instead concerns a particular event type \(k\), define the event type \(J_i\) and use a cause-specific cumulative incidence target:

$$ P(T_i\leq t_0+h,\ J_i=k\mid T_i>t_0,\mathcal H_i(t_0),X_i). $$

This probability accounts for the fact that a different event type may occur first and prevent event \(k\) from occurring as the first event. It is therefore not generally equal to \(1-S_i(t_0,h)\) when survival refers to being free of all event types.

Worked Example: A Landmark Changes the Prediction Window

Consider a hypothetical patient who is event-free at month 6 after diagnosis. At that visit, a model predicts conditional survival of 0.75 over the next 12 months. The event is first hospitalization for the disease under study.

Here \(t_0=6\) months and \(h=12\) months, so the endpoint status is assessed at month \(6+12=18\). The conditional event risk is $$ 1-0.75=0.25. $$ The interpretation is a 25% predicted probability of first hospitalization between the month-6 visit and month 18, conditional on being event-free at month 6 and on the prediction information.

If the prediction were instead described as “12-month risk from diagnosis,” its window would end at month 12 and it would answer a different question. The same patient and the same numerical probability do not make these targets interchangeable: the origin, conditioning group, and endpoint time differ.

Assumptions and Alignment Checks

Defining the target does not require that the joint model be correct. It does require a coherent, observable prediction question. To interpret model output as a prediction for that question, check the following conditions:

  • Eligibility is explicit. The population at risk at \(t_0\) is defined, including the requirement that members be event-free at the origin.
  • Time is aligned. The time scale, origin, and horizon use compatible units, and the horizon is measured forward from the stated origin.
  • Information is available at prediction time. Every marker value and covariate used in the prediction is available by \(t_0\); future measurements do not leak into the prediction.
  • The endpoint is consistently defined. Event criteria, event types, and the handling of multiple possible event types match the stated target.
  • The target population matches the intended use. A model developed in one population does not automatically target a different population with different eligibility rules or distributions of histories and covariates.

Outcome ascertainment and censoring matter when estimating how well predictions perform, but they do not change the definition of the target. The target must first say what event status would count at \(t_0+h\); an evaluation procedure must then account appropriately for outcomes that are not fully observed. A target population also cannot be made representative merely by naming it: if the evaluation data differ from the intended population, the resulting evidence may not apply without additional justification.

Worked Example: Separating All-Event Survival From Cause-Specific Risk

Suppose a hypothetical model predicts, for an eligible patient at \(t_0\), a 12-month probability of remaining free of either disease-related death or another-cause death of 0.88. The model also gives a 12-month conditional cumulative incidence of 0.07 for disease-related death and 0.05 for another-cause death.

The probability of either first death during the 12-month window is the complement of event-free survival: $$ 1-0.88=0.12. $$ The two cause-specific probabilities sum to that total: $$ 0.07+0.05=0.12. $$ Therefore, 0.12 is the predicted risk of death from either cause, while 0.07 is the predicted risk of disease-related death as the first event. Reporting 0.12 as the disease-related death risk would confuse all-event risk with cause-specific risk. The relevant target depends on which event the evaluation is intended to assess.

Alternatives, Diagnostics, and Limitations

Several nearby quantities are easy to confuse with the target conditional survival probability. The distinctions below are about the question being asked, not about which quantity is universally preferable.

QuantityConditioning or time originQuestion answered
Conditional survival \(S_i(t_0,h)\)Event-free at \(t_0\); information available at \(t_0\); horizon \(h\) after \(t_0\)What is the probability this eligible person remains event-free through the specified future window?
Survival from a fixed baselineTime measured from the baseline origin, without updating to a later landmarkWhat is the probability of surviving from baseline to a specified time?
Cause-specific cumulative incidenceEvent-free at \(t_0\); one event type is specified while other event types may occur firstWhat is the probability that the specified event type occurs first by the end of the window?

A useful target audit is to write the population, eligibility, event, origin, horizon, and conditioning information in one sentence, then translate that sentence into the estimand. Warning signs include a reported “12-month risk” with no origin; a prediction using measurements collected after its stated origin; an evaluation group containing people who had already experienced the event before that origin; and a cause-specific interpretation of the complement of all-event survival. Each is a mismatch between the prediction and its intended target.

A clearly defined target is necessary, but it does not guarantee accurate predictions. The joint model may be misspecified, the marker history may be sparse or measured with error, or the target population may differ from the population in which the model was developed. A target may also be difficult to estimate if the event is rarely observed over the specified window. These are limitations of the prediction or the evidence available for evaluating it; they do not justify changing the target definition after seeing results.

Key takeaway: A conditional survival prediction is defined by who is eligible, when the prediction is made, which event counts, and how far ahead the prediction extends. State these elements before comparing predictions with outcomes.

Check Your Understanding

For each question, identify the target implication before reading the solution.

  1. A model predicts survival of 0.90 for 6 months from a visit 4 months after diagnosis. At what month after diagnosis does the window end, and what is the conditional event risk?
  2. Why is a person who experienced the target event before the prediction origin not part of the usual event-free landmark target at that origin?
  3. For a target of death from a particular cause, why may the complement of all-cause survival be the wrong risk?
  4. A model uses a marker measurement taken two weeks after the stated prediction origin. What target-alignment problem does this create?

Solutions. (1) The window ends at month \(4+6=10\). The conditional event risk is \(1-0.90=0.10\), or 10%, among those event-free at month 4 and conditional on the stated information. (2) The landmark target is conditional on being event-free at the origin; someone who already had the event is not eligible for that risk set and cannot experience a first event after the origin. (3) All-cause event risk includes every event type. If another cause can occur first, the target-cause cumulative incidence is only the probability of the specified cause occurring first, which can be smaller than the probability of any death. (4) The prediction uses information unavailable at the stated origin. It therefore does not represent a prediction that could have been made using only information available at that time.

Clinical Biostats