Temporal Validation of Joint-Model Predictions

Why Validate Predictions Over Calendar Time?

A joint model may perform well when assessed among patients drawn from the same period used to develop it, yet work less well after clinical practice or the patient population changes. A model’s predictions can be affected by shifts in disease severity, laboratory measurement, treatment, follow-up, or the way patients enter care. Temporal validation asks whether predictions from a model developed in an earlier calendar period remain accurate for eligible patients seen later.

The key design feature is chronology: all choices that define the model must be made using information available before the validation period begins. The fitted model is then applied to the later cohort without using their future outcomes to choose or alter it. This evaluates performance in a relevant later setting; it does not, by itself, show why performance changed or establish that a change in care caused a change in outcomes.

Specify the Prediction Target Before Scoring

Let \(t_0\) be the prediction landmark and \(\tau\) the prediction horizon after that landmark. For patient \(i\), let \(T_i\) denote the event time, \(C_i\) the censoring time, and \(H_i(t_0)\) the longitudinal history available by \(t_0\). A fixed joint model produces a predicted event risk \(r_i(\tau)=P(T_i-t_0\leq\tau\mid T_i>t_0,H_i(t_0))\). The target is the risk over the next \(\tau\) time units among patients who are event-free and eligible at \(t_0\). The landmark, eligibility rules, event definition, and horizon must be the same in development and validation, unless a difference is explicitly part of the evaluation question.

For example, a model developed using annual marker measurements cannot be evaluated as if a later cohort had the same information if its markers are collected every three months or only when symptoms worsen. The prediction procedure must be implementable using the later patient’s information available at the landmark, with measurement units and timing handled consistently. If data from after \(t_0\) enter the predictor, the evaluation has information leakage and does not represent a genuine prediction at \(t_0\).

A later cohort needs enough follow-up to evaluate the chosen horizon. Censoring before \(\tau\) means that the patient’s horizon outcome may be unknown; it is not correct to label that patient event-free through \(\tau\). The censoring process must therefore be addressed when estimating performance.

Scoring Predictions With Censored Follow-Up

Define the residual event time \(U_i=T_i-t_0\), residual censoring time \(V_i=C_i-t_0\), observed residual follow-up \(\tilde U_i=\min(U_i,V_i)\), and event indicator \(\delta_i=I(U_i\leq V_i)\). The observed time \(\tilde U_i\), rather than the unknown \(U_i\), is essential: for a patient censored before an event, the true event time is not available. Let \(G(u\mid Z_i)\) be the probability of remaining uncensored through residual time \(u\), conditional on \(Z_i\). An inverse-probability-of-censoring-weighted Brier score at horizon \(\tau\) is

An observed event by \(\tau\) has horizon outcome one, so its squared prediction error is \((1-r_i(\tau))^2\). A patient observed event-free past \(\tau\) has outcome zero, giving squared error \(r_i(\tau)^2\). A patient censored before \(\tau\) without an event contributes neither observed-outcome term: their horizon outcome is unknown. The weights compensate for the fact that, under the censoring assumptions, observed cases represent patients with similar \(Z_i\) who were censored.

Why the Censoring Weights Work

The weighting argument can be seen by considering the event contribution. Conditional on an event time \(U_i=u\) and censoring variables \(Z_i\), the probability that the event is observed is \(G(u-\mid Z_i)\), using the left limit to handle event-time conventions. Thus the conditional expected weighted event indicator is

The equality uses conditional independent censoring and the definition of \(G\). Similarly, for someone event-free past \(\tau\), observation beyond \(\tau\) occurs with probability \(G(\tau\mid Z_i)\), so weighting by its reciprocal recovers the event-free-through-\(\tau\) contribution. Multiplying these indicators by their squared errors and averaging therefore estimates the horizon Brier loss under the stated assumptions. If some \(G\) values are near zero, the weights can be very large, making the estimate unstable; weighting does not create information where almost no comparable patients remain under observation.

Read Temporal Performance in Context

Use complementary summaries rather than treating one score as a complete verdict. Calibration asks whether predicted risks agree with event frequencies; the Brier score measures squared error and reflects both calibration and separation of risks. A log score is another proper scoring rule, but can be especially sensitive to predictions close to zero or one. The earlier material on calibration and prediction scores defines these measures in detail; the temporal question is whether they remain satisfactory in the later cohort.

When outcomes are fully observed at the horizon, the mean predicted risk can be compared with the observed event proportion. With censoring, observed-to-expected comparisons and calibration assessments require suitable censoring adjustment. A rise in average predicted risk alone does not imply worse calibration: it may reflect a higher-risk case mix. Conversely, similar average predictions can hide important miscalibration across the range of risk. Report the cohort’s event incidence and follow-up context alongside the chosen calibration and overall accuracy summaries.

Investigate What Changed, Without Overinterpreting

A temporal performance change is a signal to investigate, not a diagnosis. Review the later cohort’s risk distribution and clinical characteristics; the timing, frequency, units, and missingness of longitudinal measurements; treatment patterns; outcome ascertainment; and censoring. For example, treatment may have become more common after a high-risk prediction. If the model does not account for that treatment change, predicted risks may differ from subsequent observed risks. That discrepancy is predictive, not evidence that treatment caused the change: treatment selection and disease severity may be related.

Performance can also appear to change because measurement procedures or eligibility definitions changed. A marker measured by a new assay may have a different scale; a more intensive schedule may reveal worsening sooner; or a revised event definition may alter incidence. These are not merely statistical nuisances. They determine whether the predictions being evaluated correspond to the same inputs and outcome target.

A temporal split differs from a random split or ordinary cross-validation: it deliberately tests forward in calendar time, where practice and case mix may drift. It is not the same design as evaluating a model in an independent cohort. If performance is inadequate, investigate the source and plan any model updates separately; evaluating an updated model on the same later outcomes used to tune it would no longer provide an untouched temporal assessment.

Diagnostics, Failure Modes, and Limitations

• Information leakage: Using post-landmark measurements, later outcome information, or validation outcomes to select or modify the model makes the evaluation optimistic. Reconstruct the prediction using only information available at \(t_0\), and freeze all model choices before scoring.
• Misaligned targets: Different landmarks, horizons, eligibility rules, event definitions, or measurement units can make the two calendar periods incomparable. State and verify these definitions before comparing performance.
• Unstable censoring weights: Very small estimated \(G\) values produce large weights and noisy scores. Inspect the censoring survival estimates and weight distribution, and report when the horizon is poorly supported by follow-up. Truncating weights changes the estimator and should be disclosed, not treated as a cure for absent follow-up.
• Case mix mistaken for model drift: A changed event rate or mean predicted risk may result from a different patient population. Describe the later cohort and interpret calibration and prediction scores alongside those changes.
• Uncertain explanations: Calendar time bundles many changes. Temporal validation can detect performance differences but cannot isolate a causal effect of measurement, treatment, or policy without an appropriate causal design.

Temporal validation is limited to the later population, care setting, follow-up, and prediction target observed. A single later cohort may not represent future practice, and a censoring-adjusted score remains dependent on its censoring assumptions. These limitations do not make temporal validation unhelpful: its value is that it tests a frozen prediction procedure against a realistic forward-time challenge.