Temporal Validation Across Calendar Periods

Why Validate Across Calendar Time?

A prediction model can perform well in the data used to develop it and still be poorly calibrated when applied later. Clinical practice, treatment, event rates, measurement systems, and who enters care may change. A temporal validation asks a focused question: how well do predictions from a fixed model agree with outcomes in a later calendar period?

The model must be fixed before evaluating the later cohort. That includes its coefficients, baseline transition risks, prediction horizon, eligibility rules, and handling of predictors. If any of these are changed after inspecting later-period outcomes, the resulting analysis is no longer a straightforward validation of the original model. Temporal validation differs from a random split or random cross-validation because it preserves the direction of time: the model is assessed on people who enter later, rather than on a random mixture of earlier and later records.

For example, suppose a model predicts the probability of a direct transition from state \(r\) to state \(s\) within \(\tau\) time units, for people in state \(r\) at their prediction origin. The later-period cohort should use the same origin definition and the same \(\tau\). A person who makes a different transition out of \(r\) before the target transition is not a case for the target transition by \(\tau\). Such a competing transition is part of the outcome definition, not ordinary loss to follow-up.

Conditions for a Meaningful Temporal Comparison

The central design requirement is that the prediction procedure be frozen before the later outcomes are examined. Predictor definitions and units must also remain compatible. A laboratory result measured on a different scale, for instance, cannot simply be passed into the old model without checking that it represents the same quantity. The evaluation cohort must include people who meet the model's intended eligibility criteria; a validation among people in a different state or at a different prediction origin changes the question.

Choose a common prediction horizon and ensure that outcome ascertainment supports that horizon. With complete follow-up through \(\tau\), each person can be classified directly. With right censoring before \(\tau\), their target-transition status at \(\tau\) is not known. Estimating observed incidence then requires an appropriate competing-risks or multi-state estimator, such as the Aalen–Johansen estimator, or censoring weights. These approaches require assumptions about censoring. In particular, the censoring process must be independent of the relevant event process, either unconditionally for a marginal estimator or conditional on variables used by a valid censoring adjustment. The adjustment also requires adequate probability of remaining observed in the relevant covariate groups. Competing transitions are observed outcomes; they must not be treated as censoring.

If follow-up is systematically shorter in the later cohort, a crude event proportion can make the later event risk look lower simply because fewer people were observed long enough. Compare at a horizon supported in both periods, account for censoring appropriately, and report how much follow-up and how many target and competing transitions were observed. A common horizon does not by itself guarantee comparable cohorts, but it prevents one basic mismatch.

Calibration at a Fixed Horizon

For person \(i\), let \(\hat{p}_i(\tau)\) be the model's predicted probability of the target transition by \(\tau\), conditional on the person's state and prediction origin. In a cohort of size \(n\), the average predicted risk is \(E/n\), where \(E=\sum_{i=1}^{n}\hat{p}_i(\tau)\). With complete outcome ascertainment, let \(O\) be the number who make the target transition by \(\tau\). The observed proportion is \(O/n\). The observed-versus-expected ratio \(O/E\) and the difference \(O/n-E/n\) summarize calibration in the large: a ratio above 1 or a positive difference indicates more target transitions than predicted on average.

With censoring, \(O/n\) is generally not a valid observed risk. Replace it with an estimate of the target-transition cumulative incidence by \(\tau\), computed for the validation cohort using a suitable estimator and censoring assumptions. The comparison must use the same target transition, origin, horizon, and eligible population as the predictions. These aggregate summaries can hide important miscalibration in particular risk ranges; they are not a complete description of calibration.

Discrimination for the Same Transition

Calibration concerns agreement between predicted and observed risks; discrimination concerns ranking. At horizon \(\tau\), define cases as people who make the specified target transition by \(\tau\). Define controls as people who do not make that target transition by \(\tau\), including those with a competing transition and those who remain free of the target transition through \(\tau\). For a score \(\hat{p}(\tau)\), the transition-specific area under the receiver operating characteristic curve (AUC) is the probability that a randomly selected case has a higher score than a randomly selected control, with half-credit for a tie.

With complete follow-up, the empirical AUC is the fraction of case–control pairs ranked in the correct direction, counting each tied pair as one-half. Under censoring before \(\tau\), case or control status can be unknown; an estimator that accounts for censoring is needed, together with its assumptions. Simply dropping everyone censored before \(\tau\) can bias the result if censoring is related to risk. An AUC assesses ranking for this defined endpoint; it does not show that predicted probabilities are numerically accurate.

Comparing Calendar Periods

A single later-period estimate answers whether performance is acceptable in that cohort, but not whether performance has changed over time. For monitoring, prespecify multiple entry periods and calculate comparable metrics in each: the same transition, horizon, eligibility criteria, and prediction procedure. Report cohort sizes, event counts, follow-up, average predicted risk, observed risk, and discrimination estimates. Uncertainty matters: a difference between two point estimates can arise from sampling variation, especially when the target transition is rare.

A temporal difference is descriptive, not automatically an effect of calendar time. Later cohorts may differ in measured or unmeasured case mix, care pathways, data collection, or censoring. A lower AUC in a later period, for example, may reflect a narrower distribution of risk rather than a failure of the model's ranking mechanism in every subgroup. Further analyses can investigate these explanations, but the initial temporal comparison should not claim to identify their cause.

Diagnostics, Alternatives, and Limitations

Before interpreting a temporal metric, check that the later cohort really is later than the development data, that no later-period outcomes influenced model choices, and that the model was applied without undocumented changes. Confirm that the prediction origin, state eligibility, transition definition, horizon, and predictor measurements match. Tabulate competing transitions and censoring, and check whether enough target events occurred to support stable estimates. Sparse events can make calibration ratios and AUCs highly variable; a striking point estimate from a small number of cases should be reported with uncertainty, not treated as definitive.

For intervals or comparisons, resampling people within each calendar cohort can quantify sampling uncertainty while keeping the model fixed. If people are clustered within hospitals or other units, resampling should respect that structure. With two independent cohorts, resample within each cohort separately when estimating the uncertainty in their difference. Such intervals reflect sampling variation under the resampling design; they do not repair biased outcome ascertainment, incompatible definitions, or informative censoring.

Random cross-validation is useful for estimating performance under a setting where observations are exchangeable across folds, but it can mix calendar periods and conceal deterioration over time. Temporal validation instead tests transport forward in calendar time, though it may be less precise because the later cohort is smaller. Repeating validation over successive periods can show whether performance is consistently stable or varies over time. Neither a temporal split nor a favorable metric establishes performance in a different population, and neither alone explains why metrics changed.

A common mistake is to calculate the later cohort's observed event proportion using only people who completed follow-up, then compare it directly with model predictions for everyone. The incorrect reasoning is that those with complete records are a representative subset. If completion of follow-up is related to event risk, that comparison can be biased; it also discards information. The correction is to use a prespecified horizon and an estimator appropriate for censoring, state its assumptions, and distinguish competing transitions from censoring. Another mistake is to treat a high AUC as evidence of accurate risks. A model can rank people well and still systematically overpredict or underpredict.