External Validation in an Independent Cohort

What External Validation Evaluates

A joint model uses longitudinal information and an event-time model to produce predictions that may change as new measurements become available. External validation asks whether predictions from a prespecified, frozen model perform as intended in a separate cohort. The purpose is to evaluate those predictions—not to fit the model again using the validation outcomes.

Specify the prediction question before calculating a performance measure. For example, at landmark time \(s\), among individuals still event-free at \(s\), the question might be the probability of experiencing the event within the next \(u\) time units, given their longitudinal history through \(s\). Let \(Y_i(s)\) denote the history available for individual \(i\) through that landmark. A fitted joint model supplies a conditional survival prediction \(\hat S_i(s+u\mid s,Y_i(s))\), and hence the predicted event risk \(r_i(u\mid s)=1-\hat S_i(s+u\mid s,Y_i(s))\).

“Frozen” includes more than the model’s estimated coefficients. It includes the longitudinal and event submodels, association structure, baseline hazard, covariate transformations, measurement units, rules for handling missing predictors, and prediction algorithm. Any standardization or imputation learned during model development must be applied as specified, not newly estimated using validation outcomes. A prediction process that uses future measurements or information unavailable at the landmark is not a valid prediction process for that landmark.

The validation cohort should be independent of the data used to develop or select the model. Its eligibility criteria, event definition, landmark, and prediction horizon should be defined in advance. The measurement process must also be sufficiently comparable to apply the model’s input definitions. Independence does not guarantee that the cohorts are otherwise alike: differences in case mix, follow-up, or measurement practice can affect performance and require careful interpretation.

Calibration at a Fixed Horizon

Calibration concerns agreement between predicted risks and event frequencies. At a fixed horizon, let \(D_i=1\) indicate an event by \(s+u\), for an individual eligible at \(s\). If every outcome is known through the horizon, a basic summary is the observed-to-expected event ratio. The expected event count is the sum of predicted risks, and the observed count is the sum of event indicators.

An O/E ratio of 1 indicates agreement in the aggregate: observed and expected event counts match. A ratio above 1 means more events were observed than expected, while a ratio below 1 means fewer were observed. This summary cannot reveal whether predictions are too high for some people and too low for others, so it should not be treated as a complete calibration assessment.

Calibration-in-the-large can also be expressed by fitting a logistic model with the logit of each prediction as a fixed offset. The intercept \(a\) measures an overall shift: \(a=0\) corresponds to no shift. A separate calibration regression estimates an intercept \(a\) and slope \(b\), using the validation outcome as the response and the predicted logit as the predictor. In ideal calibration, \(a=0\) and \(b=1\). These regressions summarize the validation data; they do not authorize changing the original model’s predictions during validation.

For incomplete follow-up, an individual censored before the horizon does not have a known event-free outcome through that horizon. Treating such an individual as a non-event can bias calibration and discrimination estimates. Suitable survival methods include estimating observed event risk with a censoring-aware procedure or using inverse-probability-of-censoring weights. Such weighting requires a correctly specified censoring distribution conditional on the variables used for weighting, and positive probability of remaining uncensored over the relevant follow-up. The observed-to-expected formula above applies directly when horizon outcomes are completely observed; with censoring, its observed component must be estimated appropriately.

Discrimination at the Same Horizon

Discrimination describes whether people who experience events tend to receive higher predicted risks than people who remain event-free. For a cumulative/dynamic horizon definition, cases are individuals who experience the event by \(s+u\), and controls are those who remain event-free beyond \(s+u\). With complete outcome ascertainment, the time-dependent area under the ROC curve can be interpreted as the probability that a randomly selected case receives a higher risk prediction than a randomly selected control, with ties counted as one-half.

Here \(n_1\) and \(n_0\) are the numbers of cases and controls. This measure concerns ranking, not the accuracy of the risk values themselves. A model can rank individuals well while systematically overpredicting their absolute risks. With censoring before the horizon, a censoring-aware estimator is needed to handle unknown case or control status; the simple pair-count calculation is not valid if censored observations are simply labeled controls.

Uncertainty, Diagnostics, and Comparisons

A validation estimate is subject to sampling variability. Confidence intervals should accompany key performance estimates, with the validation individual as the resampling unit. If individuals contribute repeated landmark predictions, resampling must keep each individual’s records together. When predictions are genuinely fixed in advance, uncertainty for validation performance can be estimated without refitting the original model in each resample. This describes uncertainty conditional on that fitted model; it does not include uncertainty from developing the model.

Inspect the data and prediction process before interpreting a metric. Useful checks include verifying that all predictions lie between 0 and 1, confirming that landmark eligibility and event coding are correct, and checking that validation measurements use the model’s expected units and timing. Plot observed event frequencies against predicted risks, using a censoring-aware estimate when necessary. Large gaps, implausibly extreme risks, or poor ordering can signal problems, although a noisy plot in a small cohort is not conclusive.

The main alternatives answer different questions. Temporal validation evaluates a model in a later cohort or period, whereas external validation uses a separate cohort that may differ in more than calendar time. Cross-validation and bootstrap validation reuse development data through resampling to estimate performance while addressing optimism; they do not replace evaluation in an independent cohort. Refitting or recalibrating with validation outcomes can sometimes be a practical model-updating step, but performance measured on the same outcomes used for that update is no longer a clean estimate of the frozen model’s external performance. A further independent assessment would be needed to evaluate the updated procedure.

External validation is informative only for the population, prediction time, horizon, outcome definition, and measurement process represented by the study. A single cohort cannot establish performance in every intended setting. Sparse events, extensive censoring, a narrow range of predicted risks, or substantial missingness can make estimates unstable or biased. Report these limitations alongside discrimination and calibration, and avoid treating one summary statistic as a complete judgment of clinical usefulness.