External Validation Across Populations

Why Validate Across Populations?

A prediction model may perform well in the population used to develop it and less well when applied elsewhere. The new population may differ in clinical setting, eligibility, follow-up, or the distribution of patient characteristics. External validation evaluates the model’s predictions in data from a distinct population to learn how well the model performs there. The model is applied as specified; its coefficients and prediction rules are not estimated again from the validation outcomes.

For a multi-state model, evaluation must be specific about the transition or state-occupation probability being predicted, the time origin, and the horizon. A predicted probability of a direct transition from state \(r\) to state \(s\) is not interchangeable with the probability of occupying state \(s\) at a later time, which may include indirect paths. The target in the validation data must match the model output. The earlier material on transition probabilities versus state-occupation probabilities provides the distinction; here, it is a design requirement for a fair external evaluation.

The target population is defined by who could be included in the validation cohort and the setting in which the predictions would be used. Before looking at outcomes, specify eligibility, the prediction time, the state occupied at that time, the transitions of interest, the prediction horizon, and how censoring will be handled. A validation cohort is not automatically representative of every population in which the model might later be used.

Lock the Prediction and Define the Target

Suppose the model was developed to predict, for a person in state \(r\) at a specified time origin, the probability of a direct transition to state \(s\) within horizon \(h\). Write the model’s prediction for validation subject \(i\) as \(q_i(h)\). For this direct-transition target, define \(Y_i(h)=1\) if the target transition occurs by \(h\) before any competing first transition, and \(Y_i(h)=0\) if a competing first transition occurs by \(h\) or the person remains in the origin state through \(h\). If the prediction instead concerns being in state \(s\) at \(h\), the outcome indicator must represent that state-occupation event, including any paths allowed by the model. These are different validation questions.

The prediction must be produced using the original model and the same input definitions. This includes variable coding, units, missing-data rules, time origin, and any transformations required by the model. For example, changing the definition of a baseline covariate or estimating a new coefficient from validation outcomes means the evaluated procedure is no longer simply the fixed original model. Record predictions before examining validation outcomes where practicable, and retain enough information to reproduce them.

A valid comparison also requires that the validation data measure the relevant states and transition times consistently with the target. Subjects should enter at the prediction time and in the state to which the prediction applies. Follow-up must be long enough to observe outcomes at the chosen horizon, or censoring must be addressed appropriately. In a multi-state setting, competing transitions are outcomes that alter the probability of the target transition; they must not be treated as ordinary losses to follow-up.

Transition-Specific Performance Measures

For fully observed binary outcomes at a fixed horizon, the Brier score is the average squared difference between predicted probability and observed outcome. For \(n\) validation subjects with predictions \(q_i(h)\) and outcomes \(Y_i(h)\), it is

The score ranges from 0 to 1, with smaller values indicating better probability predictions. It combines aspects of calibration and discrimination, so it should not be interpreted as measuring either one alone. For multiple transitions, calculate or summarize performance separately for each transition target and horizon; a single pooled number can conceal poor performance on a clinically important, less frequent transition.

A useful descriptive comparison is the mean predicted probability against the observed event proportion for the same endpoint. With complete follow-up, these are \(\bar q(h)=n^{-1}\sum_i q_i(h)\) and \(\bar Y(h)=n^{-1}\sum_i Y_i(h)\). A difference \(\bar Y(h)-\bar q(h)\) above zero means the observed event proportion exceeds the average prediction in this cohort. This is an overall summary, not proof that every part of the probability range is well calibrated.

For a binary target with fully known outcomes, transition-specific discrimination can be summarized by an AUC: the probability that a randomly selected case has a higher predicted probability than a randomly selected non-case, with half credit for ties. The cases and non-cases must be defined for the same transition and horizon as the predictions. A high AUC does not establish that the probability values are accurate: predictions can rank individuals well while systematically overpredicting or underpredicting risk.

The Brier score has a useful justification as a probability score. If the true event probability for a subject with given information is \(p\), then \(Y\) is binary with \(P(Y=1)=p\). The expected squared error for a proposed prediction \(q\) is

The first term does not depend on \(q\), and the second is nonnegative, reaching its minimum when \(q=p\). Thus, in expectation, the Brier score rewards reporting the true probability rather than shifting predictions systematically away from it. In a finite external cohort, however, its value also reflects the cohort’s outcome frequency and case mix.

Censoring and Uncertainty

If a subject is censored before horizon \(h\) without a target or competing transition, \(Y_i(h)\) is unknown. Coding that subject as a non-case biases the evaluation when the unobserved outcome could have been a transition. One option is an inverse-probability-of-censoring-weighted Brier score. In outline, subjects with an observed target event before \(h\) contribute their event loss weighted by the inverse estimated probability of remaining uncensored to that event time; subjects known to be event-free at \(h\), including those with a recorded competing transition by \(h\), contribute their non-event loss weighted by the inverse probability of remaining uncensored to the relevant observation time. Subjects censored before \(h\) contribute no directly observed horizon outcome. The exact estimator and weights depend on the outcome definition and censoring scheme.

Such weighting requires a suitable censoring model, conditional independent censoring given the information used in that model, and positive probability of remaining uncensored over the relevant follow-up. If some kinds of patients are almost always censored before a transition could be observed, weights can become unstable and the validation estimate imprecise. State transitions should be ascertained consistently, and the censoring model should not silently treat a competing transition as censoring when it is a known non-case for the specified target.

Validation estimates are subject to sampling variation. Confidence intervals can be obtained with methods appropriate to the metric and design; resampling should preserve the independent sampling unit, such as resampling whole patients rather than individual transition records when patients contribute multiple records. If the external data come from a small number of centers, uncertainty from sampling centers may also matter, and a patient-only resampling scheme may not represent it. Report the number of subjects and the number of each relevant transition, as rare transitions can make transition-specific estimates unstable.

Worked Discrimination and Design Checks

A practical external validation separates two questions. First, does the locked model produce predictions for the target population using available inputs and the intended prediction time? Second, how well do those predictions agree with outcomes there? Keeping this separation clear prevents outcome information from influencing the predictions that are then evaluated.

Temporal validation across calendar periods applies a model in a later period, whereas external validation applies it in a distinct population or setting; the distinction is determined by the validation design, not merely by the date of data collection. Internal validation addresses performance in the development setting under resampling or related procedures. Neither form substitutes for measuring performance in the intended external target population. Conversely, a single external cohort does not establish performance everywhere.

Common Pitfalls and Limits

• Refitting before evaluation. Updating coefficients or baseline risks using validation outcomes changes the prediction procedure. Evaluate the frozen model first; any subsequent updating is a separate analysis.
• Changing the estimand. Treating a direct transition as equivalent to occupying a state at the horizon can misclassify outcomes. Specify the target and match it to the model output.
• Ignoring competing transitions. A competing transition may make the target event impossible as defined, but it is still an observed outcome. Do not automatically discard it as censoring.
• Calling a favorable average “calibration.” Equal overall observed and predicted proportions can conceal overprediction in one range and underprediction in another.
• Overinterpreting discrimination. AUC concerns ranking, not absolute probability accuracy, and can be uncertain when few transitions occur.
• Assuming transportability from one validation. External performance pertains to the sampled target population, its follow-up, outcome ascertainment, and prediction horizons. It is not a guarantee for other settings.

External validation is an assessment, not a causal analysis. Differences in performance between populations do not by themselves show that population membership caused the difference. They may reflect sampling uncertainty, outcome measurement, follow-up, or differences in the populations and settings. This tutorial evaluates the fixed model; changing its baseline risks or coefficients is a distinct task.