Assessing Transportability Across Populations

Why Transportability Requires More Than a Performance Score

A prediction model can perform differently in two populations even when its predictions have not changed. The populations may contain different proportions of people with high predicted risk, different longitudinal histories, or different distributions of baseline covariates. These differences in case mix can change an overall performance score. Alternatively, the relationship between a person's predictors and their event risk may differ in the new population. That is a change in the conditional outcome pattern, and it may indicate that predictions are no longer calibrated.

For a joint model, predictions often depend on both baseline covariates and a participant's longitudinal history up to a landmark time. Transportability assessment therefore needs to compare like with like: the same endpoint, prediction horizon, landmark eligibility rule, and prediction procedure, applied to clearly defined populations. Differences in follow-up and censoring also matter because outcomes may not be observed equally often in the two populations.

Let \(S\) denote a source population and \(T\) a target population. At landmark time \(s\), let \(Z\) collect the information used for prediction, such as baseline covariates and the observed longitudinal history \(\mathcal H(s)\). A frozen joint model supplies a predicted survival probability \(\hat S(t\mid s,Z)\) at a later time \(t\). For a single event endpoint, the corresponding predicted event risk by \(t\) is \(1-\hat S(t\mid s,Z)\). The assessment concerns the same frozen predictions in each population; refitting or updating the model changes the question.

A Standardization Decomposition

Consider the Brier loss at a fixed horizon. Define \(Y_t=I(T_{\mathrm{event}}>t)\), so \(Y_t=1\) if the participant remains event-free through \(t\), and \(Y_t=0\) if the event occurs by \(t\). For a prediction \(f(Z)=\hat S(t\mid s,Z)\), the individual loss is \(\{Y_t-f(Z)\}^2\). Let \(P_S\) and \(P_T\) represent the source and target conditional distributions of \(Y_t\) given \(Z\), and let \(Q_S\) and \(Q_T\) be the source and target distributions of \(Z\). Define

The first argument specifies the conditional outcome pattern; the second specifies the population mix over which that pattern is averaged. The following identity separates a change in mix from a change in conditional outcomes, using the source conditional outcome pattern as the reference:

This is an algebraic identity: adding the two right-hand terms cancels \(R(P_S,Q_T;f)\), leaving the left-hand side. The case-mix component asks how performance would change if the source conditional outcome pattern held, but the model were evaluated over the target predictor distribution. The conditional-shift component compares target and source outcome patterns while holding the target mix fixed. This reference choice is explicit; a decomposition that uses the target outcome pattern as its reference can allocate the difference differently.

The conditional-shift component is not automatically a pure measure of calibration drift. Conditional outcome variability can also change. Calibration drift specifically means that observed event risk changes relative to predicted risk—for example, the event frequency among people with a given predicted risk differs between populations. Calibration summaries should therefore accompany an overall loss comparison when the question is whether predictions are systematically too high or too low.

Estimating Target Performance by Reweighting

If individual-level source validation data are available, a target-mix performance can sometimes be estimated by weighting source participants to resemble the target on \(Z\). If the target-to-source density ratio is \(w(z)=dQ_T(z)/dQ_S(z)\), then, under overlap, the source expectation of the weighted loss equals its expectation under the target predictor distribution:

where \(L\) is the appropriately observed or censoring-adjusted prediction loss. The equality follows from changing the integration measure: \(\int w(z)E[L\mid z]dQ_S(z)=\int E[L\mid z]dQ_T(z)\). In practice, weights are estimated, often from a model distinguishing target from source membership. This estimates target-mix performance under the source conditional outcome pattern; it does not prove that the conditional outcome pattern is stable in the target.

Follow-Up and Censoring Must Be Compared at Common Horizons

Different follow-up patterns can make two populations appear to have different performance even when their underlying prediction accuracy is similar. A participant censored before \(t\) does not have a known event-free status at \(t\). Simply omitting such people and averaging losses among the observed can be biased, because the observed subset may differ systematically from the full population.

For a fixed horizon \(t\), inverse-probability-of-censoring weighting (IPCW) estimates the Brier score using observed contributions weighted by the inverse probability of remaining uncensored. Write \(\tilde T_i\) for the observed event or censoring time, \(\delta_i=1\) for an observed event, and \(\hat G(u\mid W_i)\) for the estimated probability of remaining uncensored beyond \(u\), conditional on suitable variables \(W_i\). A common estimator is

A person censored before \(t\) contributes neither term because their status at \(t\) is unknown. The estimator relies on censoring being conditionally independent of the event time given \(W_i\), and on \(\hat G\) being positive wherever contributions are needed. The censoring model and its adequacy should be assessed separately in each population; more censoring does not itself mean worse prediction.

Diagnostics, Alternatives, and Limits

A reliable assessment reports target-specific performance and makes the population mix explicit. Useful checks include:

• Overlap: compare predictor distributions and examine whether target observations fall in regions represented in the source. Extreme density ratios or target-only patterns signal extrapolation.
• Weight stability: inspect the range and concentration of weights, the effective sample size, and how estimates change under reasonable weight truncation. Truncation can reduce variance but introduces bias and does not repair absent overlap.
• Conditional calibration: compare observed and predicted event risk across relevant predicted-risk ranges or covariate strata. A pooled calibration summary may hide opposing errors across groups.
• Follow-up support: inspect censoring distributions and estimated censoring survival probabilities at each chosen horizon. Very small probabilities yield unstable contributions and weakly supported estimates.
• Definition consistency: verify that the event, time origin, prediction landmark, and measurements used in the prediction have the same meaning in both populations.

A direct external validation in the target population estimates performance under the target's actual case mix and outcome pattern, subject to its sample size and censoring. Reweighting source data can answer a case-mix standardization question when overlap and conditional stability are credible, but it cannot reveal unobserved target conditional drift. Internal-external validation across centers can show how performance varies across centers, while a source-to-target analysis focuses on a specified target population. These approaches complement each other rather than replacing target data when those data are available.

An overall score such as the Brier score or an observed-to-expected summary is useful but not sufficient to identify why performance changed. The standardization decomposition is reference-dependent, and its conditional component can reflect changes beyond calibration. Conversely, stable overall performance can conceal subgroup-specific miscalibration. Prediction transportability is also not a causal claim: it concerns the behavior of predictions in a population, not the effect of intervening on predictors.