Assessing Covariate Support and Transportability

Why Covariate Support Matters

An external prediction can be evaluated against observed concentrations, but an overall score alone may conceal where the model is being asked to extrapolate. A model developed mainly in adults with moderate renal function, for example, may be evaluated in an external sample containing many subjects with severe impairment. Poor performance in that group could matter clinically even if the overall score looks acceptable. Conversely, a different mix of otherwise well-supported subjects can change an aggregate score without indicating that the model's conditional predictions have changed.

Support assessment asks whether external subjects have covariate profiles resembling profiles represented in the development population. It is a diagnostic for the scope of the prediction claim, not a guarantee that predictions will transport. Similar covariate distributions do not establish that the concentration model, dosing, measurement process, or residual variability is the same in both populations.

Assessing Marginal and Joint Support

Begin by specifying which covariates define the prediction context: for example, age, body size, renal function, genotype, or dosing-related covariates. Use the subject as the basic unit for subject-level covariates. Repeated concentration records from one person do not create additional independent subjects with new ages or genotypes. If the external prediction target also depends on sampling times or dosing histories, inspect those design features separately rather than treating them as interchangeable with subject covariates.

For each continuous covariate, compare distributions using plots and summaries such as ranges, quantiles, and means. For categorical covariates, compare category proportions and identify absent or very small strata. Then inspect joint combinations. A subject can lie inside the observed age range and inside the observed renal-function range while having an age–renal-function combination scarcely present in development data. Marginal checks alone will miss that problem.

A standardized mean difference can summarize marginal imbalance for a continuous covariate. For development and external means \(\bar{x}_D\) and \(\bar{x}_E\), and standard deviations \(s_D\) and \(s_E\), one common version is

The sign indicates direction; the magnitude is in pooled standard-deviation units. For a binary covariate, a difference in proportions is often reported, with a chosen standardization stated explicitly. SMDs are descriptive, not hypothesis tests. No universal cutoff proves adequate overlap, and a small SMD can coexist with poor joint support or extreme tails.

From Overlap to Transportability

A useful way to clarify why overlap matters is to express prediction performance as a target-population average. Let \(D\) denote development and \(E\) the external target. Let \(Y\) be an external concentration outcome, and let \(L(Y,\widehat{F};X)\) be a specified prediction loss for the model's predictive distribution \(\widehat{F}\) at covariates \(X\). The target risk is the expected loss among external subjects.

If, conditional on the measured covariates, the outcome and prediction-loss relationship is stable between development and external populations, then the conditional expectation of loss can be shared across populations. If also \(f_E(x)=0\) wherever \(f_D(x)=0\), the target risk can be written using development covariate density:

The ratio \(f_E(x)/f_D(x)\) is a density ratio: regions common in the external target but uncommon in development receive greater weight. This identity explains why lack of overlap is consequential. If the external population has covariate values with no development representation, the ratio is undefined there, and development data cannot identify the corresponding conditional loss without additional assumptions or data. Even with formal overlap, very large ratios make estimates unstable.

The argument depends on more than matching covariate distributions. It assumes that the measured covariates capture relevant differences in conditional prediction performance, that the conditional outcome relationship is stable, and that the covariate densities are meaningfully estimated. Changes in assay, dosing practice, outcome handling, or unmeasured effect modifiers can violate transportability despite good measured-covariate overlap.

Subgroup-Specific Prediction Scores

After covariate support has been described, report external prediction scores for clinically relevant subgroups when sample sizes permit. Use a score defined for the predictive target and give its units or scale. For example, subgroup root mean squared error (RMSE) for concentration predictions is

where \(g\) denotes a subgroup, \(n_g\) is its number of external subjects or observations as defined for the prediction task, \(y_i\) is observed concentration, and \(\widehat{y}_i\) is the corresponding prediction. State whether the score is calculated over subjects, samples, or a prespecified prediction horizon. When subjects contribute repeated samples, uncertainty calculations and summaries should respect that clustering; treating every sample as an independent subject can give misleading precision.

Diagnostics, Alternatives, and Limitations

A defensible support assessment uses complementary summaries rather than a single pass/fail statistic. Common options include:

• Ranges and distribution plots: reveal obvious tail extrapolation and shifts, but do not characterize multivariable combinations.
• Stratified counts and plots: show whether important clinical groups or covariate combinations are sparse; results depend on how strata are defined.
• Standardized differences: summarize marginal imbalance compactly, but can conceal tail and joint-distribution differences.
• Multivariable distances or density ratios: can capture joint rarity, but depend on scaling, model specification, and the amount of data. Extreme estimated weights signal weak overlap and unstable reweighting.
• Domain classification: a model that distinguishes development from external subjects can reveal distributional differences. Strong discrimination is a warning, while weak discrimination does not prove overlap, especially with limited samples or an inadequate classifier.

Diagnostics should be interpreted with the prediction target in view. A covariate may be imbalanced but irrelevant to the model's conditional prediction, while a small imbalance in a strong effect modifier may be consequential. If support is weak, report which subjects or subgroups are affected, how many are involved, and what part of the target population is not well represented. Do not silently remove difficult subjects and then describe the narrower result as validation for the original target.

Support checks and subgroup scores answer related but distinct questions. Covariate checks identify where external subjects differ from development; subgroup scores show how predictions performed in observed external outcomes. Poor subgroup performance where support is sparse is consistent with an extrapolation problem, but it does not prove that sparsity caused the errors. Differences in outcome measurement, treatment, or other factors may also matter. Conversely, good scores in a small subgroup do not establish reliable performance throughout a sparsely represented region.

A common mistake is to say, “Every external covariate is within the development minimum and maximum, so the model is supported.” This reasoning checks only separate marginal ranges. It ignores joint combinations, sparse tails, and categorical strata, and it treats observed extremes as if they were densely represented. The correction is to combine marginal and joint diagnostics, state where support is weak, and interpret external prediction scores in those regions with appropriate caution.