Covariate Support and Transportability

Why Covariate Support Matters

External validation asks how a locked model performs in data collected outside its development setting. A key question is whether the external subjects resemble the development subjects in the covariates used by the model. If an external subject has a body weight, age, renal function, or covariate combination not represented in development data, the model may have to predict beyond the information that informed its fitted relationship. That is extrapolation, and it limits how confidently performance can be transported to that subject or population.

A model can return a numerical prediction for an unusual covariate value without being well supported there. For example, a linear covariate relationship may produce a plausible-looking clearance estimate outside the observed range, but the development data cannot establish that the relationship continues linearly beyond that range. Conversely, a support gap is a warning about evidential limits, not by itself proof that the model’s prediction is wrong.

The target of this assessment is the covariates that matter to the prediction: those used directly in the model, and those needed to define its intended application. Check units, coding, and measurement definitions first. A value of 90 may mean a body weight of 90 kg in one dataset and 90 lb in another; comparing the numbers without harmonizing units would give a false support assessment. Covariate support is also distinct from the prediction-performance summaries discussed in the earlier material on Prediction Error Metrics for External PK Validation and Subgroup Performance in External Validation.

From Marginal Ranges to Joint Support

For one continuous covariate, a simple initial check is to compare the development and external minima and maxima. A value outside the development range is an obvious extrapolation beyond the observed sample. But being between the minimum and maximum is not a guarantee of strong support: a value may be isolated in a sparse tail, or may occur only in a covariate combination absent from development.

With multiple covariates, assess the joint distribution as well as each marginal distribution. For example, body weight and renal function might each fall within their separate development ranges, while a low weight paired with very low renal function is scarcely or never observed in development. A model that includes both covariates, or their interaction, must predict for that combination nonetheless.

Useful summaries include overlaid histograms or density plots for continuous covariates, tables of category frequencies, scatterplots for pairs of continuous covariates, and counts for clinically meaningful combinations. In higher dimensions, displays are less complete; report the approach and its limitations rather than treating a single distance or range check as a definitive support test. The convex hull—the smallest convex region containing observed covariate vectors—is one geometric summary, but it can classify points as supported even when they are in very sparse regions. No one diagnostic replaces clinical judgment about which covariates and combinations are relevant.

Why Joint Support Can Fail When Ranges Overlap

A marginal range check can miss unsupported combinations. The following example uses a deliberately simple geometric description to make that distinction exact. Suppose the observed development covariate combinations lie within the triangle whose vertices are (50 kg, 90 units) for body weight and renal function, (90 kg, 90 units), and (90 kg, 50 units). The triangle can be described by the constraints \(50\leq \mathrm{Weight}\leq90\), \(50\leq \mathrm{RenalFunction}\leq90\), and \(\mathrm{Weight}+\mathrm{RenalFunction}\geq140\), with weight in kg and renal function in the stated units.

The triangle is only a transparent illustration. Real datasets are noisy clouds, not exact geometric boundaries. A point just outside a convex hull may be close to many observations, while a point inside it may be far from almost all observations. Therefore, a hull check should be paired with examination of local data density and relevant clinical combinations. In high dimensions, apparent support can be especially misleading because the number of possible combinations grows rapidly.

Overlap Is Not the Same as the Same Population

Even when all external covariates fall within development ranges, their distributions can differ. Support overlap means that external covariate values are represented to some degree in development data; it does not mean that the two populations have the same covariate frequencies. A development sample may be mostly younger adults and an external sample mostly older adults, while both cover the same overall age interval.

A useful conceptual distinction is between conditional prediction behavior and the covariate distribution over which performance is summarized. Let \(X\) denote the relevant covariates, let \(Q\) be their distribution in the external population, and let \(L\) be a prediction loss (a measure of prediction error). The external population’s mean loss can be written as

where \(\mathcal{S}_D\) denotes the region with development support. The first term averages conditional prediction loss over the part of the external covariate distribution within development support. The second concerns external covariates outside that region. If the external dataset includes outcome data there, it can provide direct evidence about those subjects in that dataset. But development data alone cannot establish the model’s conditional behavior there. If no external observations cover that region either, the corresponding target-population performance is not empirically assessed.

This decomposition also explains why an overall performance estimate is not automatically transportable to every population. Even if conditional prediction behavior is stable, a different distribution \(Q\) changes how much weight each covariate region contributes. If conditional behavior is not stable, overlap alone is insufficient. Transporting predictions therefore requires both adequate support for the target covariates and a defensible expectation that the model’s conditional relationships apply in the target setting.

A Practical Support Assessment

Assumptions, Diagnostics, and Limitations

A support-based transportability argument depends on several assumptions. First, the covariates must be measured comparably across datasets; unit errors or inconsistent definitions invalidate a numerical comparison. Second, the development sample must provide a useful representation of the region claimed as supported. A range based on one unusual subject does not mean that the surrounding tail is well characterized. Third, any claim that a fitted covariate relationship applies externally requires that the relevant conditional PK relationship remains sufficiently stable across settings, given the covariates included in the model. This is a substantive assumption, not a consequence of overlap.

Warning signs include external values beyond development minima or maxima, external categories never observed during development, thin development tails, and external combinations absent or rare in development. Also inspect whether a few extreme observations determine the apparent range, whether missingness patterns differ, and whether the covariates were recorded at comparable times. These checks diagnose limitations in the support argument; they do not replace assessment of observed prediction performance.

Support diagnostics have limits. Sample extrema are unstable summaries of population boundaries; small development samples can make genuine population support look absent, while a single extreme observation can make the observed range look broader than it is. Joint-support diagnostics become harder as the number of covariates increases. A convex hull, standardized distance, or density estimate is a summary that depends on choices and data size, not a universal pass/fail test. Report practical details, such as flagged counts and the covariate regions involved, rather than relying on an unexplained label of “overlap.”

The main alternative to a support-focused assessment is to summarize external prediction performance alone. That directly answers how predictions performed for the external subjects observed, but it can obscure where those subjects lie relative to development data and cannot establish performance for unobserved target regions. Conversely, a covariate-overlap analysis describes the populations but does not itself measure prediction error. These approaches answer complementary questions and should not be treated as substitutes. Re-estimating or revising the model to address a support gap changes the evaluation question; support assessment of a locked model should make the gap visible rather than conceal it through model updating.

Common Mistakes

• “Every external value is between the development minimum and maximum, so there is no extrapolation.” This checks only marginal ranges. Joint combinations can be unsupported, and values in sparse tails may have weak empirical support.
• “An unsupported prediction is necessarily wrong.” A prediction can happen to be accurate outside observed development support. The correction is that its accuracy is not justified by development observations there; direct external data may evaluate it for the observed subjects.
• “The two samples overlap, so their performance should transport unchanged.” Overlap does not imply identical covariate distributions or stable conditional relationships. Both composition and model behavior matter.
• “The model returned a finite value, so it is supported.” Numerical calculability is not empirical support. State which covariates or combinations are extrapolated and which assumptions extend the model there.