External Validation and Biological Domain Shift

Why Independent Validation Matters

A translational model can fit its development data well and still predict poorly for a new compound or study. The crucial question is not simply whether the model describes the data used to build it, but whether its predictions remain useful for the intended population of compounds, assays, and experimental conditions. External validation addresses that question by evaluating predictions on data independent of model development.

Independence here concerns the whole modeling process, not just whether a row appeared in the fitting table. If an outcome helped select the model structure, tune parameters, choose a transformation, or decide which compounds to retain, it has influenced development. Evaluating the final model on that same information does not provide an independent test. For a valid evaluation, specify the prediction task and endpoint, freeze the model and its prediction procedure, and then compare its predictions with outcomes from the external data.

The validation unit must match the intended use. If the goal is to predict for new compounds, holding out measurements while retaining other measurements from those same compounds may overstate performance. If the intended prediction is to a new study, holding out individual observations while including other observations from that study may likewise leak study-specific information. When observations are clustered by compound, laboratory, or study, the split and interpretation should reflect that structure.

Quantifying Prediction Error

For external observation \(i\), let \(y_i\) be the observed endpoint and \(\hat{y}_i\) the prediction made by the frozen model. Define the prediction error as \(e_i=y_i-\hat{y}_i\). A positive error means the model underpredicted the observed value; a negative error means it overpredicted. Errors and summaries must be reported on the endpoint scale relevant to the prediction task.

These metrics answer different questions. Bias near zero does not mean predictions are individually accurate: overpredictions and underpredictions can cancel. MAE is directly interpretable in endpoint units. RMSE is also in endpoint units, but is more affected by large errors. None alone diagnoses why a model missed, whether a few observations dominate performance, or whether predictions are reliable in a particular biological subgroup.

Diagnosing Biological Domain Shift

External data can differ from development data in ways that change the relationship the model is asked to predict. Domain shift is a difference between the development and validation domains that may affect the model’s inputs, the outcome-generating biology, or the measurement process. In translational work, relevant differences can include chemical scaffolds or target-binding modes, assay platforms and protocols, species or tissue sources, dose or sampling conditions, and laboratory procedures.

A validation error is evidence about prediction in the tested domain, not a diagnosis of its cause. To investigate shift, examine what differs between the development and external data and whether errors vary with those differences. Useful summaries include subgroup-specific bias and MAE, plots of residuals against chemical or experimental features, and comparisons of input ranges and distributions. Such analyses should be planned or interpreted cautiously when subgroups are small; repeated searching for a subgroup that “explains” poor performance can produce an unstable story.

For a continuous feature \(z\), such as a molecular descriptor or assay temperature, compare its development and validation ranges and distributions. A validation value far outside the development range signals extrapolation in that feature, but being within the range does not guarantee that the biological relationships are the same. For categorical features, such as assay platform, identify categories absent from development data. A model cannot learn a platform-specific relationship from a category it never encountered unless its structure contains a justified way to represent that difference.

Separating Chemistry, Assay, and Study Effects

Several sources of shift can occur together. For example, a new chemical series may be measured in a different assay at a different laboratory. If predictions fail on that series, the external evaluation reveals a problem for that combined domain, but it cannot identify which difference produced the problem. This is a confounding limitation of the comparison, not a flaw that can be repaired by a more elaborate error metric.

Where feasible, use bridging data: the same compounds measured under both assay methods, or comparable compounds measured under matched conditions across studies. Paired measurements can reveal systematic measurement differences while holding compound identity fixed. Replication and reference compounds can help distinguish assay drift from biological variation. These designs do not automatically establish that the model transports to a new biological setting, but they make competing explanations more testable.

Study-level clustering also matters when summarizing performance. If many observations come from one study, treating them as many independent demonstrations of transportability gives that study disproportionate influence. Report how many compounds and studies contribute, show study-specific performance where possible, and avoid interpreting a pooled metric as if every study contributed equally. The appropriate summary depends on the intended prediction task and the distribution of future compounds or studies.

This identity follows by writing \(e_i=(e_i-\bar{e})+\bar{e}\), squaring, and summing. The cross-product term is zero because \(\sum_{i=1}^{m}(e_i-\bar{e})=0\). Thus, squared RMSE combines the average squared spread of errors around their mean with squared bias. A model may therefore have substantial RMSE because of a systematic offset, variable errors, or both. This decomposition helps describe a pattern; it does not identify its biological cause.

Prediction Intervals and External Coverage

When the model provides prediction intervals, external validation can also check whether observed outcomes fall within them. For \(m\) external outcomes with intervals constructed at a nominal level \(1-\alpha\), empirical coverage is the number of observed outcomes inside their corresponding intervals divided by \(m\). Coverage near the nominal level is desirable across repeated comparable predictions, but a small validation set gives a coarse and uncertain estimate. A single miss does not prove the interval procedure is defective, and observed coverage does not verify performance in untested domains.

Validation Choices, Diagnostics, and Limits

External validation complements rather than replaces internal resampling. Random train-test splits and cross-validation estimate performance under assumptions about how held-out observations relate to development observations; they can be useful for model development, but they may not represent new compounds or studies if related observations occur on both sides of the split. Leave-one-species-out validation tests a particular form of species transfer. External validation tests performance on independent data from the domain actually assembled. None guarantees success in a different future domain.

Before interpreting poor external performance, verify data alignment and prediction provenance. Confirm that units, endpoint definitions, transformations, and compound identities are consistent; check for missing or duplicated records; and ensure that predictions were generated without using external outcomes to retune the model. Inspect individual residuals and subgroup patterns rather than relying only on a pooled score. A large error may reflect data quality, an endpoint mismatch, a model limitation, or a genuine biological shift.

If the model is recalibrated using external outcomes, report that as a distinct step. The same outcomes cannot then serve as an untouched external evaluation of the recalibrated model. A separate independent set is needed to evaluate the updated prediction procedure. External validation also cannot establish causal mechanisms for the observed prediction errors, guarantee performance under untested conditions, or compensate for an external set that poorly represents the intended use.

Common Pitfall

Wrong reasoning: “The overall mean error is zero, so the model has no systematic problem and transports well.” This is wrong because positive and negative errors can cancel, and one subgroup can be poorly predicted while another offsets its bias. In the four-compound example, mean error was zero even though individual errors ranged from \(-3\) to \(4\) pIC50 units.

Correction: Report direction and magnitude together, inspect residuals and relevant subgroups, and describe exactly which compounds, studies, and conditions were evaluated. A low overall error is useful evidence for the tested data, not proof of transportability to every biological domain.