Leave-One-Species-Out Validation

Why Withhold an Entire Species?

A translational model is useful only to the extent that it can predict beyond the data used to fit it. When the intended use is prediction in a species not used for fitting, validation should mimic that task: remove one animal species, refit the model using the others, and predict the removed species. Repeating this for each species reveals how predictions behave when a whole species is absent from model development.

The unit withheld matters. If a study contains many animals from each of a few species, randomly assigning individual animals to training and validation sets can leave animals from every species in both sets. Such a split mainly tests prediction for additional animals from represented species; it does not directly test prediction for an unrepresented species. Leave-one-species-out validation (LOSO) withholds all records from one species at once.

The procedure evaluates a modeling workflow, not merely a final equation. Any choice that uses data—such as selecting covariates, choosing among candidate structures, estimating parameters, or tuning a prediction method—must be made using only the training species within each fold. Otherwise information from the withheld species has influenced its prediction, making validation optimistic. The model form and evaluation target should be stated before inspecting the held-out outcomes.

Set the Prediction Target and Assumptions

Specify what is predicted before creating folds. For a pharmacokinetic (PK) example, the target might be species-level log clearance at a defined dose and physiological condition. For pharmacodynamic (PD) translation, it might be a defined response parameter or response summary. The outcome, units, time point or summary, and data-processing rules must be comparable across species. A fold is not interpretable if one species' outcome is defined differently from another's.

LOSO validation relies on several substantive conditions. First, each species must have enough data to define the target and evaluate its prediction. Second, the held-out species must not contribute to fitting, preprocessing, model selection, or uncertainty estimation for that fold. Third, the species used for validation should represent the kind of species-level prediction to which the result will be applied. If the withheld species is biologically very different from all training species, the exercise may still expose a limitation, but it does not guarantee performance in other species or in humans. The earlier material on diagnosing extrapolation distance across species helps interpret how far a held-out species lies from the training domain.

Finally, validation must reflect the intended prediction. If predictions concern a new animal from a known species, uncertainty should be appropriate to that target. If the target is a species-level mean or parameter, uncertainty should reflect that different estimand. A prediction interval should be generated from the training fold alone and should include the relevant sources of uncertainty for that target. LOSO coverage is an empirical summary across these folds; it does not, by itself, establish that an interval is calibrated for future species.

Worked Log-Scale Predictions

Log scales are often useful for positive PK quantities such as clearance because differences on the log scale correspond to ratios on the original scale. Define \(z=\ln(CL/(1\ \mathrm{L\,h^{-1}}))\), a dimensionless log clearance, and let \(x\) be a specified log body-mass predictor. Consider four hypothetical species with one species-level summary each. The values below are deliberately illustrative, not observations from a real study.

Summarizing Bias and Prediction Error

Let \(z_s\) be the held-out observed log outcome for species \(s\), and \(\hat z_{s,-s}\) its LOSO prediction. Define \(e_s=\hat z_{s,-s}-z_s\), so positive errors indicate overprediction on the log scale and negative errors indicate underprediction. With one summary per species, common aggregate measures are the mean error (bias), mean absolute error (MAE), and root mean squared error (RMSE):

Bias measures average direction and can be near zero when positive and negative errors cancel. MAE summarizes typical absolute error in the outcome's units. RMSE also has those units but gives relatively more influence to larger errors because it squares them before averaging. Report fold-specific errors alongside aggregates, especially when \(S\) is small.

The log-scale bias has a useful interpretation. Because \(e_s=\ln(\widehat{CL}_s/CL_s)\), averaging errors and exponentiating gives

The last expression is the geometric mean of the fold-specific predicted-to-observed ratios. This follows from the logarithm rule that a sum of log ratios is the log of their product. It does not imply that the model is unbiased on the original clearance scale.

Interval Coverage Across Folds

For each held-out species, the training-fold procedure may also produce a predictive interval \([L_s,U_s]\) on the same scale as the held-out outcome. The empirical LOSO coverage is the fraction of held-out outcomes lying inside their corresponding intervals:

where the indicator equals 1 when the condition is true and 0 otherwise. State the number of covered species as well as the fraction; for few species, the fraction changes in large increments. Coverage should be evaluated against intervals produced without using the withheld outcome. A high or low observed fraction with only a few folds is uncertain evidence, not a precise estimate of future coverage.

Diagnostics, Comparisons, and Limits

Inspect each fold rather than relying only on the aggregate. A single large error may identify a species for which the fitted relationship extrapolates poorly, while errors that grow with predictor value may suggest that the model's form does not capture the relationship over the tested range. Compare each held-out prediction with the species' observed outcome and record whether the fold is interpolation or extrapolation relative to the training species. A pattern of systematic underprediction or overprediction is more informative than a near-zero mean error produced by cancellation.

LOSO is usually more appropriate than a random animal-level split when the question is prediction for a species absent from fitting. A random split can be useful for a different target—prediction for additional animals from species already represented—but it does not answer the same cross-species question. A single fixed training/test division can reserve a genuinely independent species set, but with few species it may use the available evidence inefficiently. LOSO uses every species as a test species once, though its fold results are dependent because their training sets overlap.

These results have important limits. With only a few species, error averages and coverage fractions are unstable and cannot support strong claims about a broad population of possible species. Species may also differ in ways not represented by the model predictors; good LOSO performance among the tested animals does not guarantee prediction in humans, a new biological domain, or a different experimental setting. If a species has several studies or many animals, retain the species grouping in every fold so records from the held-out species cannot leak into training. Finally, conclusions apply to the exact outcome definition and modeling workflow tested, not automatically to a different dose, endpoint, or model.