Leave-One-Species-Out Validation of Human Predictions

Why Withhold an Entire Species?

A translational model is intended to predict in a species whose PK/PD observations were not used to estimate the model. A validation split that withholds only some observations from every species does not directly test that task: the model may already have learned each species’ characteristic differences from the records left in the training set. Leave-one-species-out validation instead withholds all outcome data for one species, fits the model using the others, and predicts the withheld species using only inputs that would actually be available when making that prediction.

For a human projection, an animal species held out in turn is a surrogate test of transport to an unseen species. It is not a substitute for validation against human outcomes. The earlier material on Diagnosing Species-Structured Model Misfit helps motivate this check: systematic species differences can undermine apparent predictive performance even when a model fits the pooled data well.

Definition: With $K$ species, leave-one-species-out validation creates $K$ folds. In fold $j$, species $j$'s PK/PD outcome data are withheld, the complete modeling procedure is fitted using the other $K-1$ species, and predictions are generated for the withheld species using only prespecified inputs available at prediction time. Comparing these predictions with the withheld outcomes gives out-of-fold validation errors.

Set Up the Validation Before Fitting

The validation target must be stated first. It might be a species-level parameter, such as a clearance or potency measure, or a collection of predicted concentrations or responses at prespecified times and conditions. Define the prediction unit and the scale on which error will be judged. Log errors are often useful for strictly positive quantities because they express proportional rather than absolute discrepancies. For a positive observation $y_{ij}$ and its out-of-fold prediction $\hat{y}_{ij}$, define

e_{ij}=\log\left(\frac{\hat{y}_{ij}}{y_{ij}}\right).

Here $i$ indexes prediction units within species $j$. A positive error means overprediction; a negative error means underprediction. This definition must be kept consistent throughout the analysis. Log errors are not appropriate for outcomes that can be zero or negative without a defensible transformation; those outcomes require a suitable scale and corresponding error definitions.

The central assumption is not that species are identical. Rather, the validation question is whether the prespecified model and translation procedure can predict a withheld species within the domain to which the result will be applied. The species, endpoint definitions, assays, and input conditions must be sufficiently comparable for the prediction error to be meaningful. In addition:

No outcome leakage: no withheld species outcome may contribute to parameter fitting, feature selection, normalization, tuning, or other data-dependent choices in its fold.
Available inputs only: predictors for the held-out species must be information that would be known when making a real prediction. Using its measured response, or a quantity calculated from that response, defeats the validation.
Repeat the full procedure: each fold must refit the model. If model choices are data-driven, those choices must be made within the training data, not by looking at the held-out outcome.
Adequate support: predictions should not be treated as reliable merely because a model returns a number. A withheld species far outside the training species’ relevant covariate or mechanistic range is an extrapolation test.

A valid fold therefore emulates the intended prediction task as closely as possible. For example, a human projection may use prespecified human physiology or body-size inputs if those would be available in practice. It cannot use human PK/PD responses, which are the outcomes the projection is intended to predict.

Summarize Bias and Precision

Across $m$ out-of-fold prediction units, the mean log error is a measure of average multiplicative bias. Exponentiating it gives the geometric bias ratio: a value of 1 indicates no average multiplicative bias, a value above 1 indicates overprediction on average, and a value below 1 indicates underprediction. The root mean squared log error (RMSLE) summarizes overall error on the log scale, combining bias and dispersion.

\bar{e}=\frac{1}{m}\sum_{r=1}^{m}e_r,\qquad \text{geometric bias ratio}=\exp(\bar{e}),\qquad \text{RMSLE}=\sqrt{\frac{1}{m}\sum_{r=1}^{m}e_r^2}.

RMSLE is nonnegative and equals zero only if every error is zero. A separate measure of centered dispersion helps distinguish inconsistent errors from a consistent bias. For the validation set treated as a finite set of prediction errors, define the centered root mean squared error as $\sqrt{\sum_r(e_r-\bar{e})^2/m}$. It describes spread around the mean error, not bias relative to zero.

Justification: Since $\sum_r(e_r-\bar{e})=0$, expanding $e_r=(e_r-\bar{e})+\bar{e}$ and summing the squares gives $$ \frac{1}{m}\sum_{r=1}^{m}e_r^2 = \frac{1}{m}\sum_{r=1}^{m}(e_r-\bar{e})^2+\bar{e}^{\,2}. $$ Thus RMSLE squared is the squared mean log error plus the centered mean squared dispersion. A small RMSLE indicates low total log-scale error; it does not by itself show whether the model is unbiased or whether errors are similar across species.

Worked Example: Bias Can Cancel While Predictions Still Err

Hypothetical data. In three leave-one-species-out folds, suppose the held-out observed values are 10, 20, and 40 concentration units, and the corresponding predictions are 20, 20, and 20 concentration units. The prediction-to-observation ratios are 2, 1, and 0.5.

The log errors are $\log(2)=0.6931$, $\log(1)=0$, and $\log(0.5)=-0.6931$. Their mean is $(0.6931+0-0.6931)/3=0$, so the geometric bias ratio is $\exp(0)=1$. The errors show no average multiplicative bias, but that does not mean the predictions are accurate for each species.

The RMSLE is $\sqrt{(0.6931^2+0^2+(-0.6931)^2)/3}$. Using $\log(2)$ before rounding, this is $\sqrt{2(\log 2)^2/3}=\sqrt{0.3203}\approx0.5660$. Exponentiating the RMSLE gives $\exp(0.5660)\approx1.761$, a multiplicative error scale; it is not a bias ratio. The large opposing errors cancel in the bias summary but remain visible in RMSLE.

Coverage of Predictive Intervals

Point predictions do not show whether the model’s uncertainty is appropriately represented. If each held-out prediction includes a predictive interval $[L_r,U_r]$, empirical coverage is the fraction of withheld outcomes falling inside their intervals:

\widehat{C}=\frac{1}{m}\sum_{r=1}^{m} I(L_r\leq y_r\leq U_r),

where $I(\cdot)$ equals 1 when the condition is true and 0 otherwise. For intervals intended to have, for example, 90% predictive coverage, compare $\widehat{C}$ with 0.90. Coverage should be interpreted with the number of species and prediction units in mind: a small validation set gives a coarse and uncertain estimate. Also check whether misses are concentrated in a particular species, time range, or response region. An overall fraction can conceal systematic undercoverage in one part of the prediction task.

The interval must match the quantity being validated. An interval for a species-level expected response does not automatically predict an individual observed response, which may also vary around that expectation. If the intended use is to predict observed data, the interval should include the relevant residual or observation variability as well as parameter uncertainty. A narrow interval that ignores important uncertainty can produce poor coverage even when its midpoint predictions look reasonable.

Worked Example: Calculating Error and Coverage Across Folds

Hypothetical data. A team withholds four species in turn and records one prespecified response summary for each. The observed and predicted values, in response units, are: species A, 10 and 12; B, 20 and 18; C, 15 and 15; D, 25 and 30. Suppose the respective 90% predictive intervals are [9, 13], [19, 23], [16, 19], and [24, 34].

The log errors are $\log(12/10)=0.1823$, $\log(18/20)=-0.1054$, $\log(15/15)=0$, and $\log(30/25)=0.1823$. The mean is $(0.1823-0.1054+0+0.1823)/4=0.0648$. The geometric bias ratio is $\exp(0.0648)=1.067$, indicating average overprediction of about 6.7% on the multiplicative scale.

The mean squared log error is approximately $(0.03324+0.01110+0+0.03324)/4=0.01940$, so RMSLE is $\sqrt{0.01940}=0.1393$. The centered error dispersion is $\sqrt{0.01940-(0.0648)^2}\approx0.1233$. The fourth observation, 15, is outside its interval [16, 19]; the other three are inside. Empirical coverage is therefore $3/4=75\%$. This is below the nominal 90%, but four folds are far too few to establish a stable coverage rate or a definitive calibration failure.

Weighting, Diagnostics, and Alternatives

When each species contributes different numbers of observations, the summary must reflect the intended prediction target. Pooling all records gives more weight to species with more records. Averaging a metric within each species and then across species gives each species equal weight. Neither choice is universally correct: record-weighting targets performance over the pooled prediction units, while equal species-weighting targets average performance across species. State the choice and, where it matters, report both. Repeated time points from the same species are not independent evidence of performance across additional species.

Worked Example: How Unequal Record Counts Change the Bias Summary

Hypothetical data. Suppose species A contributes 10 prediction records, each with log error $\log(1.2)=0.1823$, while species B contributes one record with log error $\log(0.5)=-0.6931$. The record-weighted mean log error is $[10(0.1823)-0.6931]/11=0.1027$, giving a geometric bias ratio of $\exp(0.1027)=1.108$.

If the intended target gives each species equal weight, the mean of the two species’ mean log errors is $[0.1823-0.6931]/2=-0.2554$, and the geometric bias ratio is $\exp(-0.2554)=0.7746$. These summaries differ because they answer different questions: the first weights individual prediction records equally; the second weights the two species equally. Neither calculation is an arithmetic mistake, but silently switching between them changes the estimand.

Diagnostics should examine fold-specific predictions and errors, not only a pooled score. Plot predicted versus observed values on the chosen scale and inspect log errors by species, outcome magnitude, and time. Look for one species driving the summary, errors that change sign across the range, implausibly narrow intervals, and predictions based on extrapolation. Confirm that every fold used the same endpoint definition and prediction procedure. If a fold fails to fit or produces unstable estimates, report that failure rather than silently omitting the species.

A random split of individual records is easier to implement, but it usually answers a different question: prediction for new records from species already represented in training. It can be optimistic for translation to an unseen species. Leave-one-species-out validation addresses the unseen-species task more directly, but with few species its performance estimates are unstable and strongly dependent on which species are included. External validation in a genuinely new data set is more independent evidence when available. These approaches are complementary rather than interchangeable.

A common mistake is to read a near-zero mean log error as proof that the model predicts well. In the first worked example, the mean is zero even though predictions are twice as high as observed in one fold and half as high in another. The correction is to report bias alongside a measure of dispersion or total error, and to examine species-specific results. Similarly, a high observed coverage fraction from a handful of intervals does not prove that nominal coverage is calibrated.

Key takeaway: Leave-one-species-out validation is credible only when each withheld species is genuinely excluded from all outcome-informed modeling choices and its predictions use realistic available inputs. Report the bias direction, error magnitude, interval coverage, weighting rule, and fold-level results; interpret them as evidence about transportability to an unseen species, not as proof of human predictive accuracy.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions.

In a fold withholding one species, may the model use that species’ observed concentration-response data to select among candidate models?
A set of log errors has mean $\bar{e}=0.10$. What does $\exp(\bar{e})$ summarize, and what does it not establish?
Three of five held-out outcomes fall within their predictive intervals. Calculate empirical coverage and give one reason not to treat it as a precise estimate.
Why might a random record-level split overstate performance for prediction in a species absent from training?
When one species has many more records than another, what should be considered before pooling all errors?

Solutions. (1) No. Using its outcomes for model selection leaks information from the held-out species; selection must be based on the training species within that fold. (2) The geometric bias ratio is $\exp(0.10)\approx1.105$, indicating about 10.5% average multiplicative overprediction. It does not establish low error for individual predictions or good performance in every species. (3) Coverage is $3/5=60\%$. With only five outcomes, each outcome changes the estimate by 20 percentage points, so the estimate is coarse and uncertain. (4) Records from the same species may remain in training, letting the model learn its species-specific pattern before predicting its held-out records. (5) Decide whether the target gives equal weight to species or to records. Pooling records weights species according to their record counts, which may not match the intended translational question.