Predictive Cross-Validation for Mechanism Selection

Why Hold Out Data When Comparing Mechanisms?

A mechanistic model can describe the data used to calibrate it without predicting new observations well. This is especially relevant when competing QSP models encode different biological hypotheses but have different parameter flexibility or respond differently to sparse measurements. Predictive cross-validation compares the mechanisms by repeatedly fitting them without some observations and scoring how well they predict those held-out data.

The comparison is only meaningful when the held-out units represent the prediction task of interest. Predicting a later time point for a participant whose earlier measurements are available is not the same task as predicting a new participant with no individual measurements. The fold design must reflect that distinction.

For mechanisms \(M_1\) and \(M_2\), a comparison requires a common outcome definition, data partition, loss function, and calibration protocol. The biological structures can differ; the information available for fitting and the rules used to score predictions should not. The information-criteria material introduced earlier compares fit and complexity through a specified criterion. Cross-validation instead evaluates predictions on data excluded from each fit; neither approach makes a mechanism biologically true merely because it scores better.

Scoring Held-Out Predictions

Let \(y_i\) be a held-out observation and \(\hat{y}_{i,-k}\) the prediction for it from a model fitted without fold \(k\). For a continuous outcome with meaningful squared deviations, the squared-error loss is \((y_i-\hat{y}_{i,-k})^2\). A common cross-validation mean squared error is the average of these losses over all held-out observations. Its square root is the root mean squared error (RMSE), in the original outcome units.

Here \(F_k\) is the set of observations in fold \(k\), and \(N\) is the total number scored. MSE has squared outcome units; RMSE has outcome units. Other losses may be appropriate for other observation types or scientific targets, but the choice must be made consistently across mechanisms and interpreted according to the loss. This tutorial uses squared error for its numerical examples.

Worked Example: Comparing Two Mechanisms on Held-Out Concentrations

Choose Folds to Match the Prediction Task

In longitudinal QSP data, measurements from the same subject are usually related through that subject’s evolving biological state and shared characteristics. Randomly assigning individual time points to folds can place early and late observations from one subject in both training and test sets. That design may evaluate interpolation within a partially observed trajectory, while appearing to evaluate prediction for a new subject. It can also leak subject-specific information into the fit.

If the intended task is prediction of future measurements for subjects whose earlier data are available, use a time-ordered split: train on measurements available through a specified time and score later measurements. If the task is prediction for a new subject, keep all observations from each subject together and hold out entire subjects. If the intended task is prediction for new subjects at future times, a design must withhold subjects and respect time ordering; the test data should not influence fitting or any preprocessing.

Multi-Subject Folds and the Meaning of the Average

When the prediction target is a new subject, a natural design is leave-one-subject-out cross-validation: fit on all but one subject, predict the omitted subject, and repeat until every subject has been held out. For larger studies, subjects can instead be divided into \(K\) groups, with all measurements from a subject assigned to one group. A subject must not appear on both sides of a fold if the goal is genuinely new-subject prediction.

The way held-out errors are aggregated also defines the target. Averaging all squared errors gives more weight to subjects contributing more observations. Averaging each subject’s mean squared error gives subjects equal weight, regardless of their number of measurements. Neither is universally preferable; use the one matching the intended prediction target and state it.

Why Cross-Validation Estimates Predictive Error

A short justification explains both the value and the limitation of the procedure. Suppose \(Z_1,\ldots,Z_n\) are exchangeable independent prediction units, and let \(L(Z_i,\hat f_{-i})\) be the loss for unit \(i\) under a predictor fitted without that unit. Leave-one-unit-out cross-validation averages these losses. By exchangeability, each omitted unit has the same expected loss as a fresh unit evaluated after fitting on \(n-1\) units. Linearity of expectation then gives

This equality depends on the stated exchangeability and on holding out the true prediction unit. For multi-subject prediction, \(Z_i\) should represent a subject, not an individual measurement from a subject. The result concerns a predictor trained on \(n-1\) units, not exactly the final model trained on all \(n\). With \(K\)-fold cross-validation, each fit uses fewer than all observations, so its estimated performance likewise corresponds approximately to that training size. Dataset shifts, dependent units, poor fold construction, and highly variable scores can undermine the intended interpretation.

Diagnostics, Alternatives, and Common Pitfalls

Before interpreting a ranking, check that each fold’s training data contain enough information to fit each mechanism, that the optimization procedure converged, and that all models were scored on the same held-out outcomes. Record fold-level scores as well as the aggregate. If one fold dominates the total, or if rankings change sharply across folds, the evidence for a stable preference is weak. When the available subjects are few, the variability of cross-validation scores can itself be substantial.

Any data-dependent operation must be performed within each training fold. This includes estimating scaling constants, selecting covariates, choosing a model variant, or tuning a fitting setting. If those choices use held-out outcomes, the reported score no longer represents prediction on untouched data. When the same cross-validation results are used to choose among many mechanisms or tuning choices, the minimum score may be optimistically low. A nested design can address this: use inner folds for choices and outer held-out folds for evaluation. An untouched external test set is another option when enough suitable data are available.

• Common mistake: random point-wise folds automatically represent new-subject prediction. They do not if the same subjects contribute to training and test sets. Group all observations from each subject when the target is a new subject.
• Common mistake: the lowest score proves the mechanism is true. It establishes only a relative predictive result under the specified data, loss, folds, and fitting procedure.
• Common mistake: more observations per subject should always mean more influence. That follows only for an observation-weighted target. An equal-subject target requires averaging subject-level losses.
• Common mistake: fit each model once, then score the same fit on held-out data. Each fold requires a fresh fit using only its training data; otherwise the held-out outcomes may already have influenced the predictions.

Information criteria offer a computationally convenient alternative when their assumptions and likelihood formulation are appropriate, while cross-validation directly assesses a chosen held-out prediction task. Cross-validation can be computationally demanding, especially when QSP calibration is costly, and its result depends on the fold design and score. It does not repair misspecified observation assumptions or ensure that the held-out data represent future applications. A lower cross-validation loss is evidence about prediction under a defined target, not a substitute for biological interpretation or experimental judgment.