Why Test Mechanisms Outside Their Calibration Conditions?
Two QSP mechanisms can reproduce the same calibration data while implying different responses to a new perturbation, dose, or biological context. A fit to the conditions used to estimate parameters therefore does not by itself establish that a mechanism will predict beyond those conditions. A useful test is to compare the mechanisms on evidence that was not used to calibrate them, asking whether their predictions carry over to the new conditions.
This is a test of predictive generalization, not a direct proof of biological truth. A mechanism may predict well because it captures a relevant causal structure, because its flexible parameterization approximates the observed response, or because the held-out conditions happen not to expose its weaknesses. The result is strongest when the held-out evidence challenges meaningful differences between mechanisms and represents the contexts where predictions will be used.
The distinction is about information use, not simply about labeling some rows “test data.” A new dose measured in the same assay may test interpolation or dose extrapolation, depending on where it lies relative to calibration doses. A perturbation of a different pathway may test transfer to a new intervention. A change in disease state or species may additionally challenge assumptions about which model structure and parameters transfer. These are different scientific challenges, and they should be named rather than treated as interchangeable.
Set Up a Fair Held-Out Comparison
Before looking at held-out outcomes, specify the candidate mechanisms, the calibration data, the held-out conditions, the outcomes to be scored, and the scoring rule. Calibrate each mechanism using only the calibration data, then freeze its structure, parameter estimates, and relevant analysis choices. Generate predictions for the held-out conditions without using their observed outcomes to revise the model. This separation is what makes performance on the held-out data informative.
Comparisons also require commensurable predictions and observations. Define the outcome, units, measurement time, assay, and context consistently. If a prediction concerns a biomarker concentration at 24 hours, it should not be compared with an observed concentration at a different time without a clearly specified mapping. Use the same observation and error assumptions for candidate mechanisms unless there is a defensible reason not to; otherwise, differences in score may reflect different scoring choices rather than different biological predictions.
For observed values \(y_i\) and predictions \(\hat{y}_i\) at \(n\) held-out observations, a simple summary is the root mean squared error (RMSE):
RMSE is expressed in the outcome's units, and smaller values indicate closer predictions on average under squared-error weighting. Squaring gives larger errors disproportionate influence; the square root returns the summary to the original units. RMSE does not account for differences in measurement precision or predictive uncertainty, so it is most interpretable when the scored outcomes are on a common scale with comparable uncertainty.
For outcomes with different units or prediction uncertainties, use standardized predictive residuals. If \(\mu_i\) is a model's predicted mean and \(s_i\) is the predictive standard deviation for observation \(i\), define \(z_i=(y_i-\mu_i)/s_i\). A simple aggregate score is the sum of squared standardized residuals, \(\sum_i z_i^2\), or its average across observations. The predictive standard deviation should reflect the uncertainty relevant to the prediction, including measurement error and, when available, parameter uncertainty. A small standardized residual means the observed outcome is close relative to the model's predictive uncertainty; it does not mean the prediction is accurate in every biologically relevant respect.
Worked Examples
Worked Example: Comparing Predictions for a Held-Out Perturbation
Hypothetical data. Two candidate mechanisms were calibrated using a baseline condition and a reference perturbation. Neither was fitted to a new pathway perturbation. The held-out biomarker responses are 40, 55, and 62 units at three prespecified measurement times. Mechanism A predicts 43, 51, and 68 units; Mechanism B predicts 49, 58, and 65 units. Assume the same assay and measurement definition were used at all three times.
For Mechanism A, the observed-minus-predicted errors are \(40-43=-3\), \(55-51=4\), and \(62-68=-6\) units. Thus,
For Mechanism B, the errors are \(40-49=-9\), \(55-58=-3\), and \(62-65=-3\) units, so
Mechanism A has the smaller RMSE on this held-out perturbation, by about \(5.745-4.509=1.236\) units. This is evidence that A predicted these three outcomes more accurately under squared-error scoring. It is not proof that A is the true mechanism: there are only three held-out observations, the result pertains to this perturbation and outcome, and other contexts might favor B.
Worked Example: Scoring Held-Out Doses with Known Predictive Uncertainty
Hypothetical data. Two doses not used in calibration produce observed biomarker changes of 28 and 74 percentage points. The mechanisms' predictions are A: 35 and 85 percentage points; B: 30 and 68 percentage points. Suppose the predictive standard deviation is 5 percentage points at each dose for both mechanisms. This common uncertainty makes a comparison of standardized errors straightforward.
For A, the observed-minus-predicted errors are \(-7\) and \(-11\) percentage points. Its RMSE and summed squared standardized residual are
For B, the errors are \(-2\) and \(6\) percentage points:
Both summaries favor B for these held-out doses. The standardized score additionally expresses the errors relative to the stated predictive spread: the two squared standardized residuals for B sum to 1.600, versus 6.800 for A. With only two dose observations, this comparison is limited. It indicates which mechanism better matched these data, not that B is reliable at every dose outside calibration.
Worked Example: Comparing Different Biomarkers in a New Context
Hypothetical data. In a held-out inflammatory context, the observed cytokine concentration is 80 units and receptor occupancy is 0.60. The predictive standard deviations are 10 concentration units and 0.05 occupancy units, respectively. Mechanism F predicts 65 units and 0.68; Mechanism G predicts 77 units and 0.55. These predictive standard deviations are assumed to be appropriate for comparing each observation with its prediction.
For F, the standardized residuals are \((80-65)/10=1.5\) for cytokine and \((0.60-0.68)/0.05=-1.6\) for occupancy. Therefore, the summed squared standardized residual is
For G, the standardized residuals are \((80-77)/10=0.3\) and \((0.60-0.55)/0.05=1.0\), giving
The aggregate standardized score favors G across these two outcomes. The units differ, so adding raw squared errors would not provide a meaningful common-scale comparison. The standardized score avoids that particular problem, but its interpretation depends on the predictive standard deviations being credible. A model can look artificially good if its uncertainty is made very broad; uncertainty calibration must therefore be inspected alongside the score.
Why Standardized Errors Reflect Both Bias and Noise
A simple justification shows why standardized squared error penalizes a systematically inaccurate prediction. Suppose an observation is generated as \(Y=\mu^*+\varepsilon\), where \(\mu^*\) is the true conditional mean and the error has mean zero and variance \(\sigma^2\). Let \(m\) be a fixed prediction and \(r=Y-m\). Then \(r=(\mu^*-m)+\varepsilon\). Squaring and taking expectations gives
The cross-product has expectation zero because \(E(\varepsilon)=0\). Under these assumptions, expected squared standardized error is 1 for an unbiased prediction and increases with squared prediction bias relative to the noise variance. This is a justification for the score, not a universal reference distribution for an observed score: fitted parameters, correlated observations, uncertain predictive variances, and small numbers of held-out conditions can change its sampling behavior.
Diagnostics, Alternatives, and Limits
A single aggregate score can conceal how a mechanism fails. Inspect observed-versus-predicted values, signed residuals, and errors by dose, time, perturbation, and biological context. Look for consistent overprediction or underprediction, errors that grow with dose, and a context in which one mechanism performs especially poorly. If measurements within a context are correlated, treating each as independent can make that context appear to contribute more evidence than it does. Consider summarizing performance by context or using a scoring method that represents the dependence, rather than interpreting every measurement as independent confirmation.
A held-out comparison is compromised if its outcomes influenced calibration, parameter bounds, model structure, endpoint selection, or repeated choices among scoring rules. Such information leakage makes the evaluation optimistic. If the held-out results are used to revise a mechanism, those data are now part of model development; an additional untouched evaluation set would be needed for an independent assessment. The earlier material on predictive cross-validation addresses repeated data-splitting for prediction; here, the central question is whether transfer to a scientifically distinct condition has been tested without that condition shaping the model.
| Approach | Useful for | Main caution |
|---|---|---|
| Raw RMSE | Comparing predictions for one outcome on a common scale | Large errors dominate; uncertainty and differing outcome scales are not accounted for |
| Standardized predictive error | Comparing outcomes with different units or predictive spreads | Depends on credible predictive uncertainty; can hide a poor individual outcome in an aggregate |
| Predictive log score | Evaluating the probability density assigned to observed outcomes, including uncertainty | Requires a specified predictive distribution and is sensitive to its calibration and tails |
| Calibration-data fit or information criteria | Summarizing fit-complexity trade-offs within the data used for model comparison | Does not substitute for testing transfer to an unobserved biological context |
The choice of held-out evidence matters as much as the metric. A narrow dose increment may provide a weak test of transfer if both mechanisms make similar predictions there. Conversely, a new context may differ in assay, disease state, or treatment history as well as in the biological feature of interest, making a failure difficult to attribute to the mechanism alone. State what changed and what was held constant. Also report the number and range of held-out contexts: a large number of repeated measurements in one context does not establish generalization across many contexts.
Finally, better held-out predictions support a bounded claim: one candidate predicted the specified outcomes better, under the stated conditions and scoring assumptions. They do not show that its biological explanation is uniquely correct, that the competing mechanism is impossible, or that the selected mechanism will generalize to untested populations and interventions. Mechanism selection should be interpreted alongside parameter and model-selection uncertainty, biological plausibility, and the intended scope of application.
Check Your Understanding
Answer each question, then compare with the solutions below.
- What makes an outcome genuinely held out for a mechanism comparison?
- For two held-out observations of 10 and 14 units, a model predicts 12 and 11 units. Calculate its RMSE.
- An observation is 0.40, its predicted mean is 0.50, and its predictive standard deviation is 0.05. Calculate its standardized residual and squared standardized residual.
- If held-out outcomes are used to change a model's parameters, can those same outcomes still provide an untouched evaluation of the revised model? Explain.
- Does a lower held-out score establish that a mechanism is universally correct? Why or why not?
Solutions. (1) The outcomes must not have influenced parameter estimation, model choice, or tuning decisions being evaluated. (2) The errors are \(10-12=-2\) and \(14-11=3\), so \(\mathrm{RMSE}=\sqrt{((-2)^2+3^2)/2}=\sqrt{6.5}\approx2.550\) units. (3) The standardized residual is \((0.40-0.50)/0.05=-2.000\); its square is 4.000. (4) No. Once used to revise the model, those outcomes are development data, not an untouched evaluation. (5) No. A lower score supports better prediction for the tested evidence under the stated scoring assumptions; it does not establish correctness in untested contexts or uniquely prove the mechanism.