Calibration Fit Versus Predictive Validation

Why Calibration Fit Is Not Predictive Validation

A QSP model can reproduce the observations used to estimate its parameters and still predict new observations poorly. This is not a contradiction: fitting asks how well parameter values can align the model with a particular data set, while predictive validation asks whether predictions made without access to the validation outcomes agree with independent evidence. The distinction matters whenever a model is used to support a claim beyond the data that shaped it.

For example, an elaborate model may match a measured biomarker time course by adjusting several parameters or adding mechanistic interactions. That agreement is evidence about calibration. It becomes evidence of predictive performance only when the model makes predictions for relevant observations that were not used to choose its parameters or structure, and those predictions are then compared with the observations.

Definition: Calibration is the process of estimating model parameters, and sometimes selecting among candidate structures, using specified data. Calibration fit describes agreement between the resulting model and those same data. Predictive validation evaluates predictions against observations that were withheld from calibration and were not used to select, tune, or revise the model. Validation evidence applies to the prediction context represented by those withheld observations; it does not establish universal validity.

Write the model prediction for observation \(i\) as \(f(x_i;\theta)\), where \(x_i\) denotes the conditions and inputs, and \(\theta\) is the parameter vector. A simple calibration procedure chooses parameters to minimize a loss such as the sum of squared differences between observations \(y_i\) and predictions:

L_C(\theta)=\sum_{i\in C}\left[y_i-f(x_i;\theta)\right]^2, \qquad \hat{\theta}=\arg\min_{\theta}L_C(\theta).

Here \(C\) denotes the calibration observations. When outcome measurements have different known variances, a weighted loss may be appropriate, but the weights and error model must be justified. Neither an unweighted nor a weighted loss, by itself, establishes that predictions will be accurate outside \(C\).

What a Calibration Result Does—and Does Not—Show

A low calibration loss shows that the fitted model can represent the calibration observations under the chosen metric. It does not show that the model has identified the true biological mechanism, that its parameters are uniquely determined, or that its predictions will remain accurate under new conditions. Different parameter combinations or structures can produce similar fitted outputs, especially when the calibration data cover only a narrow range of conditions.

A basic justification for expecting calibration error to be optimistic is built into the fitting procedure. For any fixed candidate parameter vector \(\theta_0\), the fitted vector minimizes the calibration loss, so:

L_C(\hat{\theta})\leq L_C(\theta_0).

This inequality follows directly from the definition of \(\hat{\theta}\) as a minimizer. If the fitted parameters were chosen to reduce calibration discrepancies, evaluating those same observations rewards that optimization. A separate validation set, by contrast, did not determine \(\hat{\theta}\); its error can reveal weaknesses that the calibration loss concealed. The inequality does not claim that every realized calibration error must be numerically smaller than every validation error. Rather, parameter tuning creates an advantage on the data used for tuning, and that advantage can make in-sample fit an optimistic estimate of predictive performance.

This reasoning assumes that the validation observations are genuinely independent of decisions made during calibration and are relevant to the intended prediction claim. Independence is often at the level of experimental units: if several measurements come from one subject, placing some of that subject’s measurements in calibration and others in validation may leak subject-specific information. The data-generating setting, assay, outcome definition, and input conditions must also be sufficiently aligned with the intended use for the comparison to be informative.

Worked Example: Fitting a Longitudinal Response and Checking a Withheld Time

Suppose a hypothetical biomarker response is measured in arbitrary units at days \(t=0,1,2\). The calibration observations are 10, 7, and 6 units. Consider the model \(y=B+A e^{-kt}\), with the decay rate fixed at \(k=\ln(2)\) per day from information specified before calibration. Thus \(z=e^{-kt}\) takes values 1, 0.5, and 0.25, and fitting \(A\) and \(B\) is a linear least-squares problem. A separate day-3 observation, 4.8 units, is withheld.

The calibration means are \(\bar{z}=0.583333\) and \(\bar{y}=7.666667\). The least-squares slope and intercept are:

A=\frac{\sum_i(z_i-\bar{z})(y_i-\bar{y})}{\sum_i(z_i-\bar{z})^2} =\frac{1.583333}{0.291667}\approx5.42857, \qquad B=\bar{y}-A\bar{z} =7.666667-(5.42857)(0.583333)\approx4.500.

The fitted model is therefore \(y=4.500+5.42857z\). At the three calibration times, the predictions are approximately 9.9286, 7.2143, and 5.8571 units. The squared errors are approximately 0.005102, 0.045918, and 0.020408 units squared, giving calibration sum of squared errors \(0.07143\) units squared.

At day 3, \(z=e^{-3\ln(2)}=0.125\). The prediction is \(4.500+5.42857(0.125)=5.17857\) units. The withheld observation is 4.8 units, so the prediction error, defined here as observed minus predicted, is \(4.8-5.17857=-0.37857\) units. Its squared error is approximately \(0.14332\) units squared. The calibration fit is close, but the withheld point does not coincide with the prediction. One withheld observation is limited evidence, not a definitive verdict; it does demonstrate why a small calibration error alone cannot answer the predictive question.

Flexible Mechanistic Structure Can Overfit

Overfitting is not limited to statistical models with many arbitrary coefficients. It can also occur when mechanistic structure is more flexible than the data can support. Adding a pathway, interaction, feedback term, or condition-dependent parameter may improve calibration fit. If the data do not independently constrain that added structure, the improvement may reflect accommodation of the observed points rather than a mechanism that predicts correctly elsewhere.

Worked Example: An Interaction Term Fits Three Points Exactly

Consider hypothetical calibration observations of an outcome \(y\), in units, measured under two dimensionless inputs \(x\) and \(z\): \((x,z,y)=(0,0,6),(1,0,10),(1,1,8)\). A proposed mechanistic model includes an interaction term:

$$ f(x,z)=c+a x+b xz. $$

Using the first observation gives \(c=6\). The second gives \(6+a=10\), so \(a=4\). The third gives \(6+4+b=8\), hence \(b=-2\). The fitted model is \(f(x,z)=6+4x-2xz\), which predicts 6, 10, and 8 units at the three calibration points. Its calibration sum of squared errors is zero.

Now suppose a withheld experiment has \(x=2\), \(z=1\), and an observed outcome of 7 units. The model predicts \(6+4(2)-2(2)(1)=10\) units. The validation error is \(7-10=-3\) units, with squared error \(9\) units squared. Exact calibration agreement did not protect this prediction.

The example is deliberately small: three coefficients were determined by three observations, leaving no calibration residual degrees of freedom for assessing lack of fit. It does not prove that the interaction is biologically false. It shows that exact agreement at the fitted points is insufficient evidence that the added term captures a relationship that generalizes. Mechanistic plausibility, parameter identifiability, and independent prediction evidence are distinct considerations.

Comparing Calibration and Validation Performance

Calibration and validation errors should be summarized with a defined metric and interpreted in the outcome’s units. For \(m\) observations, the root mean squared error is the square root of the mean squared prediction error:

\operatorname{RMSE}= \sqrt{\frac{1}{m}\sum_{i=1}^{m}(y_i-\hat{y}_i)^2}.

It is expressed in the same units as the outcome and gives relatively greater weight to large errors than the mean absolute error. A useful comparison uses the same outcome scale and a prespecified, relevant validation set. Calibration and validation RMSEs need not match; a gap can be a warning that the fitted performance does not carry over, but its interpretation depends on the validation design, sample size, noise, and conditions represented.

Worked Example: Comparing Error Summaries on the Same Outcome Scale

A hypothetical model is calibrated on three observations. The observed and predicted values, in biomarker units, are 10 and 9.8, 8 and 8.1, and 6 and 6.1. The calibration errors (observed minus predicted) are 0.2, \(-0.1\), and \(-0.1\) units. Thus:

\operatorname{RMSE}_C =\sqrt{\frac{0.2^2+(-0.1)^2+(-0.1)^2}{3}} =\sqrt{\frac{0.06}{3}} \approx0.1414\ \text{units}.

For three withheld observations, the observed values are 5, 4, and 3 units, while the corresponding predictions are 5.4, 4.6, and 3.8 units. The errors are \(-0.4\), \(-0.6\), and \(-0.8\) units. Therefore:

\operatorname{RMSE}_V =\sqrt{\frac{(-0.4)^2+(-0.6)^2+(-0.8)^2}{3}} =\sqrt{\frac{1.16}{3}} \approx0.6218\ \text{units}.

In this hypothetical comparison, the withheld-data RMSE is larger than the calibration RMSE. The difference signals weaker agreement on these withheld observations, but three points do not precisely characterize future performance. The result applies to the validation observations and their represented conditions; it should not be turned into a claim about conditions they do not cover.

Preserving the Predictive Test

The validation comparison is informative only if calibration decisions are completed without using validation outcomes. If a modeler examines a validation residual, changes a parameter or structure, and then reports the revised model’s error on the same observations, those observations have become part of calibration. A fresh independent data set is then needed for an independent validation assessment. The introductory material on holdout design and leakage prevention covers how to plan such separations; the central point here is that the label “validation” does not make data independent after they have influenced model development.

A single holdout can also be unrepresentative or too small to reveal unstable performance. Cross-validation repeatedly withholds portions of a data set and can help estimate internal predictive performance, but it does not substitute for evidence from an independent setting when the prediction claim concerns new populations, laboratories, or conditions. The choice among a holdout, resampling, or external data depends on the claim and the data available. In every case, the evaluation must reflect the unit of independence and the conditions relevant to the proposed use.

Residual diagnostics for longitudinal predictions can help identify patterns that an aggregate error metric hides. They do not change the basic distinction: inspecting calibration residuals evaluates fit to calibration data, whereas inspecting untouched validation residuals evaluates predictions on withheld data. Likewise, a close validation fit supports a bounded claim about the represented prediction context; it does not establish that every component of the model is mechanistically correct.

Common Pitfalls and Practical Interpretation

Calling a small calibration error validation. Calibration error measures agreement on data used to tune the model. Use genuinely withheld outcomes for predictive evaluation.
Revising the model after viewing holdout outcomes. Once validation data guide a revision, they are no longer independent for evaluating that revision.
Splitting repeated measurements carelessly. If measurements from the same subject or experimental unit occur in both sets, information can leak across the split. Separate independent units when that matches the prediction claim.
Equating exact fit with mechanistic truth. Flexible structure can interpolate observed points without predicting unobserved conditions accurately. Exact fit is not proof of a biological mechanism.
Overgeneralizing from a good validation result. A validation result supports predictions only to the extent that the data represent the target outcomes, inputs, and experimental context.

A sound report therefore states which observations informed parameter estimation and model selection, which were withheld, what prediction metric was used, and what conditions the validation data represent. It reports both calibration and validation performance without treating either as a complete description of model credibility. The main lesson is precise: calibration establishes how well a model can be made to fit selected data; independent validation supplies evidence about prediction beyond those data.

Check Your Understanding

Answer each question, then compare with the solutions below.

A model’s parameters and interaction structure were selected using a data set. What does a very small squared error on that same set establish, and what does it not establish?
A model predicts 12 units for a withheld observation of 10 units. Using observed minus predicted as the error, calculate the error and squared error, with units.
A researcher uses validation residuals to adjust one parameter and then reports error on those same observations. Are those observations still independent validation data for the revised model? Explain.
Why might an exact fit from a model with three freely estimated coefficients to three calibration observations provide little evidence about prediction at a new input condition?

Solutions. (1) The small error establishes close calibration fit under the selected squared-error metric. It does not establish accurate predictions on independent data, unique parameter values, or a correct mechanism. (2) The error is \(10-12=-2\) units; the squared error is \((-2)^2=4\) units squared. (3) No. The residuals influenced parameter choice, so those observations now contribute to calibration; an independent data set is needed to assess the revised model independently. (4) The three observations can determine three coefficients exactly, leaving no residual degrees of freedom to assess calibration lack of fit. The model’s agreement at those points alone does not constrain its prediction at a new condition sufficiently.