Two Questions, Not Two Names for the Same Result
A PBPK model can reproduce observations used to estimate its parameters and still predict new observations poorly. Calibration asks how well the model can be made to fit a specified set of data by choosing parameter values. Validation asks how well predictions from that fitted model agree with observations that were not used to choose those values. These questions are related, but a good answer to the first does not by itself answer the second.
For example, a model may be calibrated to concentration-time data from one study by adjusting clearance. The residuals from that study describe fit conditional on those data and the fitting choices. If the estimated parameters are then used, without adjustment, to predict concentrations in a separate study, agreement there provides a different kind of evidence: the prediction was tested against observations that did not determine the fitted parameter values.
Let \(C_{\mathrm{obs},i}\) be an observed concentration at a specified time and \(C_{\mathrm{model},i}(\theta)\) the corresponding PBPK prediction under parameter vector \(\theta\) and the study conditions. Calibration selects \(\hat{\theta}\) by minimizing a defined discrepancy, for example a sum of squared log residuals. The data, model structure, fixed inputs, objective function, and parameter constraints all affect the resulting estimate. Consequently, a calibration result is not a property of the model structure alone.
Here \(\mathcal{C}\) indexes the calibration observations, \(w_i\) are specified nonnegative weights, and \(\Theta\) is the allowed parameter set. This particular objective is appropriate only when the concentrations being logged are positive and when deviations on the log scale match the intended error assessment. Equal weights give equal importance to squared log differences, not necessarily equal importance to absolute concentration differences. Other objectives may be appropriate for other measurement-error assumptions or scientific purposes.
What Calibration Can Establish
Calibration establishes how the chosen model performs against the data that informed parameter selection, under the stated objective and conditions. It can help estimate parameters and reveal whether the model can reproduce the calibration profiles to an acceptable degree. It does not establish that the selected parameters are uniquely determined, that the model's mechanisms are correct, or that predictions for unobserved conditions will be accurate.
A simple derivation shows why optimizing on calibration data tends to improve the calibration objective. If \(\hat{\theta}\) is a global minimizer over \(\Theta\), and \(\theta_0\) is another allowed candidate, then by the definition of a minimum,
where \(L_{\mathcal{C}}\) is the calibration loss. This inequality says nothing about the loss \(L_{\mathcal{V}}\) on a distinct validation set \(\mathcal{V}\): those observations do not appear in the minimization. There is no corresponding guarantee that \(L_{\mathcal{V}}(\hat{\theta})\) is small. With a numerical optimizer that finds only a local solution, even the stated reduction in calibration loss is relative to the candidates actually compared, rather than a guarantee of a global minimum.
Worked Example: Fitting a Multiplicative Scale to Calibration Data
Consider a deliberately simplified, hypothetical illustration. A PBPK model produces baseline predictions for four positive concentrations. A dimensionless scale parameter \(q\) multiplies each prediction. The observed-to-baseline-prediction ratios are \(11/10\), \(11/10\), \(19/20\), and \(14/15\). Suppose calibration minimizes the sum of squared log residuals with equal weights.
For observation \(i\), the model-to-observation ratio after scaling is \(q/r_i\), where \(r_i\) is the observed-to-baseline ratio. The objective is therefore
Write \(a=\ln q\) and \(b_i=\ln r_i\). Then \(L(q)=\sum_{i=1}^{4}(a-b_i)^2\). Differentiating with respect to \(a\) gives \(2\sum_i(a-b_i)\); setting this equal to zero yields \(a=\frac{1}{4}\sum_i b_i\). The second derivative is \(8>0\), so this is the minimum. Thus the fitted scale is the geometric mean of the observed-to-baseline ratios:
Before scaling, the log-RMSE is \(\sqrt{\left[2\ln(11/10)^2+\ln(19/20)^2+\ln(14/15)^2\right]/4}\approx 0.0799360\). After fitting \(q\), the residual log ratios are centered by their mean, giving a log-RMSE of approximately \(0.07798\). Both are dimensionless. The reduction is a calibration result: the scale was chosen using these four observations. It is not evidence that the adjusted predictions will work for a new study. The scale parameter is used here to make the calculation transparent; it is not a claim that every PBPK parameter acts as a common concentration multiplier.
What Independent Validation Adds
For validation, the fitted parameter values are used to generate predictions for observations that did not determine those values. Compare prediction and observation on a scale suited to the scientific question. For positive concentrations, a useful descriptive measure is the root mean squared log error (log-RMSE), which summarizes the typical magnitude of multiplicative disagreement:
Here \(m\) is the number of evaluated observations. A value of zero means exact agreement at every evaluated point; larger values indicate greater average squared disagreement on the log scale. The metric is dimensionless and does not indicate the direction of error, so the individual ratios or signed residuals should also be examined. A single summary can conceal a systematic overprediction at some times and underprediction at others.
Worked Example: Evaluating Predictions in a Held-Out Study
Suppose a hypothetical model was calibrated using one study. With its fitted parameter values unchanged, it predicts concentrations in a separate study. At four specified sampling times, the observed concentrations are 10, 20, 25, and 15 milligrams per litre; the corresponding predictions are 10.5, 19.6, 27, and 13.95 milligrams per litre.
The prediction-to-observation ratios are \(1.05\), \(0.98\), \(1.08\), and \(0.93\). The log-RMSE is
For illustration only, suppose the evaluation plan had defined a log-RMSE below \(0.08\) and ratios between \(0.90\) and \(1.10\) as acceptable. All four ratios fall within that range, and \(0.0591<0.08\), so this hypothetical result meets those stated criteria. This supports predictive performance for the observations and conditions evaluated; it does not prove accuracy in every population, protocol, dose, or application. The numerical criteria are illustrative, not universal standards.
A held-out study contributes evidence only insofar as its observations were not used to tune the parameter values being assessed. If its outcomes influence parameter adjustment, the resulting fit is no longer an evaluation of predictions made without that adjustment. The distinction is about the role the observations played, not about whether the model output is a concentration-time curve or a summary endpoint.
Worked Example: Why Refitting Changes the Validation Question
A hypothetical fitted model predicts concentrations of 10 and 20 milligrams per litre at two observations in a separate study. The observed concentrations are 12 and 24 milligrams per litre. The prediction-to-observation ratios are both \(10/12=20/24=0.8333\), so the log-RMSE is
If a multiplier \(q=1.2\) is estimated from these same two observations, the adjusted predictions become 12 and 24 milligrams per litre, and the resulting residuals are zero. That zero error describes calibration to these observations. It cannot be reported as the original model's validation performance, because the observations have now been used to change the model parameters. The original, unadjusted log-RMSE of approximately \(0.1823\) remains the relevant assessment of the original predictions.
Diagnostics, Alternatives, and Limitations
A calibration summary should be checked against the data and the fitting process that produced it. Plot predicted and observed concentration-time profiles, inspect signed and relative residuals over time, and look for patterns such as consistent late-time underprediction. Check that the objective, weights, units, and observation matching are as intended. A small overall error can hide a poor fit in a scientifically important part of the profile. For log-based measures, zero or nonpositive values cannot be logged; values below quantification limits require an appropriate treatment rather than an arbitrary replacement with zero.
Numerical error can also obscure the distinction. If integration error is appreciable relative to the observed residuals, apparent parameter changes or close agreement may reflect solver behavior rather than the intended model. The earlier material on testing solver convergence and numerical error is relevant: predictions used for fitting or evaluation must be numerically reliable enough for the comparison being made. A failed integration is not a valid concentration prediction.
The main alternative to reporting only calibration fit is to evaluate predictions on observations not used in fitting. These approaches answer different questions, rather than competing to provide one universally superior score. Calibration is necessary when parameters must be estimated from data; independent validation adds evidence about prediction beyond those data. Calibration residual checks remain useful diagnostics, but they do not substitute for that separate predictive assessment. Validation also has limits: success in one held-out study supports the tested setting and observations, not unrestricted extrapolation. A model can perform well in both sets yet still have uncertain or compensating parameter estimates.
Check Your Understanding
Answer each question, then compare with the solutions below.
- A PBPK parameter is estimated from one study, and the model predicts a second study without parameter adjustment. What question does the second comparison address?
- Why does minimizing a calibration loss not guarantee a small loss in a separate study?
- For positive concentrations, what does a log-RMSE of zero mean, and what information can it conceal?
- A model is adjusted using observations originally set aside for evaluation. How should the result be described?
- Give one reason a small calibration error may not be reliable evidence of a good mechanistic or predictive model.
Solutions. (1) It assesses agreement of predictions with observations not used to estimate the parameter values, so it addresses validation for that evaluated study. (2) The calibration objective contains the calibration observations, whereas the separate-study loss is not part of that minimization and has no guaranteed value. (3) Zero means exact prediction-observation agreement at every evaluated point; the summary can conceal error direction or patterns across sampling times. (4) It is calibration using those observations; the adjusted fit is not validation of the original unchanged predictions. (5) For example, errors may cancel in an overall summary, numerical integration may be insufficiently accurate, or different parameter combinations may reproduce the calibration data while implying different predictions.