Why the Type of Update Matters
An external validation asks how well a model predicts data that were not used to build it. If predictions show a systematic offset, or their predictive intervals are too narrow, it can be tempting to update the model using the external observations. That may improve predictions, but the update changes what the subsequent performance calculation means. A limited recalibration adjusts a small number of prediction-level quantities; refitting changes one or more parameters of the NLME model itself. Neither should be confused with the performance of the original frozen model.
The distinction matters in pharmacometrics because an NLME model connects structural parameters, covariates, random effects, and residual error to concentration predictions. A correction applied to predictions can improve calibration without explaining why the original predictions were biased. Refitting parameters may change the model’s scientific interpretation as well as its predictions. In either case, performance measured on the same external observations used to choose the update is no longer an untouched assessment of the update.
Set Up the Comparison
For subject \(i\) and observation time \(j\), write the original NLME model schematically as \(Y_{ij}=f(t_{ij},x_i;\theta,b_i)+\epsilon_{ij}\), where \(f\) is the structural prediction function, \(x_i\) denotes covariates, \(\theta\) denotes population parameters, \(b_i\) denotes subject-specific random effects, and \(\epsilon_{ij}\) denotes residual error. The random effects and residual errors have distributions specified by the model. External predictions must use the prediction target that was defined for the validation: for example, a population prediction or an individual prediction using a prespecified amount of subject data.
Let \(y_k>0\) be an external observed concentration and \(\hat y_k\) its corresponding frozen-model prediction, with \(k\) indexing prediction-observation pairs. For log-scale work, define \(r_k=\log(y_k)-\log(\hat y_k)\). A positive \(r_k\) means the observation exceeds its prediction on the log scale. The examples below use one independent prediction per subject so that the arithmetic is transparent. With multiple observations per subject, residuals are generally correlated: uncertainty calculations should account for clustering, and a subject with many samples should not automatically dominate merely because more rows were recorded.
Simple recalibration assumes the external data are sufficiently comparable to the intended prediction setting, the outcomes and predictions are correctly paired, and the chosen adjustment is adequate for the error pattern. A common log-scale offset assumes the relevant bias is approximately constant across the prediction range. A dispersion multiplier assumes that spread, rather than the mean prediction, is the part requiring adjustment. These are not assumptions that the original NLME model is correct; they are working assumptions for a particular update.
Recalibrating Bias or Dispersion
A log-scale offset \(a\) changes each prediction to \(\hat y_k^{\,\mathrm{cal}}=\hat y_k\exp(a)\). It corrects a common multiplicative bias: if observations are, on average, 10% below predictions, then the estimated offset is near \(\log(0.90)\). It does not change the time course, covariate effects, or random-effect distribution of the NLME model.
A more flexible calibration regression is \(\log(y_k)=a+b\log(\hat y_k)+e_k\). Ideal mean calibration corresponds to \(a=0\) and \(b=1\). An intercept-only correction fixes a common shift; estimating both \(a\) and \(b\) also changes how predictions vary across their range. Although this is still a prediction-level recalibration, the slope can substantially alter predictions and requires more external information than a single offset.
Dispersion recalibration is different: it leaves the point prediction unchanged and adjusts uncertainty around it. For example, if the predictive distribution is approximately normal on the log scale with standard deviation \(\sigma_{\mathrm{pred}}\), a scale factor \(c\) changes that standard deviation to \(c\sigma_{\mathrm{pred}}\). A factor above one widens predictive intervals; a factor below one narrows them. It does not, by itself, remove a systematic bias in the predicted mean.
Worked Example: Correcting a Common Multiplicative Bias
Suppose three hypothetical independent subjects have frozen predictions of 5.00, 4.00, and 2.50 mg/L, while their observed concentrations are 4.50, 3.60, and 2.25 mg/L. Assume a common log-scale shift is appropriate and estimate it by minimizing the sum of squared log residuals.
The observed-to-predicted ratios are \(0.90\), \(0.90\), and \(0.90\). Thus each log residual is \(\log(0.90)=-0.1054\), and the estimated common shift is \(\hat a=(-0.1054-0.1054-0.1054)/3=-0.1054\).
The multiplicative correction is \(\exp(\hat a)=\exp(-0.1054)=0.9000\). The recalibrated predictions are therefore \(5.00(0.9000)=4.50\), \(4.00(0.9000)=3.60\), and \(2.50(0.9000)=2.25\) mg/L. This exactly matches these three observations because the example was constructed with the same ratio for every subject. It does not establish that the original structural model’s parameters were wrong, or that this correction will work in new subjects.
Worked Example: Recalibrating Predictive Dispersion
Suppose four hypothetical independent subjects have log-scale prediction residuals \(0.0800\), \(-0.0800\), \(0.0400\), and \(-0.0400\). Assume the residuals are centered around zero, and suppose the frozen model assigns each prediction a log-scale standard deviation of \(0.0500\). Use the root mean square residual as a simple estimated log-scale dispersion.
The mean residual is zero. The mean squared residual is \((0.0800^2+(-0.0800)^2+0.0400^2+(-0.0400)^2)/4=0.00400\). Thus the estimated dispersion is \(\sqrt{0.00400}=0.06325\), and the estimated scale factor relative to the model is \(c=0.06325/0.0500=1.265\). On this small hypothetical sample, the predictive spread appears larger than the model’s stated spread. The estimate is imprecise with only four subjects; the calculation is illustrative, not a claim of reliable calibration.
For a prediction centered at 4.00 mg/L, the original approximate 95% log-scale interval is \(4.00\exp(\pm1.96(0.0500))\), or approximately 3.627 to 4.412 mg/L. With the estimated dispersion \(0.06325\), the corresponding recalibrated interval is \(4.00\exp(\pm1.96(0.06325))\), or approximately 3.534 to 4.528 mg/L. The point prediction remains 4.00 mg/L in both cases. These intervals describe predictive uncertainty under the stated approximation, not uncertainty about whether a fixed parameter lies in an interval.
Refitting Changes the NLME Model
Refitting estimates NLME parameters from data. Depending on the prespecified refit, this may involve structural parameters in \(f\), covariate effects, random-effect variances and covariances, or residual-error parameters. For example, updating an elimination-rate parameter changes the predicted concentration-time trajectory, rather than applying the same multiplicative correction at every time. A full refit can also change the estimated distribution of subject-level parameters and the predictive distribution.
A limited update can be useful when only a predictable, stable discrepancy is present and there is a substantive reason to preserve the original model. Refitting may be more appropriate when residual patterns indicate a structural mismatch, such as time-dependent errors inconsistent with a common offset. But a refit has more opportunities to adapt to the external sample, especially when it estimates many parameters. The number and kind of parameters updated should therefore be stated explicitly, rather than describing every improved prediction as “recalibration.”
Worked Example: A Negative Log-Linear Slope Is a Parameter Update
Consider a hypothetical one-compartment, log-linear concentration profile with fixed initial concentration \(C_0=5.00\) mg/L and log concentration \(\log(C_i/C_0)=\beta t_i\). Suppose external observations are 4.30, 3.70, and 2.70 mg/L at times 1, 2, and 4 hours. Estimate \(\beta\) by least squares through the origin, treating \(C_0\) as fixed.
The through-origin estimate is \(\hat\beta=\sum_i t_i\log(C_i/5.00)/\sum_i t_i^2\). The numerator is \(1\log(4.30/5.00)+2\log(3.70/5.00)+4\log(2.70/5.00)\) \(=-0.1508-0.6022-2.4647=-3.2178\). The denominator is \(1^2+2^2+4^2=21\), so \(\hat\beta=-3.2178/21=-0.1532\ \mathrm{h}^{-1}\).
The negative sign is required: concentrations decline over time in this example. The frozen model’s slope is \(-0.1200\ \mathrm{h}^{-1}\), so estimating \(\beta\) from these observations is a parameter refit, not a common bias correction. At 4 hours, the frozen prediction is \(5.00\exp(-0.1200(4))=3.094\) mg/L, whereas the refitted prediction is \(5.00\exp(-0.1532(4))\approx2.709\) mg/L. The latter is close to the observed 2.70 mg/L at 4 hours, but the update has used the external observations to change the time course. This simplified fit illustrates a structural parameter update; a real NLME refit must specify its likelihood, random-effects structure, and residual model.
Why the Estimators Work—and What They Do Not Show
For the offset correction, write \(r_k=\log(y_k)-\log(\hat y_k)\). After applying offset \(a\), the corrected log prediction is \(\log(\hat y_k)+a\), so the corrected residual is \(r_k-a\). Minimizing the sum of squared corrected residuals gives \(\sum_k(r_k-a)^2\). Differentiating with respect to \(a\) gives \(-2\sum_k(r_k-a)\). Setting this derivative to zero yields \(\hat a=\sum_k r_k/K\), the mean log residual. The second derivative is \(2K>0\), so this stationary point is the minimum. This derivation assumes the chosen squared-error criterion and a common offset; it does not establish that the offset is biologically meaningful.
For centered independent normal residuals with common standard deviation \(\sigma\), the log likelihood is proportional to \(-K\log(\sigma)-\sum_k r_k^2/(2\sigma^2)\). Differentiating with respect to \(\sigma\) and setting the derivative to zero gives \(\hat\sigma^2=\sum_k r_k^2/K\). If a mean offset is also estimated, the residuals in this expression should be centered around that estimated mean; degrees-of-freedom adjustments may be used for unbiased variance estimation. These simple formulas do not account for within-subject dependence, unequal predictive variances, censoring, or a more complicated residual distribution. Those features require an appropriate likelihood or resampling method.
The important evidence distinction is not the arithmetic but the data’s role. If an external set is used to estimate \(\hat a\), \(\hat c\), or refitted parameters and then scored on those same observations, the reported score measures fit after adaptation. It is not an independent estimate of how the update will predict future external observations. To evaluate an update, estimate it on one set and score it on new data, or use a prespecified resampling design that respects subject-level dependence. Report the original frozen-model performance separately from post-update performance.
Record predictions and performance from the frozen model before using external outcomes to estimate any correction.
Use an offset for a stable common bias, a dispersion scale for interval spread, or refit parameters when the model structure or parameter values need reassessment.
Assess the updated predictions on observations not used to choose that update, and account for clustering when subjects contribute repeated observations.
Diagnostics, Alternatives, and Limitations
Inspect residual patterns before choosing an update. A nearly constant log residual across time and predicted concentration can support considering a common offset. Residuals that grow or shrink across the prediction range may suggest a calibration slope or a dispersion problem. A systematic time trend, different patterns across covariate groups, or a few influential subjects may indicate that a single adjustment is inadequate. A dispersion increase cannot fix a biased mean; an offset cannot make overly narrow prediction intervals wider.
Also inspect the stability and plausibility of the update. A large offset or dispersion factor estimated from a small external sample may be driven by chance, data harmonization problems, or a few subjects. Check units, dose and sample times, covariates, and the prespecified prediction target before attributing the discrepancy to the model. Compare conclusions with the uncertainty in the update estimates. If the required assumptions are doubtful, report the mismatch and its pattern rather than presenting a simple correction as a solution.
The main alternatives answer different questions. Keeping the model frozen gives the clearest estimate of its performance as originally specified. Recalibration asks whether a restricted prediction-level adjustment can improve predictions in a related setting. Refitting asks whether model parameters should be re-estimated using the new data and may change scientific interpretation. A flexible refit may be warranted, but it needs adequate data and independent evaluation; a simple correction may be more stable, but it cannot repair every structural mismatch. Neither method alone establishes transportability to a different population or setting.
A common mistake is to recalibrate on the external validation set, calculate the improved error on that same set, and call the result the original model’s external validation performance. The reasoning is wrong because the outcomes have influenced the predictions being scored: the assessment is now partly in-sample for the update. The correction is to report the frozen-model score as the original external validation result, label same-set updated scores as post-update fit, and reserve independent data (or a suitable resampling design) for evaluating the updated model.
Check Your Understanding
Answer each question, then compare your reasoning with the brief solutions.
- A set of external observations is consistently 8% below positive frozen predictions. What log-scale offset would apply a common multiplicative correction?
- A model’s point predictions appear unbiased, but its predictive intervals are too narrow. Which limited update is most directly relevant, and what does it leave unchanged?
- Why is estimating a time-varying elimination slope from external concentrations a refit rather than a common bias recalibration?
- After estimating a correction on an external data set, what can be concluded from scoring the corrected predictions on that same set, and what further evaluation is needed?
Solutions. (1) The ratio is \(0.92\), so the log offset is \(a=\log(0.92)\approx-0.08338\); predictions are multiplied by \(0.92\). (2) A dispersion scale is most directly relevant; it widens predictive intervals while leaving point predictions unchanged. (3) The slope changes the predicted concentration-time trajectory as a function of time, rather than applying the same offset to every prediction. (4) The score describes performance after adapting to those observations, not independent performance of the correction. Evaluate it on new observations or with a prespecified resampling design that respects subject-level dependence.