Time- and Concentration-Dependent External Performance

Why stratify external prediction errors?

A model can have small average prediction error across an external dataset and still perform poorly at particular parts of the concentration–time profile. For example, predictions that are too low soon after a dose can be offset by predictions that are too high later. A pooled summary may then look close to unbiased even though the model misses important time-dependent behavior. Stratifying errors by sampling time and concentration makes these patterns visible.

The purpose is focused diagnosis of where predictive performance varies, not a claim that a particular model component is responsible. A pattern concentrated near early samples may be consistent with an absorption-related problem; one concentrated in an intermediate range may motivate review of distribution behavior; and one concentrated in low, late concentrations may motivate review of elimination behavior. Such patterns suggest what to investigate, but they do not establish a mechanism on their own.

The evaluation should use predictions generated under the external-validation plan, with the model locked and the prediction target held consistent. In particular, state whether predictions are population predictions or conditional predictions, and use that choice consistently across strata. The earlier material on prediction error metrics provides the broader performance context; here, the question is how those errors vary over time and concentration.

Define the errors and the strata

Definition: For external observation \(j\), let \(C_{\mathrm{obs},j}\) be the observed concentration and \(C_{\mathrm{pred},j}\) its corresponding prediction from the locked model. When both are positive and expressed in the same concentration units, define the signed log prediction error as \(e_j=\ln(C_{\mathrm{obs},j}/C_{\mathrm{pred},j})\). Positive \(e_j\) means the observation exceeds the prediction; negative \(e_j\) means the prediction exceeds the observation. A stratum is a prespecified group of observations defined by sampling time, concentration, or both.

The ratio \(C_{\mathrm{obs}}/C_{\mathrm{pred}}\) is dimensionless, so its logarithm is well-defined without carrying concentration units. Exponentiating a mean signed log error gives a geometric mean observed-to-predicted ratio. A value above 1 indicates average underprediction on this multiplicative scale; a value below 1 indicates average overprediction. This summary describes multiplicative bias and does not imply that every observation in the stratum has that error.

Choose time windows and concentration bands before reviewing the error pattern whenever possible. Time is usually measured relative to a dose or another clearly defined dosing event. Concentration bands can use clinically meaningful ranges or prespecified quantiles, with their cut points reported. If observed concentrations define the bands, say so: this is a descriptive comparison conditional on the observed outcome, not an independent grouping variable. Predicted-concentration bands can be useful as a complementary view because they do not use the observed concentration to assign the stratum.

The basic assumptions are that observations and predictions are correctly paired; concentration units and time origins are harmonized; the prediction method is the same in every stratum; and the strata contain enough independent information to support useful summaries. Repeated samples from one person are not generally independent. Treating them as independent can make uncertainty appear smaller than it is, so uncertainty calculations should preserve the subject as the sampling unit when subjects contribute multiple samples.

Summarize within strata, then compare

For a stratum containing \(m\) observations, the mean signed log error and its exponentiated form are:

\bar e=\frac{1}{m}\sum_{j=1}^{m}e_j,\qquad \exp(\bar e)=\left(\prod_{j=1}^{m}\frac{C_{\mathrm{obs},j}}{C_{\mathrm{pred},j}}\right)^{1/m}.

The second expression follows because the exponential of a sum of logarithms is the product of their arguments, and taking the \(m\)th root gives the geometric mean. Thus, if \(\exp(\bar e)=1.15\), the geometric mean observed concentration is 1.15 times the predicted concentration in that stratum. This is a concise measure of systematic multiplicative discrepancy; it is not an estimate of the error for every individual.

A useful display reports, for each time window and concentration band, the number of observations, number of subjects, mean signed log error, exponentiated mean, and a measure of spread or uncertainty. A plot of individual errors against time or concentration can complement a table, especially when a small number of cut points might hide a gradual trend. These summaries should be read alongside overall error metrics and calibration results, not as substitutes for them.

Worked Example: Opposing Time Patterns Hidden by the Pooled Mean

In a hypothetical external validation, four samples have the following observed-to-predicted concentration ratios: at 0.5 hours after dosing, 1.20 and 1.10; at 8 hours, 0.80 and 0.90. Assume all measurements are positive and comparable, and that each prediction was generated using the same locked prediction approach.

For the early samples, the signed log errors are \(\ln(1.20)=0.18232\) and \(\ln(1.10)=0.09531\). Their mean is \((0.18232+0.09531)/2=0.13882\), and exponentiating gives \(\exp(0.13882)=1.149\). On the geometric scale, the observed concentrations are about 1.149 times the predictions in this early window.

For the later samples, the errors are \(\ln(0.80)=-0.22314\) and \(\ln(0.90)=-0.10536\). Their mean is \((-0.22314-0.10536)/2=-0.16425\), and \(\exp(-0.16425)=0.8485\). The later observations are therefore about 0.8485 times the predictions on the geometric scale.

Pooling the four errors gives \((0.18232+0.09531-0.22314-0.10536)/4=-0.01272\), using the unrounded logarithms. Its exponent is approximately \(0.9874\), which is close to 1. The pooled summary suggests little average multiplicative bias, but the stratified summaries show early underprediction and later overprediction. This example illustrates why a near-neutral pooled result does not rule out systematic time-dependent error.

Worked Example: Comparing Prespecified Concentration Bands

Consider a second hypothetical external dataset. Three observations fall in a prespecified low observed-concentration band: observed concentrations are 2, 3, and 4 mg/L, and their observed-to-predicted ratios are 0.80, 0.90, and 0.90. Three more observations fall in a high band: observed concentrations are 12, 15, and 18 mg/L, with ratios 1.10, 1.20, and 1.10. All predictions use the same locked model.

In the low band, the mean log error is \(\{\ln(0.80)+\ln(0.90)+\ln(0.90)\}/3=(-0.22314-0.10536-0.10536)/3=-0.14462\). Exponentiating gives \(\exp(-0.14462)=0.8653\). The observed concentrations are about 0.8653 times the predictions on the geometric scale, indicating overprediction in this band.

In the high band, the mean is \(\{\ln(1.10)+\ln(1.20)+\ln(1.10)\}/3=(0.09531+0.18232+0.09531)/3=0.12431\). Exponentiating gives \(\exp(0.12431)=1.1324\). The observations are about 1.1324 times the predictions, indicating underprediction in the high band.

This contrast is a descriptive pattern, not proof that concentration itself causes the prediction error. Because observed concentration defines the bands, the comparison is conditional on the outcome. The result should be checked against time-stratified summaries and, where appropriate, a complementary stratification by predicted concentration.

Use joint strata carefully

Time and concentration are related: samples taken at different times often occupy different concentration ranges. Separate time and concentration summaries can therefore leave ambiguity about which pattern is present. A joint table can show whether the time pattern persists within concentration ranges, but splitting the data into many cells rapidly reduces precision. The design should have enough observations and subjects in each planned cell to make the comparisons interpretable.

Worked Example: Reading a Joint Time-by-Concentration Table

A hypothetical dataset has two time windows, early and late, and two concentration bands, low and high. The signed log errors in the four cells are as follows: early-low, 0.10 and 0.20; early-high, 0.05 and 0.15; late-low, -0.20 and -0.10; late-high, -0.05 and 0.05. Assume the windows and bands were specified before these errors were examined.

The cell means are: early-low, \((0.10+0.20)/2=0.15\); early-high, \((0.05+0.15)/2=0.10\); late-low, \((-0.20-0.10)/2=-0.15\); and late-high, \((-0.05+0.05)/2=0\). Their exponentiated means are approximately 1.162, 1.105, 0.8607, and 1.000, respectively.

The overall mean across these eight observations is \((0.10+0.20+0.05+0.15-0.20-0.10-0.05+0.05)/8=0.025\), with exponent \(\exp(0.025)=1.025\). That pooled value is close to 1, yet the early cells show positive errors and the late-low cell shows negative errors. The late-high cell is centered at zero. The joint display therefore suggests that the error pattern differs across both dimensions, while the small number of observations per cell means the pattern is uncertain and should not be overinterpreted.

Diagnostics, alternatives, and limits

Before interpreting a stratum pattern, check that the time origin, dose history, concentration units, assay conventions, and prediction method are consistent. Inspect the number of observations and subjects in each stratum; a large apparent discrepancy based on one or two subjects may be unstable. Plot the individual errors as well as summaries, and look for whether a result is driven by one extreme observation. If subjects contribute repeated samples, display or estimate uncertainty in a way that accounts for within-subject dependence. A subject-level bootstrap is one option when its resampling scheme matches the intended target of inference.

The main alternatives answer related but different questions. A pooled mean error or pooled error metric gives a compact overall summary but can hide opposing patterns. Mean absolute error and root mean squared error summarize error magnitude without retaining its direction, so they cannot alone show whether a model systematically underpredicts or overpredicts a particular phase. A calibration summary describes agreement more broadly, while external prediction interval coverage addresses how often observations fall within stated intervals. Time- and concentration-stratified signed errors complement these approaches by locating variation in average direction.

Stratification also has limits. Cut points can change the apparent pattern, sparse cells yield imprecise summaries, and multiple exploratory comparisons make chance patterns easier to find. Observed-concentration bands can create selection effects because the outcome helps determine group membership. Neither a time trend nor a concentration trend uniquely identifies a structural model defect: design differences, measurement variability, and other sources of prediction error may contribute. Treat the pattern as a signal for focused follow-up, not a diagnosis or a reason to modify the model during external validation.

Common pitfall: “The pooled error is near zero, so the model performs equally well throughout the profile.” This conclusion does not follow. Positive errors in one phase can cancel negative errors in another. Report the pooled result together with prespecified time- and concentration-stratified summaries, and describe the pattern as evidence about predictive performance rather than proof of a specific mechanism.

Specify the evaluation.
State the locked prediction approach, prediction target, time origin, and concentration units before calculating stratified errors.

Define strata.
Choose time windows and concentration bands in advance where possible, and report whether concentration bands use observed or predicted values.

Summarize within each stratum.
Report counts, signed log-error summaries, exponentiated mean errors, and uncertainty that respects repeated sampling within subjects.

Interpret cautiously.
Compare the strata with pooled metrics and complementary diagnostics; identify patterns to investigate without claiming that a pattern proves a mechanism.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions.

If a stratum has a mean signed log error of \(-0.20\), what does its exponentiated value indicate?
Why can a pooled mean signed log error near zero be misleading?
What is a key interpretive caution when concentration strata are defined using observed concentrations?
Why should uncertainty calculations account for subjects when each subject contributes multiple samples?

Solutions. (1) \(\exp(-0.20)=0.8187\), so the geometric mean observed concentration is about 0.8187 times the prediction; this indicates average overprediction on the multiplicative scale. (2) Positive and negative errors from different times or concentration ranges can cancel, concealing systematic patterns within strata. (3) Group membership depends on the outcome itself, so the comparison is descriptive and may be affected by selection; it does not establish that concentration causes the error pattern. (4) Repeated samples from the same subject are dependent, and treating them as independent can understate uncertainty by overstating the amount of independent information.