What an Error Pattern Can Tell You
A single summary score can indicate that external predictions are inaccurate, but often cannot show where or under what conditions they fail. Plotting prediction errors against time, dose, concentration, and covariates can expose a time-course mismatch, a dose-related discrepancy, or a subgroup that is systematically over- or underpredicted. Such patterns are clues for investigation, not automatic proof that the structural model is wrong.
Interpretation starts with a precise definition of the error and prediction target. An external prediction is generated using the frozen model and the information allowed by the validation target. Its error is then calculated against the corresponding observed outcome. The prediction type matters: a population prediction and a prediction conditioned on individual information answer different questions, so their error patterns should not be treated as interchangeable.
A diagnostic plot displays these errors against a feature such as time after dose, dose amount, predicted concentration, or a covariate. If the conditional mean error has no trend with a feature \(x\), it is constant across \(x\); that constant need not be zero. Errors are conditionally centered around zero only when their conditional mean is zero. This distinction separates a trend from a persistent bias.
For example, \(E(e\mid x)=1\) describes errors that remain centered around 1 concentration unit as \(x\) changes. There is no conditional mean trend, but the predictions are, on average, 1 unit too low. A horizontal zero reference line is therefore useful, but a flat pattern that sits above or below it is different from a pattern that slopes or curves.
Reading the Main Diagnostic Axes
Plot against time when the question is whether the external concentration-time profile has the right shape. A sustained rise or fall in mean error over time, a curved pattern, or a change in sign may point to a mismatch in absorption, distribution, elimination, or the timing of the prediction. Before attributing such a pattern to model structure, check that time is measured from the correct dose and that observations and predictions are aligned to the same time origin.
Plot against dose to look for errors that vary with dose amount or regimen. A trend can be consistent with dose-dependent structural behavior that the model does not capture. It can also result from incorrectly recorded dose amounts, dose units, dose timing, or administration. Check those inputs before changing the model. Dose groups may also differ in other ways, so a plot alone does not isolate a causal explanation.
Plotting against predicted concentration helps reveal whether error size or mean error changes across the prediction range. A curved mean pattern may suggest that predictions are too low at one range and too high at another. A widening vertical spread may indicate that variability changes with concentration even when the mean error does not. Plotting error against the observed concentration can also be informative, but it has a dependence built into it because the observation is used to calculate the error. Treat such a plot as exploratory and compare it with plots against predicted values or other independently defined features.
For covariates, continuous covariates can be plotted on their original scale, while categorical covariates are often clearer as grouped error distributions or group means. A subgroup shift may suggest that a covariate effect is missing, incorrectly coded, or not transportable to that group. First confirm units, category definitions, and the values supplied to the prediction workflow. A covariate-associated pattern does not by itself establish that the covariate causes the prediction error.
Worked Example: A Flat Time Pattern With Nonzero Bias
Suppose a hypothetical external subject has observations and predictions at four times after dose. Concentrations are in mg/L, and each error is calculated as observed minus predicted.
| Time (h) | Observed (mg/L) | Predicted (mg/L) | Error (mg/L) |
|---|---|---|---|
| 1 | 9 | 10 | -1 |
| 2 | 7 | 8 | -1 |
| 4 | 4 | 5 | -1 |
| 8 | 2 | 3 | -1 |
At every time, \(e=C^{obs}-C^{pred}=-1\) mg/L. The mean error is $$ \bar e=\frac{-1+(-1)+(-1)+(-1)}{4}=-1\text{ mg/L}. $$ There is no apparent time trend: the errors are constant across the four times. But they are not centered around zero. The predictions are 1 mg/L higher than the observations throughout this example. The plot suggests a constant discrepancy, not a time-dependent shape failure. In a larger external data set, investigate this offset and its uncertainty before deciding whether it reflects a systematic mismatch or merely a small sample pattern.
Separate a Model Pattern From a Workflow Mismatch
An error plot is most useful when paired with a sequence of checks. First confirm that the external observations and predictions refer to the same individuals, samples, units, times, and dosing events. Then verify that the prediction was generated with the intended frozen model, parameter values, covariate transformations, and prediction target. If the inputs or implementation are wrong, a model change would address the wrong problem.
Confirm the observation, prediction, time origin, dose record, concentration units, and covariate values correspond. Recalculate a small number of predictions independently if needed.
Review unit conversions, missing-value handling, category coding, transformations, dose history, and software implementation. Look for errors that would affect a particular time, dose, concentration range, or subgroup.
If the data and prediction workflow are consistent, ask whether a systematic shape or subgroup pattern is compatible with a limitation of the model or with differences in the external setting.
This ordering matters because both structural deficiencies and mismatches in data or implementation can produce similar plots. For example, a dose-related pattern might reflect inadequate dose-dependent model behavior, or a unit conversion error that affects predictions at higher recorded doses. A time pattern might reflect structural misspecification, or a dose-time record that starts at the wrong event. Plots help prioritize checks; they do not uniquely identify the cause.
Worked Example: Quantifying a Dose-Related Trend
In a hypothetical external evaluation, three dose levels have the following paired predictions and observations. The values are illustrative, and dose is recorded in mg while concentration is in mg/L.
| Dose (mg) | Predicted (mg/L) | Observed (mg/L) | Error (mg/L) |
|---|---|---|---|
| 50 | 4 | 5 | 1 |
| 100 | 6 | 8 | 2 |
| 150 | 8 | 11 | 3 |
The errors are \(1,2,3\) mg/L, so higher-dose observations are increasingly above their predictions. To summarize the linear trend, calculate the least-squares slope of error on dose: $$ b=\frac{\sum_k(D_k-\bar D)(e_k-\bar e)} {\sum_k(D_k-\bar D)^2}. $$ Here \(\bar D=(50+100+150)/3=100\) mg and \(\bar e=(1+2+3)/3=2\) mg/L. Thus $$ b=\frac{(-50)(-1)+(0)(0)+(50)(1)} {(-50)^2+0^2+50^2} =\frac{100}{5000} =0.0200\text{ (mg/L)/mg}. $$ Within these illustrative data, error increases by 0.0200 mg/L per mg of dose in the fitted linear summary. The positive slope indicates increasing underprediction with dose. Before interpreting it as a structural-model deficiency, check dose units and records and whether other features vary with dose. Three points are not enough to establish a general dose relationship.
Justifying the Mean-Pattern Interpretation
The basic interpretation follows directly from the error definition. At a given feature value \(x\), the conditional mean error is $$ E(e\mid x)=E(C^{obs}-C^{pred}\mid x) =E(C^{obs}\mid x)-E(C^{pred}\mid x). $$ If this difference changes with \(x\), the mean prediction error has a conditional trend. If it equals a constant \(c\) for all relevant \(x\), it has no conditional mean trend, though \(c\) may be nonzero. Only if that constant is zero are predictions conditionally unbiased in the mean over the values of \(x\) being considered.
This is a statement about the conditional mean, not about each observed error. Individual errors may vary widely around a constant mean. Nor does a flat average error pattern imply that the full predictive distribution is adequate: its spread or tails can still be wrong. In repeated-measures data, observations from the same subject are generally not independent. A visual pattern can still be useful, but formal uncertainty calculations should account for clustering rather than treating every sample as an independent subject.
Worked Example: A Covariate Group Shift
Consider hypothetical external predictions for two renal-function categories. All concentrations are in mg/L, and the covariate category is defined before reviewing the errors.
| Category | Errors (mg/L) | Mean error (mg/L) |
|---|---|---|
| Reference | -0.1, 0.1 | 0 |
| Impaired | 1.9, 2.1 | 2.0 |
The reference mean is \((-0.1+0.1)/2=0\) mg/L. The impaired-category mean is \((1.9+2.1)/2=2.0\) mg/L. The difference is \(2.0-0=2.0\) mg/L: in this small illustration, the predictions underpredict the impaired-category observations on average by 2.0 mg/L relative to the reference category. The next checks are whether renal-function values and category rules were correctly applied, and whether other characteristics differ between these groups. With only two observations per category, the pattern is a signal to investigate, not a reliable estimate of a population subgroup effect.
Diagnostics, Alternatives, and Limitations
Look for more than a changing average. A curved or changing-sign pattern can indicate that the mean relationship is not captured by a straight line. A fan-shaped spread can indicate changing error variability. Isolated large errors may warrant a check of individual records, but should not be discarded simply because they are inconvenient. Overlaying a smooth trend can help summarize a plot, but the smoothing choice and the number of observations affect what appears.
Patterns should also be examined at appropriate levels. Plotting every repeated observation can make subjects with more samples dominate the display. Subject-level summaries or separate displays by subject can reveal whether a population-level pattern is widespread or driven by a few individuals. Conversely, aggregation can conceal within-subject time patterns. Use the view suited to the prediction target, and state clearly what is being summarized.
| Approach | What it can show | What it can miss |
|---|---|---|
| Scalar prediction score | Overall magnitude of predictive error in a compact summary | Whether errors vary systematically with time, dose, concentration, or covariates |
| Error-pattern plot | Direction, shape, spread, and subgroup differences across a feature | A unique cause for the pattern or a precise uncertainty estimate by itself |
| Observed-versus-predicted display | Agreement over the range of observations | Some patterns are harder to see than on an error scale; both axes are not independent |
A standardized error can sometimes make errors at different predicted concentrations easier to compare, but only if the scale used for standardization is appropriate to the prediction target. It does not repair an incorrect mean prediction or an incorrectly specified uncertainty model. Likewise, a statistical test for a trend can complement a plot, but a small p-value does not identify whether the source is the structural model, the external data, or the prediction implementation.
The main limitation is that the same pattern can have multiple explanations, and absence of a visible pattern is not proof of adequate prediction. Sparse data, narrow covariate ranges, noise, and dependent observations can conceal real deficiencies. These plots are diagnostic tools for locating and characterizing error; they do not, by themselves, justify a specific model change.
Check Your Understanding
Answer each question, then compare with the solutions below.
- If \(E(e\mid x)=0.7\) mg/L throughout the observed range of \(x\), is there a conditional mean trend in \(x\)? Are errors conditionally centered around zero?
- At doses of 20 and 40 mg, mean errors are 0.5 and 1.5 mg/L. What pattern is suggested, and name one data or implementation check to make before attributing it to model structure.
- Why should an error plot against observed concentration be interpreted cautiously?
- If errors have a flat mean pattern around zero over time but their spread increases at later times, what has changed: the conditional mean or the conditional variability?
Solutions. (1) There is no conditional mean trend because the mean is constant in \(x\), but the errors are not centered around zero; they are centered around 0.7 mg/L. (2) The higher-dose group has a higher mean error, suggesting increasing underprediction with dose. Check dose units, dose records, and dose-time alignment before concluding that the structural model is deficient. (3) Observed concentration is part of \(e=C^{obs}-C^{pred}\), so using it on the horizontal axis creates mathematical dependence between the axes and can accentuate or distort a pattern. Compare with predicted concentration or other features as well. (4) The conditional mean remains approximately zero, while the conditional variability increases; a zero mean pattern does not imply constant spread.