Why Residual Structure Matters
A validation prediction can miss observations for several reasons. The model structure may omit a relevant process, a parameter value may be wrong, an assay may introduce bias, or ordinary variability may produce a mismatch. The residuals—the differences between observations and predictions—are useful not just for measuring the size of those mismatches, but for asking whether they have a systematic shape. A persistent temporal trend, for example, is harder to reconcile with independent random errors than a set of residuals scattered around zero.
A pattern is evidence to investigate, not a diagnosis by itself. A parameter error can create a time trend, and a biased assay can create residuals that appear consistent across time. The aim is to compare plausible explanations and find checks that distinguish them, while preserving the validation data as a test of predictions rather than using it to tune the model.
This tutorial builds on the earlier material on longitudinal residual diagnostics by focusing on how patterns across time, dose, and biomarkers inform competing explanations.
The prediction and its uncertainty must correspond to the validation target. For an individual-level prediction, that uncertainty may include observation error and relevant individual variability; for a population mean prediction, the uncertainty should reflect uncertainty for that mean. A residual divided only by assay error can look artificially extreme if the prediction also has substantial uncertainty. Conversely, a very broad predictive distribution can conceal a meaningful pattern. State what uncertainty is included before comparing standardized residuals.
Residual patterns are assessed against relevant variables: time since dose, dose or regimen, predicted response magnitude, and biomarker identity, among others. If residuals are expected to be correlated—for example, repeated measurements from one individual—then a run of same-sign residuals is less surprising than it would be under independence. Diagnostics should therefore account for the data structure rather than treating every residual as an independent replicate.
How Different Causes Can Leave Different Signatures
Random error, under a suitably specified observation model, should not repeatedly favor one sign or vary systematically with time, dose, or biomarker. A few large residuals can occur by chance. More concerning evidence is a reproducible pattern across observations or individuals, especially when it is larger than the prediction uncertainty and has a plausible mechanistic interpretation.
Structural discrepancy means that the model’s representation of the system does not capture some aspect of the data-generating process. Examples include a missing delay, feedback pathway, or dose-dependent process. Such an omission can produce a residual shape that persists even when plausible parameter values are used. In contrast, incorrect parameter values can also yield structured residuals: an incorrect elimination rate, for example, can make late predictions systematically too high or too low. Therefore, “structured” does not automatically mean “structural.”
Assay bias is another possibility. An additive assay offset can shift residuals in one biomarker by a similar amount across time and dose. A proportional bias may instead grow with the measured concentration. If several biomarkers are measured using the same assay platform, a shared pattern may suggest a common measurement issue, but it could also reflect a shared model process. Independent assay checks, quality-control results, dilution studies, or a second measurement method can help separate these explanations.
A First-Order View of Competing Explanations
A simple decomposition shows why residual structure is not a unique signature. Suppose the model prediction is \(f(x_i;\theta)\), where \(x_i\) collects relevant conditions such as time, dose, and biomarker, and \(\theta\) is the parameter vector. Let the real system differ from the model by a discrepancy \(\delta(x_i)\), let the parameter used for prediction differ from the appropriate value by \(\Delta\theta\), and let assay bias be \(b(x_i)\). Write the observation as
where \(\epsilon_i\) is random error with mean zero under the assumed observation model. If the prediction instead uses \(\theta+\Delta\theta\), a first-order Taylor expansion gives
where \(J_i\) is the row vector of sensitivities of the prediction to the parameters at observation \(i\). Including assay bias in the observation, the residual is approximately
The signs follow from the convention \(r_i=y_i-\hat{y}_i\): a parameter change that raises the prediction tends to make the residual more negative. A missing process can contribute through \(\delta(x_i)\); a wrong parameter through the sensitivity pattern \(J_i\Delta\theta\); and measurement bias through \(b(x_i)\). These terms can resemble one another or offset one another.
This approximation is useful when the parameter difference is small enough for a local linear approximation. For large changes or strongly nonlinear models, evaluate the full model over plausible parameter values instead. In either case, parameter sensitivity provides a check: if a plausible parameter change predicts a residual shape close to the observed one, parameter misspecification remains a credible explanation. If it cannot reproduce the pattern without implausible values, structural discrepancy becomes more plausible, though it is not proven.
Worked Example: A Late-Time Residual Pattern
Suppose a hypothetical validation set contains one participant’s measured concentration at five times after a dose. The predictions and observations are in mg/L. For this illustration, assume the stated predictive standard deviations include both measurement and prediction uncertainty, and that no dosing or recording errors are present.
| Time (h) | Prediction (mg/L) | Observation (mg/L) | Predictive SD (mg/L) |
|---|---|---|---|
| 0.5 | 10.0 | 10.2 | 0.50 |
| 1 | 8.0 | 8.4 | 0.50 |
| 2 | 6.0 | 7.0 | 0.60 |
| 4 | 4.0 | 6.0 | 0.70 |
| 8 | 2.0 | 5.5 | 0.80 |
At 8 h, the raw residual is \(5.5-2.0=3.5\) mg/L and the standardized residual is \(3.5/0.80=4.375\). At 0.5 h, the standardized residual is \((10.2-10.0)/0.50=0.400\). The raw residuals are \(0.2, 0.4, 1.0, 2.0,\) and \(3.5\) mg/L: they grow positive over time rather than fluctuating around zero.
This shape is consistent with the model predicting a decline that is too rapid during the later phase. It could reflect a missing slow process, but a parameter such as the elimination rate could produce a similar pattern. The next checks would include verifying the late-time sampling and dosing records, examining whether the pattern recurs in independent individuals, and assessing whether plausible parameter changes reproduce it. Because these are repeated observations from one participant, they are not five independent confirmations; the apparent trend is a diagnostic signal, not a stand-alone statistical conclusion.
Patterns Across Dose and Biomarkers
Dose-related residual patterns can be informative without, by themselves, establishing that the model fails at a particular dose. If residuals become increasingly positive with dose, predictions may be too low at higher doses. A dose-dependent missing process is one possibility; a parameter controlling potency or saturation may also be wrong. The distinction depends on whether plausible parameter changes predict the observed pattern across the full set of conditions, not merely on the direction of the trend.
Cross-biomarker patterns add another dimension. A discrepancy that appears in several mechanistically linked biomarkers with a coherent temporal ordering may implicate a shared pathway or omitted delay. A nearly constant offset isolated to one assay may instead point toward measurement bias. These are clues, not rules: assays may share errors, and a model can contain several linked discrepancies. Interpret each biomarker in its own units and account for its own uncertainty before comparing magnitudes.
Worked Example: Residuals Increasing With Dose
Consider hypothetical validation predictions for a response measured in arbitrary response units. At each of three dose levels, three independent observations are available. Assume the predictions were fixed before these validation observations were examined, and that the observations have comparable uncertainty across dose.
| Dose (mg) | Residuals (response units) | Mean residual (response units) |
|---|---|---|
| 10 | 0.1, 0.0, -0.1 | 0.000 |
| 30 | 0.3, 0.4, 0.2 | 0.300 |
| 60 | 1.0, 1.2, 0.8 | 1.000 |
The mean residuals are calculated directly: at 10 mg, \((0.1+0.0-0.1)/3=0\); at 30 mg, \((0.3+0.4+0.2)/3=0.3\); and at 60 mg, \((1.0+1.2+0.8)/3=1.0\) response units. The positive mean grows with dose, so the predictions tend to be too low at higher doses in this hypothetical sample.
This pattern could arise from a parameter value that makes the modeled dose-response too shallow, or from a missing dose-dependent process. Before favoring either explanation, check dose and response units, the administered-dose records, the uncertainty at each dose, and the sensitivity of predictions to plausible parameter changes. Three observations per dose give limited evidence about repeatability; the increasing means are a reason to investigate, not proof of a structural defect.
Worked Example: A Biomarker-Specific Offset
A hypothetical model predicts two biomarkers, A and B, at three times. Assume measurement and prediction uncertainty are comparable within each biomarker, the observations are correctly labeled, and an independent check can be made on the assay used for A.
| Biomarker | Residuals across three times | Mean residual |
|---|---|---|
| A (pg/mL) | 5, 4, 6 | 5 pg/mL |
| B (standardized units) | -0.1, 0.2, 0.0 | 0.0333 standardized units |
For A, the mean residual is \((5+4+6)/3=5\) pg/mL. For B, it is \((-0.1+0.2+0.0)/3=0.0333\) standardized units. A shows a similar positive offset at each time, while B has small residuals of mixed sign. Because the units differ, the numerical means should not be compared directly.
A nearly constant offset confined to A is compatible with an additive assay bias. It could also indicate a model discrepancy specific to A. A useful discriminating check is to remeasure suitable samples using an independently validated assay or to inspect assay controls for an offset of about 5 pg/mL. If that check finds a comparable offset, the measurement explanation gains support; if it does not, the model and other data-handling explanations still require investigation. The residual pattern alone does not decide between them.
Practical Diagnostic Sequence and Alternatives
A disciplined sequence helps prevent an appealing residual plot from becoming an unsupported causal claim. Keep the original validation predictions fixed; any re-estimation is a diagnostic exploration and should not be reported as independent validation performance on the same data.
Check units, time stamps, dose records, biomarker identity, data transformations, and correspondence between each observation and its prediction. A bookkeeping error can create a convincing but meaningless pattern.
Inspect residuals against time, dose, prediction magnitude, and biomarker. Use raw residuals to retain physical units and standardized residuals to account for expected uncertainty. Look for trends, offsets, curvature, changing spread, and clusters of same-sign values.
Ask whether the predictive distribution includes relevant sources of variability and whether repeated observations are correlated. A pattern should be judged relative to the model’s expected variation, not against an assumed universal residual size.
Examine parameter sensitivities, assay controls, and mechanistic predictions. Prefer evidence that could distinguish alternatives, such as an independent assay check or a pattern predicted by a specific parameter change.
Describe the observed pattern, its uncertainty and recurrence, plausible explanations, and unresolved ambiguity. Do not treat exploratory refitting to the validation data as confirmation.
| Approach | What it contributes | Main limitation |
|---|---|---|
| Residual plots and summaries | Reveal shape, direction, and covariate dependence in the mismatch. | Patterns are descriptive and can be sensitive to small or selected samples. |
| Formal residual tests | Can quantify evidence against a specified null pattern under stated assumptions. | Dependence, heteroscedasticity, and multiple diagnostic checks can undermine simple interpretations; a test does not identify the cause. |
| Parameter-sensitivity analysis | Shows whether plausible parameter changes could produce the observed residual shape. | Local sensitivities can be inadequate for large changes, and several parameters may have similar effects. |
| Assay and data-quality checks | Directly probe measurement or recording explanations. | Passing checks does not establish that the model structure is adequate. |
Common Pitfalls and Limits
- Calling every trend structural discrepancy. A wrong parameter can create a coherent temporal or dose-related pattern. Compare the pattern with parameter sensitivities before making that attribution.
- Calling a pattern random because the average residual is near zero. Positive early residuals and negative late residuals can cancel in the overall mean. Examine residuals conditionally on time, dose, and biomarker.
- Comparing raw residual sizes across unlike outcomes. A difference of 5 pg/mL and a difference of 0.2 standardized units are not directly comparable. Use suitable units and uncertainty scaling.
- Using validation data to tune, then calling the same data confirmation. Refitting can make residuals smaller without showing that the model predicts new data. Keep diagnostic refits distinct from the original validation assessment.
- Ignoring dependence or selection. Repeated observations from one individual are not automatically independent, and a pattern noticed after trying many plots is weaker evidence than a prespecified, independently replicated pattern.
Residual analysis has important limits. Multiple causes may coexist; structural discrepancy can be hidden by offsetting parameter error; and a small validation set may not show a pattern even when one exists. The strongest conclusion is often conditional: the residuals are inconsistent with a particular prediction under stated assumptions and motivate a specific follow-up check. Attribution becomes more credible when independent evidence supports the same explanation.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions below.
- A model predicts 4.0 mg/L and the observed concentration is 3.2 mg/L. What is the raw residual, and what does its sign mean?
- Residuals become increasingly positive late after dosing. Give two plausible explanations and one check that could help distinguish them.
- A biomarker has a similar positive residual at every time, while other biomarkers do not. Why might an assay offset be considered, and why is it not a definite diagnosis?
- Why can repeated same-sign residuals from one participant not necessarily be treated as independent evidence?
- What is the main risk of changing parameters using validation observations and then reporting improved residuals as validation success?
Solutions. (1) \(r=3.2-4.0=-0.8\) mg/L; the observation is 0.8 mg/L below the prediction. (2) A slow process may be missing, or a parameter such as the elimination rate may be incorrect. Check whether plausible parameter changes reproduce the shape and whether it recurs in independent individuals. (3) A stable biomarker-specific offset is compatible with additive assay bias, but a biomarker-specific model discrepancy or another measurement problem could produce it too; assay controls or an independent method are needed. (4) Measurements within one participant may be correlated, so a run of residuals can be more likely than under independence. (5) The same data have been used both to adjust the model and to assess it, so the smaller residuals are not an independent test of predictive performance.