Why Diagnose Sequential Forecast Errors?
An individual Bayesian forecast is useful only to the extent that it predicts observations that have not yet been used to construct it. A model may describe past concentrations reasonably well yet still forecast the next sample too high, too low, or with errors that grow at particular times or concentration levels. Such patterns matter in individualized dosing: a sequence of biased forecasts can affect an exposure estimate or a dose decision even when each individual error looks modest.
Residual diagnostics examine the size, direction, and ordering of forecast errors. The key is to use a forecast that was available before the observation being checked. A residual calculated after incorporating that observation into the posterior can look reassuring simply because the update has already responded to the discrepancy. Sequential residuals instead ask whether the model could predict each new observation from the information available at that point.
For a Bayesian forecast, the predictive distribution averages over uncertainty in the individual parameters as well as observation error. Consequently, \(v_i\) should be the variance of the full posterior predictive distribution, not only the residual variance conditional on one estimated parameter value. If the model is specified on the log-concentration scale, diagnostics can also use log residuals, such as \(\log(Y_i)-E[\log(Y_i)\mid\mathcal{F}_{i-1}]\), when \(Y_i>0\). The scale must be stated: a residual of 2 mg/L and a log residual of 0.2 describe different kinds of discrepancy.
What the Residuals Should Show
Under a correctly specified predictive model, the conditional mean of the next forecast error is zero:
This is a statement about the forecast distribution, not a guarantee that a short series of observed residuals will balance perfectly. A run of positive residuals means the observations have repeatedly exceeded their forecasts; a run of negative residuals means they have repeatedly fallen below them. A trend over time may indicate that forecast bias changes during the dosing interval or across successive samples. A pattern in residual size across predicted concentration may indicate that the error scale is not represented appropriately.
Standardization helps compare errors with different predictive uncertainty. A residual of 2 mg/L is more surprising when the predictive standard deviation is 0.5 mg/L than when it is 5 mg/L. If the predictive distribution is approximately normal, standardized residuals near 0 indicate forecasts close to the observations, while values well above or below 0 indicate observations several predictive standard deviations higher or lower. With skewed or otherwise non-normal predictive distributions, \(z_i\) remains a useful signed scale measure, but it should not be treated as an exact normal score.
Why Predictive Variance Includes Posterior Uncertainty
Suppose \(\theta\) denotes uncertain individual parameters. Conditional on the information available before sample \(i\), the forecast distribution averages over the posterior distribution of \(\theta\). The law of total variance gives the decomposition below, where the expectations and variances over \(\theta\) are taken with respect to that posterior.
The first term averages observation variance conditional on the parameters. The second represents variation in the conditional mean across plausible parameter values. Omitting the second term can make a forecast look too precise, particularly when individual data are sparse. This decomposition also explains why residuals standardized using only a conditional observation-error standard deviation can be misleading.
The zero conditional mean result follows directly from the definition of \(m_i\): it is the conditional expectation of \(Y_i\), so subtracting it leaves a random variable with conditional expectation zero. This justification depends on using the predictive distribution appropriate to the information set \(\mathcal{F}_{i-1}\). It does not establish that the model is correct; it states what residual behavior the model implies if its conditional predictions are correct.
Worked Example: Detecting a Time-Ordered Bias
Suppose four hypothetical concentrations are forecast sequentially for one individual. Each forecast is made before its corresponding observation, and the predictive standard deviation is 2 mg/L at every time.
| Sample | Forecast mean (mg/L) | Observed (mg/L) | Residual (mg/L) | Standardized residual |
|---|---|---|---|---|
| 1 | 8 | 9 | 1 | 0.50 |
| 2 | 10 | 12 | 2 | 1.00 |
| 3 | 12 | 15 | 3 | 1.50 |
| 4 | 14 | 18 | 4 | 2.00 |
For each sample, \(e_i=Y_i-m_i\) and \(z_i=e_i/2\). For example, at sample 3, \(e_3=15-12=3\) mg/L and \(z_3=3/2=1.50\). The mean signed residual is \((1+2+3+4)/4=2.50\) mg/L. All four residuals are positive and increase in order, so this sequence indicates that the forecasts increasingly underpredict these observations. The pattern deserves investigation; four points do not, by themselves, prove a particular mechanism or establish a formal error rate.
Time-Ordered Patterns
Plot residuals against actual sampling time or sample order, retaining their chronological order. Include a horizontal reference at zero. A sequence of one sign, a steady drift, or repeated cycles can be more informative than a single large error. Also consider the clinical and dosing timeline: a pattern aligned with time since dose differs from one aligned with calendar time, and either may suggest a different explanation.
For example, a persistent positive run could reflect underprediction due to an individual parameter value not yet learned, a change in bioavailability, an unrecorded dose, or a structural mismatch in the absorption model. A persistent negative run could have analogous explanations in the other direction. If positive residuals are followed by negative residuals, the model may be predicting the timing or shape of a concentration profile poorly, even if its average level is reasonable.
Successive forecasts use evolving information, so residuals are not simply a collection of unrelated errors. A posterior update may reduce a stable discrepancy after a new sample; a forecast that continues to show the same signed error after updates is a stronger warning that the issue may not be a single fixed parameter mismatch. Even then, the residual sequence alone cannot identify the cause. Check sample timing, dose records, assay information, covariates, and the model assumptions before changing a dose or model.
Worked Example: A Sequence That May Reflect a Changing Process
Consider four hypothetical forecasts for a declining concentration profile. The displayed predictive standard deviations are the full predictive standard deviations, including parameter uncertainty.
| Sample | Forecast mean (mg/L) | Observed (mg/L) | Predictive SD (mg/L) | Standardized residual |
|---|---|---|---|---|
| 1 | 12 | 12 | 1.0 | 0.00 |
| 2 | 10 | 8 | 1.5 | -1.33 |
| 3 | 8 | 5 | 1.5 | -2.00 |
| 4 | 6 | 3 | 1.5 | -2.00 |
At sample 4, \(e_4=3-6=-3\) mg/L and \(z_4=-3/1.5=-2.00\). The later forecasts repeatedly exceed the observations, with standardized residuals becoming more negative after the first sample. One possibility is that clearance increased during the sequence, a process not represented by a model with stable individual clearance. Other explanations include a timing error or a misspecified concentration-time shape. Updating a stable clearance parameter may move subsequent predictions downward, but continued negative one-step-ahead residuals after updating would suggest that a stable-parameter explanation is not sufficient. This pattern is a prompt for investigation, not proof of changing clearance.
Concentration-Dependent Patterns and Choice of Scale
Residuals should also be examined against the forecast mean or another relevant measure of concentration. If absolute residuals grow with predicted concentration, that may be expected under a proportional error model, but it may signal a problem under a model that assumes constant additive error. Conversely, approximately constant absolute errors alongside changing concentration may be consistent with an additive error scale. A funnel shape in a residual plot is a visual warning that the error variance may depend on concentration.
The scale matters because an error can be large in absolute units yet modest relative to the concentration, or small in absolute units yet large near the assay’s lower range. For positive concentrations, log-scale residuals describe multiplicative discrepancy: a constant difference on the log scale corresponds to a constant ratio on the original scale. Choose a scale that matches the observation model, and do not use a transformation merely to make a plot look more regular.
Worked Example: Proportional Discrepancy Across Concentrations
Four hypothetical observations have forecast means of 5, 10, 20, and 40 mg/L. Their observed concentrations are 6, 12, 24, and 48 mg/L. Suppose, for illustration, that the predictive standard deviations are 1, 2, 4, and 8 mg/L, respectively.
The raw residuals are \(6-5=1\), \(12-10=2\), \(24-20=4\), and \(48-40=8\) mg/L. Their magnitudes rise with concentration, which could look like increasing error on a raw-residual plot. But each observation is 1.20 times its forecast mean: for example, \(12/10=1.20\). The log residual is therefore \(\log(1.20)\approx0.1823\) at each concentration. The standardized residuals are also all \(1\): \(1/1=1\), \(2/2=1\), \(4/4=1\), and \(8/8=1\).
These calculations show a consistent proportional underprediction, rather than a worsening discrepancy relative to the assumed predictive scale. The sequence still has a positive bias: the observed values exceed every forecast. If the specified error model is additive with a constant standard deviation, the increasing residual magnitudes would also raise a variance concern. If the model is multiplicative with uncertainty that increases proportionally, the standardized values give a different and relevant comparison. Interpretation depends on the stated observation model.
Diagnostics, Comparisons, and Failure Modes
A practical review combines plots with the data-generating context. Inspect signed residuals against time, sample order, forecast mean, and—when relevant—time since dose. Inspect their magnitudes or standardized values as well as their signs. Check whether an apparent outlier is associated with a recording issue, dose deviation, sample-time error, or assay limitation. With very few observations for one individual, visual patterns are inherently uncertain; do not treat every run or large residual as decisive evidence.
Sequential residual diagnostics complement, rather than replace, posterior predictive checks for individual PK fits. Posterior predictive checks compare features of observed data with replicated data generated under a fitted model. Sequential residuals focus on the signed forecast error at each observation, conditional on what was known before that observation. In-sample fitted residuals are another alternative, but they can be smaller because the observations being checked helped determine the fitted parameters. For assessing forecasting behavior, pre-observation errors are the relevant target.
A common mistake is to update the posterior with all observations first, calculate residuals from that final posterior, and then conclude that the model forecasted well because the residuals are small. That reasoning evaluates fit after learning from the data, not the ability to forecast them in sequence. The correction is to preserve each pre-observation forecast, calculate its error against the later observation, and only then update for the next forecast.
Residuals are diagnostic signals, not causal explanations or automatic dosing rules. A pattern may reflect model structure, parameter learning, data quality, or a changing individual process. Small residuals do not establish that predictive uncertainty is calibrated, and large residuals do not, alone, establish that the dose should change. In particular, do not infer that a posterior parameter update will resolve a discrepancy simply because a parameter can be adjusted to reduce past errors: assess the next sequential forecasts and the plausibility of the individual parameters and model assumptions.
For each observation, record its predictive distribution using only information available before that sample.
Compute signed residuals and, when useful, standardize by the full predictive standard deviation. Use a log-scale residual only when positive observations and the model make that scale meaningful.
Plot residuals in time order and against forecast concentration. Identify runs, trends, changes in spread, and discrepancies aligned with dosing or sampling events.
Check data and assumptions, then decide whether a stable individual parameter, a model feature, or a changing process is a plausible explanation. Residual patterns alone do not determine the cause or the dose.
Check Your Understanding
Answer each question using the definitions and reasoning above. Solutions are included after the questions.
- A forecast mean is 15 mg/L and the observed concentration is 12 mg/L. The full predictive standard deviation is 1.5 mg/L. Calculate the residual and standardized residual, and interpret their signs.
- Why should a residual for sample \(i\) be calculated from a forecast made before observing sample \(i\)?
- Across several samples, raw residual magnitudes increase with forecast concentration, but standardized residuals remain near 1. What distinction should be considered before calling this worsening forecast performance?
- Repeatedly negative sequential residuals remain after posterior updates. Name two broad explanations and state why the residuals do not identify which explanation is correct.
Solutions. (1) \(e=12-15=-3\) mg/L and \(z=-3/1.5=-2.00\). The observation is 3 mg/L below the forecast mean, or two predictive standard deviations below it. (2) The pre-observation forecast prevents the sample being checked from first influencing the posterior used to predict it; otherwise the diagnostic measures fit after updating rather than sequential forecast performance. (3) Consider whether the predictive uncertainty itself increases with concentration, as in a proportional error model, and whether the raw or standardized scale matches the observation model. (4) Possible explanations include a changing individual process and a structural or data problem, such as misspecified concentration-time behavior or incorrect timing. The residual pattern indicates systematic overprediction but cannot distinguish among causes without further evidence.