Why the Shape of the Curve Matters
A residual is the difference between an observed concentration and the value predicted by a model. Large residuals may indicate imprecise measurements, but they may also indicate that the model predicts the wrong concentration at particular times. If the fitted curve misses the early decline, terminal slope, or another systematic feature of the concentration-time profile, that discrepancy enters the residuals. A variance model can then appear inadequate even when the main problem is the curve’s shape.
The key diagnostic question is therefore not only whether residuals are large, but whether they show a pattern tied to time, concentration, or another aspect of the design. Residuals that repeatedly lie on one side of zero over a time interval, or trace a smooth sequence within subjects, are evidence to examine the structural model before simply increasing residual variance or adding residual correlation.
Under a correctly specified mean curve, the discrepancy is zero (or sufficiently negligible for the intended analysis), and the residual pattern is governed by the observation-error model. Under a misspecified curve, the discrepancy can vary systematically over time or concentration. This is distinct from residual variance misspecification: the former concerns the conditional mean, while the latter concerns the conditional spread or dependence of observations around that mean.
What Structural Misspecification Does to Residuals
Suppose the observation errors have conditional mean zero and variance \(v_{ij}\), given the subject effects and sampling design. If the true mean differs from the fitted curve, then
when the discrepancy is fixed given \(b_i\) and the design. Thus a nonzero expected residual is a mean-model problem, not an increase in conditional observation-error variance. In a finite data set, however, a sequence of nonzero discrepancies contributes to the observed residual sum of squares and can make a fitted residual variance look too large.
A smooth mean discrepancy can also create apparent serial correlation. For distinct observations \(j\) and \(k\), if the observation errors are conditionally uncorrelated given subject effects and design, and the discrepancies are fixed given those same quantities, then
This conditional covariance statement does not imply that the realized residuals will look patternless. A fixed sequence of discrepancies can produce runs of positive or negative residuals, and a sample correlation computed from such a sequence can be large. If the discrepancy also varies with a shared random subject effect, marginal covariance can arise as well: averaging over \(b_i\) can make the discrepancies at different times covary. Therefore, a residual correlation statistic alone does not establish that the observation errors themselves are serially correlated.
Worked Example: A Trend That Inflates the Apparent Error
Consider hypothetical concentrations from one subject at four times. The fitted curve predicts 6, 5, 4, and 3 mg/L, while the observed concentrations are 8, 6, 4, and 2 mg/L. Assume for illustration that an additive observation-error model with standard deviation 0.5 mg/L is being considered.
| Time (h) | Prediction (mg/L) | Observation (mg/L) | Residual (mg/L) | Standardized residual |
|---|---|---|---|---|
| 1 | 6 | 8 | 2 | 4 |
| 2 | 5 | 6 | 1 | 2 |
| 4 | 4 | 4 | 0 | 0 |
| 8 | 3 | 2 | -1 | -2 |
The residuals are \(2,1,0,-1\) mg/L. Their mean is \((2+1+0-1)/4=0.5\) mg/L, and they decline with time rather than fluctuating without a visible pattern. Their sample variance is
This sample variance is much larger than the assumed observation-error variance, \(0.5^2=0.25\;(\text{mg/L})^2\). But the four observations do not prove that the assay error variance is too large: the smooth downward residual pattern suggests the fitted curve may be missing a time-dependent feature. A residual variance calculated from such data combines random error and the realized structural discrepancy.
Targeted Residual Patterns
Residual plots are most useful when their horizontal axis targets a plausible feature of the mean model. Plot residuals against time within subjects and across subjects, and against predicted concentration. Examine signed residuals as well as standardized residuals. Standardization helps account for an error model whose variance changes with prediction, but it does not remove a misspecified mean.
- Runs over time: several positive residuals followed by several negative ones can indicate a misplaced peak, an incorrect absorption phase, or a terminal slope that is too shallow or too steep.
- Curvature over time: residuals that first rise and then fall, or show a repeated curved shape across subjects, suggest that the fitted time course may have the wrong shape.
- Patterns against prediction: residuals that change systematically with predicted concentration can indicate a mean discrepancy, a variance function problem, or both. The signed pattern matters: a consistent sign over a concentration range is not explained merely by allowing a larger variance there.
- Subject-specific repetition: similar patterns across subjects support a shared structural explanation. Patterns restricted to an occasion or subject may instead point toward occasion or subject effects, or sparse and influential data.
These clues are not unique diagnoses. An omitted covariate, an unsuitable dose or timing record, an influential observation, or an incorrect random-effects structure can also produce patterned residuals. The earlier material on diagnosing residual variance misspecification and distinguishing residual correlation from occasion variability is relevant when evaluating those alternatives. Here the specific concern is that a systematic feature of the expected concentration-time curve has been represented incorrectly.
Worked Example: Apparent Lag-One Correlation From a Smooth Pattern
Suppose hypothetical residuals from one subject, in time order, are \(-2,-1,1,2\) mg/L. These are not evidence by themselves that the observation errors are correlated; they are a small sequence that changes smoothly from negative to positive. To quantify the visual pattern, compute the Pearson correlation between adjacent pairs, \((-2,-1,1)\) and \((-1,1,2)\).
The means of the two vectors are \(-2/3\) and \(2/3\) mg/L. The centered cross-product sum is \(({-4}/{3})({-5}/{3})+({-1}/{3})({1}/{3})+({5}/{3})({4}/{3})=13/3\). Each centered sum of squares is \(14/3\). Therefore,
The adjacent-pair correlation is high, but there are only three pairs, and the residuals have a clear time trend. The calculation describes this sequence; it does not distinguish stochastic serial correlation from a deterministic curve discrepancy. If similar smooth patterns recur across subjects, inspect the structural predictions before interpreting this statistic as a reason to add a correlated-error model.
Separating Mean and Variance Explanations
A practical comparison is between residual patterns and the predictions made by candidate explanations. A larger additive variance should make residuals more dispersed on the concentration scale; a proportional variance should make their spread increase with prediction. Neither variance change, by itself, shifts the conditional mean from one side of zero to the other at particular times. A mean curve change can shift expected residuals over time while leaving the underlying observation-error variance unchanged.
The distinction is clearest when several subjects show the same signed time pattern. For example, if residuals are positive early and negative late across subjects, a global increase in residual variance may accommodate their magnitudes but leave the directional pattern unexplained. Conversely, residuals scattered around zero without a consistent time trend, but with spread increasing at higher predictions, are more directly suggestive of a variance function that does not match the data.
Worked Example: Standardization Does Not Repair a Wrong Curve
For a second hypothetical profile, predictions at four times are 10, 7, 4, and 2 mg/L, and observations are 10.5, 7.4, 3.3, and 0.8 mg/L. The residuals are \(0.5,0.4,-0.7,-1.2\) mg/L. Compare an additive standard deviation of 0.5 mg/L with a proportional model having a 10% coefficient of variation, so that the standard deviation at prediction \(F\) is \(0.10F\) mg/L.
Under the additive model, the standardized residuals are \(0.5/0.5=1\), \(0.4/0.5=0.8\), \(-0.7/0.5=-1.4\), and \(-1.2/0.5=-2.4\). Under the proportional model, the standard deviations are 1, 0.7, 0.4, and 0.2 mg/L, giving standardized residuals \(0.50\), \(0.4/0.7\approx0.571\), \(-0.7/0.4=-1.75\), and \(-1.2/0.2=-6.00\).
The proportional model changes how unusual each residual looks relative to its assumed scale, especially at the final low prediction. It does not remove the change from positive early residuals to negative late residuals. This pattern calls for examination of the mean curve as well as the variance assumption; the numerical comparison alone cannot identify which model is correct.
A Short Justification for the Diagnostic Distinction
The decomposition \(R_{ij}=\delta_{ij}(b_i)+\epsilon_{ij}\) gives the distinction directly. Conditional on subject effects and design, suppose \(E(\epsilon_{ij}\mid b_i,\text{design})=0\) and \(\operatorname{Var}(\epsilon_{ij}\mid b_i,\text{design})=v_{ij}\). Because the discrepancy is fixed under that conditioning,
Thus the discrepancy changes the residual mean, not the conditional variance. If instead one summarizes residuals over observations without accounting for that mean, the squared residual includes both components:
This identity explains how systematic misspecification can inflate an apparent residual sum of squares. Further, when discrepancies vary randomly across subjects, their variation may contribute to marginal covariance. By the covariance decomposition,
Under conditionally uncorrelated observation errors, the first term is zero for \(j\ne k\); the second need not be. This is why covariance statements must specify whether subject effects are conditioned on. Also, fitted residuals often use estimated individual effects rather than known \(b_i\); their patterns can be affected by that estimation. Residual plots are diagnostics, not a direct decomposition of the latent error sources.
Workflow, Alternatives, and Limitations
Plot observed-minus-predicted residuals against time and prediction. Look for repeated runs, curvature, or sign changes shared across subjects.
Compare residual spread with the specified variance function, and inspect standardized residuals. A changing spread without a systematic signed pattern is more consistent with variance misspecification than a repeated mean shift.
When a plausible structural feature is missing, compare a scientifically justified curve change with a variance or correlation change. Judge whether the residual pattern is reduced, not only whether the residual sum of squares changes.
Examine predictions and residuals across subjects and occasions. A changed curve can alter standardized residuals, estimated variance components, and the apparent need for residual dependence.
The main alternative explanations are a different residual variance function, genuinely correlated observation errors, and between-occasion or subject variability. They answer different questions: variance models describe changing spread, correlation models describe dependence remaining after the mean is accounted for, and occasion effects describe systematic differences among occasions. They can coexist with structural misspecification, so the aim is not to select one explanation from a plot alone.
Residual diagnostics also have limits. Sparse sampling may not reveal the curve’s shape; low concentrations may be dominated by assay limitations; and a flexible curve can absorb noise rather than meaningful structure. A small number of subjects or observations makes residual trends and correlations unstable. Any candidate structural change should be scientifically defensible and assessed with the design and concentration-time context, not introduced solely to make a plot look flatter.
Common Mistake
Wrong reasoning: “The residuals are large and adjacent residuals are similar, so the residual variance must be too small and the errors must be serially correlated.” This confuses a visible pattern in fitted residuals with the conditional distribution of observation error. A misspecified curve can generate both large residuals and a high sample lag correlation even when the observation errors are conditionally independent.
Correction: First ask whether the signed residuals show a repeatable time- or concentration-related pattern. Then compare a justified structural change with the proposed variance or correlation change. State correlation claims with the conditioning made explicit; dependence induced by random subject-specific discrepancies is not the same as conditional correlation in observation errors.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions below.
- If \(R_{ij}=\delta_{ij}(b_i)+\epsilon_{ij}\), with the discrepancy fixed given subject effects and design, what does a nonzero conditional mean of \(R_{ij}\) indicate?
- Why can a smooth time trend produce a high lag-one residual correlation even if observation errors are conditionally uncorrelated?
- Residuals repeatedly increase from negative to positive across the same time range in several subjects. Is simply increasing an additive residual variance a complete explanation? Why or why not?
- What extra covariance term may appear when covariance is calculated over subjects rather than conditional on their random effects?
Solutions. (1) Since \(E(\epsilon_{ij}\mid b_i,\text{design})=0\), \(E(R_{ij}\mid b_i,\text{design})=\delta_{ij}(b_i)\); a nonzero value indicates discrepancy between the true and fitted conditional mean. (2) A smooth deterministic sequence can make adjacent realized residuals similar. Its sample correlation describes the sequence and does not by itself establish correlation in the observation errors. (3) No. Increasing variance can accommodate larger spread but does not explain a repeated directional pattern in the conditional mean; the curve’s time course should be examined. (4) The covariance of the discrepancies across times, \(\operatorname{Cov}\{\delta_{ij}(b_i),\delta_{ik}(b_i)\mid\text{design}\}\), can contribute to marginal covariance.