Reading a Goodness-of-Fit Plot as Evidence, Not a Verdict
A goodness-of-fit (GOF) plot is a view of selected aspects of a fitted model, not a single pass-or-fail test. In an NLME analysis, the same observations can be compared with population predictions or with predictions that use estimated individual effects. Residuals can likewise be plotted against time, predictions, or covariates. Each display answers a different question, and each can conceal features that another display reveals.
Let \(y_{ij}\) denote the observation for subject \(i\) at observation time \(t_{ij}\). A population prediction uses the population parameter values, while an individual prediction also uses an estimate of that subject's random effects. Define a raw residual as observed minus predicted; its sign therefore indicates whether the observation lies above or below the prediction. Weighted or standardized residuals scale discrepancies using an error model, but their exact definitions depend on the model and software. Do not assume that differently named residuals have identical properties.
Interpret a plot by asking what was put on each axis, how the quantities were calculated, and what feature the display can diagnose. For example, observed-versus-population-prediction plots assess agreement at the population level, while residual-versus-time plots can reveal systematic discrepancies over a dosing interval. An apparently good individual fit is not independent confirmation of population adequacy: the individual prediction uses subject-specific estimates informed by that subject's observations.
Misreading 1: High Correlation Means Unbiased Fit
Correlation describes the strength of a linear association, not whether predictions agree in level or scale with observations. If every prediction is too low by the same amount, the correlation can still be perfect. A useful observed-versus-predicted display therefore needs a reference line of equality, and its interpretation should consider bias and scale as well as correlation.
Worked Example: Perfect Correlation With a Systematic Bias
Suppose four hypothetical population predictions of concentration are 10, 20, 30, and 40 mg/L. The corresponding observations are 20, 30, 40, and 50 mg/L. The observed values equal the predictions plus 10 mg/L at every point.
The prediction mean is 25 mg/L, and the observation mean is 35 mg/L. Using the usual sample correlation, \(r=\sum_j(x_j-\bar{x})(y_j-\bar{y})/\sqrt{\sum_j(x_j-\bar{x})^2\sum_j(y_j-\bar{y})^2}\), the prediction deviations and observation deviations are identical: \(-15,-5,5,15\). The numerator is \(225+25+25+225=500\), and each sum of squared deviations in the denominator is also 500. Thus \(r=500/\sqrt{500\times500}=1\).
However, each residual is \(y-\text{prediction}=10\) mg/L. The mean residual is \(10\) mg/L, and the fitted values sit 10 mg/L above the equality line. The predictions are strongly associated with the observations but systematically low. The correlation alone misses this bias.
The reason is general. If \(y_j=x_j+c\) for a constant \(c\), then \(y_j-\bar{y}=x_j-\bar{x}\). The centered values, and therefore the correlation, are unchanged by the shift. A multiplicative scale mismatch can also preserve a perfect correlation when the multiplier is positive. Inspect the equality line, residuals, and relevant scales rather than treating a large correlation as a calibration result.
A related error is to treat a cloud that roughly follows the equality line as proof of adequate fit. A pooled plot may obscure time-dependent bias, subgroup differences, or clusters from subjects with many observations. Examine the plot's structure and the sampling design, not only its overall appearance.
Misreading 2: Residual Normality Proves Adequacy
A residual histogram or normal quantile plot concerns the distribution of residual values after a particular calculation. It does not show whether residuals depend on time, predicted concentration, dose, or subject characteristics. Nor does a distribution that looks approximately normal prove that the structural model or residual-error model is correct. A useful diagnostic question is not only “What is the pooled distribution?” but also “Does the residual behavior change systematically with something the model should explain?”
Worked Example: A Symmetric Residual Distribution With a Time Trend
Consider ten hypothetical standardized residuals ordered by time after a dose: \(-1.2,-1.0,-0.8,-0.5,-0.2,0.2,0.5,0.8,1.0,1.2\). The values are symmetric around zero, so their overall mean is zero. A histogram that ignores their order would show a symmetric distribution.
Now divide the observations into the first five and last five by time. The early-time mean is \((-1.2-1.0-0.8-0.5-0.2)/5=-3.7/5=-0.74\). The late-time mean is \((0.2+0.5+0.8+1.0+1.2)/5=3.7/5=0.74\). The residuals therefore increase systematically over time despite having a centered, symmetric pooled distribution.
The histogram has discarded the time ordering that reveals the pattern. A residual-versus-time plot would show early negative and later positive residuals, prompting investigation of time-dependent structural or error-model misfit. This example does not establish which model change is needed; it shows why a distributional display cannot replace conditional diagnostics.
Residuals in NLME models are also estimated quantities. They depend on fitted parameters, the assumed residual-error model, and—in conditional or individual residuals—the estimated random effects. Consequently, residuals need not behave like independent observations from a simple reference distribution. A formal normality test or a visually unremarkable histogram should be treated as limited evidence, not a model-adequacy certificate.
Misreading 3: One Isolated Point Decides the Diagnosis
An isolated point deserves attention, but its presence alone does not identify its cause. It may reflect a data-entry or assay issue, a rare but valid observation, an unmodeled subject feature, a sampling-time discrepancy, or a model limitation. Conversely, the absence of a dramatic point does not rule out a systematic pattern spread across many observations.
Worked Example: Following Up an Isolated Standardized Residual
Suppose a hypothetical residual-versus-time plot contains 99 standardized residuals between \(-1.8\) and \(1.9\), and one residual of \(+3.1\) at 2 hours after a dose. Under a standard normal reference, the two-sided tail probability beyond 3.1 is approximately \(2[1-\Phi(3.1)]\approx0.0019\), where \(\Phi\) is the standard normal cumulative distribution function. This calculation signals that the value is unusual relative to that reference; it does not prove the observation is erroneous or the model is wrong.
The next checks are contextual: verify the recorded value, units, dosing and sampling times, and assay information; inspect the subject's full concentration-time profile; and compare the point with other observations at similar times and predictions. Also check whether the plotted residual is conditional or population-based and how its standardization was calculated. If the value is valid, determine whether it is isolated or part of a recurring pattern. Do not remove it solely because it is inconvenient to the fit.
A plot can make an unusual point visually prominent, but visual prominence is not the same as influence on the fitted model. Influence depends on how the observation affects parameter estimates and model conclusions, not just on its distance from a reference line. Any decision to exclude data requires a defensible reason independent of merely improving the plot, with the analysis and its consequences documented.
Misreading 4: Ignoring Shrinkage and Sampling Design
Estimated individual random effects are often pulled toward zero when the data for a subject provide limited information. This phenomenon, called shrinkage, can make estimated effects appear less variable than the underlying between-subject variability and can weaken visible patterns in plots of estimated effects against covariates. A flat plot of estimated effects against a covariate is therefore not conclusive evidence that no covariate relationship exists, especially when shrinkage is substantial.
Sampling design matters as well. A pooled point-level summary gives more weight to subjects with more observations. That may be appropriate for some questions about observations, but it is not the same as giving each subject equal weight. Unequal sampling can cause dense profiles to dominate a display and can hide patterns in subjects with sparse data. The earlier material on eta distributions and shrinkage and on unequal sampling and subject-weighted summaries provides the relevant diagnostic context.
Worked Example: How Unequal Sampling Changes a Summary
Suppose hypothetical standardized residuals are recorded for two subjects. Subject A has nine residuals, all equal to 0. Subject B has one residual, equal to 3. The point-level mean across all ten observations is \((9\times0+3)/10=0.30\). The subject means are 0 for A and 3 for B, so their equal-weight average is \((0+3)/2=1.50\).
The two summaries differ because the point-level mean gives nine times as much weight to Subject A as to Subject B. Neither summary is automatically the right one for every diagnostic question. The point-level view describes the average across recorded observations; the subject-level summary makes the two subjects contribute equally. Inspecting both, alongside profiles and sampling counts, makes the weighting visible rather than allowing the pooled display to decide silently.
Shrinkage can further complicate interpretation. For illustration, suppose the estimated standard deviation of empirical Bayes estimates is 0.20 while the estimated population standard deviation of the corresponding random effect is 0.40. Under the common descriptive approximation \(1-\mathrm{SD}(\hat{\eta})/\omega\), the shrinkage estimate is \(1-0.20/0.40=0.50\), or 50%. This indicates substantial compression of estimated effects relative to the population variability; it does not mean that the true biological variability is known to be half as large.
A Practical Reading Sequence
No single panel should carry the entire diagnostic argument. A disciplined reading sequence makes the limits of each display explicit and helps separate recurring patterns from isolated features.
Determine whether predictions are population-based or individual, whether residuals are raw or standardized, and which scales and units are used.
For observed-versus-predicted plots, compare with the equality line and inspect residual direction and magnitude. Do not substitute correlation for agreement.
Inspect residuals against time, predictions, dose, and relevant covariates, and examine individual profiles when the sampling design permits.
Check how many observations each subject contributes, whether sparse profiles limit individual interpretation, and whether shrinkage may obscure patterns in estimated random effects.
Verify unusual records and compare complementary displays before deciding whether a pattern is isolated, design-related, or consistent with model misfit.
These plots complement one another rather than compete. A pooled residual histogram summarizes distributional shape but loses ordering and subject labels. A residual-versus-time plot preserves time structure but may be hard to read when observations overlap. Individual fits expose within-subject trajectories but may look reassuring when estimates use the same observations being displayed. Population-level plots emphasize the typical model behavior but can conceal subject-specific variation. Numerical summaries can make differences comparable, but compressing data into a statistic can also hide patterns. Use the display that matches the question, then check whether its conclusion is compatible with other relevant views.
Common Mistakes and Their Corrections
- “The correlation is near one, so the model is unbiased.” Correlation is unchanged by an additive shift. Check the equality line and residuals for systematic offset or scale mismatch.
- “The residual histogram looks normal, so the model is adequate.” A histogram does not show ordering or conditional structure. Inspect residuals against time, predictions, and relevant covariates.
- “One large residual proves the model failed.” A single point signals a follow-up question, not a conclusion. Check the record and profile, then look for a recurring pattern.
- “A flat eta-versus-covariate plot rules out a relationship.” Shrinkage can compress estimated effects toward zero. Interpret the plot in light of information per subject and the degree of shrinkage.
- “The pooled mean represents the typical subject.” Subjects with many observations contribute more to a point-level mean. Compare point-level and subject-level views when unequal sampling matters.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- Predictions are 5, 10, and 15 mg/L; observations are 8, 13, and 18 mg/L. What is the mean residual, and what can their perfect linear association not establish?
- Why can a symmetric residual histogram coexist with a systematic residual-versus-time pattern?
- A single standardized residual is unusually large. Name two checks to perform before proposing that the observation be excluded.
- Why might a flat estimated-eta-versus-covariate plot be inconclusive when shrinkage is high?
- One subject contributes eight residuals with mean 0, and another contributes two with mean 2. Calculate the point-weighted mean of all ten residuals and the equal-subject-weighted mean of the two subject means.
Solutions: (1) The residuals are \(8-5=3\), \(13-10=3\), and \(18-15=3\) mg/L, so the mean is 3 mg/L. The perfect linear association does not establish agreement or unbiased predictions; all three are shifted upward by 3 mg/L. (2) A histogram discards observation order. Residual values can be symmetric overall while negative values occur earlier and positive values later, creating a conditional time trend. (3) Verify the recorded value and units, and inspect the subject's full profile and dosing or sampling times; a large residual alone is not a reason for exclusion. (4) Shrinkage pulls estimated effects toward zero, weakening visible covariate patterns even when the underlying relationship may not be absent. (5) The point-weighted mean is \((8\times0+2\times2)/10=0.40\); the equal-subject-weighted mean is \((0+2)/2=1.00\). They differ because the point-level calculation gives the first subject more weight.