Reading a Summary as Evidence, Not a Verdict
An automated diagnostic summary can make a report easier to scan: it may count large residuals, summarize interval coverage, or flag panels that meet a programmed rule. But a flag is only the result of that rule. It does not, by itself, establish that a model is inadequate. Conversely, an unflagged summary cannot guarantee that the model is adequate: aggregation may hide a pattern, and a rule may not examine the feature that matters.
Interpretation therefore requires reading the summary alongside the plots and the information that defines them. Ask what observations were included, how the metric was calculated, what its denominator is, and whether the apparent pattern is consistent across relevant views of the data. In pharmacometrics, an overall residual average, for example, may conceal a time trend or a concentration-dependent pattern. The diagnostic question is not simply “Did the metric cross a threshold?” but “What model behavior would produce this combination of summaries and plotted evidence?”
Before interpreting a flag, establish what it summarizes. A standardized residual, for example, is an observed discrepancy divided by a specified estimate of its standard deviation. Its meaning depends on how the prediction and standard deviation were obtained; residual definitions are not interchangeable. Likewise, “coverage” is incomplete unless the report identifies the interval, the observations assessed, and the denominator. A count of large residuals is uninterpretable without knowing the threshold and whether repeated observations from the same subject are included.
A useful reading assumes that the reported diagnostic quantities are aligned with the intended analysis: the data set, model run, observation records, units, and plotted labels correspond; missing quantities and exclusions are accounted for; and the metric is calculated as its label describes. These are conditions for interpreting the report, not reasons to assume the model is correct. If the report does not establish them, the first conclusion should be that the evidence is incomplete.
A Structured Review
Review a summary in a sequence that moves from what was calculated to what the results might mean. Preserve the distinction between observation and interpretation at each step.
Read the metric definition, threshold, denominator, units, and relevant exclusions. Check that a displayed count is not being confused with a proportion or a rate.
Use the corresponding plot or stratified summaries to see where the flagged observations occur: across time, concentration, prediction, dose group, or subjects.
Compare the global metric with local patterns. A near-zero overall mean can coexist with opposing trends; a modest count can be concentrated in a consequential subset.
Describe the pattern and its uncertainty. Identify what needs closer examination, without converting a descriptive threshold into a formal test or an automatic instruction to change the model.
The assumptions behind any comparison matter. If a report uses a normal-reference expectation for standardized residuals, that comparison presumes the residuals are meaningfully standardized under the stated definition and that the reference distribution is a reasonable approximation. Repeated measurements within a subject are generally dependent, so a simple independent-observation calculation can be a rough reference, not a calibrated test. For simulation-based intervals, interpretation also depends on the simulated quantities, binning, and number of observations represented. The report’s graphical evidence and design context remain essential.
Worked Example: A Residual Count That Needs Context
Suppose a hypothetical report summarizes 120 standardized residuals. Its rule flags each residual with absolute value greater than 2, and it reports 8 flagged observations. The overall mean standardized residual is 0.03. The plot shows that 6 of the 8 flagged observations belong to one subject, which has 18 observations.
Start with the denominator. The flagged fraction is \(8/120=0.0667\), or 6.67%. The mean of 0.03 is close to zero, but that average does not reveal the distribution of the flagged observations across subjects.
Use the reference cautiously. If standardized residuals were independent standard normal quantities, the reference probability of exceeding 2 in absolute value would be approximately \(P(|Z|>2)=0.0455\). That would give an expected count of \(120(0.0455)=5.46\). The observed count of 8 is higher than 5.46, but this comparison alone is not a test: residuals may be dependent, and the standard-normal approximation may not match the residual definition or fitting procedure.
Read the plot with the count. Six of the eight flagged values occur in one subject, so that subject contributes \(6/18=0.333\), or 33.3%, flagged observations. The other 102 observations contribute only 2 flags. This concentration is more informative for follow-up than the overall count alone. It suggests examining that subject’s time course, dosing and observation records, and the relevant predictions. It does not establish whether the cause is data, model structure, or chance.
A measured conclusion is: “Eight of 120 standardized residuals exceed the report’s threshold; six occur in one subject. The overall mean is 0.03, but the subject-level concentration warrants review of that subject’s records and diagnostic plots.” This states the evidence without treating the flag as a verdict.
Why Global Metrics Can Hide Patterns
An aggregate metric compresses information. Compression is useful for scanning, but it can remove ordering and grouping. In particular, positive and negative discrepancies can cancel in an overall mean. The following calculation shows why a mean near zero is not evidence that residuals lack a systematic pattern.
Consider five equally sized time bins whose mean standardized residuals are \(-0.8,-0.4,0,0.4,0.8\). Their average is
Yet the bin means rise steadily with time. If the bin positions are \(x=0,1,2,3,4\), their mean is 2, and the least-squares slope is
Thus a zero aggregate mean coexists with a positive trend of 0.4 standardized-residual units per bin position. This is a descriptive derivation, not a significance test; it demonstrates why plots or stratified summaries are needed to retain ordering. A fitted trend or a smooth curve should also be read cautiously when there are few bins or sparse data.
Worked Example: Interval Coverage by Prediction Range
In a hypothetical visual predictive check, a report divides observations into prediction-range bins and compares the observed proportion within a simulation-based 90% prediction interval. In a high-prediction bin, 11 of 20 observations fall within the interval; in the other nine bins, coverage is near the expected 90%.
Calculate the local coverage. The high-bin coverage is \(11/20=0.55\), or 55%. The count outside the interval is \(20-11=9\), or 45%. Since the interval is labelled 90%, this bin merits attention; it is not appropriate to replace the local result with the overall impression that most bins look satisfactory.
Consider the amount of information. With only 20 observations, one observation changes the displayed proportion by \(1/20=0.05\), or 5 percentage points. Under a simplified independent Bernoulli reference with coverage probability 0.90, the standard deviation of the coverage proportion would be \( \sqrt{0.90(0.10)/20}=0.0671\), or 6.71 percentage points. This calculation illustrates the scale of sampling variability under that simplified reference; it is not a formal test of the simulation-based diagnostic, especially if observations are clustered within subjects.
Combine rather than substitute evidence. Check whether the plotted observed values lie systematically above or below the simulated interval, whether the pattern appears in adjacent bins, and how the report formed the bins. If this is a prediction-corrected display, confirm that the plotted quantity and interval are on the stated corrected scale. A single sparse bin may be unstable; a coherent discrepancy across neighboring ranges is stronger evidence for targeted investigation. Neither pattern alone identifies the source.
When Summaries Disagree
Different diagnostic views answer different questions. A global mean describes average signed discrepancy; a count above a threshold describes frequency beyond that threshold; a plot against time or prediction can reveal ordering or curvature; a subject-level display can reveal concentration. Agreement across complementary views increases confidence that a pattern is real in the analyzed output. Disagreement is not necessarily a reporting error, but it is a reason to ask what each summary retains and what it discards.
Worked Example: A Near-Zero Mean With a Time Pattern
A hypothetical report gives mean standardized residuals of \(-0.8,-0.4,0,0.4,0.8\) in five equal-sized time bins. It also reports an overall mean of 0.00 and no flag because its flagging rule checks only whether the absolute overall mean exceeds 0.5.
Check the aggregate. The five bin means sum to zero, so their average is \(0/5=0.00\). The automated rule therefore does not flag the overall mean.
Check the ordered pattern. The bin means increase by 0.4 standardized-residual units at each step. As shown by the slope calculation above, their descriptive least-squares slope is 0.4 units per bin position. Thus, the no-flag result is exactly what the programmed rule implies, but it does not address the time pattern.
The appropriate summary is not “the diagnostic passed.” It is: “The overall mean is 0.00 and does not meet the report’s flagging condition; the binned residual means nevertheless increase from \(-0.8\) to \(0.8\) over time, warranting closer examination of the time-course plot and related model predictions.” This keeps the automated outcome and the visual evidence distinct.
Diagnostics, Failure Modes, and Alternatives
A summary is most useful when it is treated as a navigation aid: it points to plots and strata that deserve attention. It is least useful when it is treated as a universal score of model quality. Before drawing a conclusion, look for these common failure modes:
- Threshold treated as a universal standard. A flag threshold is a programmed rule, not automatically a scientifically justified boundary. State what rule was used and interpret its result in context.
- Counts reported without denominators. “Eight flagged” means little without the number assessed, the missing or excluded observations, and the grouping used.
- Dependence ignored. Repeated observations from one subject do not generally provide the same independent information as observations from different subjects. A pooled count can obscure within-subject clustering.
- Aggregation mistaken for absence of structure. A near-zero mean can result from cancellation. Inspect ordered and stratified displays as well as global metrics.
- One unusual panel treated as a diagnosis. Sparse data, a small bin, or a few influential records can produce unstable patterns. Verify the underlying observations and compare related views before assigning a cause.
- Missing diagnostics interpreted as reassuring. A quantity omitted because it could not be calculated is not evidence of good fit. Establish whether missingness changes the population or denominator being summarized.
Automated rules, visual review, and formal statistical procedures are alternatives only in a limited sense; they do not answer identical questions. Rules are efficient and reproducible for screening but inherit their chosen thresholds and may miss unprogrammed patterns. Plots preserve structure and support contextual judgment, but interpretation can vary and sparse displays can mislead. Formal tests can quantify evidence relative to specified assumptions, but they require a suitable test, account for the analysis design, and do not by themselves establish practical importance or identify a corrective model change. Use them as complementary approaches rather than substitutes.
The main limitation is that a diagnostic summary describes the analyzed data and model outputs under particular definitions and settings. It cannot, by itself, determine whether a discrepancy reflects the model, data processing, study design, or random variation. Nor does an apparently clean summary prove that all relevant assumptions hold. A responsible report distinguishes direct observations (“11 of 20 fell within the interval”) from interpretation (“this high-range pattern merits follow-up”) and from a claim about cause (“the model structure is inadequate”), which needs additional evidence.
Check Your Understanding
Answer each question by distinguishing the reported quantity from the conclusion it supports.
- A report finds 6 threshold exceedances among 100 observations, all from one subject with 10 observations. What are the overall and subject-specific flagged proportions, and what should be reviewed?
- Five equally sized bins have mean residuals of \(-0.6,-0.3,0,0.3,0.6\). What is their overall mean, and why is that not sufficient to conclude that there is no pattern?
- A prediction interval is labelled 90%, and 16 of 20 observations in one bin lie within it. Calculate the bin’s coverage and explain one limitation of interpreting that number.
- What is the difference between an automated flag and evidence that a model is inadequate?
- A summary reports no flags, but a plot shows a consistent increase in residuals over time. Give a suitable interpretation.
Solutions. (1) Overall, \(6/100=6\%\); within the subject, \(6/10=60\%\). Review the subject’s time course and associated records, and compare with other diagnostic views; concentration does not establish a cause. (2) The mean is \((-0.6-0.3+0+0.3+0.6)/5=0\). Opposing values cancel, so inspect their ordered pattern rather than relying on the aggregate. (3) Coverage is \(16/20=80\%\). With 20 observations, each observation changes the proportion by 5 percentage points; dependence and binning also limit simple interpretations. (4) A flag says a programmed condition was met; inadequacy is a broader conclusion requiring contextual evidence from the defined diagnostics and relevant model behavior. (5) State that the programmed rules did not flag the report, but the plot shows an increasing time pattern that warrants closer examination; do not claim that the model has passed or failed on that basis alone.