Checking Diagnostic Patterns Across Subgroups

Why Review Diagnostics by Subgroup?

A model can look satisfactory overall while predicting poorly for a clinically important part of the population. For example, overprediction in one group and underprediction in another can cancel in a pooled mean residual. A pooled error measure can also be dominated by the largest group, leaving weak performance in a smaller group hard to see. Reviewing diagnostics by subgroup is a way to look for these masked patterns, not a substitute for the overall review.

The subgroups should have a clinical or design rationale: examples include age bands, renal-function categories, body-size ranges, dose levels, formulation, or study site. Where a covariate is continuous, clinically justified ranges can make a comparison interpretable, but categorizing it discards information and results may depend on the chosen cut points. Define groups before examining the diagnostic pattern when possible; selecting cut points after seeing the results can make chance differences look persuasive.

This review builds on residual diagnostics, individual and population prediction diagnostics, and visual predictive checks. The goal here is narrower: compare the relevant diagnostic summaries across groups, check whether group sizes support those comparisons, and decide whether a pattern merits investigation.

Choose Comparable Diagnostics

For observation \(j\) from individual \(i\), let \(y_{ij}\) be the observed outcome and \(\hat y_{ij}\) the prediction being reviewed. State which prediction is used. A population prediction and a prediction informed by an individual's observations answer different questions, so results based on them should not be silently combined. Define the signed residual as

A positive residual means the observation exceeds the prediction; a negative residual means the prediction exceeds the observation. Its units are the same as the outcome. Within subgroup \(g\), useful summaries include the mean signed residual, mean absolute error, and root mean squared error:

Here \(n_g\) is the number of observations in subgroup \(g\). The mean signed residual indicates direction of average error; MAE and RMSE describe error magnitude, with RMSE more affected by large errors. These are not interchangeable. Opposing errors may make \(\bar e_g\) small even when MAE is large. Always retain the outcome units when interpreting these summaries.

When outcomes span a wide range, relative errors or observed-to-predicted ratios may help compare scale, but they need care: ratios become unstable when predictions are near zero, and a relative error is not directly comparable to an absolute error. Prediction-interval coverage is another useful diagnostic. If an interval is intended to contain a specified proportion of outcomes under the model, calculate the fraction of observed outcomes inside it for each group. Coverage summaries should state the nominal target and the number of observations used.

Why Pooled Summaries Can Mislead

The masking in the example follows directly from the weighted mean. If subgroups partition the observations, the pooled mean residual is the subgroup-size-weighted average of the subgroup means:

To verify this, write the sum of all residuals as the sum within each subgroup. By definition, the residual sum in group \(g\) is \(n_g\bar e_g\). Dividing the sum of those group totals by the total number of observations gives the equation above. Consequently, a large subgroup can dominate the pooled result, and subgroup means with opposite signs can cancel. An overall mean is therefore not evidence that every subgroup is adequately predicted.

This identity also explains why raw counts belong beside each diagnostic summary. An unweighted average of subgroup means answers a different question from the pooled observation-level mean when group sizes differ. Neither summary is universally preferable: report the one that matches the review question and make the weighting explicit.

Check Counts, Clustering, and Coverage

The count \(n_g\) alone may not describe how much information a subgroup contains. In longitudinal pharmacometric data, many observations can come from a small number of individuals. If one individual contributes numerous measurements, an observation-level summary may primarily reflect that individual’s record. Report both the number of observations and the number of individuals, and inspect whether the pattern is concentrated in one or two individuals. If the review question concerns performance for a typical individual, consider summaries at the individual level as well as observation-level summaries. State how those individual summaries were formed.

Prediction-interval coverage has a simple definition: count an observation as covered if it falls between the lower and upper bounds of its stated prediction interval. Divide the covered count by the number of eligible observations in that subgroup. Coverage can reveal a group with too many outcomes outside the intervals even when average residuals look modest. It is a descriptive comparison, not proof that the interval-generating procedure is correct. The nominal coverage, interval construction, and whether the intervals were intended for individual outcomes must be clear.

A Practical Review Sequence

Diagnostics, Alternatives, and Limitations

A subgroup table is compact and makes counts and numerical summaries easy to compare. Plots can reveal structure that a table obscures: for example, residuals consistently increasing with time in one group, or one extreme observation accounting for a large RMSE. A useful review often pairs a table of counts and summaries with a plot that preserves the underlying observations. Do not rely on a single summary statistic to describe a complex pattern.

The main alternative to subgrouping is to inspect diagnostics continuously against a covariate such as age or body weight. This avoids dependence on arbitrary category boundaries and can show gradual trends, but it can be harder to summarize and can obscure a clinically meaningful threshold. Subgroup summaries are easier to communicate but discard within-group variation. These approaches complement each other: use clinically justified groups for focused comparison and retain continuous diagnostic displays when the underlying covariate is continuous.

Common failure modes include interpreting a small group’s unstable summary as a reproducible model defect; treating repeated observations from the same individual as independent evidence; pooling incompatible prediction types; and searching many subgroup definitions until a striking pattern appears. A subgroup with only a few observations may show extreme MAE, RMSE, or coverage because one observation changes the result substantially. Conversely, a reassuring mean can coexist with substantial individual errors or errors at particular times.

Subgroup diagnostics are descriptive and do not automatically supply a formal test of subgroup differences. A large number of comparisons increases the chance of finding an unusual pattern by chance. Nor can the diagnostic alone distinguish model misspecification from data-processing errors, differences in observation timing, sparse sampling, or other study features. Review the pattern against the model’s intended use and the recorded data, and communicate uncertainty. A subgroup review can identify where further examination is needed; it cannot, by itself, establish causation or guarantee performance for future patients.