Why Compare Diagnostic Patterns Across Subgroups?
A diagnostic display can look different across subgroups for two distinct reasons: the model may behave differently in those groups, or the displays may have different amounts of information. A small subgroup often produces a jagged residual pattern, while a large subgroup can show a modest but persistent trend clearly. Neither visual smoothness nor visual prominence alone establishes a model problem.
Subgroup comparisons are useful when the groups were chosen for a scientific or design reason, such as dose level, study, or a prespecified covariate category. The aim is to ask whether a diagnostic feature is consistent, localized, or too uncertain to distinguish across groups. A diagnostic plot can flag where model review may be warranted; by itself, it does not prove why a pattern occurred or establish a subgroup treatment effect.
Set Up a Fair Comparison
Use the same diagnostic quantity and construction in every subgroup. For example, if the display uses standardized residuals, use the same residual definition, horizontal reference at zero, and smoothing or binning rule throughout. A residual is the observed value minus its corresponding prediction; a standardized residual expresses that discrepancy on a scale adjusted for its expected variability. The interpretation still depends on which residual and prediction were selected.
For direct visual comparison, begin with common axis limits, units, transformations, and reference lines. Consistent plot scales make differences in location or spread easier to see; changing scales panel by panel can make a small difference look large or a large difference look small. Earlier material on population goodness-of-fit panels and residual diagnostics provides the relevant display types. Here the focus is how their patterns compare across groups.
Before interpreting a contrast, check what constitutes an independent unit. Multiple observations from one subject are usually related. Thus, 100 residual points from 10 subjects do not necessarily provide as much independent information as 100 residual points from 100 subjects. Repeated observations can make a panel look dense without making its pattern correspondingly precise.
The reason for the simple formula is direct. If \(R_1,\ldots,R_n\) are independent residuals, each with variance \(\sigma^2\), then the mean is \(\bar{R}=(R_1+\cdots+R_n)/n\). Independence makes the variance of the sum equal to the sum of the variances:
Replacing the unknown \(\sigma\) by the sample standard deviation \(s\) gives the estimated standard error \(SE=s/\sqrt{n}\). This derivation explains why a smaller independent sample tends to produce a less stable mean, even if the underlying residual behavior is the same. It does not justify treating repeated measurements as independent.
A Practical Reading Sequence
Check that the residual definition, prediction type, axes, units, reference lines, and display construction match across subgroups.
Look at how many subjects and observations contribute to each panel, and note imbalance, clustering, and sparse regions.
Describe whether the apparent issue is a shift from zero, a trend with time or prediction, a change in spread, or a few isolated points.
Ask whether the feature occurs in multiple groups and whether it persists across the supported range, rather than relying on one noisy region.
Report what the display suggests and what it cannot resolve. Treat a contrast as a prompt for targeted review, not automatic evidence of a subgroup-specific model failure.
Worked Comparisons
Worked Example: A Noisier Small Subgroup
Suppose two hypothetical subgroups each have residuals centered near zero with a standard deviation of about \(0.80\). Subgroup A has 100 independent subjects and a mean residual of \(0.12\); subgroup B has 16 independent subjects and a mean residual of \(0.30\). The plotted difference in means is \(0.30-0.12=0.18\).
For this illustration, assume one residual per subject, so the independence condition is reasonable. The estimated standard errors are
Under independent sampling between groups, the standard error of the difference is approximately
The observed difference, \(0.18\), is smaller than this approximate standard error. A rough two-standard-error range for the difference is \(0.18\pm2(0.215)\), or approximately \([-0.250,\,0.610]\) residual units. This is not a formal model comparison, but it shows why the higher mean in subgroup B is not compelling evidence of a distinct pattern on its own. The small panel is expected to fluctuate more. If the data instead contained repeated observations per subject, these calculations would need to reflect that dependence.
Worked Example: A Direction Shared Across Groups
In a hypothetical residual-versus-time display, two prespecified groups show positive residual means in the later part of follow-up. In an early window, group A has mean residual \(0.02\) from 64 independent subjects and group B has mean residual \(0.04\) from 25 independent subjects. In a later window, their means are \(0.24\) and \(0.28\), respectively. Suppose the residual standard deviation is \(0.80\) in each window.
For the later window, the standard errors are \(0.80/\sqrt{64}=0.100\) for A and \(0.80/\sqrt{25}=0.160\) for B. The increases from early to late are \(0.24-0.02=0.22\) for A and \(0.28-0.04=0.24\) for B. The direction and approximate magnitude of change are similar, though the smaller group is less precise.
This comparison supports a cautious description: both displays suggest a positive residual pattern later in time, and the difference in subgroup sample sizes affects how stable each panel looks. It does not establish that the time trend is identical, nor does it show that time itself causes the residual pattern. The key diagnostic observation is that the feature recurs in both groups rather than appearing only in the smaller panel.
Worked Example: One Point Can Dominate a Small Panel
Consider a hypothetical subgroup with eight independent standardized residuals: seven are \(0.0\) and one is \(2.4\). Its mean is
A larger comparison subgroup has 80 residuals with mean \(0.05\). The small subgroup’s mean is visibly higher, but the calculation shows that one observation accounts for the entire positive shift. The appropriate visual reading is not immediately “the model underpredicts for this subgroup.” First inspect whether the point is valid, whether it belongs to a particular subject or design region, and whether nearby residuals support a broader pattern.
For the eight-value subgroup, the sample standard deviation is approximately \(0.849\): the deviations from the mean are seven values of \(-0.300\) and one value of \(2.100\), so
The large spread relative to the mean reinforces that the panel is dominated by an isolated point, not a uniformly shifted cloud. This does not make the point irrelevant; it makes the conclusion less certain and directs attention to its context.
Distinguishing Pattern From Display Artifact
Unequal sample sizes affect a diagnostic display in several ways. A large group tends to show a denser cloud and can reveal subtle structure; a small group may show gaps, irregular local features, or an apparently extreme point simply because there are fewer observations to average over. Overplotting can also hide dense regions, while jitter or transparency can change visibility without changing the underlying data. Such display effects should not be mistaken for a change in model behavior.
Compare the shape and location of patterns, not just how smooth or visually dominant they appear. If a trend recurs in several subgroups in the same region and direction, it is more credible as a shared diagnostic feature than a trend visible in just one sparse panel. If the pattern appears only where one group has little data, state that the display is inconclusive there. A lack of visible structure in a small panel is not strong evidence that the structure is absent.
The main alternatives each answer a different question. Separate panels with shared scales make within-group structure easy to inspect and preserve a fair visual scale. An overlay can make alignment more direct, but overlapping symbols may obscure group membership, especially with unequal sizes. Free scales can help reveal within-panel detail, but should not be the sole basis for comparing magnitudes. A pooled diagnostic can show the overall pattern while concealing subgroup-specific behavior. These views can complement each other; none removes the need to consider how observations are distributed across subjects and the range of the predictor.
Common Mistakes and Limits
- Calling the largest-looking panel the worst-fitting group. A larger group may look smoother or more prominent because it contains more observations. Compare pattern direction and location, not visual density alone.
- Interpreting a noisy small-group feature as a confirmed difference. A jagged curve or isolated point may be chance variation. Check whether the feature is supported across the group’s observed range and by more than one subject.
- Comparing panels with different scales. A plot with a narrower vertical range can exaggerate variation. Use common scales for direct magnitude comparisons, and clearly identify any supplementary rescaled view.
- Counting observations as independent subjects. Repeated residuals within a subject are dependent. Dense panels may contain less independent information than their point counts suggest.
- Searching many subgroups and reporting only the most striking contrast. Selective comparison increases the chance of finding an accidental pattern. Prespecify meaningful groups where possible, and describe exploratory comparisons as exploratory.
Subgroup diagnostic displays have limits. They are descriptive evidence, not a substitute for evaluating the model, data quality, or study design. Unequal exposure to the predictor range can make groups incomparable in some regions, and unexplained patterns may reflect data issues or model structure not visible in a single panel. The sound conclusion is often deliberately restrained: identify the feature, indicate where it occurs, note the information available, and say whether it appears consistently across groups.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- Two groups have the same residual standard deviation. One has 36 independent subjects and the other has 9. What is the ratio of their estimated standard errors for the mean residual?
- A positive residual trend appears in both subgroups, but the smaller group’s panel is much more jagged. What is a defensible interpretation?
- Why can a panel with many residual points still provide limited independent information?
- One subgroup has an apparent shift that is driven by a single extreme residual. What should be checked before describing it as a subgroup-wide pattern?
Solutions. (1) The standard errors are proportional to \(1/\sqrt{n}\). Their ratio, smaller-group SE divided by larger-group SE, is \((s/\sqrt{9})/(s/\sqrt{36})=6/3=2\); the smaller group’s estimated standard error is twice as large. (2) State that both groups suggest a positive trend, while the smaller panel is less stable; do not treat jaggedness alone as evidence of different model behavior. (3) Multiple observations may come from the same subjects and be correlated, so the number of points can exceed the amount of independent information. (4) Check the observation and subject context, data validity, and whether other residuals in that subgroup support the shift; one point alone does not establish a broad pattern.