Residual Patterns Across Categorical Groups

Why Compare Residuals Across Groups?

A fitted regression relationship can look reasonable overall while systematically underpredicting for one category and overpredicting for another. For example, a common relationship between treatment dose and a measured response might fit the full sample reasonably well, yet leave mostly positive residuals in one clinic and mostly negative residuals in another. Such a pattern suggests that the model may be missing a group-specific feature.

Residuals are the observed responses minus the corresponding fitted values. Comparing them across categories can reveal a possible group-specific offset (a difference in average residual level) or a trend (a residual pattern that changes with a predictor within a group). These comparisons are diagnostic: they indicate where the fitted mean relationship may need investigation, but they do not by themselves establish a cause or provide a formal test.

Definition: For observation $i$, the residual is $e_i=y_i-\hat y_i$, where $y_i$ is the observed response and $\hat y_i$ is the fitted response. For a categorical group $g$ containing $n_g$ observations, its mean residual is $$ \bar e_g=\frac{1}{n_g}\sum_{i:G_i=g}e_i. $$ A positive $\bar e_g$ means that, on average, the model underpredicts responses in that group; a negative $\bar e_g$ means that it overpredicts them.

What the Group Comparisons Can Show

Suppose the fitted model uses predictors $X_i$, but does not allow its fitted relationship to vary by category $G_i$. A general way to describe that common-relationship model is

Y_i=f(X_i;\beta)+\epsilon_i.

Here $f(X_i;\beta)$ is the modeled mean relationship and $\epsilon_i$ is the unobserved error. For the model to describe the conditional mean correctly, its central mean assumption is $E(\epsilon_i\mid X_i,G_i)=0$. In other words, after accounting for the predictors in the model, the expected response does not systematically depend on the category. Residuals are observable estimates of the unobserved errors, so they can suggest departures from this assumption but do not equal the errors themselves.

For this diagnostic comparison, first identify the categorical variable and the fitted model being assessed. Then compare residual averages and distributions by category. A plot of residuals by group can show shifts, unusual clusters, asymmetry, or differences in spread. If the model includes a quantitative predictor, also check how residuals vary with that predictor within each group. A group average can conceal a trend, especially when residuals rise in some groups and fall in others.

These are exploratory checks, not a procedure that requires normally distributed errors. Independence and the mean-model assumptions matter when interpreting the fitted regression and its errors; a residual plot alone cannot verify them. Nor should a group difference automatically be attributed to the category: groups may have different predictor values, so a crude comparison can reflect that imbalance rather than a group-specific departure from the fitted relationship.

Worked Example: A Possible Group-Specific Offset

A hypothetical analyst fits a common regression relationship without including clinic as a predictor. The fitted values and residuals for eight patients are shown below. The two clinic groups have the same number of observations.

Clinic	Residuals (response units)
A	-2, -1, 0, 1
B	0, 0, 1, 1

For Clinic A, the mean residual is

\bar e_A=\frac{-2+(-1)+0+1}{4}=\frac{-2}{4}=-0.500.

For Clinic B, it is

\bar e_B=\frac{0+0+1+1}{4}=\frac{2}{4}=0.500.

Thus the common model underpredicts by an average of 0.500 response units in Clinic B and overpredicts by an average of 0.500 response units in Clinic A. The opposing averages warrant checking whether a clinic-specific offset is missing. They do not show that clinic caused the difference, and the summaries alone do not account for possible differences in the patients’ predictor values.

Residual Constraints Matter

Least-squares fitting imposes some constraints on residuals. With an intercept in the fitted model, residuals sum to zero across the full data set. This does not mean that their averages must be zero in every category. However, if group indicators are included in the model, the residuals also sum to zero within each represented group. In that case, a group mean residual is not a useful check for a group offset on the same fitted data: it is zero by construction.

The zero-sum result follows directly from least squares. With an intercept, the sum of squared residuals is minimized as the fitted coefficients vary. If the intercept is $b_0$, changing it shifts every fitted value by the same amount. At the least-squares minimum, the derivative of the residual sum of squares with respect to $b_0$ must be zero:

\frac{\partial}{\partial b_0}\sum_{i=1}^{n}(y_i-\hat y_i)^2 =-2\sum_{i=1}^{n}(y_i-\hat y_i) =-2\sum_{i=1}^{n}e_i=0.

Therefore $\sum_i e_i=0$. If the model also contains an indicator for a group, changing that indicator’s coefficient shifts the fitted values for observations in that group only. The corresponding derivative is $-2\sum_{i:G_i=g}e_i$; setting it to zero gives $\sum_{i:G_i=g}e_i=0$. Consequently, inspect patterns that the fitted terms do not force to vanish, such as residual trends with a quantitative predictor within a group.

Worked Example: Opposing Trends Within Groups

A hypothetical common linear model includes a quantitative predictor $x$ but no group-specific slopes. The predictor values are $-1$ and $1$ in each of two groups. Suppose the fitted residuals are as follows:

Group	Residual at $x=-1$	Residual at $x=1$
A	-1	1
B	1	-1

Within Group A, the residual changes from $-1$ to $1$, an increase of $2$ residual units as $x$ increases by $2$ units. The change per unit of $x$ is $2/2=1$. Within Group B, it changes from $1$ to $-1$, a decrease of $2$, or $-2/2=-1$ residual units per unit of $x$.

The group mean residuals are both zero:

\bar e_A=\frac{-1+1}{2}=0,\qquad \bar e_B=\frac{1+(-1)}{2}=0.

Even the overall linear association between these residuals and $x$ cancels:

\sum_i x_i e_i=(-1)(-1)+(1)(1)+(-1)(1)+(1)(-1)=0.

So an overall residual average, or even a single overall linear trend summary, would miss the opposing within-group patterns. A residual-versus-$x$ plot separated by group would reveal them. This pattern suggests investigating whether the groups have different slopes; it does not, by itself, prove that a group-specific slope is needed.

Compare Distributions, Not Only Averages

A mean residual is a useful summary of average over- or underprediction, but it cannot describe the full residual distribution. Two groups can have the same mean residual and very different variability or shapes. Conversely, a group average near zero can conceal a pattern of positive residuals at some predictor values and negative residuals at others. Use plots alongside numerical summaries, and make the comparison in the context of the predictors included in the model.

Worked Example: Equal Means but Different Residual Spread

A hypothetical fitted model produces these residuals, in response units, for two categories:

Category	Residuals
A	-1, 0, 0, 1
B	-3, -1, 1, 3

Both categories have mean residual zero:

\bar e_A=\frac{-1+0+0+1}{4}=0,\qquad \bar e_B=\frac{-3+(-1)+1+3}{4}=0.

Their sample standard deviations differ. For Category A, the sum of squared deviations from its mean is $1+0+0+1=2$, so

s_{e,A}=\sqrt{\frac{2}{4-1}}=\sqrt{\frac{2}{3}}\approx0.8165.

For Category B, the sum is $9+1+1+9=20$, giving

s_{e,B}=\sqrt{\frac{20}{4-1}}=\sqrt{\frac{20}{3}}\approx2.582.

The average residuals do not suggest an offset, but Category B’s residuals are much more spread out in this small example. A plot or distribution summary would make that difference visible. It is evidence to investigate, not enough on its own to identify why the spread differs or to conclude that the mean relationship is wrong.

A Practical Comparison

A careful group comparison keeps the fitted model, the categories, and the predictor values in view. For a compact diagnostic workflow:

State the model and grouping variable.
Identify which predictors define the common fitted relationship and which categorical variable is being used to organize the residuals.

Compare group averages and distributions.
Look for consistently positive or negative residuals, unusual clusters, or marked differences in spread or shape.

Check trends within groups.
If the model uses a quantitative predictor, plot residuals against it separately by category to see whether patterns differ across groups.

Interpret cautiously.
Check whether groups occupy comparable predictor ranges and whether fitted terms force the observed group averages to zero. Describe a pattern as a diagnostic clue, not as proof of a cause or a formal finding.

Common Mistakes

Reading a near-zero overall mean as evidence of no group pattern. Positive and negative group residuals can cancel. Compare groups separately.
Using group means alone. Opposing within-group trends can exist even when every group mean is zero. Inspect residuals against relevant quantitative predictors within each category.
Treating zero group means as evidence that offsets are absent. If group indicators are already in the least-squares model, group residual sums are zero by construction. Use diagnostics for patterns not forced away by the fitted terms.
Calling a residual difference a causal group effect. A residual pattern is evidence that the fitted relationship may not capture some structure. It does not establish what produced that structure.
Ignoring predictor imbalance. If categories have different predictor values, their raw residual averages may not be directly comparable. Examine the residual patterns over the predictor ranges the groups share.

Key takeaway: Compare residual averages to look for group-specific offsets, and compare within-group residual patterns to look for group-specific trends. Always consider the fitted terms, the predictor values represented in each group, and the limits of what a diagnostic pattern can establish.

Check Your Understanding

Use these questions to check your interpretation of residual patterns across categories.

A category has a mean residual of $-0.7$ response units. What does this say about the model’s predictions for that category on average?
Why can an overall mean residual of zero coexist with different mean residuals across two categories?
In a least-squares model with group indicators, why is the in-sample mean residual within each represented group zero?
Two groups have mean residuals of zero, but residuals rise with $x$ in one group and fall with $x$ in the other. What diagnostic information would an overall mean fail to show?
Why should an analyst check the predictor values in each category before interpreting differences in group residual summaries?