Residuals Across Categorical Groups

Why Examine Residuals by Category?

A model can have acceptable overall error while making systematically high or low predictions for a particular kind of observation. For example, a hypothetical pricing model might underpredict one coverage tier and overpredict another. If those errors offset when all observations are combined, an overall average can conceal the group-level pattern.

A categorical-group residual analysis asks whether prediction errors differ across levels of a factor, such as region, product tier, or customer channel. The aim is diagnostic: identify patterns that deserve investigation, not to conclude from a plot alone that the model is fair, unfair, or causally wrong. The factor levels are labels; their order on a plot does not imply a numerical scale.

Definition: For observation $i$, define the residual as $e_i=y_i-\hat{y}_i$, where $y_i$ is the observed response and $\hat{y}_i$ is the model prediction on the same scale. A positive residual means the prediction was too low (underprediction); a negative residual means it was too high (overprediction). For group $g$, its mean residual is $\bar{e}_g=(1/n_g)\sum_{i\in g}e_i$, where $n_g$ is the number of observations in that group.

This sign convention must be stated because some software or analysts define residuals in the opposite direction. Here, a group mean above zero indicates average underprediction, while a group mean below zero indicates average overprediction. A group mean near zero does not establish that the individual errors are small: positive and negative errors may cancel.

Group Summaries and Unequal Group Sizes

For each factor level, start with the number of observations and the mean residual. The count tells the reader how much data underlies the summary. Consider also a measure of error magnitude, such as the group mean absolute error (MAE), $(1/n_g)\sum_{i\in g}|e_i|$. Mean residual addresses direction; MAE addresses typical absolute size without cancellation. These summaries answer different questions and should not be substituted for one another.

The overall mean residual is not generally the simple average of the group means. When each observation has equal weight, it is the group-size-weighted average. This identity follows by partitioning the full sum of residuals into its group sums:

\bar{e} =\frac{\sum_g\sum_{i\in g}e_i}{\sum_g n_g} =\frac{\sum_g n_g\bar{e}_g}{\sum_g n_g}.

The equality holds because each group sum is $n_g\bar{e}_g$. It is an arithmetic identity, not a claim that the pooled result is always the most useful business summary. If the question concerns the average observation in the validation data, weighting by the observed group counts is appropriate. If the question concerns performance in a future population with a different group mix, the relevant target weights may differ. An equal-weight average of group means answers a different question: the average across factor levels, giving each level the same importance regardless of its count.

Worked Example: Pooled Bias Versus Equal-Group Bias

A hypothetical model predicts repair costs for two service regions. Residuals, in dollars, are North: $-2, 1, 0, 1$; South: $6, 8$. Calculate each group mean, the pooled mean, and the equal-weight average of group means. State what the signs imply. Use the method of summarizing by group before pooling; the common mistake is to give each region equal weight when the target is the average validation observation.

North. There are $n_N=4$ observations, with residual sum $-2+1+0+1=0$. Therefore $\bar{e}_N=0/4=0$ dollars.

South. There are $n_S=2$ observations, with residual sum $6+8=14$. Therefore $\bar{e}_S=14/2=7$ dollars. The model underpredicts South costs by an average of 7 dollars in these data.

Pooled mean. The full residual sum is $0+14=14$, and the total count is $4+2=6$. Thus $\bar{e}=14/6=2.333\ldots$ dollars, or $2.33 to the nearest cent. The weighted calculation gives the same result: $(4(0)+2(7))/6=14/6$.

Equal-weight group mean. Giving each region one-half weight gives $(0+7)/2=3.50$ dollars. This is not the pooled observation-level mean because the regions have different counts. The calculation checks: North contributes no net residual, and South contributes 14 dollars across all six observations, so the pooled average must be positive and less than South’s 7-dollar average.

Choosing and Reading a Grouped Residual Plot

A useful first plot places factor levels on the horizontal axis and residuals on the vertical axis, with a horizontal reference line at zero. A jittered point plot shows individual residuals without stacking identical values on top of one another. An overlaid group-mean marker makes average direction easier to compare. Labeling each level with its count, or showing the count in a companion table, prevents a mean based on only a few records from looking as well-supported as a mean based on many records.

A boxplot by group can compactly show the median and spread, but it does not replace the mean when the diagnostic question concerns average signed error. Likewise, a mean-only bar chart hides the individual observations and can make a small group look more definitive than it is. Keep the zero line visible: the distance from zero conveys direction and size of average signed error.

Inspect both the mean residual and an error-magnitude summary. A group can have a mean near zero and still have large errors in both directions. Conversely, a group can have a small MAE but a nonzero mean if most modest errors point in the same direction. Group summaries help locate patterns; they do not by themselves explain why the patterns arose.

Define the residual and group.
Use one sign convention and state the response scale. Confirm that each observation belongs to the intended factor level.

Calculate counts and summaries.
For each level, report $n_g$, mean residual, and an appropriate magnitude measure such as MAE.

Plot around zero.
Show individual residuals or a distribution summary, include a zero reference, and make group counts visible.

Interpret with the target population in mind.
Distinguish observation-weighted performance from equal-level comparisons, and treat sparse-group patterns cautiously.

Worked Comparisons of Direction and Magnitude

Worked Example: A Large Group Can Dominate the Pooled Result

A hypothetical model predicts monthly equipment costs in two customer channels. Channel A has $n_A=20$, residual sum $-40$, and sum of absolute residuals $80$. Channel B has $n_B=5$, residual sum $20$, and sum of absolute residuals $30$. Calculate group mean residuals and MAEs, then calculate the pooled mean residual and pooled MAE. The method is to divide each group total by its own count before interpreting it; a common mistake is to compare raw totals without accounting for group size.

Channel A. Its mean residual is $-40/20=-2$ dollars, indicating average overprediction by 2 dollars. Its MAE is $80/20=4$ dollars.

Channel B. Its mean residual is $20/5=4$ dollars, indicating average underprediction by 4 dollars. Its MAE is $30/5=6$ dollars.

Pooled mean residual. The total residual is $-40+20=-20$, and the total count is $20+5=25$. Therefore the pooled mean is $-20/25=-0.80$ dollars. The pooled result indicates slight average overprediction overall, even though Channel B is underpredicted.

Pooled MAE. The total absolute residual is $80+30=110$, so pooled MAE is $110/25=4.40$ dollars. As a check, the count-weighted group MAE is $(20(4)+5(6))/25=(80+30)/25=4.40$. The group means reveal opposing directions that the pooled signed mean partly cancels; the MAEs show that Channel B has the larger average absolute error.

The equal-weight average of the two group means would be $(-2+4)/2=1$ dollar, not the pooled $-0.80$ dollars. Neither is an arithmetic error: they describe different weightings.

Worked Example: Reading a Small-Group Pattern Carefully

A hypothetical model is assessed across three membership tiers. The summaries are: Tier A, $n=100$, mean residual $0.20$ dollars and total absolute residual $400$ dollars; Tier B, $n=8$, mean residual $-3$ dollars and total absolute residual $40$ dollars; Tier C, $n=2$, mean residual $12$ dollars and total absolute residual $24$ dollars. Calculate pooled signed mean and MAE, then describe what a grouped residual plot should communicate. The method is to combine totals and counts for pooled metrics, while retaining group-level summaries in the plot; the trap is allowing the pooled average to erase the sparse Tier C pattern.

Recover the group residual totals. Tier A contributes $100(0.20)=20$ dollars; Tier B contributes $8(-3)=-24$ dollars; Tier C contributes $2(12)=24$ dollars. Their total is $20-24+24=20$ dollars.

Pooled signed mean. The total count is $100+8+2=110$. Thus the pooled mean is $20/110=0.1818\ldots$ dollars, or $0.18 to the nearest cent. The positive result is close to zero, but it does not mean every tier is well calibrated: Tier B’s mean is negative, while Tier C’s is strongly positive.

Pooled MAE. The total absolute error is $400+40+24=464$ dollars. Therefore pooled MAE is $464/110=4.2181\ldots$, or $4.22 to the nearest cent. The group MAEs are Tier A: $400/100=4$ dollars; Tier B: $40/8=5$ dollars; and Tier C: $24/2=12$ dollars. These values show that Tier C has the largest average absolute error, but it is based on only two observations.

A suitable plot would show the zero line, the residuals or group summaries for all three tiers, and counts $100$, $8$, and $2$. Tier C’s mean of 12 dollars is worth investigating, but the plot should make its very small count unmistakable. The summaries do not establish whether the pattern will persist in future data.

Common Traps and Practical Limits

Reversing the residual sign. With $e_i=y_i-\hat{y}_i$, positive means underprediction. State the definition rather than relying on an unlabeled axis.
Averaging group means without checking the target. Equal group weights answer a different question from the average across all observations when group counts differ.
Interpreting a near-zero mean as small error. Opposing residuals can cancel. Pair signed summaries with an absolute-error measure or inspect the plotted points.
Overstating sparse-group evidence. A count is essential context, but count alone does not quantify uncertainty. The variability of residuals and the way observations were sampled also matter.
Reading category order as numerical distance. Put factor levels in a deliberate, clearly labeled order. A visual slope across labels is not automatically a trend in a quantitative predictor.

The analysis is most useful when connected to the intended prediction task. A persistent group-specific mean residual can point to a missing predictor, an unsuitable model structure, a data-processing difference, or a change in the population. These are possible explanations to investigate, not conclusions that follow from the grouped plot alone. The approach is aligned with the SOA Exam PA (Predictive Analytics) syllabus.

Check Your Understanding

For each question, use residuals defined as observed minus predicted, and distinguish pooled observation-level summaries from equal-weight group summaries where relevant.

A factor has Group A residuals $-1, 0, 2$ and Group B residuals $3, 5$. Find both group means and the pooled mean residual. (Answer: Group A $=1/3$, Group B $=4$, pooled $=9/5=1.8$.)
Group A has $n=10$ and residual sum $-15$; Group B has $n=5$ and residual sum $20$. Find the pooled mean residual and the equal-weight average of the group means. (Answer: pooled $=5/15=1/3$; equal-weight average $=(-1.5+4)/2=1.25$.)
Group A has four residuals with mean $0$; Group B has one residual of $10$. Find the pooled mean and the equal-weight average of group means. Explain which is appropriate for the mean across all five observations. (Answer: pooled mean $=2$; equal-weight group mean $=5$; the pooled mean is appropriate for the average across all five observations.)
A group’s mean residual is $-2.50$ in the stated sign convention. Interpret the direction of average prediction error. (Answer: predictions are too high on average by $2.50; this is average overprediction.)
Group A has $n=4$ and total absolute residual $12$; Group B has $n=2$ and total absolute residual $10$. Find each group MAE and the pooled MAE. (Answer: Group A MAE $=3$, Group B MAE $=5$, pooled MAE $=(12+10)/(4+2)=22/6\approx3.67$.)

What You'll Learn