Comparing Error Distributions Across Groups

Why Compare Error Distributions?

A single error metric can be useful for comparing overall predictive performance, but it compresses many individual errors into one number. That compression can hide important differences between groups. Two groups might have the same mean absolute error (MAE), for example, even though one has mostly small errors and a few large ones while the other has errors of similar size throughout. For an actuarial decision, those patterns can matter: a rare, large miss may have a different consequence from a small, routine miss.

Comparing group-specific distributions means examining the collection of errors in each group, not just comparing one aggregate metric. Useful summaries include the mean signed error, MAE, RMSE, median, quartiles, and range. A dot plot, histogram, or boxplot can show how the errors are arranged. These methods describe the observed errors; by themselves, they do not establish that a difference between groups is statistically significant or will persist in future data.

Definition: For observation \(i\), define the signed prediction error as \(e_i=y_i-\hat{y}_i\), where \(y_i\) is the observed value and \(\hat{y}_i\) is its prediction. A positive error means the prediction was below the observation; a negative error means it was above. For group \(g\) with \(n_g\) observations, the empirical error distribution is the set of its observed errors, with each observation assigned mass \(1/n_g\). Always use the same error definition and measurement units for every group being compared.

Summaries That Describe Different Features

The mean signed error, \(\bar e_g\), indicates the direction and average size of errors with their signs retained. Errors of opposite signs can cancel. MAE averages the absolute sizes, so it does not cancel over- and under-predictions. RMSE gives relatively more influence to larger errors because it squares each error before averaging. These measures answer different questions; none describes the full distribution.

\bar e_g=\frac{1}{n_g}\sum_{i\in g}e_i,\qquad \operatorname{MAE}_g=\frac{1}{n_g}\sum_{i\in g}|e_i|,\qquad \operatorname{RMSE}_g=\sqrt{\frac{1}{n_g}\sum_{i\in g}e_i^2}.

The median and quartiles describe locations within the ordered errors. The interquartile range, \(Q_{3,g}-Q_{1,g}\), describes the spread of the middle half. The minimum and maximum show the observed range, though they can be sensitive to individual observations. Report the quantile convention if exact quartiles matter: software packages can use different interpolation rules. In hand calculations below, quartiles are the medians of the lower and upper halves, excluding the overall median when the group size is odd.

A useful distinction is between a group’s center and its shape. A median near zero does not say whether errors are tightly clustered or widely dispersed. A symmetric-looking distribution may have a different tail from a skewed one. Read numeric summaries alongside a plot rather than treating one statistic as a complete description.

Worked Comparisons

Worked Example: Same MAE, Different Spread

A hypothetical pricing model is evaluated on five policies in each of two regions. The signed errors, in dollars, are Region A: \(-2,-1,0,1,2\), and Region B: \(-3,0,0,0,3\). Compare their centers and spreads using the mean signed error, MAE, RMSE, and quartiles.

Method. Calculate each measure from the individual signed errors, then compare the ordered values. This avoids assuming that equal MAE means the error distributions are alike.

For Region A, the sum of errors is \(-2-1+0+1+2=0\), so \(\bar e_A=0/5=0\). The absolute errors sum to \(2+1+0+1+2=6\), giving \(\operatorname{MAE}_A=6/5=1.2\). The squared errors sum to \(4+1+0+1+4=10\), so \(\operatorname{RMSE}_A=\sqrt{10/5}=\sqrt{2}\approx1.414\).

For Region B, the sum of errors is \(-3+0+0+0+3=0\), so \(\bar e_B=0\). The absolute errors sum to \(3+0+0+0+3=6\), giving \(\operatorname{MAE}_B=6/5=1.2\). The squared errors sum to \(9+0+0+0+9=18\), so \(\operatorname{RMSE}_B=\sqrt{18/5}\approx1.897\).

Both medians are 0. For Region A, the lower half is \((-2,-1)\) and the upper half is \((1,2)\); their medians are \(Q_{1,A}=-1.5\) and \(Q_{3,A}=1.5\), so the IQR is 3. For Region B, the lower half is \((-3,0)\) and the upper half is \((0,3)\); hence \(Q_{1,B}=-1.5\), \(Q_{3,B}=1.5\), and the IQR is also 3. The quartiles alone do not distinguish the patterns: Region B has three errors exactly at zero and two more extreme errors.

Check. Both groups have mean error 0 and MAE 1.2, as required by the equal signed and absolute sums. Region B’s higher RMSE is consistent with its larger squared-error total. A dot plot would reveal the concentration at zero and the two distant values more clearly than the shared median and IQR.

Worked Example: A Shared MAE Can Hide a Large Error

Two hypothetical service areas each have five forecast errors, in thousands of dollars. Area C has errors \(0,0,0,0,10\); Area D has errors \(2,2,2,2,2\). Compare the mean signed error, MAE, RMSE, and median.

For Area C, the errors sum to 10, so the mean signed error is \(10/5=2\). The absolute errors also sum to 10, giving an MAE of \(10/5=2\). The squared errors sum to \(100\), so the RMSE is \(\sqrt{100/5}=\sqrt{20}\approx4.472\). The sorted values have median 0.

For Area D, the errors sum to \(2+2+2+2+2=10\), so the mean signed error is \(10/5=2\). The absolute errors sum to 10, and the MAE is \(10/5=2\). The squared errors sum to \(5(2^2)=20\), so the RMSE is \(\sqrt{20/5}=2\). The median is 2.

Interpretation and check. Both areas have the same mean signed error and MAE, but Area C’s RMSE is about 4.472 thousand dollars, compared with 2 thousand dollars for Area D. Area C has four zero errors and one large positive error; Area D has the same moderate error each time. A dot plot makes this contrast immediate. The common mistake is to conclude from the shared MAE that the two areas have similar risk of large misses. Their individual error patterns show otherwise.

Worked Example: Comparing Asymmetry With Matching Mean and MAE

A hypothetical forecasting team compares seven errors in each of two product groups, measured in dollars. Group E has errors \(-2,-1,0,0,1,1,1\). Group F has errors \(-1,-1,-1,0,0,0,3\). Compare their mean signed errors, MAEs, RMSEs, medians, and observed extremes.

For Group E, the signed errors sum to \(-2-1+0+0+1+1+1=0\), so the mean is 0. The absolute values sum to \(2+1+0+0+1+1+1=6\), so the MAE is \(6/7\approx0.8571\). The squared values sum to \(4+1+0+0+1+1+1=8\), so the RMSE is \(\sqrt{8/7}\approx1.069\). The median is the fourth ordered value, 0; the observed minimum and maximum are \(-2\) and 1.

For Group F, the signed errors sum to \(-1-1-1+0+0+0+3=0\), so the mean is 0. The absolute values sum to \(1+1+1+0+0+0+3=6\), giving MAE \(6/7\approx0.8571\). The squared values sum to \(1+1+1+0+0+0+9=12\), giving RMSE \(\sqrt{12/7}\approx1.309\). The median is 0; the minimum and maximum are \(-1\) and 3.

Interpretation and check. The groups match on mean signed error, MAE, and median, but Group F has a more prominent positive extreme and a higher RMSE. Its observations are concentrated at \(-1\) and 0, with one error at 3; Group E’s observed values are more spread across both sides of zero. The sums verify that the signed errors cancel in both groups, while the squared-error totals verify why Group F’s RMSE is larger. A dot plot, with a common horizontal scale, would reveal the different asymmetry and tail pattern.

Why RMSE Responds to Spread

There is an exact identity that helps explain what RMSE captures. Define the empirical variance of the errors in group \(g\) using divisor \(n_g\), not \(n_g-1\), as \(V_g=n_g^{-1}\sum_{i\in g}(e_i-\bar e_g)^2\). Then:

\begin{aligned} \operatorname{RMSE}_g^2 &=\frac{1}{n_g}\sum_{i\in g}e_i^2\\ &=\frac{1}{n_g}\sum_{i\in g}\bigl[(e_i-\bar e_g)+\bar e_g\bigr]^2\\ &=\frac{1}{n_g}\sum_{i\in g}(e_i-\bar e_g)^2 +\frac{2\bar e_g}{n_g}\sum_{i\in g}(e_i-\bar e_g) +\bar e_g^2\\ &=V_g+\bar e_g^2. \end{aligned}

The cross-term is zero because \(\sum_{i\in g}(e_i-\bar e_g)=\sum_{i\in g}e_i-n_g\bar e_g=0\). Thus, for this empirical definition, squared RMSE combines squared average signed error with the variance of errors. This identity explains why RMSE can be elevated by a nonzero average error, by substantial spread, or by both. It does not mean that RMSE identifies the shape of the distribution: different collections of errors can have the same mean and variance.

Choosing and Reading Plots

Use a plot that makes the relevant feature visible, and put groups on a common scale. A dot plot is especially informative for small groups because it shows repeated values and isolated observations. Histograms can compare the broad shape of larger collections, but bin width and bin boundaries affect their appearance. Boxplots compactly compare medians and central spread, but they do not show how many observations occupy each value or reveal every feature of the tails. If a tail is important, inspect the actual extremes or use a plot that displays individual observations.

When plotting errors, a reference line at zero helps distinguish positive from negative errors. If comparing absolute errors instead, zero is a boundary and direction has been discarded; use absolute errors when magnitude is the question, not when over- versus under-prediction matters. Keep the same units and axes across groups. Separate axis ranges can make similar spreads look different, while a shared range makes visual comparisons more direct.

Common Mistakes and Practical Interpretation

Equating zero mean error with accuracy. Errors \(-4\) and \(4\) have mean zero, but MAE is \((4+4)/2=4\) and RMSE is \(\sqrt{(16+16)/2}=4\). The errors cancel in the signed mean; they do not disappear.
Calculating RMSE from the group mean alone. For errors \(0,0,0,0,10\), taking the absolute mean error of 2 as the RMSE is wrong. The correct calculation is \(\sqrt{(0+0+0+0+100)/5}=\sqrt{20}\approx4.472\).
Reading a boxplot as the full distribution. A boxplot summarizes selected quantiles and may not make clusters or individual tail values apparent. Check a dot plot or the underlying observations when those features matter.
Ignoring group size and context. A group with few observations may have an unstable-looking distribution simply because each observation has a large visual presence. State the number of observations and avoid treating a descriptive difference as proof of a persistent difference.
Changing error definitions between groups. Reversing the sign convention changes the interpretation of positive and negative errors. Mixing signed and absolute errors makes comparisons invalid.

For a clear comparison, report enough information to distinguish direction, typical magnitude, spread, and unusual extremes. For example, pair mean signed error with MAE or RMSE, state each group’s sample size, and support the summaries with a common-scale plot. These steps help ensure that a shared average does not conceal a consequential pattern. This approach is aligned with the SOA Exam PA (Predictive Analytics) syllabus.

Check Your Understanding

For each problem, calculate the requested quantities before checking the final answer.

A hypothetical claims team records four signed forecast errors, in dollars: \(-1,0,1,4\). Calculate the mean signed error, MAE, RMSE, and median. What feature would a dot plot make visible? (Answer: mean \(=1\), MAE \(=1.5\), RMSE \(=\sqrt{4.5}\approx2.1213\), median \(=0.5\); the isolated error at 4.)
Group A has errors \(-2,0,2\), and Group B has errors \(-1,-1,2\). For each group calculate the mean signed error and MAE. Explain why the means alone do not establish equal accuracy. (Answer: both means are 0; both MAEs are \(4/3\); the signed means conceal the nonzero error magnitudes.)
Two groups each have four errors. Group C has \(0,0,0,8\); Group D has \(2,2,2,2\). Calculate each group’s MAE and RMSE, and identify which has the larger observed extreme. (Answer: both MAEs are 2; Group C RMSE is \(\sqrt{16}=4\), Group D RMSE is 2; Group C has the larger maximum error, 8.)
Group E has errors \(-3,-1,1,3\), and Group F has errors \(-2,-2,2,2\). Calculate the mean signed error and RMSE for each. What do these summaries say about their centers and squared-error magnitudes? (Answer: both means are 0; both RMSEs are \(\sqrt{(9+1+1+9)/4}=\sqrt{5}\approx2.2361\); both have zero average signed error and equal squared-error averages.)
A seven-observation group has sorted errors \(-2,-1,0,0,0,1,6\). Using the stated median-of-halves convention, calculate the median, \(Q_1\), \(Q_3\), IQR, MAE, and RMSE. (Answer: median \(=0\), \(Q_1=-1\), \(Q_3=1\), IQR \(=2\), MAE \(=10/7\approx1.4286\), RMSE \(=\sqrt{42/7}=\sqrt{6}\approx2.4495\).)

What You'll Learn