Weighted and Unweighted Prediction Metrics

Why Weighting Changes a Prediction Summary

An overall prediction metric can answer different questions depending on how cases are combined. If one group contributes many more evaluated cases than another, pooling every case gives the larger group more influence. Giving each group equal influence instead answers a different question: how well does the method perform across groups when the groups, rather than the individual cases, are treated equally?

Neither choice is automatically correct. The appropriate summary depends on what set of cases the evaluation is meant to represent and on which cases should count most in the decision being made. This tutorial develops the distinction for mean absolute error (MAE) and root mean squared error (RMSE), using groups with unequal sizes.

Definitions: Individual Errors and Group Metrics

For each evaluated case $i$, let $y_i$ be the observed outcome and $\hat{y}_i$ its prediction. Define the prediction error as $e_i=y_i-\hat{y}_i$. Absolute errors measure the size of misses without regard to sign; squared errors give relatively more influence to large misses.

Definition: For a group $g$ containing $n_g$ evaluated cases, its mean absolute error and root mean squared error are $$ MAE_g=\frac{1}{n_g}\sum_{i\in g}|e_i|, \qquad RMSE_g=\sqrt{\frac{1}{n_g}\sum_{i\in g}e_i^2}. $$ For all $N=\sum_g n_g$ cases pooled together, the corresponding metrics are $$ MAE_{\text{pooled}}=\frac{1}{N}\sum_g\sum_{i\in g}|e_i|, \qquad RMSE_{\text{pooled}}=\sqrt{\frac{1}{N}\sum_g\sum_{i\in g}e_i^2}. $$ Both metrics are expressed in the outcome’s units.

These are descriptive summaries of prediction errors on the evaluated cases. They do not require errors to follow a particular probability distribution. For a fair comparison, the outcomes, units, prediction task, and evaluation-case definition should be consistent. The weights should also be stated, because a weighted summary describes a different target from an unweighted one.

Here, “unweighted group average” means that the group metrics are averaged with equal weight, regardless of how many cases each group contains. “Pooled” means that all individual cases contribute once. Thus, pooling inherently weights a group in proportion to its case count. If every group has the same size, these two approaches agree for MAE; with unequal sizes, they generally do not.

Pooling Cases and Weighting Group Metrics

For MAE, the pooled value can be calculated by weighting each group’s MAE by its number of cases. Let $w_g=n_g/N$ be the group’s share of all evaluated cases. Then the pooled MAE is $\sum_g w_g MAE_g$. An equal-group MAE instead gives each of the $G$ groups weight $1/G$.

RMSE requires an additional step. The group RMSE is the square root of a group’s mean squared error, so averaging group RMSE values does not in general recover the pooled RMSE. To pool correctly, first combine the group mean squared errors using the desired weights, then take the square root.

Key result: With case-count weights $w_g=n_g/N$, $$ MAE_{\text{pooled}}=\sum_g w_g MAE_g, \qquad RMSE_{\text{pooled}}=\sqrt{\sum_g w_g RMSE_g^2}. $$ The second formula weights group mean squared errors, not group RMSE values themselves.

The identities follow by substituting the group definitions. For MAE, multiplying $MAE_g$ by $n_g/N$ cancels the group denominator $n_g$. For RMSE, $RMSE_g^2$ is the group mean squared error, so the same cancellation occurs before taking the square root.

\sum_g\frac{n_g}{N}MAE_g = \sum_g\frac{n_g}{N}\left(\frac{1}{n_g}\sum_{i\in g}|e_i|\right) = \frac{1}{N}\sum_g\sum_{i\in g}|e_i|. $$ $$ \sqrt{\sum_g\frac{n_g}{N}RMSE_g^2} = \sqrt{\sum_g\frac{n_g}{N}\left(\frac{1}{n_g}\sum_{i\in g}e_i^2\right)} = \sqrt{\frac{1}{N}\sum_g\sum_{i\in g}e_i^2}.

The final expressions are precisely the pooled definitions. This justifies both the weighting formulas and the important distinction between averaging RMSE values and averaging their squares.

Worked Example: Pooling Unequal Groups

Suppose a hypothetical evaluation has two groups. Group A has two prediction errors, $1$ and $4$ outcome units. Group B has four errors, $6$, $6$, $0$, and $0$ outcome units. Calculate the group metrics, the equal-group averages, and the pooled metrics.

Group MAE values. For Group A, $MAE_A=(|1|+|4|)/2=5/2=2.5$ units. For Group B, $MAE_B=(|6|+|6|+|0|+|0|)/4=12/4=3$ units. Giving each group equal weight produces $$ MAE_{\text{equal-group}}=\frac{2.5+3}{2}=2.75\text{ units}. $$

Pooled MAE. There are six cases, and their absolute errors sum to $5+12=17$. Therefore $$ MAE_{\text{pooled}}=\frac{17}{6}\approx 2.833\text{ units}. $$ Equivalently, the group weights are $2/6$ and $4/6$, giving $(2/6)(2.5)+(4/6)(3)=2.833$ units. The pooled result is larger because Group B, with four of the six cases, has more influence.

Group RMSE values. The sum of squared errors in Group A is $1^2+4^2=17$, so $RMSE_A=\sqrt{17/2}\approx2.915$ units. For Group B, the sum is $6^2+6^2+0^2+0^2=72$, so $RMSE_B=\sqrt{72/4}=\sqrt{18}\approx4.243$ units. The equal-group average of these group RMSE values is approximately $(2.915+4.243)/2=3.579$ units.

Pooled RMSE. Across all cases, the squared errors sum to $17+72=89$. Thus $$ RMSE_{\text{pooled}}=\sqrt{\frac{89}{6}}\approx 3.851\text{ units}. $$ The same result comes from $\sqrt{(2/6)RMSE_A^2+(4/6)RMSE_B^2}$. It is not the equal-group average of the group RMSE values, nor is it the case-count-weighted arithmetic average of those RMSE values. The calculations show how both group sizes and large errors affect the reported summary.

Equal Groups Can Tell a Different Story

A pooled metric describes average performance over the evaluated cases when each case counts equally. That can be the desired target, for example when the evaluation set reflects the population of cases that will actually be encountered. But it can mask poor performance in a small group. An equal-group average prevents large groups from dominating, though it gives a small group the same total influence as a large one.

Worked Example: A Small Group With Larger Errors

Consider another hypothetical evaluation. Group A has eight cases, each with an absolute error of $2$ units. Group B has two cases, each with an absolute error of $8$ units. Find the MAE summaries and compare how they weight the groups.

Group A has $MAE_A=2$ units, and Group B has $MAE_B=8$ units. Giving each group equal weight gives $$ MAE_{\text{equal-group}}=\frac{2+8}{2}=5\text{ units}. $$

For the pooled result, the total absolute error is $8(2)+2(8)=16+16=32$ units across $10$ cases. Therefore $$ MAE_{\text{pooled}}=\frac{32}{10}=3.2\text{ units}. $$ The pooled metric gives Group A weight $8/10$ and Group B weight $2/10$. Although the two groups contribute the same total absolute error here, Group A’s eight cases make the pooled average lower. The equal-group value, by contrast, treats a typical group as the unit of comparison and gives the small, higher-error group half of the total weight.

Weights for a Target Population

Case-count weights are appropriate when the evaluated cases themselves represent the target mix and each case should count equally. Sometimes the evaluation sample has a different mix from the population or operational setting of interest. In that situation, a stated target share can be used as the group weight. The weights must be nonnegative and sum to one. For MAE, average the group MAEs using these weights; for RMSE, average group mean squared errors and then take the square root.

For example, a sample might contain many cases from one group simply because it was easier to recruit them. If the goal is to describe performance in a setting where the other group is more common, sample-size weights would answer the wrong question. Target weights do not make the raw sample larger or change the individual errors; they change the relative contribution of each group to the summary.

Worked Example: Applying Target-Population Weights

Return to the first hypothetical evaluation, where Group A has $MAE_A=2.5$ and $RMSE_A^2=17/2=8.5$, while Group B has $MAE_B=3$ and $RMSE_B^2=18$. Suppose the intended target mix is 25% Group A and 75% Group B, rather than the observed sample mix of two-thirds Group B.

Target-weighted MAE. The target-weighted value is $$ MAE_{\text{target}}=(0.25)(2.5)+(0.75)(3)=0.625+2.25=2.875\text{ units}. $$ This is above the sample-pooled MAE of $2.833$ units because the target mix gives more weight to Group B, whose MAE is higher.

Target-weighted RMSE. First combine the group mean squared errors: $$ MSE_{\text{target}}=(0.25)(8.5)+(0.75)(18)=2.125+13.5=15.625\text{ units}^2. $$ Then take the square root: $$ RMSE_{\text{target}}=\sqrt{15.625}\approx3.953\text{ units}. $$ The result is not $(0.25)RMSE_A+(0.75)RMSE_B$, because that averages square roots rather than mean squared errors. The target-weighted result describes the specified target mix, not the observed sample mix.

Common Mistakes and Reporting Choices

Calling an equal-group average “the pooled metric.” With unequal group sizes, an equal-group average usually does not equal the metric computed over all cases. State which one is reported.
Averaging group RMSE values to obtain pooled RMSE. This is generally wrong. Combine group MSE values using the weights, then take the square root.
Treating sample-size weighting as universally fair. It is fair to each observed case when cases represent the target mix. It may not reflect a different target population or a goal of comparing groups equally.
Reporting weights without their meaning. Say whether weights are group case shares, target-population shares, or another prespecified importance scheme. The interpretation depends on that choice.
Assuming an overall value describes every group. A single summary can hide markedly different group errors. When relevant, show group metrics alongside the overall value so readers can see how the summary was formed.

Weighting is not merely a technical adjustment: it defines what “overall performance” means. Case-count weighting summarizes errors across cases in the evaluated mix. Equal-group weighting summarizes group-level performance with each group contributing equally. Target weighting summarizes performance under a specified group mix. None of these changes the predictions or the observed errors; each changes which cases dominate the summary.

Key takeaway: For MAE, a weighted group average combines group MAEs. For RMSE, combine weighted group MSEs and then take the square root. Always identify the groups, weights, and target represented by the reported metric.

Check Your Understanding

Use the definitions and weighting distinctions above to answer these questions.

Two groups have sizes $n_1=3$ and $n_2=9$, with MAEs of $4$ and $2$ outcome units. Calculate the equal-group MAE and the pooled MAE.
Why does a case-count-weighted average of group RMSE values generally not equal the pooled RMSE? State the correct calculation.
A small group has much higher errors than a large group. How could the pooled MAE and equal-group MAE differ in their representation of those groups?
What does a target-weighted metric describe, and what condition must the group weights satisfy?
When reporting an overall prediction metric, what information about the weighting should be included so that its interpretation is clear?