What Aggregate Performance Metrics Conceal

Why an Overall Score Is Not the Whole Story

A single performance number is useful: it gives a compact summary that can be compared across models or evaluation periods. But the summary combines errors across cases. A low overall error does not establish that errors are small for every type of case, and a change in the overall score does not, by itself, explain what changed.

For an actuarial prediction, the distinction can matter even when the metric is calculated correctly. A model may perform well for the many ordinary cases that dominate the evaluation data but poorly for a small group of large predicted values. Or the average error may change because the evaluation set contains a different mix of cases, even though performance within each type of case is unchanged. The metric describes the cases included and the way their errors were aggregated; interpreting it requires that context.

Definition: For observed outcomes $y_i$ and corresponding predictions $\hat{y}_i$, define the residual $e_i=y_i-\hat{y}_i$. On the same set of $n$ evaluated cases, the mean absolute error is $\operatorname{MAE}=\frac{1}{n}\sum_{i=1}^n |e_i|$, the mean squared error is $\operatorname{MSE}=\frac{1}{n}\sum_{i=1}^n e_i^2$, and the root mean squared error is $\operatorname{RMSE}=\sqrt{\operatorname{MSE}}$. MAE and RMSE are nonnegative and are zero only when every evaluated prediction is exact. RMSE is in the outcome's units; MSE is in squared units.

These definitions describe empirical performance on a specified evaluation set. They do not, on their own, establish how the model will perform on future cases. They also do not identify which groups or prediction ranges have large errors, whether errors tend to be over- or underpredictions, or why any observed difference arose. Those questions require additional context and appropriate breakdowns.

How Pooled Metrics Weight Cases

A pooled metric combines all the evaluation cases. If the cases are divided into nonoverlapping groups, the pooled MSE is the weighted average of the group MSEs, with group sizes determining the weights. This makes it possible for a small group with large errors to contribute substantially to the total squared error, while remaining a small fraction of the cases.

\operatorname{MSE}_{\text{pooled}} =\frac{\sum_g n_g\operatorname{MSE}_g}{\sum_g n_g} =\sum_g w_g\operatorname{MSE}_g, \qquad w_g=\frac{n_g}{\sum_h n_h}.

Here $n_g$ is the number of cases in group $g$, and $\operatorname{MSE}_g$ is the mean squared error among those cases. The weights $w_g$ are nonnegative and sum to one. Pooled RMSE is the square root of pooled MSE, not the weighted average of group RMSEs. Pooled MAE is likewise the size-weighted average of group MAEs, because absolute errors are averaged directly.

Worked Example: A Small Group Drives Most of the Squared Error

In a hypothetical validation set of 100 insurance claims, 90 ordinary claims each have an absolute prediction error of $1, while 10 unusually large claims each have an absolute prediction error of $5. Calculate the pooled MAE and RMSE, then determine how much of the total squared error comes from the 10 large claims.

Method. Calculate the absolute-error and squared-error totals for each group, combine the totals, and divide by all 100 cases. This avoids the mistake of treating a group's percentage of cases as its percentage of error.

For the ordinary claims, the absolute-error total is $90(1)=90$, and the squared-error total is $90(1^2)=90$. For the 10 large claims, those totals are $10(5)=50$ and $10(5^2)=250$.

The pooled absolute-error total is $90+50=140$, so the pooled MAE is $140/100=\$1.40$. The pooled squared-error total is $90+250=340$, so the pooled MSE is $340/100=3.4$ dollars squared. Therefore, the pooled RMSE is $\sqrt{3.4}=\$1.844$, rounded to three decimal places.

The 10 large claims are 10% of the cases but contribute $250/340=0.7353$, or 73.53%, of the total squared error. The aggregate RMSE is mathematically correct, but it conceals that concentration unless the group contribution is also reported.

Check. The group RMSEs are $1 and $5. Weighting their squared values gives $0.9(1^2)+0.1(5^2)=3.4$, which agrees with the pooled MSE. Averaging the group RMSEs instead would give $0.9(1)+0.1(5)=1.4$, which is not the pooled RMSE.

Why Residual Direction Can Also Disappear

MSE and RMSE square residuals, so they do not preserve whether predictions are too high or too low. Even an overall mean residual can conceal opposing errors: positive residuals in one group can offset negative residuals in another. The following decomposition makes one part of this issue precise.

Result: For a group $g$, let $\bar e_g$ be its mean residual and let $s_g^2=\frac{1}{n_g}\sum_{i\in g}(e_i-\bar e_g)^2$ be its residual variance using divisor $n_g$. Then the pooled MSE is the weighted average of each group's residual variance plus its squared mean residual: $\operatorname{MSE}_{\text{pooled}}=\sum_g w_g(s_g^2+\bar e_g^2)$.

To justify the result, write each residual as $e_i=(e_i-\bar e_g)+\bar e_g$ for a case in group $g$, and square it. Summing within that group gives

\frac{1}{n_g}\sum_{i\in g}e_i^2 = \frac{1}{n_g}\sum_{i\in g}(e_i-\bar e_g)^2 +2\bar e_g\frac{1}{n_g}\sum_{i\in g}(e_i-\bar e_g) +\bar e_g^2 =s_g^2+\bar e_g^2.

The middle term is zero because $\sum_{i\in g}(e_i-\bar e_g)=0$ by the definition of the group mean. Weighting the group identities by $w_g=n_g/n$ gives the pooled identity. This shows that pooled squared error reflects both residual variation within groups and systematic average residuals within groups. Opposing group means can cancel in the overall mean residual, but their squared contributions do not cancel in MSE.

Worked Example: Opposing Errors Produce a Zero Overall Mean

A hypothetical prediction system is evaluated on 40 cases: 20 cases from one type of policy have residual $+\$3$ each, and 20 cases from another type have residual $-\$3$ each. Calculate the overall mean residual and RMSE, and explain what the mean residual misses.

Method. Sum the signed residuals to calculate mean residual, then square the residuals to calculate RMSE. This avoids interpreting a zero signed average as evidence of zero prediction error.

The signed residual total is $20(3)+20(-3)=60-60=0$. Dividing by 40 gives an overall mean residual of $0. The positive residuals mean actual outcomes exceed predictions by $3; the negative residuals mean predictions exceed actual outcomes by $3.

Every squared residual is $3^2=9$, so the total squared error is $40(9)=360$. The MSE is $360/40=9$ dollars squared, and the RMSE is $\sqrt{9}=\$3$. The overall mean residual indicates no net signed error, but it conceals two groups with opposite average errors of $3 in magnitude.

Check. The overall MSE is also $0.5(9+9)=9$, consistent with the direct calculation. A zero mean residual does not imply that individual predictions are accurate.

Changing Case Mix Can Change the Aggregate

The weights in a pooled metric depend on the evaluation cases. If the proportions of different case types change, the aggregate can change even when each type's performance remains constant. Conversely, improvement in one group can be offset by a larger share of difficult cases, leaving an aggregate score nearly unchanged. An overall comparison across periods therefore combines two questions: did performance within comparable types of cases change, and did the composition of the evaluated cases change?

Worked Example: A Higher Error Caused by a Different Mix

A hypothetical forecasting model is evaluated in two periods. In each period, the absolute error is consistently $2 for routine cases and $8 for complex cases. Period 1 contains 90 routine and 10 complex cases; Period 2 contains 50 of each. Calculate the MAE and RMSE for each period and decide whether these aggregates show that performance within either case type deteriorated.

Method. Compute each period's error totals using that period's case counts. This avoids comparing pooled scores as if they had identical case composition.

In Period 1, the absolute-error total is $90(2)+10(8)=180+80=260$. With 100 cases, MAE is $260/100=\$2.60$. The squared-error total is $90(2^2)+10(8^2)=360+640=1{,}000$, so MSE is $1{,}000/100=10$ and RMSE is $\sqrt{10}=\$3.162$, rounded to three decimal places.

In Period 2, the absolute-error total is $50(2)+50(8)=100+400=500$, giving MAE $500/100=\$5.00$. The squared-error total is $50(4)+50(64)=200+3{,}200=3{,}400$, giving MSE $3{,}400/100=34$ and RMSE $\sqrt{34}=\$5.831$, rounded to three decimal places.

Both pooled metrics are higher in Period 2, but the stated error for each case type is unchanged: $2 for routine cases and $8 for complex cases in both periods. The example therefore demonstrates a composition effect, not evidence of within-type deterioration. In practice, aggregate scores alone cannot establish which explanation applies; comparable group summaries and information about the evaluation populations are needed.

Check. The complex-case share rises from $10/100=10\%$ to $50/100=50\%$. Since complex cases have larger errors in both periods, giving them more weight raises both pooled measures.

Errors in Important Prediction Ranges

A metric gives the same formal weight to every case's error, but the cases need not be equally consequential for the decision being supported. For example, predictions at the high end of a claim-cost range may be important to review even if such cases are rare. A pooled score does not show where in the prediction range its errors occur. Comparing the overall metric with clearly defined range-specific summaries can reveal whether a small number of high predictions contributes disproportionately to the error.

Worked Example: Rare High Predictions Dominate Squared Error

In a hypothetical set of 100 claim-cost predictions, 95 cases have absolute error $1. The remaining five cases are all in a prespecified high-prediction range and each has residual $-\$10$, meaning the prediction exceeds the observed outcome by $10. Calculate overall MAE and RMSE, then quantify the high-range contribution to total squared error.

Method. Sum absolute errors and squared errors separately, then isolate the high-range terms. This avoids relying on the overall RMSE to describe where or in which direction the errors occur.

The absolute-error total is $95(1)+5(10)=95+50=145$, so overall MAE is $145/100=\$1.45$. The squared-error total is $95(1^2)+5(10^2)=95+500=595$. Thus MSE is $595/100=5.95$ dollars squared and RMSE is $\sqrt{5.95}=\$2.439$, rounded to three decimal places.

The five high-range cases are 5% of the evaluation set but account for $500/595=0.8403$, or 84.03%, of total squared error. Their residuals are all negative, so the predictions in this range are systematically higher than the observed outcomes in this example. The overall RMSE reports the combined magnitude; it does not reveal that range or direction.

Check. The high-range MAE is $10, while the other 95 cases have MAE $1. Combining them gives $0.05(10)+0.95(1)=1.45$, agreeing with the overall MAE. A range-specific result is still a summary: it does not explain the cause of the errors or establish future performance.

How to Report an Aggregate Metric Responsibly

An aggregate measure is not misleading simply because it is aggregate. It is misleading when it is treated as a complete description despite not answering the question at hand. A compact report can preserve the usefulness of the overall score while making its scope and limits explicit.

Define the evaluation set.
State the outcome, prediction target, evaluation period, number of cases, and any important inclusion conditions. A metric without its evaluation population has limited interpretation.

Name the metric and its units.
Distinguish MAE, MSE, and RMSE. For example, report RMSE in dollars, not as though it were an average signed error or an error guaranteed for each case.

Check how errors are distributed.
Compare summaries for relevant case types or prediction ranges, and report whether a small number of cases contributes a large share of the aggregate error. Choose breakdowns that are justified by the application and data.

Interpret comparisons with case mix in view.
When comparing periods or populations, establish whether their composition is comparable. If it is not, describe the aggregate difference without attributing it solely to changed model performance.

These checks do not replace a broader assessment. In particular, MAE and RMSE measure error magnitude, not every aspect of predictive adequacy, and a small evaluation group may provide an unstable estimate. Explain what the selected metric supports, what it does not establish, and which material concentrations or composition changes were examined. This makes an aggregate result useful without presenting it as a guarantee of uniformly good performance.

Key takeaway: A pooled score is a weighted summary of the evaluated cases, not a description of every case or group. Report it with its evaluation context, and examine whether error concentration, case-mix changes, or important prediction ranges alter its interpretation.

Check Your Understanding

For each problem, calculate the requested aggregate and interpret what the summary can and cannot establish.

A hypothetical test set has 32 routine cases with RMSE $2 and 8 complex cases with RMSE $6. Calculate pooled MSE and RMSE, and find the percentage of total squared error from complex cases. (Answer: pooled MSE $10.4$ dollars squared; pooled RMSE $\$3.225$ to three decimals; complex cases contribute $288/416=69.23\%$ of squared error.)
A model has MAE $3 for standard cases and $9 for specialist cases in two periods. Period A has 80 standard and 20 specialist cases; Period B has 40 of each. Calculate both pooled MAEs and state whether the increase alone demonstrates within-group deterioration. (Answer: Period A MAE $\$4.20$; Period B MAE $\$6.00$; no, the given within-group MAEs are unchanged and the case mix changed.)
There are 30 residuals: 15 equal $+\$4$ and 15 equal $-\$4$. Calculate mean residual, MAE, and RMSE. What is concealed by the mean residual? (Answer: mean residual $\$0$, MAE $\$4$, RMSE $\$4$; the signed mean conceals equal-sized errors in opposite directions.)
In a 200-case evaluation, 190 cases have absolute error $2 and 10 high-range cases have absolute error $12. Calculate overall MAE and the share of total squared error contributed by the high-range cases. (Answer: MAE $\$2.50$; high-range squared-error share $1{,}440/(760+1{,}440)=65.45\%$.)
Two periods have equal numbers of cases. In each period, a low-prediction group has RMSE $1 and a high-prediction group has RMSE $5. The high-prediction group's share changes from 20% to 60%. Calculate the two pooled RMSEs and state whether the aggregate comparison alone establishes deterioration within either range. (Answer: first RMSE $\sqrt{0.8(1)+0.2(25)}=\sqrt{5.8}=\$2.408$; second RMSE $\sqrt{0.4(1)+0.6(25)}=\sqrt{15.4}=\$3.924$, both rounded to three decimals; no, the aggregate comparison alone does not establish within-range deterioration.)

What You'll Learn