Extreme Errors and Their Metric Contributions

Why an Extreme Error Deserves Investigation

A model can have acceptable average performance while making one prediction that is far from the observed outcome. That case may dominate a metric, signal a data problem, represent a legitimate rare event, or expose a serious limitation in the model. The size of the error alone cannot tell you which explanation is right.

This tutorial focuses on tracing individual errors into mean absolute error (MAE) and root mean squared error (RMSE), then investigating their context. The calculations assume a fixed evaluation set of $n$ observations, one prediction per observation, and equal weight for every observation. Predictions and outcomes must use the same units. If the evaluation uses weights, the corresponding weighted metric—not an unweighted calculation—must be used.

Definition: For observation $i$, let $y_i$ be the observed value, $\hat{y}_i$ its prediction, and $e_i=y_i-\hat{y}_i$ the signed prediction error. The absolute error is $|e_i|$, and the squared error is $e_i^2$. For an equally weighted evaluation set, $\operatorname{MAE}=\frac{1}{n}\sum_{i=1}^n |e_i|$, $\operatorname{MSE}=\frac{1}{n}\sum_{i=1}^n e_i^2$, and $\operatorname{RMSE}=\sqrt{\operatorname{MSE}}$.

The sign of $e_i$ indicates direction: a positive error means the observed value exceeds the prediction, while a negative error means the prediction exceeds the observed value. MAE and RMSE do not preserve that direction. They summarize error magnitude, in the original units; MSE is in squared units.

How One Case Enters Each Metric

For MAE, observation $i$ adds $|e_i|/n$ to the average. Its share of the total absolute error is $|e_i|/\sum_j|e_j|$, provided the total is positive. For MSE, it adds $e_i^2/n$; its share of the total squared error is $e_i^2/\sum_j e_j^2$, provided that total is positive. The share is useful for locating what drives a metric, but it does not by itself explain why the error occurred.

RMSE is the square root of MSE. Its individual squared-error inputs are additive before the square root, but RMSE itself is not a sum of per-observation contributions. In particular, it is not correct to say that a case contributes $|e_i|/n$ to RMSE. To see the effect of deleting a case from a calculation, compare the full RMSE with the RMSE on the other $n-1$ observations. This is a diagnostic comparison, not a justification for excluding the case.

\operatorname{RMSE}_{-i} = \sqrt{\frac{\sum_{j=1}^n e_j^2-e_i^2}{n-1}}

This expression follows directly from the definition: removing observation $i$ subtracts its squared error $e_i^2$ from the total and changes the denominator from $n$ to $n-1$. Comparing $\operatorname{RMSE}_{-i}$ with the full RMSE reveals how dependent the reported metric is on that case. It does not estimate what performance would have been on a different population.

Squaring also explains why a very large error can dominate RMSE. If one error has magnitude $k$ times another, its squared-error input is $k^2$ times as large. An error ten times larger therefore contributes one hundred times as much to the sum of squared errors, while its contribution to the sum of absolute errors is only ten times as large.

Worked Example: Measuring the Influence of One Large Error

A hypothetical evaluation of four equipment-repair cost predictions has signed errors, in dollars, of $2$, $-3$, $4$, and $20$. Calculate MAE and RMSE, then quantify the last observation’s contribution. The method is to calculate the error totals first and compare the observation’s shares; this avoids treating RMSE as if it were an additive average of individual root errors.

MAE. The absolute errors are $2$, $3$, $4$, and $20$. Their sum is $2+3+4+20=29$ dollars, so $\operatorname{MAE}=29/4=\$7.25$.

RMSE. The squared errors are $2^2=4$, $(-3)^2=9$, $4^2=16$, and $20^2=400$. Their sum is $4+9+16+400=429$ dollars squared. Thus $\operatorname{MSE}=429/4=107.25$ dollars squared and $\operatorname{RMSE}=\sqrt{107.25}=\$10.36$, rounded to the nearest cent.

Shares attributable to the $20$-dollar error. Its share of absolute error is $20/29=0.6897$, or $68.97\%$. Its share of squared error is $400/429=0.9324$, or $93.24\%$. The case is therefore much more dominant for the squared-error calculation.

Deletion comparison. With that case omitted, the remaining absolute-error sum is $2+3+4=9$, giving MAE $9/3=\$3.00$. The remaining squared-error sum is $4+9+16=29$, giving RMSE $\sqrt{29/3}=\$3.11$, rounded to the nearest cent. These large changes show that both reported metrics are sensitive to the case, especially RMSE. They do not show that it should be removed: first determine whether its record and outcome are valid.

Investigate Before Deciding What the Error Means

A disciplined investigation separates three questions: is the observation correct, is it part of the population the model is intended to predict, and is the resulting prediction failure consequential? A large residual can justify checking these questions; it cannot answer them.

Verify the record.
Check the source value, units, identifiers, data joins, and whether the target was recorded for the right observation and time period. Correct a confirmed error and recalculate on the corrected data. Do not alter a value merely because it is unusual.

Confirm the evaluation scope.
Check that the case meets the stated inclusion rules and that its outcome is defined consistently with the prediction target. A valid case outside the intended scope may require a scope decision, not a quiet deletion.

Assess whether it is a valid rare case.
If the record is sound and the case belongs in scope, consider whether its circumstances are rare but plausible. Such cases can be important precisely because they occur infrequently.

Assess model and decision consequences.
Inspect what information was available for prediction and what action would follow from the error. A large miss that changes an important decision may deserve attention even when it has limited influence on a portfolio average.

Report sensitivity transparently.
Show the metric on the stated evaluation set and, when informative, a clearly labelled sensitivity calculation. Explain any verified correction or scope exclusion; never present a selectively altered score as the original evaluation result.

Worked Example: A Data-Entry Error or a Genuine Prediction Miss?

A hypothetical claim-cost evaluation is recorded in thousands of dollars. The signed errors are $-1$, $2$, $-2$, $3$, and $215$. The last prediction was $25$, while the stored observed outcome was $240$. A source record is checked and confirms that the outcome should have been $24$, not $240$. Calculate the metrics before and after correction. The method is to verify the source first, then replace only the confirmed incorrect value; this avoids labelling a valid large loss an error simply because it has a large residual.

Before correction. The absolute errors sum to $1+2+2+3+215=223$ thousand dollars, so MAE is $223/5=44.60$ thousand dollars. The squared errors sum to $1+4+4+9+215^2=18+46{,}225=46{,}243$ thousand-dollars-squared. Thus MSE is $46{,}243/5=9{,}248.6$ thousand-dollars-squared, and RMSE is $\sqrt{9{,}248.6}=96.17$ thousand dollars, rounded to two decimal places.

After correction. For that observation, the corrected error is $24-25=-1$ thousand dollars. The corrected error list is $-1$, $2$, $-2$, $3$, and $-1$. The absolute errors sum to $1+2+2+3+1=9$, so MAE is $9/5=1.80$ thousand dollars. The squared errors sum to $1+4+4+9+1=19$, so MSE is $19/5=3.8$ thousand-dollars-squared and RMSE is $\sqrt{3.8}=1.95$ thousand dollars, rounded to two decimal places.

The change is dramatic because the stored value was wrong, not because a large but valid outcome was inconvenient. The corrected observation must remain in the evaluation: its corrected error is $-1$ thousand dollars. The correction and its source should be documented alongside the recalculated result.

Worked Example: A Valid Rare Case With a Consequential Miss

In a hypothetical evaluation of five specialized repair events, the signed errors, in thousands of dollars, are $2$, $-1$, $3$, $-2$, and $30$. The last event had an observed cost of $38$ thousand dollars and a prediction of $8$ thousand dollars. Source checks confirm the outcome and prediction inputs. A review is triggered when predicted cost exceeds $20$ thousand dollars. Calculate the overall metrics and assess the last case. The method is to calculate metric contributions and separately check the decision rule; this avoids assuming that metric importance and practical consequence are identical.

Overall metrics. The absolute errors sum to $2+1+3+2+30=38$ thousand dollars, so MAE is $38/5=7.60$ thousand dollars. The squared errors sum to $4+1+9+4+900=918$ thousand-dollars-squared. Hence MSE is $918/5=183.6$ thousand-dollars-squared and RMSE is $\sqrt{183.6}=13.55$ thousand dollars, rounded to two decimal places.

Contribution of the last case. Its share of the absolute-error total is $30/38=0.7895$, or $78.95\%$. Its share of the squared-error total is $900/918=0.9804$, or $98.04\%$. If the case were omitted for a sensitivity calculation, the other four absolute errors sum to $8$, giving MAE $8/4=2.00$ thousand dollars. Their squared errors sum to $18$, giving RMSE $\sqrt{18/4}=\sqrt{4.5}=2.12$ thousand dollars, rounded to two decimal places.

Decision context. The prediction of $8$ thousand dollars does not exceed the $20$-thousand-dollar review threshold, while the observed cost of $38$ thousand dollars does. The model therefore failed to trigger the hypothetical review for this event. Because the record is verified and in scope, the error should not be deleted as an outlier. Its rarity and cause merit investigation, and the missed review should be reported as a consequential limitation even though the overall metrics describe performance across all five cases.

Common Metric Traps

A common mistake is to confuse MSE with RMSE. In the first worked example, $107.25$ is MSE in dollars squared; $\sqrt{107.25}=\$10.36$ is RMSE in dollars. Another mistake is to call $400/429$ the “RMSE share” of the $20$-dollar error. It is the case’s share of the sum of squared errors, and therefore of MSE; RMSE applies a square root to the overall average, so it has no directly additive case shares.

Also avoid removing a large error just to improve the score. A deletion comparison is useful for explaining sensitivity, but an observation belongs in the evaluation if it is valid and in scope under the stated rules. Conversely, keeping a confirmed data defect without correction can make the metric describe a recording process rather than prediction performance. State the data decision and its rationale.

Metric contribution is not the same as decision importance. A case can contribute little to portfolio-wide MAE yet still cause a costly missed action. Conversely, a large residual might have limited operational impact in a particular setting. Interpret the metric alongside the target, intended use, and consequence of error. This analysis is aligned with the SOA Exam PA (Predictive Analytics) syllabus.

Key takeaway: Calculate how much an extreme error contributes, then investigate its validity, scope, and consequence. Use sensitivity comparisons to describe dependence on a case, not to justify excluding it without a defensible reason.

Check Your Understanding

For each problem, calculate the requested metric quantities before interpreting the result.

A three-observation evaluation has errors $1$, $-2$, and $9$, in dollars. Find MAE, RMSE, and the largest error’s shares of the absolute-error and squared-error totals. (Answer: MAE $=\$4.00$; RMSE $=\sqrt{86/3}=\$5.35$ to the nearest cent; shares $=75.00\%$ and $81/86=94.19\%$.)
Four errors are $2$, $2$, $-2$, and $10$, in thousands of dollars. Find MAE and RMSE, and the $10$-thousand-dollar error’s share of the squared-error total. (Answer: MAE $=4.00$ thousand dollars; RMSE $=\sqrt{28}=5.29$ thousand dollars; squared-error share $=100/112=89.29\%$.)
Errors in thousands of dollars are $-1$, $1$, $2$, and $18$. Find full-set MAE and RMSE, then calculate RMSE after omitting the $18$-thousand-dollar case for sensitivity analysis. (Answer: full MAE $=5.50$ thousand dollars; full RMSE $=\sqrt{82.5}=9.08$ thousand dollars; omitted-case RMSE $=\sqrt{2}=1.41$ thousand dollars.)
A verified, in-scope observation has an error of $40$ dollars, and the remaining three errors are $1$, $-2$, and $3$ dollars. What are the full-set MAE and RMSE, and should the $40$-dollar case be removed solely because it is extreme? (Answer: MAE $=46/4=\$11.50$; RMSE $=\sqrt{1614/4}=\$20.09$ to the nearest cent; no, not solely because it is extreme.)
A record has prediction $12$ thousand dollars and stored outcome $120$ thousand dollars. A source check confirms the outcome was $20$ thousand dollars. What is its corrected signed error, and what is the appropriate next step for the evaluation? (Answer: corrected error $=20-12=8$ thousand dollars; document the verified correction, recalculate the metric with this in-scope observation retained, and assess its decision consequence.)

What You'll Learn