Integrated Error Analysis and Metric Selection

Why Integrated Error Analysis Matters

A model can have a low average error and still be unreliable for a particular purpose. Positive and negative errors can cancel in a signed average; a large group can dominate a pooled metric; and a few large misses can have a disproportionate effect on squared-error summaries. Numeric prediction therefore calls for more than one calculation: select metrics to match the task, compare important subgroups, and inspect how errors vary across predictions and observations.

This tutorial integrates those checks in a hypothetical claim-cost prediction setting. The calculations are aligned with the SOA Exam PA (Predictive Analytics) syllabus. They assess predictive performance, not whether a relationship is causal or whether a model is suitable for every business decision.

Definition: For observation $i$, let $y_i$ be the observed numeric outcome and $\hat{y}_i$ its prediction. Define the residual as $e_i=y_i-\hat{y}_i$; a positive residual means the prediction was too low. The absolute error is $|e_i|$, and the squared error is $e_i^2$. Unless weights are explicitly introduced, each of the $n$ observations receives equal weight.

Select Metrics for the Prediction Task

Three summaries answer different questions. Mean absolute error (MAE) gives the average miss in the outcome's units. Root mean squared error (RMSE) is also in those units, but penalizes large errors more heavily. Mean signed error describes average direction: with the residual convention above, a positive value indicates average underprediction. A signed average near zero does not establish that individual predictions are accurate.

\operatorname{MAE}=\frac{1}{n}\sum_{i=1}^{n}|e_i|,\qquad \operatorname{RMSE}=\sqrt{\frac{1}{n}\sum_{i=1}^{n}e_i^2},\qquad \bar e=\frac{1}{n}\sum_{i=1}^{n}e_i

MAE is often a useful summary when a typical dollar miss matters and the impact of an error grows roughly in proportion to its size. RMSE is more sensitive to extreme misses, so it can be more informative when very large errors are especially costly. Neither metric is universally preferable: the choice should reflect how predictions will be assessed or used. It is often useful to report both, alongside signed error, rather than letting one number stand in for every aspect of performance.

Justification: Squaring gives a large miss more influence than a small one: an error of 40 contributes $1{,}600$, while an error of 10 contributes $100$, four times as much despite being only four times larger. RMSE takes the square root of mean squared error to return the result to the outcome's units; it does not remove that greater influence. Also, if $e_i$ are residuals and $\bar e$ is their mean, expansion of the square gives

\frac{1}{n}\sum_{i=1}^{n}e_i^2 = \frac{1}{n}\sum_{i=1}^{n}(e_i-\bar e)^2+\bar e^2.

The cross-product term vanishes because $\sum_i(e_i-\bar e)=0$. Thus mean squared error combines variation in residuals around their mean with squared average bias. A zero mean residual removes the bias term, but the remaining variation can still be substantial.

Worked Example: Selecting a Metric for Claim-Cost Predictions

A hypothetical insurer compares two models on four claims. Observed costs are $100, $100, $300, and $300. Model A predicts $120, $80, $240, and $360; Model B predicts $100, $100, $260, and $340. Calculate MAE, RMSE, and mean residual for each model. Then identify which model performs better under each summary.

Method. Compute residuals as observed minus predicted. Then average absolute residuals, average squared residuals before taking the square root, and average the signed residuals. This avoids confusing error direction with error size.

For Model A, the residuals are $-20,\ 20,\ 60,\ -60$. Their absolute values sum to $20+20+60+60=160$, so $\operatorname{MAE}=160/4=\$40$. Their squared values sum to $400+400+3{,}600+3{,}600=8{,}000$. Thus $\operatorname{RMSE}=\sqrt{8{,}000/4}=\sqrt{2{,}000}=\$44.72$, rounded to the nearest cent. The signed residuals sum to zero, so $\bar e=\$0$.

For Model B, the residuals are $0,\ 0,\ 40,\ -40$. The absolute values sum to 80, giving $\operatorname{MAE}=80/4=\$20$. The squared values sum to $0+0+1{,}600+1{,}600=3{,}200$, giving $\operatorname{RMSE}=\sqrt{3{,}200/4}=\sqrt{800}=\$28.28$. The signed residuals again sum to zero, so $\bar e=\$0$.

Model B has lower MAE and RMSE on these four observations; both models have zero mean residual. The check is that the units of MAE and RMSE are dollars, while the signed mean says only that the errors balance in direction. Common mistake avoided: declaring Model A unbiased because its mean residual is zero, then assuming its predictions are accurate. Its errors still average $40 in absolute size.

Compare Overall and Subgroup Errors

An overall metric describes the evaluated observations under a particular weighting scheme. In a pooled MAE, each observation contributes equally, so groups with more observations have more influence. An equal-subgroup average instead gives each subgroup the same influence regardless of its size. These answer different questions: expected error for a randomly selected observation from the combined data versus the average of subgroup-level errors when groups are intentionally treated equally.

Subgroup comparisons should use groups that are relevant to the prediction task and supported by adequate data. Small groups can have unstable estimates. When error direction matters, compare signed residuals as well as absolute or squared errors: a group may be systematically underpredicted even if the overall signed error is close to zero.

Worked Example: A Pooled Metric Hides a Group Difference

In a hypothetical validation sample, subgroup A has four claims, each with a prediction that is $10 above its observed cost. Subgroup B has two claims, each with a prediction that is $60 below observed cost. Calculate subgroup MAE and mean residual, pooled MAE, equal-subgroup MAE, pooled RMSE, and pooled mean residual.

Method. Use $e=y-\hat y$, so subgroup A has residuals $-10$ on all four claims and subgroup B has residuals $+60$ on both. Calculate pooled summaries from all six observations, not by averaging subgroup summaries unless the requested summary gives groups equal weight.

Subgroup A has MAE $(4\times \$10)/4=\$10$ and mean residual $-\$10$. Subgroup B has MAE $(2\times \$60)/2=\$60$ and mean residual $+\$60$. The pooled absolute error sum is $4\times10+2\times60=160$, so pooled MAE is $\$160/6=\$26.67$, rounded to the nearest cent. The equal-subgroup MAE is $(\$10+\$60)/2=\$35$.

The pooled squared error sum is $4\times10^2+2\times60^2=400+7{,}200=7{,}600$. Therefore pooled RMSE is $\sqrt{7{,}600/6}=\sqrt{1{,}266.6667}=\$35.59$, rounded to the nearest cent. The pooled mean residual is $(4\times-10+2\times60)/6=80/6=\$13.33$, indicating average underprediction overall.

The pooled MAE is lower than the equal-subgroup MAE because the lower-error subgroup is larger. The subgroup figures reveal the much larger errors and underprediction in B. Common mistake avoided: treating the equal average of subgroup MAEs as the pooled MAE; the two weight groups differently.

Read Residual and Calibration Patterns Together

A residual-versus-prediction plot places predictions on the horizontal axis and residuals on the vertical axis. A horizontal reference at zero helps identify direction. Residuals scattered around zero without a systematic pattern are more reassuring than a trend, curve, changing spread, or cluster on one side of zero. A residual plot is a diagnostic, not a proof that a model is correct; its interpretation depends on the observations and prediction task.

A numeric calibration comparison asks whether observed outcomes agree with predictions, either overall or within prediction ranges or relevant subgroups. In a binned calibration table or plot, compare the mean observed outcome with the mean prediction for each bin. Close agreement in the overall mean alone is insufficient: overprediction in one range can cancel underprediction in another. Binning can also conceal variation within bins, especially when bins are wide or contain few observations.

Worked Example: Overall Agreement Conceals a Calibration Pattern

For six hypothetical claims, predictions are $90, $110, $190, $210, $290, and $310. Corresponding observed costs are $120, $140, $190, $210, $260, and $280. Pair claims into three prediction bands: the first two, middle two, and last two. Calculate the mean prediction and mean observed cost in each band, then calculate overall MAE, RMSE, and mean residual. Describe the diagnostic pattern.

Method. Calculate residual as observed minus predicted for each claim. For each band, compare average observed with average predicted; then calculate overall metrics from all six residuals. This checks whether agreement in the overall average also holds across prediction levels.

The residuals are $30,\ 30,\ 0,\ 0,\ -30,\ -30$. In the first band, mean prediction is $(90+110)/2=\$100$, while mean observed cost is $(120+140)/2=\$130$. The model underpredicts that band by $30 on average. In the middle band, both means are $(190+210)/2=\$200$, so the band-level difference is zero. In the last band, mean prediction is $(290+310)/2=\$300$, while mean observed cost is $(260+280)/2=\$270$; the model overpredicts by $30 on average.

The absolute residual sum is $30+30+0+0+30+30=120$, giving MAE $=120/6=\$20$. The squared residual sum is $4\times900=3{,}600$, so RMSE $=\sqrt{3{,}600/6}=\sqrt{600}=\$24.49$, rounded to the nearest cent. The signed residual sum is $30+30-30-30=0$, so mean residual is $0. Overall mean prediction and observed outcome are both $200, but low predictions are too low and high predictions are too high.

A residual-versus-prediction plot would show positive residuals at the low end, values near zero in the middle, and negative residuals at the high end. A binned calibration plot would show the observed means above predictions in the first band, equal in the middle, and below predictions in the last. Common mistake avoided: treating zero overall bias as proof of calibration throughout the prediction range.

A Practical Integrated Review

For a numeric prediction assessment, a disciplined sequence helps prevent a headline metric from controlling the conclusion:

Define the error and evaluation set.
State which value is observed, which is predicted, and the residual sign convention. Check that the observations and outcome units match the intended prediction task.

Select complementary summaries.
Use MAE for average absolute miss, RMSE when larger misses deserve more influence, and signed mean residual to assess average direction. Explain what each does and does not show.

Compare consequential subgroups.
Show group-level errors and clarify whether the overall figure weights observations or groups equally. Interpret group direction as well as magnitude.

Inspect patterns, not just totals.
Check residuals across prediction levels and groups, then compare observed and predicted averages in useful calibration ranges. Note that sample size and binning affect how much can be inferred.

State the limits of the evidence.
A validation result summarizes performance on the evaluated data. It does not establish that future performance will be identical, nor does predictive accuracy alone establish fairness, causality, or suitability for an action.

A common exam trap is to report only the metric that makes a model look best. A low MAE can coexist with important extreme errors; a low RMSE overall can conceal poor performance in a small subgroup; and near-zero signed error can result from cancellation. Another trap is comparing metrics computed on different observations or with different weighting rules as if they were directly comparable. A sound conclusion names the metric, its weighting, the relevant subgroup or prediction range, and the main diagnostic limitation.

Key takeaway: Choose metrics to match the error consequences, report direction as well as magnitude, and examine subgroup and prediction-level patterns. No single overall score establishes that a numeric prediction model performs adequately in every relevant part of the data.

Check Your Understanding

For each problem, use residual $e=y-\hat y$ unless the question asks specifically for prediction minus observed outcome.

Observed values are 40, 60, and 100; predictions are 50, 40, and 110. Calculate MAE, RMSE, and mean residual. (Answer: MAE = 13.33; RMSE = 14.14; mean residual = 0.00.)
Subgroup X has two absolute errors, 5 and 15; subgroup Y has three absolute errors, each 20. Calculate pooled MAE and equal-subgroup MAE. (Answer: pooled MAE = 16; equal-subgroup MAE = 15.)
Two predictions are 80 and 120, and both observed values are 100. Calculate mean residual, MAE, and RMSE. What does the zero mean residual fail to show? (Answer: mean residual = 0; MAE = 20; RMSE = 20; the signed errors cancel despite two nonzero errors.)
A validation goal assigns much greater concern to occasional very large misses than to several small misses. Which should generally receive more attention, MAE or RMSE, and why? (Answer: RMSE, because squaring gives large errors disproportionately greater influence.)
In three prediction bands, mean predicted values are 50, 100, and 150, while mean observed values are 70, 100, and 130. What pattern should appear in residuals defined as observed minus predicted, and why is the overall mean difference insufficient? (Answer: positive residuals in the low band, about zero in the middle, and negative residuals in the high band; the band differences cancel overall and hide calibration differences across prediction levels.)

What You'll Learn