Why Regression Metrics Can Disagree
Two regression models can make predictions for the same validation cases and still receive different rankings under different error metrics. One model may be close on most cases but miss a rare, large outcome; another may make moderate errors more consistently. A metric that emphasizes large errors can prefer the second model, while a metric that summarizes typical absolute error can prefer the first.
The comparison is meaningful only when the models are evaluated on the same cases, against the same observed response, and on the same response scale. If cases have unequal importance for the intended use, the evaluation must also specify whether and how those cases are weighted. These choices define what “better prediction” means; the metrics do not make that decision automatically. This comparison is aligned with the SOA Exam PA (Predictive Analytics) syllabus.
Mean error describes average signed bias, not the typical size of an error. Overpredictions and underpredictions can cancel. MAE and RMSE are nonnegative summaries of error magnitude, expressed in the same units as the response. MAE gives each absolute error a linear contribution. RMSE squares errors before averaging and then takes a square root, so large errors have greater influence. Neither measure, by itself, says whether an error is operationally acceptable.
What Each Summary Emphasizes
MAE is often useful when a dollar of error has roughly the same importance regardless of its size. RMSE may be more appropriate when unusually large misses deserve extra concern. That extra sensitivity is not automatically a virtue: if a single extreme observation is a data error or is irrelevant to the intended prediction task, RMSE may overemphasize it. The analyst should investigate why errors differ rather than treating one metric as universally correct.
The median absolute error is the median of \(|e_1|,\ldots,|e_n|\). It describes a central error size and is less affected by an extreme error than MAE or RMSE. Mean error complements these summaries by indicating direction. For example, a model can have mean error near zero because positive and negative errors cancel while still having substantial MAE and RMSE.
A relative summary such as mean absolute percentage error can be hard to interpret when observed responses are zero or close to zero: division by a zero response is undefined, and division by a very small response can produce a huge percentage. Relative metrics also answer a different question from errors measured in the target’s original units. Use them only when their denominator and interpretation fit the data and decision.
Why RMSE Is at Least MAE
Write \(a_j=|e_j|\), so each \(a_j\) is nonnegative. The average squared deviation of these absolute errors from their mean cannot be negative:
Therefore, the mean of the squared absolute errors is at least the square of their mean. Taking nonnegative square roots gives \(\sqrt{\frac{1}{n}\sum a_j^2}\geq\bar{a}\), which is exactly RMSE \(\geq\) MAE. Equality holds when every \(a_j\) equals \(\bar{a}\), meaning all absolute errors are identical. This derivation explains the relationship between the metrics for one model, but gives no general ranking between two models: each model has its own distribution of errors.
Worked Example: A Single Large Miss Reverses the Ranking
A hypothetical insurer compares two models on the same four validation policies. In thousands of dollars, Model A has signed errors \(0,0,0,8\), while Model B has signed errors \(3,3,3,3\). Calculate mean error, MAE, and RMSE for each model, and identify the metric-dependent ranking.
Method. Calculate each summary directly from the four errors. This avoids treating the error on the largest claim as if it had the same effect under every metric.
For Model A, the error sum is \(0+0+0+8=8\), so mean error is \(8/4=2\). Its absolute-error sum is \(8\), giving MAE \(=8/4=2\). The squared-error sum is \(0+0+0+64=64\), giving RMSE \(=\sqrt{64/4}=\sqrt{16}=4\).
For Model B, the error sum is \(3+3+3+3=12\), so mean error is \(12/4=3\). Its absolute-error sum is also \(12\), giving MAE \(=12/4=3\). The squared-error sum is \(9+9+9+9=36\), giving RMSE \(=\sqrt{36/4}=\sqrt{9}=3\).
Model A has the lower MAE (\(2\) versus \(3\)), while Model B has the lower RMSE (\(3\) versus \(4\)). Model A is closer on three cases but has one larger miss. Mean error indicates that both models underpredict on average in this sample; it is not a substitute for either error-magnitude summary. All amounts are in thousands of dollars. The calculations are exact, so no rounding is needed.
Check. Each RMSE is at least its model’s MAE: \(4\geq2\) for A and \(3\geq3\) for B. The equality for B occurs because all four absolute errors are identical. The common mistake this example avoids is declaring one model the universal winner from a single metric without saying what kind of error the metric emphasizes.
Interpreting Skewed Errors and Robust Summaries
When the response has a long right tail, some validation cases may be much larger than the rest. A model can fit common outcomes closely while making a substantial error on a rare large outcome. The resulting error distribution can also be skewed, but a skewed response does not by itself determine which model wins. The relevant calculations are the errors on the evaluation cases and how the chosen metric summarizes them.
Median absolute error adds useful context when a few large misses could dominate the mean-based summaries. It should not be used to dismiss those misses: a rare, high-cost underprediction may be the most important error for the application. Report the metric that matches the objective and, when useful, another summary that reveals a different feature of the error distribution.
Worked Example: Typical Error Versus Tail-Sensitive Error
For a hypothetical portfolio, two models predict claim amounts measured in thousands of dollars. On the same five validation claims, Model A’s signed errors are \(0,1,1,2,11\), and Model B’s are \(3.5,3.5,3.5,3.5,3.5\). Calculate MAE, RMSE, and median absolute error. Explain why the summaries differ.
Method. Since each listed error is nonnegative, its absolute error is the same number. Calculate the arithmetic mean of absolute errors, the square root of the mean squared errors, and the median of the sorted absolute errors.
For Model A, the absolute-error sum is \(0+1+1+2+11=15\), so MAE \(=15/5=3\). The squared-error sum is \(0+1+1+4+121=127\), so RMSE \(=\sqrt{127/5}=\sqrt{25.4}\approx5.040\). The sorted absolute errors are \(0,1,1,2,11\), so the median absolute error is \(1\).
For Model B, the absolute-error sum is \(5(3.5)=17.5\), so MAE \(=17.5/5=3.5\). The squared-error sum is \(5(3.5^2)=5(12.25)=61.25\), so RMSE \(=\sqrt{61.25/5}=\sqrt{12.25}=3.5\). Its five absolute errors are all \(3.5\), so its median absolute error is \(3.5\).
Model A has lower MAE (\(3\) versus \(3.5\)) and lower median absolute error (\(1\) versus \(3.5\)); Model B has lower RMSE (\(3.5\) versus approximately \(5.040\)). Model A is more accurate on a typical case, but its one error of \(11\) has a strong effect on RMSE. RMSE is rounded to three decimal places; all other results are exact. The common mistake this example avoids is reporting the median as if it described the tail: Model A’s median absolute error is \(1\), but its largest error is \(11\).
Weights and Fair Comparisons
If validation cases have unequal weights, first state what the weights represent. For example, a weight may represent exposure or the frequency with which a case type occurs in the intended population. For nonnegative weights \(w_j\) with positive total weight, weighted MAE and weighted RMSE are:
These summaries give more influence to cases with larger weights. A weighted and an unweighted comparison can therefore rank models differently. Use the same justified weights for every candidate model; choosing weights after seeing which model they favor makes the comparison difficult to defend.
Worked Example: Exposure Weights Change the Preferred Model
A hypothetical validation set has three case types with exposure weights \(1,1,4\). Model P has errors \(0,0,5\); Model Q has errors \(2,2,2\). Calculate unweighted and weighted MAE and RMSE. Treat the weights as defining the target population for the weighted comparison.
Method. First calculate ordinary summaries with each case counted once. Then use the specified weights in both the numerator and denominator of each weighted metric.
For Model P, the unweighted absolute-error sum is \(0+0+5=5\), so unweighted MAE \(=5/3\approx1.667\). Its squared-error sum is \(0+0+25=25\), so unweighted RMSE \(=\sqrt{25/3}\approx2.887\). For Model Q, the absolute-error sum is \(6\) and the squared-error sum is \(12\), so unweighted MAE \(=6/3=2\) and RMSE \(=\sqrt{12/3}=2\). Unweighted MAE favors P, while unweighted RMSE favors Q.
The total weight is \(1+1+4=6\). For P, weighted absolute error is \(1(0)+1(0)+4(5)=20\), so weighted MAE \(=20/6\approx3.333\). Weighted squared error is \(1(0^2)+1(0^2)+4(5^2)=100\), so weighted RMSE \(=\sqrt{100/6}\approx4.082\). For Q, weighted absolute error is \(1(2)+1(2)+4(2)=12\), so weighted MAE \(=12/6=2\). Weighted squared error is \(1(4)+1(4)+4(4)=24\), so weighted RMSE \(=\sqrt{24/6}=2\).
Both weighted metrics favor Q, because the case type on which P has an error of \(5\) receives four times the weight of either other case. Unweighted values are rounded to three decimal places; weighted values are rounded to three decimal places. The common mistake this example avoids is treating a weighted score as an unexplained technical adjustment: its interpretation depends on what the weights represent.
A Reliable Comparison Process
Compare predictions against the same observed responses. Different validation samples can create apparent metric disagreements that are really sample differences.
Specify the response units and whether cases are weighted. Use consistent choices across candidate models.
Use MAE or RMSE for error magnitude, mean error for signed bias, and median absolute error when a robust summary of typical error is informative.
Inspect the error pattern. A small number of large errors can influence RMSE more than MAE, while a consistently moderate error can produce a different ranking.
State which error feature matters for the prediction task and disclose meaningful limitations. Do not describe a metric-dependent result as a universal winner.
Common Exam Traps
- Calling mean error accuracy. Signed errors can cancel. A mean error of zero does not imply zero prediction error.
- Forgetting the square root. Mean squared error is not RMSE. For errors \(0,0,0,8\), the squared-error mean is \(64/4=16\), but RMSE is \(\sqrt{16}=4\), not \(16\).
- Comparing scores from different cases or scales. A lower score is not a fair model comparison if the models were assessed on different validation cases or differently transformed responses without a clear reason.
- Assuming one metric is always best. MAE and RMSE encode different sensitivities to large errors. Explain the trade-off in context.
- Ignoring the meaning of weights. Weighted summaries describe a different evaluation target from unweighted summaries when the weights vary.
For a defensible comparison, report the metric, its units or scale, the validation cases and weighting basis, and the reason that metric is relevant. When rankings conflict, describe the error pattern that produces the conflict rather than hiding one of the results.
Check Your Understanding
For each problem, use the stated error convention \(e=y-\hat{y}\). Show your calculations before checking the final answer.
- On four validation cases, Model R has errors \(0,2,2,2\), while Model S has errors \(1.5,1.5,1.5,1.5\). Calculate each model’s MAE and RMSE. Which model is preferred by each metric? (Answer: R has MAE \(1.5\) and RMSE \(\sqrt{3}\approx1.732\); S has MAE \(1.5\) and RMSE \(1.5\). MAE ties; RMSE favors S.)
- A model has signed errors \(-3,-1,2,2\). Calculate its mean error, MAE, and RMSE. What does the mean error indicate? (Answer: Mean error \(=0\), MAE \(=2\), and RMSE \(=\sqrt{4.5}\approx2.121\); the signed errors cancel, but the model still has nonzero error magnitude.)
- Two models have errors P: \(1,1,6\) and Q: \(3,3,3\), with case weights \(1,1,2\). Which model has lower unweighted MAE, and which has lower weighted MAE? (Answer: Unweighted MAE is \(8/3\approx2.667\) for P and \(3\) for Q, so P is lower. Weighted MAE is \(14/4=3.5\) for P and \(12/4=3\) for Q, so Q is lower.)
- A model’s absolute errors are \(1,1,1,9,10\). Calculate its median absolute error, MAE, and RMSE. Which summary is most affected by the two largest errors? (Answer: Median absolute error \(=1\), MAE \(=22/5=4.4\), and RMSE \(=\sqrt{184/5}=\sqrt{36.8}\approx6.066\); RMSE is most affected by the largest errors.)