Comparing MAE and RMSE

Why Compare MAE and RMSE?

Two models can have the same average absolute error yet make very different kinds of mistakes. One may miss by a similar amount on every case; another may predict almost all cases closely but miss badly on one. The mean absolute error (MAE) and root mean squared error (RMSE) summarize these patterns differently. Comparing them helps explain what a model's errors look like, not just how large one average score is.

Both measures apply to a set of \(n\) observed outcomes and predictions. For observation \(j\), define the prediction error as \(e_j=y_j-\hat{y}_j\), where \(y_j\) is the observed value and \(\hat{y}_j\) is the prediction. The sign indicates direction, but MAE and RMSE use the error's magnitude. A negative error and a positive error of the same magnitude contribute equally.

Definition: For errors \(e_1,\ldots,e_n\) evaluated on the same \(n\) observations, the mean absolute error is \(\operatorname{MAE}=\frac{1}{n}\sum_{j=1}^{n}|e_j|\). The root mean squared error is \(\operatorname{RMSE}=\sqrt{\frac{1}{n}\sum_{j=1}^{n}e_j^2}\). Both are nonnegative and are reported in the units of the outcome.

These are descriptive performance measures, not probability models. They require no assumption that errors follow a normal or other particular distribution. For a fair comparison, calculate both measures on the same observations, use the same outcome definition and units, and apply the same weighting convention. Otherwise, a difference in scores may reflect a difference in the evaluation data or procedure rather than a difference between models.

What the Two Measures Emphasize

MAE first takes the absolute value of each error and then averages. Each additional unit of error adds one unit to its absolute-error contribution. RMSE squares each error before averaging and then takes a square root. Squaring makes large errors count disproportionately more: an error with magnitude 8 contributes 64 to the sum of squared errors, while an error with magnitude 2 contributes 4. The first squared contribution is 16 times the second, although the first absolute-error contribution is only four times the second.

The square root in RMSE returns the result to the outcome's original units. It does not undo the relative emphasis that squaring gave to large errors. Therefore, RMSE is often more sensitive than MAE to a small number of unusually large misses. That sensitivity can be desirable when large misses are especially consequential, but it should not be mistaken for a universal rule that RMSE is the better metric.

Worked Example: Equal MAE, Different RMSE

Two hypothetical models predict the same four claim amounts, measured in thousands of dollars. Model C has errors \(2,2,2,2\); Model D has errors \(0,0,0,8\). Calculate both metrics for each model and interpret the comparison.

Method. Calculate the absolute errors and squared errors separately, then average the relevant values. This avoids treating MAE as though it were an average of squared errors, or vice versa.

For Model C, the absolute errors are \(2,2,2,2\), with sum \(2+2+2+2=8\). Thus \(\operatorname{MAE}_C=8/4=2\). The squared errors are \(4,4,4,4\), with sum \(16\). Thus \(\operatorname{RMSE}_C=\sqrt{16/4}=\sqrt{4}=2\).

For Model D, the absolute errors are \(0,0,0,8\), with sum \(8\). Thus \(\operatorname{MAE}_D=8/4=2\). The squared errors are \(0,0,0,64\), with sum \(64\). Thus \(\operatorname{RMSE}_D=\sqrt{64/4}=\sqrt{16}=4\).

The check is that both models have total absolute error 8, so their MAEs must match. Model D concentrates that error in one large miss, so its RMSE is larger. Both scores are in thousands of dollars. The common mistake is to conclude that the models perform identically because their MAEs match; the RMSE calculation reveals a difference in their error patterns.

Why RMSE Cannot Be Below MAE

For the same observations and equal weights, RMSE is always at least as large as MAE. The result follows from a standard inequality, and does not depend on any assumptions about the errors' distribution.

Result: \(\operatorname{RMSE}\geq\operatorname{MAE}\) for any finite set of errors evaluated with the definitions above. Equality holds exactly when all absolute errors are equal.

To justify the result, let \(a_j=|e_j|\). Each \(a_j\) is nonnegative, and the inequality \((\sum_{j=1}^{n}a_j)^2\leq n\sum_{j=1}^{n}a_j^2\) follows from the Cauchy–Schwarz inequality applied to the vectors \((a_1,\ldots,a_n)\) and \((1,\ldots,1)\). Dividing by \(n^2\) gives \((\frac{1}{n}\sum a_j)^2\leq\frac{1}{n}\sum a_j^2\). Both sides are nonnegative, so taking square roots gives \(\frac{1}{n}\sum a_j\leq\sqrt{\frac{1}{n}\sum a_j^2}\), which is MAE \(\leq\) RMSE. Equality in Cauchy–Schwarz requires the two vectors to be proportional, meaning every \(a_j\) is the same.

This result is a useful arithmetic check. If a calculation using the same observations and equal weights gives RMSE less than MAE, check for an omitted square, an incorrect square root, or an inconsistent denominator. The equality condition also explains why the measures match in the special case where every error has the same magnitude.

Worked Example: Different Rankings for Two Hypothetical Models

A hypothetical insurer evaluates two models on five claim amounts, each stated in thousands of dollars. The observed values are \(10,12,14,16,18\). Model A predicts \(10,12,14,16,14\), and Model B predicts \(9,11,13,15,17\). Use observed minus predicted as the error, calculate both metrics, and state which model each metric favors.

Method. Calculate each model's errors against the same five observed values before computing the metrics. This keeps the comparison on a common evaluation set.

For Model A, the errors are \(10-10=0\), \(12-12=0\), \(14-14=0\), \(16-16=0\), and \(18-14=4\). The absolute-error sum is \(0+0+0+0+4=4\), so \(\operatorname{MAE}_A=4/5=0.8\). The squared-error sum is \(0+0+0+0+16=16\), so \(\operatorname{RMSE}_A=\sqrt{16/5}=\sqrt{3.2}\approx1.789\).

For Model B, the errors are \(10-9=1\), \(12-11=1\), \(14-13=1\), \(16-15=1\), and \(18-17=1\). The absolute-error sum is \(5\), so \(\operatorname{MAE}_B=5/5=1\). The squared-error sum is also \(5\), so \(\operatorname{RMSE}_B=\sqrt{5/5}=1\).

Round the final metric values to three decimal places: Model A has MAE \(0.800\) and RMSE \(1.789\); Model B has MAE \(1.000\) and RMSE \(1.000\), in thousands of dollars. The calculation checks because each model's RMSE is at least its MAE. MAE favors Model A, whose average absolute error is smaller; RMSE favors Model B, which avoids the single large miss. The common mistake is to report one metric as if it settled every comparison. The preferred model depends on how the evaluation values a concentrated large error relative to several modest errors.

Reading the Difference Between the Scores

The gap between RMSE and MAE can help flag that large errors are influencing the squared-error summary. It does not, by itself, identify which observations produced those errors or prove that an error is an outlier. To understand the cause, inspect the individual errors in context. For example, a large miss could reflect a data problem, an unusual but valid outcome, or a part of the population that the model predicts poorly.

MAE and RMSE also answer different practical questions. MAE expresses the average size of an error in the outcome's units. RMSE expresses a squared-error-sensitive summary in those same units. If avoiding occasional large misses is particularly important, RMSE makes those misses more influential in the score. If the desired summary gives each unit of absolute error a constant incremental contribution, MAE follows that rule. Neither metric says which error sizes are acceptable; that judgment depends on the prediction task.

An RMSE-to-MAE comparison is meaningful only when both use the same errors. Do not compare one model's MAE to another model's RMSE as though the two numbers represented the same quantity. Likewise, comparing scores from different outcome scales or different evaluation samples needs care: the numerical values can change for reasons unrelated to the models' relative error patterns.

Common Exam Trap

A frequent error is to calculate \(\sqrt{\text{mean}(|e_j|)}\) and label it RMSE. That is not the RMSE formula. The order matters: RMSE squares each error, averages the squared errors, and then takes the square root. Another mistake is to average the squared errors and stop; that quantity is mean squared error, which is in squared outcome units, not RMSE.

For example, with errors \(1,1,1,5\), the absolute-error sum is \(1+1+1+5=8\), so MAE is \(8/4=2\). The squared-error sum is \(1+1+1+25=28\), so RMSE is \(\sqrt{28/4}=\sqrt{7}\approx2.646\). The incorrect calculation \(\sqrt{\text{MAE}}=\sqrt{2}\approx1.414\) is not RMSE: it takes the square root of the wrong average and does not follow the definition.

Key takeaway: MAE summarizes average absolute error; RMSE gives larger errors extra influence through squaring. For a valid comparison, calculate both on the same observations, check that RMSE is at least MAE, and choose the measure whose penalty pattern fits the evaluation objective.

Worked Example: How One Larger Error Changes Both Scores

A hypothetical forecast team compares two four-period error sequences, measured in thousands of dollars. Sequence P is \(1,1,1,2\), while Sequence Q is \(1,1,1,8\). Calculate both measures and compare how changing the final error affects the summaries.

Method. Hold the first three errors fixed and calculate the contribution of the changed fourth error to each total. This isolates how the two formulas respond to a larger miss.

For P, the absolute-error sum is \(1+1+1+2=5\), so \(\operatorname{MAE}_P=5/4=1.25\). Its squared-error sum is \(1+1+1+4=7\), so \(\operatorname{RMSE}_P=\sqrt{7/4}=\sqrt{1.75}\approx1.323\).

For Q, the absolute-error sum is \(1+1+1+8=11\), so \(\operatorname{MAE}_Q=11/4=2.75\). Its squared-error sum is \(1+1+1+64=67\), so \(\operatorname{RMSE}_Q=\sqrt{67/4}=\sqrt{16.75}\approx4.093\).

The final error increased from 2 to 8, a fourfold increase. Its absolute contribution increased from 2 to 8, also fourfold; its squared contribution increased from 4 to 64, sixteenfold. Accordingly, MAE rose by \(2.75-1.25=1.50\), while RMSE rose by approximately \(4.093-1.323=2.770\). Both are in thousands of dollars, and the rounded values are to three decimal places. The check that RMSE exceeds MAE holds for each sequence. This comparison does not establish that the larger miss is invalid; it shows why RMSE responds more strongly to it.

Check Your Understanding

For each problem, calculate or compare the requested measures using the stated observations and units.

A model has errors \(-2,2,-2,2\). Calculate MAE and RMSE. (Answer: MAE \(=2\), RMSE \(=2\).)
Two models have errors C: \(0,0,0,6\) and D: \(1.5,1.5,1.5,1.5\). Calculate both metrics and identify which model each metric favors. (Answer: C has MAE \(1.5\), RMSE \(3\); D has MAE \(1.5\), RMSE \(1.5\). MAE ties them, while RMSE favors D.)
Errors are \(1,-2,3\), in the outcome's units. Calculate MAE and RMSE, rounding to three decimal places. (Answer: MAE \(=2.000\), RMSE \(=\sqrt{14/3}\approx2.160\).)
On the same four observations, Model M has errors \(0,0,0,5\), and Model N has errors \(1,1,1,1\). Calculate MAE and RMSE for each and state the rankings. (Answer: M has MAE \(1.25\), RMSE \(2.5\); N has MAE \(1\), RMSE \(1\). Both metrics favor N.)
A model's errors are \(2,2,2,10\). A candidate reports MAE \(=4\) and RMSE \(=\sqrt{4}\). Identify the correct values and the calculation error. (Answer: MAE \(=(2+2+2+10)/4=4\); RMSE \(=\sqrt{(4+4+4+100)/4}=\sqrt{28}\approx5.292\). The reported value takes the square root of MAE instead of averaging squared errors first.)

What You'll Learn