Choosing Between MAE and RMSE

Why the Choice of Error Metric Matters

Two prediction models can make errors of similar size most of the time yet differ sharply on a few cases. One model might be slightly off for many observations; another might be accurate on most observations but make an occasional very large miss. Which model is better depends partly on how prediction errors are summarized.

The mean absolute error (MAE) and the root mean squared error (RMSE) both measure the typical size of prediction errors, in the outcome's units, with smaller values indicating smaller errors overall. They do not, however, treat large errors in the same way. Comparing both on the same observations can reveal whether a model's ranking depends on how strongly large misses are penalized.

Definitions and Comparison Conditions

Suppose there are $n$ observed outcomes, $y_1,\ldots,y_n$, and one prediction $\hat{y}_i$ matched to each outcome $y_i$. Define the prediction error for case $i$ as $e_i=y_i-\hat{y}_i$. For MAE and RMSE, only the error's magnitude matters: an error of $5$ and an error of $-5$ contribute equally.

Definition: For $n$ matched outcomes and predictions, the mean absolute error is the average of the absolute prediction errors, and the root mean squared error is the square root of the average squared prediction errors: $$ \mathrm{MAE}=\frac{1}{n}\sum_{i=1}^{n}|e_i| \qquad\text{and}\qquad \mathrm{RMSE}=\sqrt{\frac{1}{n}\sum_{i=1}^{n}e_i^2}. $$ Both measures are nonnegative and have the same units as the outcome. A lower value means smaller errors according to that particular metric.

A fair comparison requires predictions and observed outcomes to be paired correctly and scored on the same cases. The models must also predict the same outcome in the same units. If one model is evaluated on a different set of cases, a difference in MAE or RMSE may reflect which cases were included rather than a genuine difference in prediction quality. These metrics describe the errors in the evaluated cases; by themselves, they do not establish how well a model will perform on future cases.

Worked Example: One Large Miss Reverses the Ranking

Four hypothetical delivery orders each take $50$ minutes. Model A predicts $50, 50, 50,$ and $40$ minutes; Model B predicts $47, 47, 47,$ and $47$ minutes. Compare their MAE and RMSE on these same four orders.

Using $e_i=y_i-\hat{y}_i$, Model A's errors are $0,0,0,10$ minutes. Its MAE is $$ \mathrm{MAE}_A=\frac{|0|+|0|+|0|+|10|}{4} =\frac{10}{4}=2.5\text{ minutes}. $$ Its RMSE is $$ \mathrm{RMSE}_A=\sqrt{\frac{0^2+0^2+0^2+10^2}{4}} =\sqrt{\frac{100}{4}}=\sqrt{25}=5\text{ minutes}. $$

Model B's errors are $3,3,3,3$ minutes. Therefore, $$ \mathrm{MAE}_B=\frac{3+3+3+3}{4}=3\text{ minutes}, \qquad \mathrm{RMSE}_B=\sqrt{\frac{3^2+3^2+3^2+3^2}{4}} =\sqrt{\frac{36}{4}}=3\text{ minutes}. $$

By MAE, Model A ranks better: $2.5<3$ minutes. By RMSE, Model B ranks better: $3<5$ minutes. Model A is closer on three orders but misses the fourth by $10$ minutes. Squaring makes that large error weigh heavily, so RMSE favors the model with the more even set of errors. Neither ranking is an arithmetic contradiction; the metrics emphasize different features of the same errors.

Why RMSE Cannot Be Smaller Than MAE

Although the two metrics can rank models differently, RMSE is never less than MAE when both are computed from the same set of errors. A short algebraic argument shows why.

Justification: Let $a_i=|e_i|$, so each $a_i$ is nonnegative, and let $\bar{a}=\frac{1}{n}\sum_{i=1}^{n}a_i=\mathrm{MAE}$. The average squared deviation from $\bar{a}$ cannot be negative: $$ \frac{1}{n}\sum_{i=1}^{n}(a_i-\bar{a})^2\geq 0. $$ Expanding the left side and using $\frac{1}{n}\sum_{i=1}^{n}a_i=\bar{a}$ gives $$ \frac{1}{n}\sum_{i=1}^{n}a_i^2-\bar{a}^{\,2}\geq 0. $$ Since $a_i^2=e_i^2$, this says $\mathrm{RMSE}^2\geq\mathrm{MAE}^2$. Both quantities are nonnegative, so $\mathrm{RMSE}\geq\mathrm{MAE}$. Equality holds exactly when all the absolute errors are equal.

This inequality applies to the two metrics for one model on one fixed set of cases. It does not mean that the model with the smaller RMSE must have the smaller MAE compared with another model. The worked example showed that cross-model rankings can reverse.

Worked Example: Calculating Both Metrics for One Set of Predictions

In a hypothetical demand-forecasting exercise, four observed demands are $20,25,30,$ and $40$ units. A model predicts $19,27,27,$ and $32$ units. Calculate and interpret both metrics.

The signed errors, observed minus predicted, are $1,-2,3,$ and $8$ units. The absolute errors are $1,2,3,$ and $8$, so $$ \mathrm{MAE}=\frac{1+2+3+8}{4} =\frac{14}{4}=3.5\text{ units}. $$

The squared errors are $1,4,9,$ and $64$ square units. Thus $$ \mathrm{RMSE}=\sqrt{\frac{1+4+9+64}{4}} =\sqrt{\frac{78}{4}} =\sqrt{19.5}\approx4.416\text{ units}. $$

For these four cases, the average absolute prediction error is $3.5$ units, while the RMSE is about $4.416$ units. RMSE exceeds MAE because the absolute errors are not all equal, and the $8$-unit error contributes $64$ to the sum of squared errors. The squared-error calculation is in square units, but taking the square root returns RMSE to the original unit, demand units.

Choosing a Metric to Match the Consequences

The choice between MAE and RMSE is not just a mathematical preference. It should reflect what counts as a costly prediction mistake in the intended use. MAE assigns each additional unit of absolute error the same additional amount in its sum. For example, increasing an error from $2$ to $3$ units adds $1$ unit to the absolute-error total, just as increasing an error from $9$ to $10$ units does.

Squared errors grow faster. An error of $10$ units contributes $100$ squared units, whereas an error of $2$ units contributes $4$ squared units: the first contributes $25$ times as much to the squared-error total. RMSE then takes the square root of the average, so its final value is not a simple sum of per-case penalties. Still, the squaring step makes large errors exert relatively more influence on RMSE than on MAE.

Worked Example: A Tie in MAE but Not in RMSE

Suppose two hypothetical clinics compare forecasts of patient waiting times, scored on the same four days. Model C has absolute errors of $1,1,1,$ and $7$ minutes. Model D has absolute errors of $2,2,2,$ and $4$ minutes. Compare the metrics.

For Model C, $$ \mathrm{MAE}_C=\frac{1+1+1+7}{4}=\frac{10}{4}=2.5\text{ minutes}, \qquad \mathrm{RMSE}_C=\sqrt{\frac{1+1+1+49}{4}} =\sqrt{13}\approx3.606\text{ minutes}. $$

For Model D, $$ \mathrm{MAE}_D=\frac{2+2+2+4}{4}=\frac{10}{4}=2.5\text{ minutes}, \qquad \mathrm{RMSE}_D=\sqrt{\frac{4+4+4+16}{4}} =\sqrt{7}\approx2.646\text{ minutes}. $$

The models tie on MAE because their absolute errors have the same total. Model D has the lower RMSE because its errors are less concentrated in one large miss. If the clinic mainly cares about average absolute deviation, the MAE tie is relevant. If unusually large forecast errors are especially undesirable, RMSE distinguishes the models in favor of D. The numbers alone do not determine which concern should take priority; that depends on the use of the forecasts.

In practice, use MAE when an average absolute deviation is a clear and useful summary, or when the goal is to avoid giving large misses extra emphasis merely because they are large. RMSE is useful when large errors should count disproportionately more in model comparison. This can be appropriate when a very large miss has much greater consequences than several small misses. Even then, RMSE is a particular mathematical penalty, not a guarantee that it exactly matches the real-world cost of an error. If consequences are asymmetric—for example, overprediction is more harmful than underprediction—neither MAE nor RMSE alone represents that asymmetry, because both treat errors of equal magnitude with opposite signs equally.

Common Mistakes and a Practical Check

Assuming the metrics must rank models the same way. They need not: squaring changes how much influence large errors have. Calculate both on the same cases if the choice of metric matters.
Calling RMSE the average absolute error. MAE is the average of absolute errors. RMSE is the square root of the average of squared errors; it is generally different from MAE.
Comparing metrics with different units or cases. MAE and RMSE have the outcome's units, but their values are comparable across models only when the outcome scale and evaluated cases are the same.
Choosing a metric after seeing which model it favors. That makes the comparison hard to interpret. Decide which pattern of errors matters for the intended use, then report the metric that reflects that choice. Reporting both can show how sensitive the ranking is.
Treating a lower score as proof of better future performance. The metrics summarize errors in the evaluation data. Their value for future predictions depends on how representative that evaluation is and how the predictions were produced.

Key takeaway: MAE summarizes average absolute error, while RMSE gives relatively greater influence to large errors. Both are measured in outcome units and should be compared on the same cases. Choose between them according to the consequences of errors, not merely according to which one gives a preferred ranking.

Check Your Understanding

Try these questions before moving on.

A model's absolute errors on three cases are $2,2,$ and $8$ units. Calculate its MAE and RMSE.
For the same set of cases, can RMSE be smaller than MAE? Explain using the nonnegativity of squared deviations or the stated inequality.
Two models have errors with the same average absolute magnitude, but one has a single much larger error and several smaller errors. Which metric is more likely to distinguish them, and why?
Why is it not a fair model comparison to calculate one model's MAE on one set of cases and another model's RMSE on a different set?
Give a practical reason to prefer MAE over RMSE, and a different practical reason to prefer RMSE over MAE.

What You'll Learn