Why Forecast Error Metrics Matter
A forecast is useful only in relation to the errors it makes and the decision those errors inform. A model that is usually close but occasionally far off may be preferable for one application and unacceptable for another. Similarly, an error of 10 units may be minor for a large-volume series but consequential for a small one. Forecast-error metrics summarize performance, but they do not all summarize it in the same way.
This tutorial develops three common measures: mean absolute error (MAE), root mean squared error (RMSE), and mean absolute percentage error (MAPE). Each is calculated from actual observations and forecasts over the same evaluation periods. The comparison is meaningful only if the models are scored on the same targets and using the same periods.
Three Metrics, Three Summaries
MAE averages the absolute sizes of the errors. RMSE squares the errors before averaging, then takes the square root. MAPE averages absolute errors relative to the observed values. These definitions use \(n\) evaluation periods and assume the same periods are included in each metric.
MAE and RMSE are expressed in the same units as the observed series. MAPE is expressed as a percentage and is defined by this formula only when every evaluated \(y_t\) is nonzero. If an actual value is zero, its percentage error has a zero denominator; the MAPE over the full set is therefore undefined. Even when all actuals are nonzero, a very small actual can make its percentage error extremely large.
All three measures discard the direction of the error: an overforecast and an underforecast of the same size contribute equally. If the decision assigns different consequences to overforecasting and underforecasting, these three symmetric metrics alone do not represent that difference.
Why RMSE Responds More to Large Errors
Squaring makes a large absolute error count disproportionately more than several smaller errors of the same total size. The square root at the end restores the result to the original units, but it does not undo the relative emphasis created by squaring.
RMSE is at least as large as MAE on the same observations. To see why, set \(a_t=|e_t|\), so \(a_t\geq0\). The mean of the squares is at least the square of the mean: \(\frac{1}{n}\sum a_t^2\geq(\frac{1}{n}\sum a_t)^2\). Taking nonnegative square roots gives RMSE \(\geq\) MAE. Equality holds when all absolute errors are equal. This is a useful arithmetic check, not a rule that RMSE must be preferred.
Worked Calculations
Worked Example: Scoring Weekly Energy-Demand Forecasts
A hypothetical facility compares a forecast with observed energy demand, measured in megawatt-hours. For three weeks, the actual values are 120, 150, and 90, and the forecasts are 110, 165, and 105. Calculate MAE, RMSE, and MAPE, rounding final results to two decimal places.
Method. Calculate signed errors as actual minus forecast, then use their absolute values for MAE and MAPE and their squares for RMSE.
The errors are \(120-110=10\), \(150-165=-15\), and \(90-105=-15\). Thus the absolute errors are 10, 15, and 15, and the squared errors are 100, 225, and 225.
MAE: \((10+15+15)/3=40/3=13.33\) megawatt-hours.
RMSE: \(\sqrt{(100+225+225)/3}=\sqrt{550/3}=\sqrt{183.3333\ldots}=13.54\) megawatt-hours.
MAPE: The three absolute percentage errors are \(100(10/120)=8.3333\ldots\%\), \(100(15/150)=10\%\), and \(100(15/90)=16.6666\ldots\%\). Their mean is \((8.3333\ldots+10+16.6666\ldots)/3=11.6666\ldots\%=11.67\%\).
Check and common mistake avoided. RMSE exceeds MAE, as expected. The signed errors must not be averaged before taking absolute values: doing so would let errors in opposite directions cancel and would not calculate MAE.
Worked Example: Comparing Scores When One Forecast Misses Badly
A hypothetical service team forecasts the number of incoming requests for four shifts. The actual counts are 100, 100, 100, and 100. Model A forecasts 90, 110, 100, and 160; Model B forecasts 85, 115, 100, and 130. Calculate MAE and RMSE for each model, then explain how the scores differ. Round final results to two decimal places.
Method. Compare absolute and squared errors on the same four shifts. The actual counts are positive, so MAPE could also be calculated, but the task is to see the effect of the unusually large miss on Model A.
For Model A, the signed errors are 10, \(-10\), 0, and \(-60\); the absolute errors are 10, 10, 0, and 60. Therefore, \(\operatorname{MAE}_A=(10+10+0+60)/4=20\) requests. The squared errors are 100, 100, 0, and 3,600, so \(\operatorname{RMSE}_A=\sqrt{(100+100+0+3600)/4}=\sqrt{950}=30.82\) requests.
For Model B, the signed errors are 15, \(-15\), 0, and \(-30\); the absolute errors are 15, 15, 0, and 30. Thus \(\operatorname{MAE}_B=(15+15+0+30)/4=15\) requests. The squared errors are 225, 225, 0, and 900, giving \(\operatorname{RMSE}_B=\sqrt{(225+225+0+900)/4}=\sqrt{337.5}=18.37\) requests.
Model B has both the lower MAE and the lower RMSE here. Model A’s 60-request miss has a particularly strong effect on its RMSE because it contributes 3,600 to the squared-error total. The other three shifts do not erase that contribution.
Check and common mistake avoided. The square root is applied after averaging the squared errors, not to each error before averaging. Choosing only the model with the best typical absolute error can also understate the importance of a rare, very large miss when such misses are especially costly.
Worked Example: A Zero Actual and a Small Actual
A hypothetical weekly output series has actual values 0 and 20 units, with forecasts 5 and 18 units. Calculate MAE and RMSE, and determine whether MAPE is defined. Round the final RMSE to four decimal places.
Method. First calculate the two absolute errors and squared errors. Then inspect every actual value in the MAPE denominators before attempting a percentage calculation.
The signed errors are \(0-5=-5\) and \(20-18=2\). The absolute errors are 5 and 2, so \(\operatorname{MAE}=(5+2)/2=3.5\) units. The squared errors are 25 and 4, so \(\operatorname{RMSE}=\sqrt{(25+4)/2}=\sqrt{29/2}=\sqrt{14.5}=3.8079\) units.
MAPE is undefined for the two-period evaluation because the first actual is zero, making that period’s percentage error divide by zero. The second period alone has absolute percentage error \(100(2/20)=10\%\), but reporting 10% as the two-period MAPE would silently drop one of the evaluated periods.
Check and common mistake avoided. The absolute errors sum to \(5+2=7\), so the MAE is \(7/2=3.5\), not 3. A calculation that reports 3 has misadded or misaveraged the errors. Removing the zero-actual period does not repair the stated two-period MAPE; it changes the evaluation sample.
Choosing a Metric for the Decision
There is no universally best metric among MAE, RMSE, and MAPE. Select a measure based on the scale of the series, the cost of different errors, and whether relative or absolute performance is the relevant comparison.
| Metric | What it emphasizes | Useful when | Main caution |
|---|---|---|---|
| MAE | Average absolute error, with errors weighted linearly | A typical error in the series’ original units is interpretable | Does not emphasize a very large error as strongly as RMSE |
| RMSE | Large errors, through squared contributions | A decision places extra weight on avoiding large misses | A small number of large errors can dominate the score |
| MAPE | Absolute error as a percentage of each actual value | Relative errors are meaningful and actual values are safely nonzero | Undefined at zero and can be dominated by small actual values |
A unit-based measure such as MAE or RMSE is often easier to interpret when all observations are on the same scale. MAPE is unitless, which can help with a relative interpretation, but it is not automatically a fair comparison across series. Its denominator gives small actual values much more influence than large ones, regardless of whether that weighting matches the decision.
For example, suppose two periods have actuals of 2 and 100, and each forecast is wrong by 2. The absolute errors are equal, but the percentage errors are \(100\%\) and \(2\%\). MAPE treats the first period as far worse in relative terms. That may be appropriate if proportional accuracy matters, but it may not be if the absolute number of missed units drives the cost.
When comparing models, report the metric in context: name its evaluation periods and units, and state whether it is an average over those periods. Do not compare a model’s MAE calculated on one set of periods with another model’s RMSE or MAPE on a different set and call the smaller number a better forecast. The measures have different definitions and, in the case of MAE and RMSE, different behavior despite sharing units.
Common Exam Traps
- Allowing signed errors to cancel. MAE uses \(|e_t|\), not the signed mean error. Opposite forecast biases can produce a small signed average while individual forecasts remain inaccurate.
- Forgetting the final square root. The mean squared error is in squared units; RMSE takes its square root to return to the original units.
- Using a zero actual in MAPE. A zero denominator makes the period’s percentage error undefined. Do not silently delete that period unless the problem explicitly defines a different evaluation sample.
- Overlooking small denominators. MAPE can be very large for a modest absolute error when the actual value is small.
- Assuming a lower score always means a better decision. A metric is informative only to the extent that its weighting of errors matches the consequences of the decision.
Check Your Understanding
Calculate each requested score using the stated periods, and check whether the metric is defined before reporting it.
- A hypothetical monthly count has actual values 40, 50, and 60 and forecasts 44, 45, and 66. Find MAE, RMSE, and MAPE. (Answer: MAE \(=5\) counts; RMSE \(=\sqrt{77/3}\approx5.0662\) counts; MAPE \(=10.00\%\).)
- A hypothetical two-day demand series has actuals 20 and 40, and forecasts 10 and 50. Find MAE, RMSE, and MAPE. (Answer: MAE \(=10\) units; RMSE \(=10\) units; MAPE \(=37.50\%\).)
- A hypothetical output series has actuals 0 and 10, and forecasts 2 and 8. Find MAE and RMSE, and state whether MAPE is defined over both periods. (Answer: MAE \(=2\) units; RMSE \(=2\) units; two-period MAPE is undefined because an actual is zero.)
- On three hypothetical periods, actuals are 100, 100, and 100. Model A forecasts 90, 100, and 130; Model B forecasts 85, 115, and 105. Find the MAE and RMSE for each, and identify which model has the lower score on both measures. (Answer: Model A MAE \(=13.33\), RMSE \(=\sqrt{1000/3}\approx18.2574\); Model B MAE \(=11.67\), RMSE \(=\sqrt{475/3}\approx12.5831\); Model B is lower on both.)
- A hypothetical series has actuals 2 and 100, with forecasts 4 and 90. Find MAE, RMSE, and MAPE. Briefly state why the MAPE may not reflect the same priorities as an absolute-error measure. (Answer: MAE \(=6\) units; RMSE \(=\sqrt{52}\approx7.2111\) units; MAPE \(=55.00\%\). The 2-unit error at actual 2 contributes \(100\%\), so the small actual has strong influence on MAPE.)