Why Measure Forecast Errors?
A forecast is useful only to the extent that its errors are acceptable for the intended decision. Comparing a forecast with observed values gives a set of errors, but there is more than one way to summarize them. Mean absolute error and root mean squared error are expressed in the units of the observations, while mean squared error is expressed in squared units. Mean absolute percentage error expresses errors relative to the actual values, but it cannot be calculated in the usual way when an actual value is zero.
The choice of measure can change which forecast appears better. The right measure depends on the decision: a large error may be especially costly, relative errors may matter more than dollar errors, or the observations may include zeros. The calculations below use hypothetical time-series observations and forecasts. No probability distribution is needed to calculate these descriptive measures.
The sign convention matters when interpreting individual errors: \(e_t>0\) means the actual value exceeded the forecast, so the forecast was too low; \(e_t<0\) means the forecast was too high. The four measures above use absolute values or squares, so they summarize error size rather than direction. They do not reveal whether forecasts tend to be systematically high or low.
Calculating and Interpreting the Measures
MAE averages the absolute errors. It is in the same units as the observations and gives each unit of error the same weight. MSE averages squared errors. Squaring makes larger errors count disproportionately more and gives MSE squared units. RMSE takes the square root of MSE, returning the measure to the original units while retaining the extra emphasis on large errors.
MAPE divides each absolute error by its corresponding actual value before averaging. It is a percentage, so multiplying all actual values and forecasts by the same positive constant leaves MAPE unchanged. That scale-free feature can help compare series measured on different scales, but the denominator also makes MAPE sensitive to small actual values and undefined for any zero actual value.
Worked Example: Calculating All Four Measures
A hypothetical weekly service-volume series has actual values \(100, 120, 80, 100\). A forecasting procedure predicted \(90, 130, 100, 90\), respectively. Calculate the four measures using all four weeks. Report MAE, RMSE, and MAPE to two decimal places, and MSE to two decimal places.
Method. First calculate \(e_t=y_t-\hat{y}_t\), then calculate absolute errors, squared errors, and absolute percentage errors.
| Week | Actual \(y_t\) | Forecast \(\hat{y}_t\) | Error \(e_t\) | \(|e_t|\) | \(e_t^2\) | \(100|e_t/y_t|\) |
|---|---|---|---|---|---|---|
| 1 | 100 | 90 | 10 | 10 | 100 | 10.00% |
| 2 | 120 | 130 | -10 | 10 | 100 | 8.33% |
| 3 | 80 | 100 | -20 | 20 | 400 | 25.00% |
| 4 | 100 | 90 | 10 | 10 | 100 | 10.00% |
The sum of the absolute errors is \(10+10+20+10=50\), so \(\operatorname{MAE}=50/4=12.50\) observations. The sum of the squared errors is \(100+100+400+100=700\), so \(\operatorname{MSE}=700/4=175.00\) squared observations. Therefore, \(\operatorname{RMSE}=\sqrt{175}=13.2288\ldots\), or \(13.23\) observations to two decimal places.
The percentage errors sum to \(10.00+8.33+25.00+10.00=53.33\%\), using unrounded values for the calculation. Thus \(\operatorname{MAPE}=53.3333\ldots\%/4=13.33\%\) to two decimal places. The week-three error contributes most to both the absolute and squared totals.
Check. RMSE \(13.23\) exceeds MAE \(12.50\), as expected when a larger error receives extra weight under squaring. The MAPE calculation uses each week's own actual value as its denominator.
Common mistake avoided: The percentage error for week two is \(10/120\), not \(10/130\). MAPE divides by the actual value, not the forecast.
Why RMSE and MSE Rank Forecasts the Same Way
For a fixed set of \(n\) observations, MSE is nonnegative. The square-root function is strictly increasing on nonnegative numbers, so taking the square root cannot reverse the ordering of two MSE values. RMSE therefore gives the same ranking as MSE when both are calculated on the same observations, although it is easier to interpret in the original units.
This result does not mean that RMSE and MAE must rank forecasts identically. Squaring gives disproportionate influence to a large error, so a forecast with a few large errors may look worse under RMSE than it does under MAE.
Worked Example: How a Large Error Changes the Comparison
Two forecasting procedures predict a hypothetical quantity observed as \(10, 10, 10, 100\). Procedure A forecasts \(9, 9, 9, 70\); Procedure B forecasts \(6, 6, 6, 95\). Calculate MAE, MSE, RMSE, and MAPE for both, then compare their rankings.
Method. For A, the errors \(y_t-\hat{y}_t\) are \(1, 1, 1, 30\). The absolute errors sum to \(33\), giving \(\operatorname{MAE}_A=33/4=8.25\). The squared errors sum to \(1+1+1+900=903\), giving \(\operatorname{MSE}_A=903/4=225.75\) and \(\operatorname{RMSE}_A=\sqrt{225.75}=15.0249\ldots\), or \(15.02\).
For A, the absolute percentage errors are \(10\%,10\%,10\%,30\%\). Their mean is \(\operatorname{MAPE}_A=60\%/4=15.00\%\).
For B, the errors are \(4,4,4,5\). The absolute errors sum to \(17\), giving \(\operatorname{MAE}_B=17/4=4.25\). The squared errors sum to \(16+16+16+25=73\), giving \(\operatorname{MSE}_B=73/4=18.25\) and \(\operatorname{RMSE}_B=\sqrt{18.25}=4.2720\ldots\), or \(4.27\).
B's percentage errors are \(40\%,40\%,40\%,5\%\), so \(\operatorname{MAPE}_B=125\%/4=31.25\%\). MAE, MSE, and RMSE favor B because each is smaller for B. MAPE favors A because its relative errors, on average, are smaller. The different rankings reflect different ways of weighting errors; they are not a calculation contradiction.
Check. A's single error of \(30\) drives its MSE upward, while B's errors of \(4,4,4\) are large relative to the actual value \(10\), raising its MAPE. All values use the same four observations.
Common mistake avoided: Do not assume that one metric's ranking must hold for every other metric. The measure should match the loss or comparison that matters.
Scale, Zero Values, and Fair Comparisons
MAE and RMSE depend on the scale and units of the series. If a series and all its forecasts are converted from units to thousands of units, MAE and RMSE are divided by 1,000, and MSE is divided by \(1{,}000{,}000\). MAPE is unchanged, provided the actual values are nonzero. Consequently, raw MAE values from two series with different units or very different typical sizes are not automatically comparable.
MAPE has a more serious limitation when an actual value is zero: the term \(|e_t/y_t|\) is undefined. If even one actual value is zero, the usual MAPE over the full evaluation set is undefined. Omitting that observation and averaging the remaining percentages changes the calculation and can give a misleading summary. Very small actual values can also produce extreme percentage errors even when the absolute errors are modest.
Worked Example: Zero Actual Values and MAPE
A hypothetical weekly count has actual values \(0, 10, 100\), with forecasts \(5, 8, 110\). Calculate MAE, MSE, and RMSE, and determine whether standard MAPE is defined.
Method. The errors are \(0-5=-5\), \(10-8=2\), and \(100-110=-10\). The absolute errors are \(5,2,10\), which sum to \(17\). Therefore \(\operatorname{MAE}=17/3=5.6666\ldots\), or \(5.67\).
The squared errors are \(25,4,100\), which sum to \(129\). Therefore \(\operatorname{MSE}=129/3=43.00\), and \(\operatorname{RMSE}=\sqrt{43}=6.5574\ldots\), or \(6.56\).
For the first observation, the MAPE term would require \(100\%|{-5}/0|\), which is undefined. Hence standard MAPE for all three observations is undefined. The other two percentage errors are \(100\%(2/10)=20\%\) and \(100\%(10/100)=10\%\), but their average, \(15\%\), is not the MAPE over the stated three observations.
Check. MAE, MSE, and RMSE can still be computed because none divides by an actual value. MAPE cannot be computed over the complete set because one actual value is zero.
Common mistake avoided: Do not silently discard the zero observation and report the average of the remaining percentages as if it were the original MAPE.
Using the Metrics Responsibly
A score describes forecast errors on the observations included in its calculation; it does not by itself establish how well the forecasting procedure will perform on future data. For an honest forecast comparison, evaluate forecasts against observations not used to fit or select the procedure, or use a suitable time-ordered evaluation design. The compared procedures must be assessed on the same target observations and forecast horizons. Otherwise, differences in the score may reflect different evaluation cases rather than different forecast quality.
A practical interpretation checklist is:
- Use MAE when average absolute error in the original units is directly meaningful.
- Use MSE or RMSE when large errors should receive substantially more weight; RMSE is usually easier to interpret because it has the original units.
- Use MAPE only when actual values are nonzero and percentage error is meaningful for the application.
- Before comparing scores, check that the observations, forecast horizons, units, and evaluation periods match.
- Do not treat any single metric as a complete description: a score does not show the timing or direction of errors.
Check Your Understanding
Calculate the requested measures and check that each comparison uses the stated observations.
- A hypothetical process has actual values \(50,60,40\) and forecasts \(45,63,42\). Find MAE, MSE, and RMSE, rounding each to three decimal places. (Answer: MAE \(=3.333\), MSE \(=12.667\), RMSE \(=3.559\).)
- Actual values are \(25,50,100\), and forecasts are \(20,55,90\). Find MAPE and RMSE, rounding to two decimal places. (Answer: MAPE \(=13.33\%\); RMSE \(=7.07\).)
- All actual values and forecasts in a data set are multiplied by \(10\), with no actual value equal to zero. How do MAE, MSE, RMSE, and MAPE change? (Answer: MAE and RMSE are multiplied by \(10\), MSE by \(100\), and MAPE is unchanged.)
- Actual values are \(0,20,80\), and forecasts are \(4,18,88\). Find MAE, MSE, and RMSE to two decimal places, and state whether standard MAPE is defined. (Answer: MAE \(=4.67\), MSE \(=28.00\), RMSE \(=5.29\); standard MAPE is undefined.)
- Actual values are \(10,10,100\). Procedure A forecasts \(8,8,80\), and procedure B forecasts \(5,5,95\). Which procedure has the lower MAE, and which has the lower MAPE? (Answer: B has lower MAE, \(5\) versus \(8\); A has lower MAPE, \(20\%\) versus \(35\%\).)