Why the Error Metric Is Part of the Decision
Two forecasts can look equally good under one summary and quite different under another. A forecast that misses a large-volume series by a few hundred units may matter more to total dollars, while a forecast that misses a small series by the same percentage may matter more to a relative service target. Near-zero actual values create a further complication: a small absolute miss can become an enormous percentage error.
An error metric is not a neutral finishing step. It encodes how misses are counted and what kinds of misses matter. The appropriate choice depends on the decision being evaluated, such as controlling total forecast dollars, limiting typical unit errors, or comparing proportional accuracy across products. These comparisons should use forecasts evaluated on the same held-out observations; the metric does not replace a sound time-series validation design.
The sign convention \(e_t=y_t-\hat{y}_t\) makes a positive error an underforecast and a negative error an overforecast. MAE and RMSE remove that sign before aggregation, so overforecasts do not cancel underforecasts. The mean signed error, \(\frac{1}{n}\sum e_t\), can indicate directional bias, but it is not a measure of overall error size: large positive and negative errors can cancel.
What MAE and RMSE Penalize
MAE assigns a penalty proportional to the size of each miss. RMSE squares the errors before averaging, so a large error has more influence than several smaller errors; taking the square root restores the original units. Because the square root is increasing, ranking forecasts by RMSE gives the same ranking as ranking them by mean squared error, but the RMSE is easier to interpret in the series' units.
This result gives a useful interpretation, not a blanket rule for fitting every time-series model: under squared-error loss, forecasts are judged in a way that emphasizes large misses; under absolute-error loss, each additional unit of miss has the same marginal penalty. If the actual operational cost is asymmetric—for example, underforecasting creates a penalty that overforecasting does not—neither symmetric metric fully represents that cost. In that case, evaluate the decision using the relevant cost function as well.
Worked Example: Same MAE, Different RMSE
A hypothetical service team compares two forecasts for four weekly call counts. The actual counts are \(50,60,40,50\). Forecast A gives \(40,60,40,60\), and Forecast B gives \(45,55,45,55\). Calculate MAE and RMSE for each and identify what the comparison says.
Method. Calculate each error as actual minus forecast, then average absolute errors for MAE and squared errors for RMSE. This avoids letting signed errors cancel.
For Forecast A, the errors are \(50-40=10\), \(60-60=0\), \(40-40=0\), and \(50-60=-10\). The absolute errors sum to \(10+0+0+10=20\), so \(\operatorname{MAE}_A=20/4=5\) calls. The squared errors sum to \(10^2+0^2+0^2+(-10)^2=200\). Thus \(\operatorname{RMSE}_A=\sqrt{200/4}=\sqrt{50}\approx7.071\) calls.
For Forecast B, the errors are \(50-45=5\), \(60-55=5\), \(40-45=-5\), and \(50-55=-5\). The absolute errors sum to \(20\), so \(\operatorname{MAE}_B=20/4=5\) calls. The squared errors sum to \(25+25+25+25=100\), so \(\operatorname{RMSE}_B=\sqrt{100/4}=5\) calls.
Check. The forecasts tie on MAE, since both have five calls of average absolute error. Forecast B has lower RMSE because it spreads error evenly rather than making two ten-call misses. The common mistake is to conclude that equal MAE means equal error behavior; the squared-error calculation shows the difference. Round only the final RMSE for A, to three decimals.
Percentage-Based Metrics and Their Limits
Percentage-based metrics express errors relative to actual values. They can support comparisons among series measured on different scales, but the denominator matters. In particular, MAPE is not defined when an actual value is zero and can be dominated by values close to zero.
MAPE gives each observation's percentage error equal weight, regardless of the size of its actual value. WAPE instead divides total absolute error by total absolute actual value; observations with larger actual values therefore tend to have more influence. Neither is universally superior. WAPE is undefined if all actual values sum to zero in absolute value. MAPE cannot be computed as written if any actual value is zero; dropping those observations changes the evaluation set and can bias the summary. sMAPE avoids division by the actual alone, but it can still give small-volume observations substantial influence, and its value depends on the chosen convention.
Worked Example: Competing Forecasts Across Two Scales
A hypothetical analyst evaluates weekly demand for two products over three weeks. Product X has actual values \(1000,1200,800\); Product Y has actual values \(10,12,8\). Model A forecasts X as \(900,1100,900\) and Y as \(8,10,10\). Model B forecasts X as \(850,1050,650\) and Y as \(9,11,9\). Compare the models using pooled MAE, pooled RMSE, MAPE, and WAPE, and explain why the rankings differ.
Method. Pool the six observations for the absolute and squared unit errors. For MAPE, calculate all six percentage errors and give each observation equal weight. For WAPE, divide the pooled absolute error by the pooled actual total.
For Model A, the absolute errors are \(100,100,100\) for X and \(2,2,2\) for Y. Their sum is \(300+6=306\), giving \(\operatorname{MAE}_A=306/6=51\) units. The squared errors sum to \(3(100^2)+3(2^2)=30000+12=30012\), so \(\operatorname{RMSE}_A=\sqrt{30012/6}=\sqrt{5002}\approx70.725\) units.
The X percentage errors for Model A are \(100/1000=10\%\), \(100/1200\approx8.333\%\), and \(100/800=12.5\%\). The Y percentage errors are \(2/10=20\%\), \(2/12\approx16.667\%\), and \(2/8=25\%\). Their sum is approximately \(92.5\%\), so \(\operatorname{MAPE}_A=92.5\%/6\approx15.417\%\).
For Model B, the absolute errors are \(150,150,150\) for X and \(1,1,1\) for Y. Their sum is \(453\), giving \(\operatorname{MAE}_B=453/6=75.5\) units. The squared errors sum to \(3(150^2)+3(1^2)=67500+3=67503\), so \(\operatorname{RMSE}_B=\sqrt{67503/6}=\sqrt{11250.5}\approx106.068\) units.
The X percentage errors for Model B are \(15\%,12.5\%,18.75\%\); the Y errors are \(10\%,100/12\%\approx8.333\%,12.5\%\). These sum to approximately \(77.083\%\), so \(\operatorname{MAPE}_B\approx77.083\%/6=12.847\%\). The pooled actual total is \(1000+1200+800+10+12+8=3030\). Therefore, \(\operatorname{WAPE}_A=100\%(306/3030)\approx10.099\%\), while \(\operatorname{WAPE}_B=100\%(453/3030)\approx14.950\%\).
Check. MAE, RMSE, and WAPE favor Model A; MAPE favors Model B. Model A is better in total unit error, while Model B's errors are smaller as a share of each observation's actual value on average. The common mistake is to call one model simply “more accurate” without specifying the comparison's weighting and purpose. Round final percentages to three decimals and final unit RMSEs to three decimals.
Near-Zero Values Can Distort Relative Errors
When the series contains zeros or values close to zero, a percentage may not represent a decision-relevant miss. For instance, a one-unit error on an actual value of \(0.5\) is a \(200\%\) MAPE contribution, even though the absolute error is only one unit. A zero actual makes the MAPE term undefined rather than merely large. Always inspect the actual values and the chosen denominator before interpreting a percentage summary.
Worked Example: A Zero and a Near-Zero Actual
A hypothetical low-volume monitoring series has actual values \(100,0.5,0,2\) and forecasts \(90,1,1,1.5\). Calculate MAE and RMSE, explain why MAPE is not directly available for all four observations, and calculate sMAPE and WAPE using the stated definitions.
Method. First compute unit errors and absolute errors. For sMAPE, use \(2|e_t|/(|y_t|+|\hat{y}_t|)\) for each observation. For WAPE, aggregate the absolute errors and actual values before dividing.
The errors are \(10,-0.5,-1,0.5\), so the absolute errors are \(10,0.5,1,0.5\). Their sum is \(12\), giving \(\operatorname{MAE}=12/4=3\) units. The squared errors sum to \(100+0.25+1+0.25=101.5\), so \(\operatorname{RMSE}=\sqrt{101.5/4}=\sqrt{25.375}\approx5.037\) units.
MAPE terms for the nonzero actuals are \(10/100=10\%\), \(0.5/0.5=100\%\), and \(0.5/2=25\%\). Their average would be \(45\%\) if the zero observation were omitted, but that is not MAPE over the original four observations. The zero-actual term divides by zero, so the four-observation MAPE is undefined.
The sMAPE terms are \(200(10)/(100+90)\approx10.526\%\), \(200(0.5)/(0.5+1)\approx66.667\%\), \(200(1)/(0+1)=200\%\), and \(200(0.5)/(2+1.5)\approx28.571\%\). Their average is approximately \(76.441\%\). For WAPE, the absolute actual total is \(100+0.5+0+2=102.5\), so \(\operatorname{WAPE}=100\%(12/102.5)\approx11.707\%\).
Check. The zero does not prevent this WAPE calculation because the pooled denominator is positive, but it makes MAPE undefined. The very large sMAPE term for the zero actual illustrates that avoiding division by the actual alone does not make every percentage summary stable near zero. The common mistake is to omit the zero silently and report the resulting average as if it covered all observations. Round final values to three decimals.
Match the Metric to the Decision
Before comparing forecasts, state what one observation represents, what the decision-maker loses when a forecast is wrong, and whether errors should be weighted equally by observation or by volume. Then choose a metric consistent with that objective. A useful interpretation is:
| Metric | What it emphasizes | Useful when | Important limitation |
|---|---|---|---|
| MAE | Typical absolute miss in the original units | A unit of error has roughly constant cost | Does not strongly distinguish one large miss from several smaller errors with the same total absolute size |
| RMSE | Large misses through squared penalties | Large errors are especially costly or undesirable | Can be dominated by a small number of unusually large errors |
| MAPE | Average observation-level percentage miss | Actuals are positive and relative accuracy per observation matters | Undefined at zero and unstable near zero |
| sMAPE | Symmetric scaled errors under a stated convention | A relative comparison is desired and the chosen convention is appropriate | Variants differ; near-zero values can still have large influence |
| WAPE | Total absolute error as a share of total actual volume | Total-volume relative accuracy is relevant | Large-volume observations carry more weight; undefined if total absolute actual value is zero |
A metric also determines how series are combined. Pooled MAE gives each evaluated observation equal weight in unit error; it does not give each product equal weight if products have different numbers of observations. Averaging a separate metric for each product and then taking an unweighted average gives each product equal weight instead. These answer different questions. Report the aggregation rule when a comparison includes multiple series.
No summary eliminates the need to inspect errors in context. A low overall RMSE can conceal a repeated operationally important miss, while a low MAPE can result from treating tiny actual values as equally important as large ones. If the decision has a specific loss—such as a fixed cost per unit of shortage plus a different cost per unit of excess—calculate that loss directly where possible, rather than assuming MAE, RMSE, or a percentage metric is an exact substitute.
Check Your Understanding
For each original problem, select or calculate the requested metric. Treat the stated observations as the complete evaluation set.
- Actual values are \(20,30,40\), and forecasts are \(18,34,40\). Calculate MAE and RMSE. (Answer: MAE \(=2\); RMSE \(=\sqrt{20/3}\approx2.582\).)
- Actual values are \(50,100\), and forecasts are \(40,120\). Calculate MAPE. (Answer: \((20\%+20\%)/2=20\%\).)
- Actual values are \(0,5,15\), and forecasts are \(2,4,18\). Calculate WAPE and state whether MAPE is defined for all three observations. (Answer: WAPE \(=100\%(6/20)=30\%\); MAPE is undefined because one actual is zero.)
- Actual values are \(100,200\). Model A forecasts \(90,190\); Model B forecasts \(80,200\). Compare MAE and RMSE and say which model RMSE favors. (Answer: Both MAEs are \(10\); Model A RMSE \(=10\), Model B RMSE \(=\sqrt{200}\approx14.142\), so RMSE favors A.)
- Two series have actual values \(10,20\) and \(1000,2000\). Model A has absolute errors \(2,2\) on the first series and \(0,0\) on the second. Model B has errors \(0,0\) on the first and \(100,100\) on the second. Which model has lower pooled WAPE, and which has lower MAPE? (Answer: Model A has lower WAPE: \(4/3030\times100\%\approx0.132\%\), versus \(200/3030\times100\%\approx6.601\%\). Model B has lower MAPE: \(3.75\%\), versus \(7.5\%\) for Model A.)