Metric Choice Under Asymmetric Prediction Costs

Why the Choice of Error Metric Matters

A prediction can be close to the outcome on most cases but miss badly on a few. Or it can tend to be slightly too high, even when its errors are usually small. Which summary is most useful depends on what a prediction will be used for: a large miss may be especially costly, or overprediction may have different consequences from underprediction. A metric makes some features of errors more visible than others, so it should be selected with the decision in mind.

This tutorial compares three summaries: mean absolute error (MAE), root mean squared error (RMSE), and mean signed error. The first two summarize error magnitude in different ways; the third summarizes average direction. None automatically represents every practical cost.

These formulas apply to a set of paired predictions and outcomes on the same quantitative scale. For a fair comparison of two prediction methods, calculate their metrics on the same cases and use the same units and case-weighting scheme. If the cases do not represent the setting where predictions will be used, a correctly calculated metric may still be a poor guide to future performance. These are conditions for interpreting the comparison, not assumptions about a probability distribution.

MAE and RMSE are nonnegative; zero means every prediction exactly matches its observed outcome. Smaller values indicate smaller errors according to the metric used. Mean signed error can be positive, negative, or zero. A value near zero means the errors balance on average, not necessarily that individual predictions are close.

What Each Metric Emphasizes

MAE gives each case weight in proportion to the absolute size of its error. An error of 4 outcome-units contributes twice as much as an error of 2 units. This makes MAE a direct summary when a typical miss measured in the outcome's units is the main concern. However, it does not distinguish overprediction from underprediction: errors of \(4\) and \(-4\) contribute equally.

RMSE also has the outcome's units, but it squares each error before averaging and taking a square root. As a result, a large error contributes disproportionately. An error of 4 contributes \(16\) to the average squared error, while an error of 2 contributes \(4\): four times as much, despite being only twice as large. RMSE can be useful when unusually large misses deserve particular attention. It is still symmetric: an overprediction of a given magnitude and an underprediction of the same magnitude contribute equally.

Mean signed error retains direction. Under the convention \(e_i=y_i-\hat{y}_i\), a positive mean indicates average underprediction and a negative mean indicates average overprediction. But errors can cancel: a few large positive errors may offset large negative errors. Therefore, mean signed error alone does not tell us how far predictions tend to be from outcomes.

A Relationship Between Direction and RMSE

The link between mean signed error and RMSE helps clarify what each tells us. Squaring RMSE gives the mean squared error, which can be separated into a component due to the mean signed error and a component due to variation in errors around that mean.

The middle term vanishes because \(\sum_{i=1}^{n}(e_i-\bar{e})=0\), and the final term equals \(\bar{e}^{\,2}\). Taking square roots gives \(\mathrm{RMSE}=\sqrt{\bar{e}^{\,2}+\frac{1}{n}\sum_{i=1}^{n}(e_i-\bar{e})^2}\). Thus, RMSE reflects both average signed error and how much individual errors vary around it. A small mean signed error alone cannot establish that RMSE is small.

When Large Misses Matter More

If the practical concern is the size of a typical miss on the original scale, MAE is often the more straightforward choice. If a rare large error could have much greater consequences than several small errors, RMSE gives those large errors more influence. This is a difference in emphasis, not a universal rule that one metric is always better. The choice should reflect the consequences of prediction errors in the application.

The following hypothetical comparison shows how the metrics can rank predictions differently. Suppose errors are measured in thousands of dollars, and predictions are assessed on four cases.

When Overprediction and Underprediction Have Different Costs

MAE and RMSE treat positive and negative errors of equal size identically. If overprediction is more harmful than underprediction, neither metric by itself encodes that difference. Mean signed error helps reveal a tendency toward one direction, but it does not measure the harm caused by that direction: the same mean can arise from very different individual errors.

For example, a staffing forecast that is too high might incur costs different from a forecast that is too low. In such settings, first identify which direction is costly and how the cost changes with error size. A metric designed to reflect that cost may be needed; alongside it, MAE or RMSE can still describe symmetric error magnitude, and mean signed error can help monitor directional tendency. If only the three metrics here are available, use them for those limited purposes rather than treating any one as a complete measure of asymmetric cost.

A Practical Choice and a Common Mistake

Before calculating a metric, check that outcomes and predictions are paired case by case, expressed in the same units, and evaluated on cases relevant to the intended use. Then match the summary to the question:

• Use MAE when the average size of an absolute miss is a useful summary and each additional unit of error should count proportionally.
• Use RMSE when large errors should receive substantially more influence than small ones.
• Use mean signed error to describe average direction, while remembering that positive and negative errors can cancel.
• For unequal directional costs, do not assume that any of these three alone captures the relevant consequences. State the cost difference and assess performance accordingly.

A common mistake is to conclude that a mean signed error close to zero proves predictions are accurate. That reasoning overlooks cancellation: errors of \(+10\) and \(-10\) have mean signed error zero, even though both predictions miss by 10 units. The correction is to use an error-magnitude metric as well when closeness matters. The reverse mistake is to interpret a low MAE or RMSE as evidence of no directional bias. Those metrics discard the sign, so inspect mean signed error when direction matters.