How Outcome Transformations Change Error Metrics

Why the Outcome Scale Matters

A prediction error is a difference between an observed outcome and a prediction. But a difference depends on the scale on which it is calculated. If outcomes are analyzed after a transformation, such as taking logarithms, errors measured on that transformed scale answer a different question from errors measured in the outcome’s original units. The distinction matters when reporting prediction accuracy: a small error in log units is not a small error of the same numerical size in dollars, kilograms, or another original unit.

This tutorial compares mean absolute error (MAE) and root mean squared error (RMSE) across outcome scales. The calculations use paired observed outcomes and predictions for the same cases. For an outcome that is logged, predictions on the log scale must be exponentiated before they can be compared with observed outcomes on the original scale.

Define the Errors on Each Scale

Suppose \(y_i\) is the observed outcome for case \(i\), and \(\hat{y}_i\) is its prediction on the original scale. The signed prediction error is observed minus predicted. MAE averages the absolute errors, while RMSE takes the square root of the average squared errors. Both are nonnegative, and smaller values indicate smaller errors according to the corresponding measure.

Definition: For \(n\) observed outcomes and predictions on the original scale, the mean absolute error and root mean squared error are

\operatorname{MAE}_{\text{original}}=\frac{1}{n}\sum_{i=1}^{n}|y_i-\hat{y}_i|, \qquad \operatorname{RMSE}_{\text{original}}= \sqrt{\frac{1}{n}\sum_{i=1}^{n}(y_i-\hat{y}_i)^2}.

If \(g\) is a transformation of the outcome, the corresponding errors on the transformed scale compare \(g(y_i)\) with a prediction \(\hat{z}_i\) made on that scale:

\operatorname{MAE}_{g}=\frac{1}{n}\sum_{i=1}^{n}|g(y_i)-\hat{z}_i|, \qquad \operatorname{RMSE}_{g}= \sqrt{\frac{1}{n}\sum_{i=1}^{n}(g(y_i)-\hat{z}_i)^2}.

These formulas describe errors in the units of the scale used in the calculation. For example, if \(y_i\) is measured in kilograms, original-scale MAE and RMSE are in kilograms. If \(g(y)=\ln(y)\), errors are in log units, which express differences between logarithms rather than differences in kilograms. A transformed-scale metric and an original-scale metric should not be compared as though their numerical values had the same meaning.

The calculations themselves require matched observations and predictions. To use a validation set to estimate performance on future cases, that set should also be separate from the data used to fit or choose the model and should be relevant to the intended prediction setting. The error metrics summarize performance on the cases evaluated; they do not, by themselves, establish how well a model will work in every other setting.

Worked Example: Comparing Log-Scale and Original-Scale Errors

Suppose three hypothetical positive outcomes are 10, 20, and 40 units. A model predicts their natural logarithms as \(\ln(12)\), \(\ln(18)\), and \(\ln(50)\). These values can be exponentiated to obtain original-scale predictions of 12, 18, and 50 units. Calculate the MAE and RMSE on both scales.

Original-scale errors. Observed minus predicted, the errors are \(10-12=-2\), \(20-18=2\), and \(40-50=-10\) units. Therefore,

\operatorname{MAE}_{\text{original}} =\frac{|{-2}|+|2|+|{-10}|}{3} =\frac{14}{3} \approx 4.667\text{ units}.

The squared errors sum to \(4+4+100=108\) square units, so

\operatorname{RMSE}_{\text{original}} =\sqrt{\frac{108}{3}} =\sqrt{36} =6.000\text{ units}.

Log-scale errors. Because the predicted log values are \(\ln(12)\), \(\ln(18)\), and \(\ln(50)\), subtracting predictions from observed log outcomes gives \(\ln(10/12)\), \(\ln(20/18)\), and \(\ln(40/50)\). The signed values are approximately \(-0.1823\), \(0.1054\), and \(-0.2231\). Thus,

\operatorname{MAE}_{\log} =\frac{0.1823+0.1054+0.2231}{3} \approx 0.1703\text{ log units}.

The squared log errors sum to approximately \(0.03324+0.01110+0.04979=0.09414\). Hence

\operatorname{RMSE}_{\log} =\sqrt{\frac{0.09414}{3}} \approx 0.1771\text{ log units}.

The values \(4.667\) units and \(0.1703\) log units measure different things; neither is a converted version of the other. The original-scale values summarize absolute and squared differences in outcome units. The log-scale values summarize differences between the logarithms of outcomes and predictions.

What Log-Scale Errors Mean

For positive outcomes and predictions, a log-scale error has a useful ratio interpretation. If the signed log error is \(\ln(y_i)-\ln(\hat{y}_i)\), the logarithm rules give

\ln(y_i)-\ln(\hat{y}_i)=\ln\left(\frac{y_i}{\hat{y}_i}\right).

So the error is determined by the ratio of observed to predicted outcome, not by their absolute difference. For instance, predicting 12 when the outcome is 10 gives a ratio of \(10/12\), while predicting 120 when the outcome is 100 gives the same ratio. Both cases therefore have the same absolute log error, \(|\ln(10/12)|\), even though their original-scale differences are 2 and 20 units.

Key result: If the outcome and prediction are both multiplied by the same positive constant \(c\), their log error is unchanged:

\ln(cy_i)-\ln(c\hat{y}_i) =\ln(y_i)-\ln(\hat{y}_i).

Their original-scale difference, however, is multiplied by \(c\): \(|cy_i-c\hat{y}_i|=c|y_i-\hat{y}_i|\).

A log error is not itself a percentage error. Rather, its exponential corresponds to an observed-to-predicted ratio. For example, an absolute log error of \(\ln(1.2)\) corresponds to a factor of 1.2 between the larger and smaller value. Log errors treat reciprocal ratios symmetrically: a prediction that is 1.2 times the outcome and an outcome that is 1.2 times the prediction have the same absolute log error.

Back-Transforming Predictions

For the natural-log transformation, the inverse transformation is exponentiation. If a prediction on the log scale is \(\hat{z}_i\), its direct back-transformed prediction is \(\hat{y}_i=\exp(\hat{z}_i)\). To calculate original-scale MAE or RMSE, compare this back-transformed prediction with the original observed outcome, using the original outcome’s units.

Back-transforming a prediction does not make the transformed-scale error an original-scale error. It gives a prediction expressed on the original scale, after which the original-scale error must be calculated as \(y_i-\exp(\hat{z}_i)\). Also, if a model prediction represents a mean on the log scale, simply exponentiating it does not generally produce the conditional mean on the original scale. The interpretation of the prediction depends on what the model predicts; the metric calculation must use the predictions defined for the intended task.

Worked Example: A Change in Model Ranking Across Scales

Consider two hypothetical positive outcomes, 1 and 100 units. Model A predicts 2 and 90; Model B predicts 1.5 and 80. Compare the models using MAE on the original scale and on the natural-log scale. Each log-scale prediction is the logarithm of the stated original-scale prediction.

On the original scale, Model A’s absolute errors are \(|1-2|=1\) and \(|100-90|=10\). Its MAE is \((1+10)/2=5.500\) units. Model B’s absolute errors are \(|1-1.5|=0.5\) and \(|100-80|=20\), giving an MAE of \((0.5+20)/2=10.25\) units. By original-scale MAE, Model A has the smaller error.

For Model A, the absolute log errors are \(|\ln(1/2)|=\ln(2)\approx0.6931\) and \(|\ln(100/90)|=\ln(10/9)\approx0.1054\). Thus,

\operatorname{MAE}_{\log,A} =\frac{0.6931+0.1054}{2} \approx 0.3993\text{ log units}.

For Model B, the absolute log errors are \(|\ln(1/1.5)|=\ln(1.5)\approx0.4055\) and \(|\ln(100/80)|=\ln(1.25)\approx0.2231\). Therefore,

\operatorname{MAE}_{\log,B} =\frac{0.4055+0.2231}{2} \approx 0.3143\text{ log units}.

By log-scale MAE, Model B has the smaller error. The ranking changes because the metrics weight discrepancies differently: original-scale MAE gives more influence to large absolute differences, while log-scale MAE assesses multiplicative discrepancies. Neither ranking is universally correct; the relevant scale depends on the purpose of the predictions and the costs of errors.

Units Change Original-Scale Metrics

The same outcome can be recorded in different units. When both observations and predictions are converted using a multiplicative conversion, original-scale errors change by that conversion factor, so both MAE and RMSE change by the same factor. Their numerical values therefore cannot be compared across units without accounting for the conversion. In contrast, for a log transformation, multiplying all outcomes and predictions by the same positive factor leaves log errors and their MAE and RMSE unchanged.

Worked Example: Rescaling Outcomes by a Factor of Ten

Suppose two hypothetical outcomes are 10 and 20 units and their predictions are 12 and 18 units. The signed original-scale errors are \(-2\) and \(2\) units, so

\operatorname{MAE}_{\text{original}}=\frac{2+2}{2}=2\text{ units}, \qquad \operatorname{RMSE}_{\text{original}} =\sqrt{\frac{4+4}{2}}=2\text{ units}.

Now express both outcomes and predictions in a scale ten times larger: the outcomes become 100 and 200, and the predictions become 120 and 180. The signed differences are now \(-20\) and \(20\), so

\operatorname{MAE}_{\text{original,new}}=\frac{20+20}{2}=20, \qquad \operatorname{RMSE}_{\text{original,new}} =\sqrt{\frac{400+400}{2}}=20.

The original-scale errors have each been multiplied by 10. But the log errors have not changed: the observed-to-predicted ratios remain \(10/12\) and \(20/18\) after rescaling. The log-scale MAE is \((|\ln(10/12)|+|\ln(20/18)|)/2\approx0.1438\) before and after the change. This is the distinction: multiplying the original outcome and prediction by 10 changes a difference of 2 units to 20, while the log error remains unchanged.

Choosing and Reporting an Error Scale

Choose the scale to match the question the metric is meant to answer. If a decision depends on absolute discrepancies in the outcome’s units, report original-scale errors. If proportional differences are more relevant, a log-scale metric may be informative for positive outcomes. RMSE gives relatively more influence to large errors than MAE because it squares each error before averaging; that property applies on whichever scale is used.

A clear report names the metric and scale, states the units or describes the log scale, and identifies how predictions were obtained. If both scales are relevant, reporting both can be useful, provided each is interpreted separately. For example, “original-scale MAE was 4.667 units” and “log-scale MAE was 0.1703 log units” are meaningful reports; describing the second value as 0.1703 outcome units is not.

Common Mistakes

Comparing metric values across scales as if they had common units. A log-scale MAE of 0.2 and an original-scale MAE of 2 do not show that one error is ten times the other. They summarize different discrepancies.
Comparing original outcomes directly with log predictions. Exponentiate log predictions first, then compare the back-transformed predictions with original outcomes.
Assuming a transformation cannot change which model looks better. A nonlinear transformation changes the relative size of errors, and the worked example shows that model rankings can reverse.
Claiming original-scale differences stay the same when units are multiplied. Multiplying both outcomes and predictions by 10 multiplies their differences, MAE, and RMSE by 10. It is the log error that remains unchanged under this shared rescaling.
Calling a log error a percentage error. It is a difference between logarithms, equivalently the logarithm of a ratio. Report it as a log-scale error or explain its ratio interpretation.

Key takeaway: MAE and RMSE are defined by both the loss formula and the scale on which errors are calculated. Back-transform predictions before calculating original-scale metrics, and interpret every reported metric in the units and scale that produced it.

Check Your Understanding

Answer each question by identifying the scale and showing the relevant calculation or reasoning.

For observed outcomes 5 and 10 and predictions 4 and 12, calculate original-scale MAE and RMSE. State their units if the outcomes are measured in kilograms.
If positive outcomes and predictions are both multiplied by 100, what happens to their natural-log errors? What happens to original-scale MAE?
Why can the model with smaller log-scale MAE have larger original-scale MAE?
A model predicts \(\ln(25)\) for an outcome observed to be 20 units. Give its back-transformed prediction and describe how to calculate its original-scale absolute error.
Explain why a log-scale RMSE of 0.15 should not be reported as an error of 0.15 units of the original outcome.