Normalized RMSE for Comparing Scales

Why Normalize RMSE?

Root mean squared error (RMSE) is expressed in the same units as the outcome. An RMSE of 200 dollars may be small for a high-value claim and large for a small benefit payment. When outcomes have different typical magnitudes or are recorded in different units, the raw RMSEs are difficult to interpret side by side. Normalized RMSE expresses RMSE relative to a stated scale of the observed outcomes.

There is no single denominator that defines every use of “normalized RMSE.” A result is interpretable only when the denominator, the data used to calculate it, and whether it is reported as a ratio or percentage are stated. Normalization can help describe relative error; it does not make different prediction problems identical in difficulty.

Definition: For \(n\) observed outcomes \(y_1,\ldots,y_n\) and corresponding predictions \(\hat{y}_1,\ldots,\hat{y}_n\), define the errors as \(e_i=y_i-\hat{y}_i\). The RMSE is \(\sqrt{\frac{1}{n}\sum_{i=1}^{n}e_i^2}\). Given a specified outcome scale \(c>0\), the normalized RMSE (NRMSE) is \(\frac{\mathrm{RMSE}}{c}\). It may be reported as a percentage by multiplying the ratio by 100.

The positive scale \(c\) must be selected and named. Common choices include the mean of the outcomes, their range, and their standard deviation. These choices answer slightly different descriptive questions and can give different numerical results. For example, a mean-based NRMSE describes RMSE relative to the average outcome; a range-based NRMSE describes RMSE relative to the observed spread from minimum to maximum.

\mathrm{NRMSE}=\frac{\sqrt{\frac{1}{n}\sum_{i=1}^{n}(y_i-\hat{y}_i)^2}}{c}, \qquad \mathrm{NRMSE\%}=100\left(\frac{\mathrm{RMSE}}{c}\right)\%.

For a mean-based version, this tutorial uses \(c=\bar{y}=\frac{1}{n}\sum_i y_i\) when \(\bar{y}>0\). A range-based version uses \(c=\max_i(y_i)-\min_i(y_i)\), which must be positive. A standard-deviation version must also state its convention. Here, when used, the outcome standard deviation is calculated with divisor \(n\): \(c=\sqrt{\frac{1}{n}\sum_i(y_i-\bar{y})^2}\). Other conventions are possible, but they must not be silently substituted.

How the Choice of Scale Affects Interpretation

An NRMSE is dimensionless because its numerator and denominator have the same units. If both are measured in dollars, dollars cancel; if both are measured in thousands of dollars, thousands cancel. This helps communicate the size of prediction error relative to an outcome scale, but it does not remove the need to define that scale.

The denominator also determines how the metric responds to changes in the data. The range can be dominated by one extreme observation. The mean is unsuitable when it is zero and can be misleading when it is close to zero; for outcomes that can be negative, a signed mean may not even be a positive scale. The standard deviation describes spread around the mean, but it is zero when all outcomes are identical. In each such case, the chosen denominator may make the NRMSE undefined or uninformative.

Worked Example: One Error Set, Three Normalizations

A hypothetical model predicts four observed payment amounts. The observed values are 20, 25, 30, and 25 dollars. The predictions are 17, 21, 30, and 25 dollars. Calculate RMSE, then calculate mean-based and range-based NRMSE. Also show why the denominator needs to be reported.

Method. Calculate each error as observed minus predicted, find RMSE, and divide by each specified scale. This avoids treating “NRMSE” as if it had one universal denominator.

The errors are \(20-17=3\), \(25-21=4\), \(30-30=0\), and \(25-25=0\) dollars. Their squared errors are \(9\), \(16\), \(0\), and \(0\), for a total of \(25\) dollars squared. Thus,

\mathrm{RMSE}=\sqrt{\frac{25}{4}}=\sqrt{6.25}=2.5\text{ dollars}.

The outcome mean is \((20+25+30+25)/4=100/4=25\) dollars. The mean-based NRMSE is \(2.5/25=0.10\), or \(10\%\). The outcome range is \(30-20=10\) dollars, so the range-based NRMSE is \(2.5/10=0.25\), or \(25\%\).

Check. Both ratios are dimensionless, and both use the same RMSE. They differ because the denominators differ: 25 dollars for the mean and 10 dollars for the range. A report that says only “NRMSE = 10%” omits information needed to reproduce or interpret the result; it should identify mean normalization.

Why Normalization Cancels a Change of Units

A useful property of some scale choices is invariance to a change of measurement units. Suppose every outcome and prediction is multiplied by the same nonzero conversion factor \(b\), such as converting dollars to cents. Each error is then multiplied by \(b\), so the RMSE is multiplied by \(|b|\). If the denominator is the range or standard deviation, it is also multiplied by \(|b|\). Their ratio is unchanged.

This argument does not apply in the same way to every denominator. If outcomes are shifted as well as rescaled, the mean changes by the shift, while the errors do not change from the shift alone. Consequently, mean-based NRMSE is not generally invariant to adding a constant to all outcomes. This matters for unit conversions that include an offset, such as converting a temperature scale.

Justification: If \(y_i^*=a+b y_i\) and \(\hat{y}_i^*=a+b\hat{y}_i\), then \(e_i^*=y_i^*-\hat{y}_i^*=b(y_i-\hat{y}_i)=b e_i\). Therefore \(\mathrm{RMSE}^*=|b|\mathrm{RMSE}\). The range and standard deviation of the transformed outcomes are also multiplied by \(|b|\), so range- and standard-deviation-based NRMSE remain unchanged when \(b\ne0\). The mean, however, becomes \(a+b\bar{y}\); the additive term \(a\) does not generally cancel. Thus mean-based NRMSE is not generally unchanged by an additive shift.

Comparing Outcomes on Different Scales

To compare error relative to outcome magnitude across two prediction tasks, calculate RMSE and NRMSE separately for each task using an explicit and consistent normalization rule. The example below uses each task’s positive outcome mean. The resulting comparison is about error relative to each mean, not about identical model difficulty, business importance, or error consequences.

Worked Example: Comparing Claim and Benefit Predictions

A hypothetical analyst evaluates two models on four held-out records per task. For claim amounts in dollars, the observed outcomes are 1,800, 1,900, 2,100, and 2,200, with prediction errors \(100,-100,200,-200\) dollars. For a separate benefit measure in thousands of dollars, the observed outcomes are 1, 2, 3, and 4, with prediction errors \(0.2,-0.2,0.4,-0.4\). Compare the two tasks using mean-based NRMSE.

Method. Calculate each task’s RMSE in its own units, divide by that task’s observed mean, and compare the resulting percentages. The common mistake this avoids is comparing the raw RMSEs without accounting for the very different outcome scales.

For claim amounts, the mean outcome is \((1{,}800+1{,}900+2{,}100+2{,}200)/4=8{,}000/4=2{,}000\) dollars. The squared errors sum to \(100^2+(-100)^2+200^2+(-200)^2=10{,}000+10{,}000+40{,}000+40{,}000=100{,}000\) dollars squared. Therefore,

\mathrm{RMSE}_{\text{claim}} =\sqrt{\frac{100{,}000}{4}} =\sqrt{25{,}000} \approx 158.11\text{ dollars}, \qquad \mathrm{NRMSE}_{\text{claim}} =\frac{158.11}{2{,}000}\times100\% \approx 7.91\%.

For the benefit measure, the mean is \((1+2+3+4)/4=2.5\) thousand dollars. The squared errors sum to \(0.2^2+(-0.2)^2+0.4^2+(-0.4)^2=0.04+0.04+0.16+0.16=0.40\) in squared thousands of dollars. Therefore,

\mathrm{RMSE}_{\text{benefit}} =\sqrt{\frac{0.40}{4}} =\sqrt{0.10} \approx 0.3162\text{ thousand dollars}, \qquad \mathrm{NRMSE}_{\text{benefit}} =\frac{0.3162}{2.5}\times100\% \approx 12.65\%.

Using mean normalization, the claim task has the smaller error relative to its mean: about \(7.91\%\), compared with \(12.65\%\) for the benefit task. This does not establish that the claim model is better for every purpose. The tasks have different outcome distributions and may have different consequences for prediction errors. The comparison is specifically relative to each task’s observed mean.

Check. The units cancel within each ratio: dollars cancel for claims, and thousands of dollars cancel for benefits. The separate denominators and the mean-based rule are stated, so the comparison can be interpreted correctly.

Reporting the Denominator and Its Basis

An outcome scale can be calculated from the evaluation outcomes, from a training set, or from a fixed reference chosen in advance. These choices are not interchangeable. Using the evaluation outcomes gives a scale tied to that evaluation sample. Using a training-set scale or a fixed reference can support consistent reporting across evaluation periods, provided the choice is applied consistently and clearly described.

When comparing models evaluated on the same outcomes using the same positive denominator, the model with lower RMSE also has lower NRMSE: dividing both RMSE values by the same positive number preserves their ordering. The normalization does not change that within-sample ranking. Its value is in expressing the error relative to a scale, especially when comparing outcomes whose units or magnitudes differ.

A concise report should identify at least the following:

the RMSE and the unit of the outcome;
the normalization rule, such as mean, range, or standard deviation;
the calculated denominator and its units;
whether the result is a ratio or percentage; and
which data were used to calculate the denominator.

Worked Example: A Range-Based NRMSE and a Denominator Trap

A hypothetical model predicts four weekly service costs. The observed costs are 40, 50, 60, and 70 dollars, and the prediction errors are \(2,-2,2,-2\) dollars. Calculate range-based NRMSE. Then assess a report that divides RMSE by the maximum outcome rather than the range.

Method. Calculate RMSE, find the range as maximum minus minimum, and divide. The common mistake is to use the maximum observed value as though it were the range.

The squared errors are \(4,4,4,4\), which sum to \(16\). Therefore \(\mathrm{RMSE}=\sqrt{16/4}=\sqrt{4}=2\) dollars. The range is \(70-40=30\) dollars, so the range-based NRMSE is \(2/30=0.0667\), or \(6.67\%\) to two decimal places.

The incorrect calculation uses \(70\) dollars as the denominator: \(2/70=0.02857\), or \(2.86\%\) to two decimal places. This is not range-based NRMSE because 70 dollars is the maximum, not the difference between the maximum and minimum. The corrected value is \(6.67\%\). A report should state “range-based NRMSE; observed range \(=30\) dollars,” not merely “normalized RMSE.”

Check. The corrected ratio uses two quantities in dollars, and \(2/30\) is approximately \(0.0667\). The incorrect denominator produces a smaller value but answers a different, unstated normalization question.

Common Interpretation Errors

Assuming NRMSE has one definition. A percentage without its denominator cannot be reproduced or fully interpreted.
Using a scale that is zero or nearly zero. A zero denominator makes the ratio undefined; a very small denominator can make it extremely large. Check whether the scale is meaningful for the outcomes.
Comparing unlike tasks as though normalization removes all differences. A dimensionless metric does not adjust for differing outcome distributions, irreducible variability, or consequences of error.
Changing the denominator rule between models. For a fair comparison, calculate NRMSE using the same stated rule and the same evaluation basis for the models being compared.
Confusing percentage with percentage points. NRMSE is a ratio that can be expressed as a percentage; it is not automatically an accuracy rate or a percentage of predictions that are correct.

Key takeaway: NRMSE is RMSE divided by a stated positive outcome scale. Always name the denominator, report how it was calculated, and interpret cross-scale comparisons as relative-error summaries rather than proof that two prediction tasks are equally difficult.

Check Your Understanding

For each problem, identify the stated scale and calculate the requested NRMSE. Round percentages to two decimal places where needed.

A model’s errors on four records are \(1,-1,2,-2\). The mean observed outcome is 50. Calculate mean-based NRMSE as a percentage. (Answer: RMSE \(=\sqrt{10/4}\approx1.5811\); NRMSE \(\approx3.16\%\).)
A validation set has RMSE of 6.4 units and an observed outcome range of 80 units. Calculate range-based NRMSE as a percentage. (Answer: \(6.4/80\times100\%=8.00\%\).)
Observed outcomes are 4, 8, and 12, and prediction errors are \(1,-1,2\). Calculate mean-based NRMSE. (Answer: RMSE \(=\sqrt{6/3}=\sqrt{2}\approx1.4142\); mean \(=8\); NRMSE \(\approx17.68\%\).)
A model has RMSE 3 and observed range 30. If both outcomes and predictions are converted to a unit 100 times as large, what is the range-based NRMSE? (Answer: \(300/3{,}000=10.00\%\), unchanged.)
A set of outcomes has mean 10 and a model has RMSE 2. If 100 is added to every observed outcome and every prediction, what is the new mean-based NRMSE? (Answer: the RMSE remains 2, the mean becomes 110, and NRMSE is \(2/110\times100\%\approx1.82\%\).)

What You'll Learn