Baseline-Relative Prediction Error

Why Compare With a Baseline?

An error value is easier to judge when it answers a practical question: is the model better than a simple alternative we could actually use? An RMSE of 12 units, for example, has limited meaning by itself. If a simple operational forecast has an RMSE of 20 units, the model may be useful; if the simple forecast has an RMSE of 8 units, the model may not justify its complexity.

A baseline is a specified reference prediction, not a claim that the reference is optimal. It might be a fixed operational forecast, a historical average available at prediction time, or another simple rule. The comparison is meaningful only when the model and baseline are evaluated on the same outcomes using the same error metric.

Definition: For a selected error metric $E$, let $E_{\text{model}}$ be the model's error and $E_{\text{baseline}}$ the baseline predictor's error on the same evaluation observations. If $E_{\text{baseline}}>0$, the baseline-relative improvement is $$ I = \frac{E_{\text{baseline}}-E_{\text{model}}}{E_{\text{baseline}}} = 1-\frac{E_{\text{model}}}{E_{\text{baseline}}}. $$ A positive $I$ means the model has lower error; $I=0$ means equal error; a negative $I$ means the model has higher error. Multiplying $I$ by 100 expresses the result as a percentage.

The metric $E$ must be stated. For example, MAE and RMSE both measure prediction error, but they do not penalize large errors in the same way. A relative improvement based on MAE answers a different question from one based on RMSE. The comparison does not make those metrics interchangeable.

What Makes a Fair Baseline Comparison?

For each evaluation observation $j$, let $y_j$ be the observed value, $\hat{y}_{j,\text{model}}$ the model prediction, and $\hat{y}_{j,\text{baseline}}$ the baseline prediction. MAE is the average absolute error, and RMSE is the square root of the average squared error. Calculate each metric for both predictors on the same set of $n$ observations.

\operatorname{MAE}_{\text{model}} =\frac{1}{n}\sum_{j=1}^n |y_j-\hat{y}_{j,\text{model}}|, \qquad \operatorname{MAE}_{\text{baseline}} =\frac{1}{n}\sum_{j=1}^n |y_j-\hat{y}_{j,\text{baseline}}|

\operatorname{RMSE}_{\text{model}} =\sqrt{\frac{1}{n}\sum_{j=1}^n(y_j-\hat{y}_{j,\text{model}})^2}, \qquad \operatorname{RMSE}_{\text{baseline}} =\sqrt{\frac{1}{n}\sum_{j=1}^n(y_j-\hat{y}_{j,\text{baseline}})^2}

The baseline must also be operationally credible: its predictions should use information that would have been available when predictions were made. If a baseline is fitted using the evaluation outcomes themselves, it is not a fair forecast comparison. A very weak baseline can make a model's relative improvement look impressive without showing that the model is useful in practice, so the baseline choice should be explained.

Specify the evaluation set.
Use the same observations for the model and baseline, with the same outcome definition.

Choose the error metric and baseline.
State what the baseline predicts and calculate both errors using the same metric.

Calculate and interpret relative improvement.
Use $1-E_{\text{model}}/E_{\text{baseline}}$, then report the sign and size in context.

Deriving the Relative Improvement Formula

The numerator $E_{\text{baseline}}-E_{\text{model}}$ is the amount by which the model reduces error, measured in the units of the error metric. Dividing that difference by the baseline error expresses the reduction as a fraction of the reference error. Algebra gives the equivalent ratio form:

\frac{E_{\text{baseline}}-E_{\text{model}}}{E_{\text{baseline}}} = \frac{E_{\text{baseline}}}{E_{\text{baseline}}} -\frac{E_{\text{model}}}{E_{\text{baseline}}} = 1-\frac{E_{\text{model}}}{E_{\text{baseline}}}.

This explains why the result is dimensionless: both errors have the same units, so their ratio cancels those units. If model error is half the baseline error, then $I=1-0.5=0.5$, a 50% reduction. If model error is 1.2 times baseline error, then $I=1-1.2=-0.2$, a 20% increase in error relative to the baseline.

Worked Example: Relative Improvement for MAE and RMSE

A hypothetical service center evaluates daily work volume on four days. The observed volumes are 10, 12, 14, and 20 jobs. A model predicts 11, 10, 15, and 18 jobs. An operational rule, set before these days, predicts 14 jobs on each day. Compare the model with the rule using MAE and RMSE. The method is to calculate each metric separately on the same four days, then calculate relative improvement. This avoids treating an improvement under one metric as an improvement under every metric.

MAE for the model. The absolute errors are $|10-11|=1$, $|12-10|=2$, $|14-15|=1$, and $|20-18|=2$. Their sum is $1+2+1+2=6$, so model MAE is $6/4=1.5$ jobs.

MAE for the baseline. Its absolute errors are $|10-14|=4$, $|12-14|=2$, $|14-14|=0$, and $|20-14|=6$. Their sum is $4+2+0+6=12$, so baseline MAE is $12/4=3$ jobs.

The MAE-based improvement is $1-1.5/3=1-0.5=0.5$, or 50%.

RMSE for the model. The squared errors are $1^2=1$, $2^2=4$, $1^2=1$, and $2^2=4$. Their sum is $10$, so model RMSE is $\sqrt{10/4}=\sqrt{2.5}\approx1.5811$ jobs.

RMSE for the baseline. The squared errors are $4^2=16$, $2^2=4$, $0^2=0$, and $6^2=36$. Their sum is $56$, so baseline RMSE is $\sqrt{56/4}=\sqrt{14}\approx3.7417$ jobs. The RMSE-based improvement is $1-\sqrt{2.5}/\sqrt{14}\approx1-0.4226=0.5774$, or 57.74%, rounded to two decimal places. The model is better under both metrics here, but the reported improvement depends on the metric. The calculations check: both model errors are smaller than their corresponding baseline errors.

Worked Example: Negative Improvement Means Worse Performance

A hypothetical equipment team observes 40, 50, 60, 70, and 80 service requests over five periods. A simple baseline predicts 60 requests each period, while a new model predicts 50 each period. Compare their MAE and RMSE. The method is the same-metric comparison on the same five periods; the trap to avoid is reporting a negative percentage as though it were a positive improvement.

Model MAE. The absolute errors are $10, 0, 10, 20, 30$. Their sum is $70$, giving MAE $70/5=14$ requests.

Baseline MAE. The absolute errors are $20, 10, 0, 10, 20$. Their sum is $60$, giving MAE $60/5=12$ requests. Thus $I=1-14/12=1-1.1667\approx-0.1667$, or -16.67% to two decimal places. Equivalently, model MAE is 16.67% higher than baseline MAE.

Model RMSE. The squared errors are $100, 0, 100, 400, 900$, with sum $1{,}500$. RMSE is $\sqrt{1{,}500/5}=\sqrt{300}\approx17.3205$ requests.

Baseline RMSE. The squared errors are $400, 100, 0, 100, 400$, with sum $1{,}000$. RMSE is $\sqrt{1{,}000/5}=\sqrt{200}\approx14.1421$ requests. Therefore $I=1-\sqrt{300}/\sqrt{200}=1-\sqrt{1.5}\approx-0.2247$, or -22.47% to two decimal places. The model is worse under both metrics. The larger deterioration under RMSE reflects the metric's greater sensitivity to large errors; the negative sign is not a calculation failure.

Build the Baseline Without Using Evaluation Outcomes

A common baseline is a constant prediction equal to the mean of the training outcomes. This can be reasonable when the mean is available before the evaluation period and represents a plausible simple forecast. The mean must be calculated from the training data, not from the evaluation outcomes. The distinction matters even if the baseline rule is simple: using evaluation outcomes to choose its prediction gives it information unavailable at prediction time.

Worked Example: A Training-Mean Baseline

A hypothetical planning team has training-period volumes of 8, 10, 12, and 14 units. In a separate evaluation period, the observed volumes are 9, 13, and 15. A model predicts 10, 12, and 14. Compare the model with a constant baseline equal to the training mean, using MAE and RMSE. The method is to fit the baseline from training data only, then evaluate both predictors on the three held-out observations. This avoids leakage from the evaluation outcomes.

Set the baseline. The training mean is $(8+10+12+14)/4=44/4=11$. The baseline therefore predicts 11 for each evaluation observation.

Compare MAE. Model absolute errors are $|9-10|=1$, $|13-12|=1$, and $|15-14|=1$, so model MAE is $3/3=1$. Baseline absolute errors are $|9-11|=2$, $|13-11|=2$, and $|15-11|=4$, so baseline MAE is $8/3\approx2.6667$. Relative improvement is $1-1/(8/3)=1-3/8=5/8=0.625$, or 62.50%.

Compare RMSE. Model squared errors are $1,1,1$, so model RMSE is $\sqrt{3/3}=1$. Baseline squared errors are $4,4,16$, so baseline RMSE is $\sqrt{24/3}=\sqrt{8}\approx2.8284$. Relative improvement is $1-1/\sqrt{8}\approx0.6464$, or 64.64% to two decimal places. Both results favor the model.

If the evaluation outcomes were incorrectly used to set the constant baseline, its prediction would be their mean, $(9+13+15)/3=37/3\approx12.3333$. That is not the specified training-mean baseline and uses information from the evaluation period. For instance, its MAE would be $(10/3+2/3+8/3)/3=20/9\approx2.2222$, a different comparison. The correct calculation above uses the pre-evaluation training mean of 11.

Common Traps and Interpretation

A relative improvement is a comparison, not a stand-alone measure of model quality. Always report the metric, the baseline, and enough context to understand the size of the errors. These checks help prevent the most common misinterpretations:

Reversing the subtraction. Use $(E_{\text{baseline}}-E_{\text{model}})/E_{\text{baseline}}$. If the model error is larger, the result should be negative.
Comparing different evaluation cases. Missing observations or different subsets can change the comparison. Use the same outcomes for both predictors.
Changing the metric mid-calculation. Compare MAE with MAE or RMSE with RMSE; do not divide model MAE by baseline RMSE.
Using an unhelpful baseline. A model can show a large relative improvement over a weak reference and still have errors too large for the intended use.
Ignoring a zero or very small baseline error. If $E_{\text{baseline}}=0$, the relative improvement formula is undefined. If it is positive but very close to zero, the ratio can be unstable and should be interpreted cautiously.

For example, a result of $-20\%$ means the model's error is 20% larger than the baseline error under the selected metric. It does not mean that the model has a 20% probability of being wrong. A result of 0% means equal measured error on the specified evaluation set, not necessarily equal performance on every future population.

Key takeaway: Relative improvement is $1-E_{\text{model}}/E_{\text{baseline}}$. It is positive when the model reduces the selected error, zero when errors are equal, and negative when the model is worse. The comparison is interpretable only when the baseline is credible and both errors are calculated consistently.

Check Your Understanding

For each problem, use the stated error metric and compare predictors on the same observations.

A baseline has MAE 5 units and a model has MAE 4 units. Calculate the baseline-relative improvement. (Answer: 20%)
Observed values are 2, 4, and 8. A baseline predicts 5 for each; a model predicts 3, 5, and 7. Calculate the RMSE-based relative improvement, rounded to two decimal places. (Answer: 60.26%)
A baseline has MAE 3 and a model has MAE 5. Calculate the relative improvement and state what it means. (Answer: -66.67%; the model's MAE is 66.67% higher than the baseline's.)
A baseline has zero RMSE on an evaluation set, and a model also has zero RMSE. Can the relative improvement formula be evaluated? (Answer: No; it has a zero denominator and is undefined.)
A constant baseline is intended to predict the training-set mean. Training outcomes are 6, 8, and 10; evaluation outcomes are 12 and 18. What baseline value should be used, and why? (Answer: 8; calculate the mean from training outcomes, not from the evaluation outcomes.)

What You'll Learn