Why Compare a Model With a Baseline?
An error metric has little meaning in isolation. A mean absolute error of 4 outcome units might be useful if a simple alternative typically misses by 10, but unimpressive if the alternative misses by 3. Comparing a regression model with a baseline prediction gives the metric a reference point: it shows whether the model improves on a reasonable, simpler way of making the same predictions.
A baseline is not necessarily a competing statistical model. It may be a constant prediction, such as the mean or median outcome in the training data, or a simple prediction based on information routinely available when a prediction is needed. The comparison is meaningful only when the model and baseline predict the same outcome for the same test cases, using information available at prediction time.
Skill Scores and Relative Improvement
A skill score expresses the model’s error relative to the baseline’s error. For either MAE or RMSE, use the same metric for the model and baseline:
The metric in the numerator and denominator must be the same: compare MAE with baseline MAE, or RMSE with baseline RMSE. A positive score means the model’s metric is smaller than the baseline’s; a score of zero means they are equal; a negative score means the model’s metric is larger. Multiplying the score by 100 expresses relative improvement as a percentage when the score is positive. For example, a score of \(0.20\) means the model’s metric is 20% lower than the baseline’s.
A skill score is a relative comparison, not an error measured in outcome units. If the model MAE is 6 and the baseline MAE is 8, the score is \(1-6/8=0.25\), or a 25% reduction in MAE. The model still misses by an average of 6 outcome units; the score does not change that absolute interpretation. If the baseline metric is zero, this ratio is undefined, so a skill score cannot be calculated by this formula.
Worked Comparisons
Worked Example: Comparing MAE With a Training-Set Median
Suppose a hypothetical model predicts delivery time in hours for four test orders. The training outcomes have median delivery time 13 hours, so a simple baseline predicts 13 hours for each test order. The model and baseline are evaluated on exactly the same test orders.
| Order | Observed time \(y_i\) (hours) | Model prediction (hours) | Baseline prediction (hours) |
|---|---|---|---|
| 1 | 10 | 11 | 13 |
| 2 | 12 | 13 | 13 |
| 3 | 15 | 14 | 13 |
| 4 | 18 | 16 | 13 |
The model’s absolute errors are \(1,1,1,\) and \(2\) hours. Therefore,
The baseline’s absolute errors are \(3,1,2,\) and \(5\) hours:
The MAE skill score is
The model’s test-set MAE is about 54.5% lower than the baseline’s MAE. In outcome units, its average absolute error is 1.25 hours rather than 2.75 hours. The comparison shows improvement on these test orders; it does not establish that the model will have exactly the same performance on every future set of orders.
Worked Example: Comparing RMSE With a Training-Set Mean
In a second hypothetical example, a model predicts a product’s weekly sales in units. The training data have mean sales of 25 units, giving a constant baseline prediction of 25 for each test week. On four test weeks, the observed sales are 20, 24, 28, and 32 units. The model predicts 21, 22, 30, and 31 units. The test outcomes were not used to calculate the training mean.
The model’s prediction errors, written as observed minus predicted, are \(-1,2,-2,\) and \(1\) units. Its RMSE is
The baseline’s errors are \(20-25=-5\), \(24-25=-1\), \(28-25=3\), and \(32-25=7\) units. Thus,
Using RMSE for both predictions, the skill score is
The model’s RMSE is about 65.5% lower than the baseline’s RMSE on these four test weeks. RMSE gives greater influence to large errors because it squares them before averaging. The model’s score therefore describes its relative performance under squared-error emphasis, not a claim that every prediction is 65.5% closer.
Choose a Baseline That Fits the Task
A baseline is useful only if it represents a fair and relevant alternative. The mean and median are common constant baselines: the training-set mean is a natural choice under squared-error loss, while the training-set median is a natural choice under absolute-error loss. These are not the only possible baselines. In some settings, a simple rule based on information available at prediction time is more informative than a constant.
For example, when predicting next month’s demand, a reasonable simple alternative might be the most recently observed month’s demand. In contrast, that rule would not be a fair baseline if the most recent value would not actually be known when the prediction is made. The prediction task determines what information is available and which simple strategy counts as a meaningful comparison.
Worked Example: Comparing With a Task-Specific Baseline
A hypothetical store wants to predict each coming week’s demand for a product. At the start of each week, the previous week’s demand is already known. A simple persistence baseline predicts that the coming week will have the same demand as the previous week. A regression model and that baseline make predictions for the same four weeks.
| Week | Observed demand (units) | Previous week’s demand, used as baseline (units) | Model prediction (units) |
|---|---|---|---|
| 1 | 40 | 38 | 41 |
| 2 | 42 | 40 | 43 |
| 3 | 41 | 42 | 44 |
| 4 | 47 | 44 | 45 |
The model’s absolute errors are \(1,1,3,\) and \(2\) units, so its MAE is
The baseline’s errors in absolute value are \(2,2,1,\) and \(3\) units. Its MAE is
The MAE skill score is
The regression model’s MAE is 12.5% lower than the persistence baseline’s MAE on these four weeks. This comparison is useful because the baseline uses information that really would have been available when the weekly forecasts were made. A training-set mean could also be calculated, but it would answer a different practical question: whether the model improves on a constant prediction rather than on the store’s simple recent-demand rule.
Fairness and Interpretation of the Comparison
The comparison should follow the same prediction and evaluation conditions for both methods. In particular, calculate both metrics on the same test cases, with the same outcome values and the same outcome units. Choose the metric before comparing results; an MAE skill score and an RMSE skill score answer different questions because the metrics treat error sizes differently.
The baseline’s rule and any values needed to apply it should be determined without using the test outcomes. For a mean or median baseline, estimate the constant from the training data. For a task-specific rule, use only inputs that would genuinely be available at the time each prediction is made. Otherwise, the baseline has information it could not have in practice, or it has been adapted to the answers. Either problem makes the comparison misleading.
A negative score is informative rather than a calculation failure. For example, if model MAE is 5 units and baseline MAE is 4 units, then \(1-5/4=-0.25\): the model’s MAE is 25% higher than the baseline’s MAE. That result suggests the model did not improve on this baseline for this metric and evaluation set. It does not, by itself, identify why.
Specify the outcome, prediction time, and information available at that time.
Select a simple strategy that could actually be used for the same task, such as a training-set mean, a training-set median, or a feasible task-specific rule.
Calculate the chosen metric for the model and baseline using the same observed outcomes.
Use \(1-\text{model metric}/\text{baseline metric}\), and state the metric, outcome units, baseline, and evaluation context.
Common Mistake: A Baseline Chosen With the Answers
A tempting but invalid procedure is to choose a constant baseline from the test outcomes themselves—for example, to use their mean or median—then describe the comparison as if that baseline had been available before the test cases were observed. This uses the evaluation outcomes to construct the predictions being evaluated. It no longer represents the intended prediction-time alternative, and it can make the comparison look better or worse than a genuinely available baseline would.
The correction is to set the baseline rule using training data or a clearly specified prediction-time rule, then apply it to the test cases without using their outcomes. After both sets of predictions are made, use the observed test outcomes to calculate the metrics. That keeps the test set in its role as an evaluation set.
Check Your Understanding
Answer each question using the definitions and comparison rules in this tutorial.
- A model has test MAE 3.6 minutes and a baseline has test MAE 4.8 minutes. Calculate the MAE skill score and interpret it in context.
- Why is it not appropriate to compare a model’s MAE with a baseline’s RMSE using the skill-score formula?
- A model has RMSE 7 units and a baseline has RMSE 5 units. Calculate the skill score and explain what its sign means.
- For forecasting next-day demand, name one possible task-specific baseline and state what information must be available to use it fairly.
- Why should a constant mean or median baseline generally be calculated from training outcomes rather than test outcomes?