Why a Performance Claim Needs a Baseline
A statement such as “the model has a mean absolute error of 4” gives a number, but not enough context to judge whether the result is useful. Is 4 better than a simple forecast, the current process, or an acceptable operational standard? A decision-maker needs a relevant point of comparison before the model’s performance can be interpreted.
A baseline is a specified reference method or standard against which model performance is assessed. It is not automatically a weak method, nor is it necessarily the method currently used in practice. The appropriate choice depends on the predictive question and intended action. A simple historical average may be a useful reference for a new demand forecast; an existing manual process may be the more relevant benchmark when deciding whether to replace that process.
The report should name the baseline plainly. “A constant forecast equal to the mean from the earlier reference period” is more informative than “the naïve model.” “The existing review process” is useful only if readers know what that process produces and how its predictions were assessed. The comparison should also say whether lower or higher values of the chosen metric indicate better performance.
Making the Comparison Fair
A performance comparison is interpretable only when the baseline and model are assessed on comparable terms. In particular, use the same observed outcomes and metric, predict the same target over the same time horizon, and respect the information that would have been available when each prediction was made. If one method receives extra information or is scored on easier cases, a difference in scores cannot be attributed simply to the methods.
For a lower-is-better error measure such as mean absolute error, let \(B\) denote the baseline’s error and \(M\) the model’s error on the assessment cases. The absolute reduction in error is \(B-M\). When \(B>0\), the relative reduction, expressed as a proportion of baseline error, is:
A positive value means the model’s error is lower; zero means the two errors are equal; and a negative value means the model’s error is higher. Multiplying by 100 expresses the result as a percentage. This percentage is relative to the baseline, not relative to the model.
Worked Comparisons
Worked Example: Comparing Two Demand Forecasts
A hypothetical service team assesses daily demand forecasts on five days. The observed demand is 40, 50, 60, 70, and 80 units. A baseline predicts 50 units each day. A model predicts 45, 50, 55, 65, and 75 units. Compare the methods using mean absolute error (MAE), then calculate the model’s relative MAE reduction.
Method. Compute each method’s absolute error on the same five days, average those errors, and use the baseline MAE as the denominator for the relative reduction. This comparison avoids treating an unscaled difference in error as a percentage improvement.
For the baseline, the absolute errors are \(|40-50|=10\), \(|50-50|=0\), \(|60-50|=10\), \(|70-50|=20\), and \(|80-50|=30\). Their sum is \(10+0+10+20+30=70\), so:
For the model, the absolute errors are \(|40-45|=5\), \(|50-50|=0\), \(|60-55|=5\), \(|70-65|=5\), and \(|80-75|=5\). Their sum is \(5+0+5+5+5=20\), so:
The absolute reduction is \(14-4=10\) units. The relative reduction is \(10/14=0.7142857\), or 71.4% when rounded to one decimal place. Thus, on these five assessed days, the model has lower MAE than the stated baseline. This result does not establish that the model will retain this advantage on future days or that acting on its forecasts will improve service outcomes.
Common mistake avoided: Dividing the reduction by the model’s MAE would give \(10/4=2.5\), or 250%. That is a comparison relative to the model error, not the conventional reduction relative to the baseline. The baseline denominator answers the question, “What fraction of the baseline error was reduced?”
Worked Example: Using a Historical Reference Forecast
A hypothetical analyst compares a model with a reference forecast that was set, before the assessment period, at 160 units for each day. On four assessment days, actual totals are 120, 150, 180, and 210 units. The model predicts 130, 155, 170, and 190 units. Compare the two methods by MAE and calculate relative reduction.
Method. Treat the stated 160-unit reference as the baseline, rather than recalculating it from the assessment outcomes. This preserves the meaning of a forecast that could have been made before those outcomes were known.
The baseline’s absolute errors are \(|120-160|=40\), \(|150-160|=10\), \(|180-160|=20\), and \(|210-160|=50\). Their sum is \(40+10+20+50=120\), giving:
The model’s absolute errors are \(|120-130|=10\), \(|150-155|=5\), \(|180-170|=10\), and \(|210-190|=20\). Their sum is \(10+5+10+20=45\), giving:
The absolute reduction is \(30-11.25=18.75\) units. Relative to the baseline, the reduction is \(18.75/30=0.625\), or 62.5%. The arithmetic supports the statement that the model has lower MAE on these assessment days. It does not establish that the model would outperform every reasonable alternative: that conclusion depends on which reference methods are relevant to the decision.
Common mistake avoided: Using the assessment outcomes to redefine the reference forecast after the fact would change the stated comparator. Keep the benchmark’s construction explicit, and do not present a reference calculated with information unavailable at the time of prediction as though it were an operational forecast.
Why the Relative-Reduction Formula Works
The relative reduction describes the absolute difference as a fraction of the starting error. If the baseline error is \(B>0\) and the model error is \(M\), then the difference is \(B-M\). Dividing that difference by the baseline gives the fraction of the baseline error that is no longer present:
For example, suppose a baseline MAE is \(50/3\) and a model MAE is 5, in the same units. The absolute reduction is \(50/3-5=35/3\). Relative to the baseline, the reduction is \((35/3)/(50/3)=35/50=0.70\), or 70%. Dividing by the model MAE instead gives \((35/3)/5=7/3=2.333\), or 233.3%. The latter calculation is a ratio of the reduction to the model error; it is not the percentage reduction relative to the baseline. In particular, it is not 7 or 700%.
A Lower Error Score Is Not Always a Better Decision
Worked Example: When the Decision Has Unequal Error Costs
A hypothetical facility forecasts a daily quantity with an actual value of 100 units on each of four assessed days. The baseline predicts 102 units on all four days. A model predicts 95, 100, 100, and 100 units. For a simple planning illustration, assume each unit of overprediction costs $1 and each unit of underprediction costs $10. Compare MAE and total cost under this stated rule.
Method. First compare the methods using MAE. Then calculate the decision cost separately, applying the stated cost to each day’s overprediction or underprediction. This avoids assuming that equal absolute errors have equal consequences when the scenario says otherwise.
The baseline’s absolute error is 2 units on each day, so its total absolute error is \(2+2+2+2=8\) units and its MAE is \(8/4=2\) units. The model’s absolute errors are 5, 0, 0, and 0 units, so its total absolute error is 5 units and its MAE is \(5/4=1.25\) units. The model’s relative MAE reduction is \((2-1.25)/2=0.375\), or 37.5%.
Now apply the cost rule. The baseline overpredicts by 2 units on each day. Its total cost is \(4\times2\times\$1=\$8\). The model underpredicts by 5 units on one day and is exact on the other three. Its total cost is \(5\times\$10=\$50\). Under this cost rule, the model has lower MAE but higher total cost.
The conclusion is not that MAE is inherently wrong. MAE answers a particular error-summary question; it does not encode the unequal costs specified here. A report supporting this decision should show a measure tied to the consequences of the action, or otherwise explain the mismatch. Common mistake avoided: claiming that lower MAE alone proves the model is preferable for an operational decision.
What Improvement Does—and Does Not—Establish
A well-chosen baseline makes a performance claim more interpretable. It shows whether a model improves on a named reference under a stated measure. But a comparison answers only that bounded question. It does not, by itself, show that the model’s predictions caused an improvement, that a proposed action is worthwhile, or that the comparison applies to every population or operating condition of interest.
A report should avoid turning “lower error than this baseline on these assessed cases” into “the model will improve results in production.” The latter claim concerns how the model is used and what happens after decisions change; it is not established by the metric comparison alone. Likewise, a model can beat a simple reference while still failing a practical requirement, such as being accurate enough for a particular action.
Before writing a performance claim, use the following sequence:
State whether the comparison is with a simple forecast, a current process, or a defined standard. Say how its predictions or threshold were set.
Confirm that the baseline and model are assessed on the same outcomes, target, horizon, and metric, with information appropriate to prediction time.
Give the baseline and model scores, their absolute difference, and—when defined and useful—the relative change with its denominator made clear.
Say what the comparison supports, then distinguish that result from a claim about deployment, causation, or decision value that the comparison alone cannot establish.
For example, a concise report might say: “On the assessed days, the model’s MAE was 4 units, compared with 14 units for the constant 50-unit reference forecast, a 71.4% reduction relative to that reference. This comparison does not by itself establish the effect of using the forecasts to change staffing.” This wording states the result, identifies the comparator, and limits the inference.
The benchmark should also be meaningful for the intended audience. A simple reference provides a useful floor for asking whether the model adds predictive value beyond a basic rule. An incumbent-process comparison addresses whether the model performs better than the method it might replace. These are different questions, so a single baseline should not be described as answering both unless it genuinely represents both.
This disciplined comparison is aligned with the SOA Exam PA (Predictive Analytics) syllabus: it connects a reported performance result to a clear reference and an appropriately limited decision claim.
Check Your Understanding
For each problem, identify the comparison being made and check the arithmetic before interpreting the result.
- On four days, actual demand is 30, 40, 50, and 60 units. A baseline predicts 40 each day; a model predicts 35, 42, 48, and 55. Find both MAEs and the model’s relative reduction from baseline. (Answer: Baseline MAE \(=10\); model MAE \(=3.5\); relative reduction \(=65\%\).)
- Actual values are 10, 10, and 10. A baseline predicts 12 each time; a model predicts 9, 10, and 13. Compare RMSE and calculate the relative RMSE reduction, rounding the RMSEs to three decimals and the percentage to one decimal place. (Answer: Baseline RMSE \(=2.000\); model RMSE \(=\sqrt{10/3}\approx1.826\); relative reduction \(\approx8.7\%\).)
- Actual values are 20, 30, and 40. A baseline predicts 25 each time; a model predicts 22, 32, and 36. Calculate both MAEs and the model’s relative reduction. (Answer: Baseline MAE \(=25/3\approx8.333\); model MAE \(=8/3\approx2.667\); relative reduction \(=68\%\).)
- A model has MAE 4, while a named reference has MAE 6 on the same assessment cases. Calculate the absolute and relative reductions, and state one conclusion that this comparison does not establish. (Answer: Absolute reduction \(=2\); relative reduction \(=2/6=33.3\%\); it does not by itself establish that adopting the model will improve operational outcomes.)
- Actual values are 50 on each of three days. A baseline predicts 52 each day, while a model predicts 45, 50, and 50. Under a hypothetical cost rule of $1 per unit overpredicted and $8 per unit underpredicted, compare MAE and total cost. (Answer: Baseline MAE \(=2\) and total cost \(=\$6\); model MAE \(=5/3\approx1.667\) and total cost \(=\$40\). The model has lower MAE but higher cost under this rule.)