Why Compare Against a Baseline?
A candidate model’s validation score is difficult to interpret in isolation. A Brier score of 0.14 or an RMSE of 1.8 may sound good, but whether it represents useful predictive value depends on the outcome and the difficulty of the prediction task. A simple baseline supplies a reference: it answers how well one could predict without using the candidate model’s explanatory structure.
For a binary outcome, a natural probability baseline predicts the event prevalence in the training data for every validation observation. For a quantitative outcome evaluated by squared error, a natural baseline predicts the training mean for every validation observation. A GLM, penalized regression, or tree ensemble adds value when it improves the chosen held-out metric relative to the matching baseline, under the same validation design.
Define the Baseline and the Comparison
Suppose a training set contains \(n_{\text{tr}}\) binary outcomes \(y_i\), where \(y_i=1\) denotes the event and \(y_i=0\) denotes no event. The training prevalence is the proportion of training outcomes equal to 1. For a held-out set of \(n_{\text{val}}\) observations, the prevalence baseline uses that single training estimate for every prediction. Its Brier score is the average squared difference between the observed outcomes and predicted probabilities.
For a quantitative training outcome \(y_i\), define the training mean \(\bar{y}_{\text{tr}}\). The mean baseline predicts \(\bar{y}_{\text{tr}}\) for each held-out case. The root mean squared error (RMSE) is the square root of the average held-out squared error.
These definitions assume the stated unweighted scoring setup. If a task uses weights or a different target definition, the baseline and metric must be adapted to that setup rather than compared as though they were unchanged. A model’s score and the baseline score must also use the same held-out observations and the same metric.
Why These Constant Predictions Are Natural
The prevalence and mean are not arbitrary reference values. Each is the constant prediction that minimizes the corresponding training squared-error criterion. The argument also explains why the training sample, not the held-out sample, determines the baseline prediction.
Let \(c\) be any constant probability prediction. If the event proportion is \(q\), the expected squared error for a binary outcome is \(q(1-c)^2+(1-q)c^2\).
Expanding gives \(q-2qc+c^2\). Its derivative with respect to \(c\) is \(2(c-q)\), and its second derivative is 2, which is positive. The minimum therefore occurs at \(c=q\). In the training data, the empirical value of \(q\) is the training prevalence.
The training sum of squared errors for a constant \(c\) is \(\sum_i(y_i-c)^2\). Differentiating with respect to \(c\) gives \(-2\sum_i(y_i-c)\). Setting this equal to zero yields \(c=\bar{y}_{\text{tr}}\); the second derivative is \(2n_{\text{tr}}>0\), so this is the minimum.
This justification concerns squared error. It does not claim that the same constant is optimal for every metric. For example, the training median, rather than the mean, minimizes the sum of absolute errors. A baseline should be chosen to match the outcome, prediction task, and metric being evaluated.
Worked Example: A Prevalence Baseline for a Binary Outcome
A hypothetical training sample contains 40 applications, of which 10 result in a claim-review event. On four held-out applications, the outcomes are \(1,0,1,0\). A candidate model predicts probabilities \(0.70,0.20,0.40,0.30\), respectively. Calculate both Brier scores and the candidate’s relative reduction in Brier score versus the baseline.
Method. Estimate prevalence from training outcomes only, then score both sets of held-out predictions with the Brier score. The baseline prediction is \(10/40=0.25\) for each held-out application.
Baseline calculation. The four squared errors are \((1-0.25)^2=0.5625\), \((0-0.25)^2=0.0625\), \((1-0.25)^2=0.5625\), and \((0-0.25)^2=0.0625\). Their sum is \(0.5625+0.0625+0.5625+0.0625=1.25\), so the baseline Brier score is \(1.25/4=0.3125\).
Candidate calculation. The squared errors are \((1-0.70)^2=0.09\), \((0-0.20)^2=0.04\), \((1-0.40)^2=0.36\), and \((0-0.30)^2=0.09\). Their sum is \(0.09+0.04+0.36+0.09=0.58\), so the candidate Brier score is \(0.58/4=0.1450\).
Relative reduction. Lower Brier scores are better. The absolute reduction is \(0.3125-0.1450=0.1675\). Relative to the baseline, the reduction is \(0.1675/0.3125=0.536\), or 53.6% to one decimal place. The candidate improves on this held-out set; that result alone does not establish how stable the improvement is on other samples.
Check and trap avoided. The candidate score is below the baseline score, as expected from the stated errors. The held-out event proportion is \(2/4=0.50\), but using 0.50 as the baseline prediction would use held-out outcomes to construct predictions. That would no longer be a valid out-of-sample baseline.
Worked Example: A Mean Baseline for a Quantitative Outcome
A hypothetical training sample has outcome values \(4,6,8,10\). The two held-out outcomes are \(5\) and \(9\). A candidate regression model predicts \(6\) and \(8\). Find the baseline and candidate RMSE values, and compare them.
Method. For RMSE, use the training mean as the constant baseline prediction. The training mean is \((4+6+8+10)/4=28/4=7\).
Baseline calculation. The held-out errors are \(5-7=-2\) and \(9-7=2\). Their squared errors are \(4\) and \(4\). Thus the baseline RMSE is \(\sqrt{(4+4)/2}=\sqrt{4}=2.0000\), in the outcome’s units.
Candidate calculation. The held-out errors are \(5-6=-1\) and \(9-8=1\). Their squared errors are \(1\) and \(1\). The candidate RMSE is \(\sqrt{(1+1)/2}=\sqrt{1}=1.0000\), in the outcome’s units.
Comparison and check. The absolute RMSE reduction is \(2.0000-1.0000=1.0000\), which is a 50% reduction relative to the baseline. The candidate’s two predictions are each one unit from the corresponding outcome, confirming the candidate RMSE of 1.0000. A common arithmetic trap is to add the baseline squared errors to get 8 and take \(\sqrt{8/2}\); this is incorrect because \(8/2=4\), so the result is \(\sqrt{4}=2\), not \(\sqrt{8}\).
Compare Candidate Models on Matched Validation Data
The baseline is useful for more than checking one fitted model. It gives a common point of reference for GLMs, penalized regressions, and tree ensembles, provided their predictions target the same outcome and are evaluated on the same held-out observations using the same metric. When validation uses multiple folds, construct each fold’s baseline from that fold’s training data and score it on that fold’s validation data.
The following hypothetical table reports Brier scores from four matched validation folds. Lower scores are better. The candidate columns represent a GLM, a penalized regression, and a tree ensemble; the baseline in each row is built from that fold’s training data.
| Validation fold | Prevalence baseline | GLM | Penalized regression | Tree ensemble |
|---|---|---|---|---|
| 1 | 0.20 | 0.16 | 0.15 | 0.14 |
| 2 | 0.24 | 0.20 | 0.19 | 0.22 |
| 3 | 0.18 | 0.17 | 0.16 | 0.15 |
| 4 | 0.22 | 0.18 | 0.19 | 0.21 |
| Mean fold score | 0.2100 | 0.1775 | 0.1725 | 0.1800 |
Worked Example: Interpreting Fold-Averaged Baseline Comparisons
Using the four-fold table, identify the lowest mean fold score, calculate each candidate’s relative reduction versus the mean baseline score, and describe what the comparison does and does not establish.
Method. Add the four fold scores for each column and divide by four. The mean baseline score is \((0.20+0.24+0.18+0.22)/4=0.84/4=0.2100\). The GLM mean is \((0.16+0.20+0.17+0.18)/4=0.71/4=0.1775\). The penalized-regression mean is \((0.15+0.19+0.16+0.19)/4=0.69/4=0.1725\). The tree-ensemble mean is \((0.14+0.22+0.15+0.21)/4=0.72/4=0.1800\).
Relative reductions versus baseline. For the GLM, the reduction is \((0.2100-0.1775)/0.2100=0.0325/0.2100\approx0.1548\), or 15.5% to one decimal place. For the penalized regression, it is \((0.2100-0.1725)/0.2100=0.0375/0.2100\approx0.1786\), or 17.9%. For the tree ensemble, it is \((0.2100-0.1800)/0.2100=0.0300/0.2100\approx0.1429\), or 14.3%.
Interpretation and check. The penalized regression has the lowest mean fold score, and all three candidates have lower mean scores than the baseline. Each candidate also beats the baseline in all four displayed folds. These results show incremental predictive value over the baseline in this comparison, but four fold averages do not by themselves show how stable the ranking would be across new samples. The mean fold scores in the table are the arithmetic averages of fold scores, not a pooled score calculated from all individual predictions.
Common Traps and Practical Interpretation
A baseline is a diagnostic reference, not a guarantee that a candidate should win. If a complex model does not improve on a sensible baseline, investigate the setup before arguing that added complexity is worthwhile. Check that the target, prediction horizon, held-out cases, and evaluation metric are aligned. Also check that baseline predictions were determined from training data rather than information from the validation outcomes.
- Using held-out prevalence or mean. This makes the baseline depend on information that would not be available when predicting new cases. Calculate prevalence or mean from the relevant training partition.
- Choosing a baseline for the wrong metric. The prevalence and mean baselines have specific squared-error motivations. Do not assume they are optimal references for every metric. In particular, an accuracy comparison based on classifying everyone as the majority class is a different baseline from a probability-prediction comparison using the Brier score.
- Comparing unmatched scores. A candidate and a baseline should be scored on the same validation cases. Otherwise, score differences may reflect differences in case mix rather than differences in predictions.
- Overstating a small gain. A lower average score is evidence about the evaluated resampling design, not proof of a large or stable improvement. Consider fold-level differences and the broader validation evidence before making an operational choice.
- Ignoring the scale of improvement. Report both the absolute score difference and, where useful, the relative reduction. A percentage change can appear large when the baseline score is small, so it should not replace the actual scores.
Check Your Understanding
For each problem, use the training sample to construct the baseline, then evaluate it on the stated held-out data. Give probabilities to four decimal places and RMSE values to four decimal places.
- A binary training sample has 50 observations and 15 events. The held-out outcomes are \(1,0,0\). What is the prevalence-baseline Brier score? (Answer: 0.2233)
- A quantitative training sample has values \(3,5,7,9\). The held-out outcomes are \(4,8\), and a candidate predicts \(5,7\). Find the baseline and candidate RMSE values. (Answer: Baseline 2.0000; candidate 1.0000, in the outcome’s units)
- A binary training sample has 20 observations and 4 events. The held-out outcomes are \(1,0,1\), and a candidate predicts \(0.50,0.10,0.80\). Find both Brier scores and the candidate’s relative reduction versus baseline. (Answer: Baseline 0.4400; candidate 0.1000; relative reduction 77.27%)
- Across three matched validation folds, the baseline RMSE values are \(5,4,6\), while a candidate’s values are \(4,4,5\). Compare their mean fold RMSE values and calculate the relative reduction from baseline. (Answer: Baseline 5.0000; candidate 4.3333; reduction 0.6667, or 13.33%)
- A baseline has a held-out Brier score of 0.16 and a candidate has a score of 0.18 on the same cases. State the absolute change and the relative change versus baseline, and interpret the direction. (Answer: The score increases by 0.02, a 12.5% increase; the candidate is worse because lower Brier score is better)