Out-of-Sample R-Squared as Baseline Improvement

Why Compare Predictions With a Baseline?

A prediction error is easier to judge when there is a clear reference point. If a model predicts patient waiting times, for example, a set of errors by itself does not say whether the model is useful. A simple alternative is to predict the same value for every case: the mean outcome in the training data. Comparing a model with this constant-mean baseline asks whether the model improves on a straightforward prediction rule when both are evaluated on the same held-out cases.

Out-of-sample R-squared expresses that comparison as a relative change in squared error. It is a prediction-performance measure, not automatically the same quantity as the in-sample R-squared reported for a fitted regression model. The defining ingredients are the held-out outcomes, the model’s predictions for them, and a stated baseline prediction.

Definition and Evaluation Setup

Definition: Let \(y_i\) be the observed outcome and \(\hat{y}_i\) the model prediction for held-out case \(i\), for \(i=1,\ldots,n_{\text{test}}\). Let \(\bar{y}_{\text{train}}\) be the mean outcome in the training data. The constant-mean baseline predicts \(\bar{y}_{\text{train}}\) for every held-out case. Define the model's held-out sum of squared errors as \(\sum_{i=1}^{n_{\text{test}}}(y_i-\hat{y}_i)^2\), and the baseline's as \(\sum_{i=1}^{n_{\text{test}}}(y_i-\bar{y}_{\text{train}})^2\). Out-of-sample R-squared is one minus the ratio of these sums.

R^2_{\text{OS}} = 1- \frac{\displaystyle\sum_{i=1}^{n_{\text{test}}}(y_i-\hat{y}_i)^2} {\displaystyle\sum_{i=1}^{n_{\text{test}}}(y_i-\bar{y}_{\text{train}})^2}

The subscript OS indicates out-of-sample evaluation. The numerator measures squared error from the model; the denominator measures squared error from the baseline. Both sums use the same held-out cases. Their ratio is unitless because both sums are measured in squared outcome units.

This definition relies on an evaluation design, not on a particular probability distribution for the outcomes. For the comparison to represent held-out performance, the model must be developed without using the held-out outcomes to fit or select it, and its predictions must be available for those cases without outcome leakage. The baseline mean must also be calculated from training outcomes, not chosen after inspecting the test outcomes. The two error sums must use the same cases and the same squared-error loss. If the denominator is zero, the held-out outcomes all equal the training mean, and this ratio is undefined.

The choice of baseline matters. Using \(\bar{y}_{\text{train}}\) gives a constant prediction that could have been made before the held-out outcomes were observed. An alternative calculation sometimes centers held-out outcomes around their own mean. That answers a different comparison: it uses information from the evaluation outcomes to construct the reference prediction. When reporting out-of-sample R-squared, state which baseline is used; here the benchmark is the training-set mean.

Worked Example: Comparing a Model With the Training Mean

Suppose a hypothetical model is trained on outcome values of 8, 10, 12, and 14 minutes. Its held-out evaluation has three cases. The observed waiting times are 9, 12, and 15 minutes, and the model predicts 10, 11, and 14 minutes. Calculate out-of-sample R-squared using the training-mean baseline.

First calculate the training mean, which is the baseline prediction for each held-out case:

\bar{y}_{\text{train}}=\frac{8+10+12+14}{4}=11\text{ minutes}

The model's squared errors are \((9-10)^2=1\), \((12-11)^2=1\), and \((15-14)^2=1\), all in squared minutes. Thus its total held-out squared error is \(1+1+1=3\) square minutes.

The baseline predicts 11 minutes for all three cases. Its squared errors are \((9-11)^2=4\), \((12-11)^2=1\), and \((15-11)^2=16\), for a total of \(4+1+16=21\) square minutes. Therefore,

R^2_{\text{OS}}=1-\frac{3}{21} =1-\frac{1}{7} \approx 0.8571

The model reduced the total squared prediction error by about 85.7% compared with predicting 11 minutes for every held-out case. This describes improvement relative to this baseline on these cases; it does not mean the model explains 85.7% of all variation in waiting times.

How to Read the Score

The formula can be rearranged to make its meaning explicit. Write \(SSE_{\text{model}}\) for the model's held-out sum of squared errors and \(SSE_{\text{base}}\) for the baseline's held-out sum. Then:

R^2_{\text{OS}} =1-\frac{SSE_{\text{model}}}{SSE_{\text{base}}} =\frac{SSE_{\text{base}}-SSE_{\text{model}}}{SSE_{\text{base}}}

The numerator in the final fraction is the amount of squared error removed by using the model instead of the baseline. Dividing by baseline error expresses that reduction relative to the baseline's error. For instance, a score of 0.30 means the model's total squared error is 30% lower than the baseline's on the evaluated cases. A score of 1 occurs when the model predicts every held-out outcome exactly, provided the baseline error is nonzero. A score of 0 means the model and baseline have equal total squared error.

The score is not measured in the units of the outcome, even though the underlying errors are. If outcomes are measured in minutes rather than seconds, both squared-error totals are multiplied by the same conversion factor squared, so their ratio and the score do not change. This makes the score useful for expressing relative performance, while MAE and RMSE retain direct outcome-unit interpretations.

Worked Example: A Model With a Different Amount of Improvement

In a hypothetical study of delivery times, the training outcomes are 18, 20, and 22 minutes. Three held-out deliveries have observed times of 18, 20, and 22 minutes, while a model predicts 19, 19, and 21 minutes. Find the score and express it as a relative error reduction.

The training mean, and therefore each baseline prediction, is:

\bar{y}_{\text{train}}=\frac{18+20+22}{3}=20\text{ minutes}

For the model, the squared errors are \((18-19)^2=1\), \((20-19)^2=1\), and \((22-21)^2=1\). Thus \(SSE_{\text{model}}=3\) square minutes. For the baseline, the squared errors are \((18-20)^2=4\), \((20-20)^2=0\), and \((22-20)^2=4\), so \(SSE_{\text{base}}=8\) square minutes.

R^2_{\text{OS}}=1-\frac{3}{8} =0.625

The model's squared error is 62.5% lower than the constant 20-minute baseline's squared error on these three held-out deliveries. The result is based on squared errors, so the two one-minute model errors and the baseline's two-minute errors contribute according to their squares, not just their absolute sizes.

Why the Baseline Must Be Specified

A baseline is part of the definition of any relative-performance score. With the training-mean baseline, the reference prediction is determined using information available during model development. Suppose instead that the held-out outcomes in the first example were used to calculate a reference mean. Their mean is \((9+12+15)/3=12\) minutes, not the training mean of 11 minutes. The baseline squared error would then be \((9-12)^2+(12-12)^2+(15-12)^2=18\) square minutes, and the resulting score would be \(1-3/18=0.8333\), rather than 0.8571. The changed score reflects a changed baseline, not a changed model.

The held-out mean is informed by the outcomes being evaluated. It can serve a descriptive comparison, but it is not a constant prediction available in advance for those cases. For a predictive evaluation asking how a model compares with a simple rule that could have been used before observing test outcomes, the training mean is the appropriate constant-mean reference.

Out-of-sample R-squared is also distinct from an in-sample regression R-squared. An in-sample measure describes fit on data used to estimate a regression; this score assesses prediction errors on data held aside for evaluation. They may use related algebra, but the data and question differ. In particular, the out-of-sample score is not automatically a measure of the proportion of population outcome variance explained by a model.

Worked Example: Checking the Calculation Before Reporting

A hypothetical clinic uses a training-set mean of 50 points as its baseline to predict a held-out health score. For three held-out people, observed scores are 45, 50, and 55; model predictions are 47, 50, and 53. Calculate the score and check whether the comparison is set up consistently.

The model errors are \(45-47=-2\), \(50-50=0\), and \(55-53=2\) points. Their squares sum to \(4+0+4=8\) square points. The baseline errors are \(45-50=-5\), \(50-50=0\), and \(55-50=5\) points, with squared errors summing to \(25+0+25=50\) square points.

R^2_{\text{OS}} =1-\frac{8}{50} =0.840

The score says that the model's squared error is 84.0% lower than the training-mean baseline's squared error on these three held-out outcomes. The check is consistent: both calculations include all three people, both use squared errors, and the baseline value 50 is identified as the training mean. Because the outcomes are scores, the sums have units of square points; the final score has no units.

Common Mistakes and Practical Checks

Using different cases in the two sums. The comparison is fair only when model and baseline errors are calculated for exactly the same held-out observations.
Forgetting to square the errors. Out-of-sample R-squared here is defined using squared errors, not absolute errors or signed errors.
Using the held-out mean without saying so. That changes the benchmark. Use the training mean when the goal is comparison with a baseline available before observing held-out outcomes.
Calling the score a percentage of variance explained. Instead, describe the relative change in squared error against the stated baseline and on the stated evaluation cases.
Reporting a number without its evaluation context. State the outcome, the held-out data, and the baseline. A score summarizes a comparison; it does not establish that a model will perform equally well in every future setting.

A useful calculation check is to write both squared-error totals separately before taking their ratio. Confirm that the denominator is the baseline error, that the same cases appear in both totals, and that the baseline value came from training data. Then interpret \(1-SSE_{\text{model}}/SSE_{\text{base}}\) as a relative comparison rather than as an isolated model property.

Key takeaway: Out-of-sample R-squared is one minus the ratio of a model's held-out squared error to the held-out squared error from a specified baseline. With a training-mean baseline, it quantifies relative squared-error improvement on the same held-out cases.

Check Your Understanding

Use the training-mean baseline unless a question explicitly specifies another reference prediction.

A model has held-out squared error 24, and the training-mean baseline has held-out squared error 40. Calculate out-of-sample R-squared and interpret it as a relative change in squared error.
Why must the model and baseline error totals be calculated on the same held-out observations?
A training set has outcomes 6, 8, and 10. What constant value does its mean baseline predict for every held-out case?
Explain why calculating the baseline mean from held-out outcomes answers a different comparison from using the training mean.
True or false: an out-of-sample R-squared of 0.70 means that 70% of population outcome variance has been explained. Give a more accurate interpretation.