Why Work Through the Calculation?
An out-of-sample \(R^2\) score compares the squared prediction errors of a model with the errors of a simple baseline on the same held-out cases. Its value depends on two calculations: how far the model’s predictions are from the observed outcomes, and how far a baseline prediction is from those same outcomes. Working through both parts makes the score easier to interpret—and helps prevent a common mistake: using information from the test set to construct the baseline.
This tutorial uses small hypothetical datasets so that every contribution to the score can be checked. The observations in a test set are outcomes the fitted model did not use to make its predictions. The baseline is a constant prediction equal to the mean outcome in the training set. Once this baseline is specified, the model and baseline errors can be compared case by case.
Definition and Calculation
The model prediction \(\hat{y}_i\) and the baseline prediction \(\bar{y}_{\mathrm{train}}\) are both compared with the same observed test outcome \(y_i\). Thus, both error totals use the same test cases, the same outcome scale, and squared differences. If outcomes are measured in points, each sum of squared errors is measured in squared points; their ratio, and therefore \(R^2_{\mathrm{OS}}\), has no units.
The training mean defines a prediction that could be made for a new case without knowing its outcome: predict the same training average for everyone. It is important that the test outcomes are not used to calculate this baseline. Otherwise, the benchmark would use information that would not be available when predicting a genuinely new outcome.
This equivalent form shows the score’s meaning. The numerator in the first fraction is the reduction in squared error relative to the baseline. Dividing that reduction by the baseline error expresses the improvement as a proportion of baseline error. For example, a score of \(0.60\) means that the model’s test-set squared error is 60% lower than the baseline’s; equivalently, the model has 40% of the baseline’s squared error. It does not mean that 60% of its predictions are correct.
Worked Calculations
Worked Example: A 75% Reduction in Held-Out Squared Error
Suppose a hypothetical training set has outcome values of 8, 10, and 12 points. A model is then evaluated on four held-out cases. The table gives the observed outcome and the model’s prediction for each case.
| Test case | Observed outcome \(y_i\) (points) | Model prediction \(\hat{y}_i\) (points) | Model squared error (points squared) | Baseline squared error (points squared) |
|---|---|---|---|---|
| 1 | 8 | 9 | \((8-9)^2=1\) | \((8-10)^2=4\) |
| 2 | 12 | 11 | \((12-11)^2=1\) | \((12-10)^2=4\) |
| 3 | 14 | 12 | \((14-12)^2=4\) | \((14-10)^2=16\) |
| 4 | 6 | 8 | \((6-8)^2=4\) | \((6-10)^2=16\) |
First calculate the training mean, which gives the baseline prediction:
The model’s held-out sum of squared errors is \(1+1+4+4=10\) points squared. The baseline’s held-out sum is \(4+4+16+16=40\) points squared. Therefore,
The model has 25% of the baseline’s held-out squared error, which is a 75% reduction in squared error compared with predicting 10 points for every test case. This statement concerns the relative squared-error totals; it is not a claim that each individual prediction improved by 75%.
Worked Example: A One-Third Reduction
In another hypothetical setting, the training outcomes are 16, 20, and 24 minutes. Three held-out outcomes and model predictions are shown below. The outcome and prediction units are minutes.
| Test case | Observed outcome \(y_i\) (minutes) | Model prediction \(\hat{y}_i\) (minutes) | Model squared error (minutes squared) | Baseline squared error (minutes squared) |
|---|---|---|---|---|
| 1 | 18 | 17 | \((18-17)^2=1\) | \((18-20)^2=4\) |
| 2 | 22 | 24 | \((22-24)^2=4\) | \((22-20)^2=4\) |
| 3 | 19 | 18 | \((19-18)^2=1\) | \((19-20)^2=1\) |
The training mean is \(\bar{y}_{\mathrm{train}}=(16+20+24)/3=60/3=20\) minutes. Summing the entries in the table gives \(\mathrm{SSE}_{\mathrm{model}}=1+4+1=6\) minutes squared and \(\mathrm{SSE}_{\mathrm{baseline}}=4+4+1=9\) minutes squared. Thus,
The model’s squared error is two-thirds of the baseline’s squared error. The reduction is one-third, or about 33.3%, relative to this baseline on these three held-out cases. Although the score is positive, it does not imply that the model beats the baseline on every case: in case 2, the model’s squared error is equal to the baseline’s.
Worked Example: Check the Baseline Before Interpreting the Score
A hypothetical training set has outcomes of 5, 7, 9, and 11 units. On four held-out cases, the observed outcomes are 6, 10, 8, and 14 units, and the model predictions are 7, 9, 9, and 12 units.
The training-set mean is
Using 8 units as the baseline prediction, the baseline squared errors are \((6-8)^2=4\), \((10-8)^2=4\), \((8-8)^2=0\), and \((14-8)^2=36\), all in units squared. Therefore, \(\mathrm{SSE}_{\mathrm{baseline}}=4+4+0+36=44\) units squared.
The model squared errors are \((6-7)^2=1\), \((10-9)^2=1\), \((8-9)^2=1\), and \((14-12)^2=4\) units squared. Therefore, \(\mathrm{SSE}_{\mathrm{model}}=1+1+1+4=7\) units squared, and
On these four held-out cases, the model’s squared error is about 15.91% of the training-mean baseline’s squared error. The reduction is about 84.09%. The large difference is partly associated with the fourth case, where the baseline misses by 6 units and the model misses by 2. Since errors are squared, that case contributes 36 units squared to the baseline total but only 4 units squared to the model total.
How to Check a Calculation
A reliable calculation keeps the baseline and model comparisons aligned. The following sequence makes the role of each quantity explicit.
Average the training outcomes only. This is the constant baseline prediction used for each test case.
For each held-out case, subtract the model prediction from the observed outcome, then square the difference. Add these values to obtain \(\mathrm{SSE}_{\mathrm{model}}\).
For those same held-out cases, subtract the training mean from each observed outcome, square each difference, and add them to obtain \(\mathrm{SSE}_{\mathrm{baseline}}\).
Divide model squared error by baseline squared error, then subtract that ratio from 1. Describe the result as relative held-out squared-error improvement or loss.
This metric is a comparison of predictions on a particular held-out set. It does not by itself establish how a model will perform on every future sample; a small test set can give an unstable estimate of predictive performance. The cases must also be paired correctly: each model prediction must correspond to the outcome for the same test case used in its baseline comparison.
Common Mistake: Using the Test-Set Mean
A tempting alternative is to calculate the baseline prediction by averaging the test outcomes. That changes the benchmark. The test-set mean uses the answers that the prediction exercise is supposed to evaluate; it is not the fixed training-mean prediction available before those outcomes are observed. Unless a different benchmark is deliberately defined, use \(\bar{y}_{\mathrm{train}}\) for every test case.
Another frequent error is to average the squared errors and then compare totals with different numbers of cases. For a single test set shared by both methods, the average squared errors divide both sums by the same \(n_{\mathrm{test}}\), so their ratio is unchanged. However, the calculation should use the same cases and the same definition on both sides. If the cases differ, the resulting ratio is no longer the stated comparison.
Check Your Understanding
Answer each question using the definitions and calculation steps above.
- A training set has outcomes 4, 6, and 8. What constant prediction does the training-mean baseline make for each held-out case?
- For two held-out cases, the model squared errors are 3 and 5, while the baseline squared errors are 6 and 10. Calculate \(R^2_{\mathrm{OS}}\) and interpret it as a relative reduction in squared error.
- Why should the baseline mean be calculated from training outcomes rather than from held-out outcomes?
- If the model and baseline have the same total squared error on the held-out cases, what is \(R^2_{\mathrm{OS}}\)?
- Explain why both error totals must use the same held-out cases and the same outcome units.