A Prediction Performance Reporting Workshop

From a Prediction Table to a Performance Summary

A prediction model can produce a number for every case and still be difficult to evaluate from a single headline metric. Typical prediction errors, unusually large errors, systematic over- or underprediction, and performance relative to a simple benchmark are different questions. A useful report combines numerical summaries with plots and states what data were used to assess performance.

This workshop uses a small, hypothetical held-out dataset. The outcomes were not used to fit or tune the model, and each outcome has one matched prediction on the same measurement scale. Those conditions matter: evaluating predictions on data used to fit or select the model can make performance look better than it will be on new cases. The calculations below describe this held-out set; by themselves, they do not establish how well the model will perform in every future population.

Definition: For held-out case $i$, let $y_i$ be the observed outcome and $\hat{y}_i$ the model's prediction. Define the prediction error as $e_i=y_i-\hat{y}_i$. Thus, a positive error means the prediction was too low (underprediction), and a negative error means it was too high (overprediction). The error has the same units as the outcome.

Worked Example: Calculate Three Error Summaries

Suppose a model predicts a hypothetical measurement in millimetres for six held-out cases. The table lists the observed outcomes, predictions, and errors using the definition above.

Case	Observed $y_i$ (mm)	Predicted $\hat{y}_i$ (mm)	Error $e_i$ (mm)	$\|e_i\|$ (mm)	$e_i^2$ (mm²)
A	10	11	-1	1	1
B	12	11	1	1	1
C	13	14	-1	1	1
D	15	14	1	1	1
E	18	17	1	1	1
F	20	19	1	1	1
	Totals	88	86	2	6	6

The mean absolute error (MAE) averages the absolute errors, so positive and negative errors do not cancel. Here the sum of absolute errors is $6$ mm across $n=6$ cases:

\operatorname{MAE}=\frac{1}{n}\sum_{i=1}^{n}|e_i| =\frac{1+1+1+1+1+1}{6} =\frac{6}{6}=1.00\text{ mm}.

The root mean squared error (RMSE) squares errors before averaging and then takes the square root. Squaring gives relatively greater influence to large errors than MAE does. For these data:

\operatorname{RMSE} =\sqrt{\frac{1}{n}\sum_{i=1}^{n}e_i^2} =\sqrt{\frac{1+1+1+1+1+1}{6}} =\sqrt{1}=1.00\text{ mm}.

The mean error is the signed average. The errors sum to $2$ mm, so the mean error is $2/6=0.3333$ mm. On average, predictions are $0.3333$ mm below the observed values in this set. This does not mean every prediction is too low: two errors are negative and four are positive, with the positive errors producing a positive signed average.

\overline{e}=\frac{1}{n}\sum_{i=1}^{n}e_i =\frac{-1+1-1+1+1+1}{6} =\frac{2}{6}=0.3333\text{ mm}.

The MAE and RMSE happen to be equal because every error has magnitude $1$ mm. With a wider spread of error sizes, RMSE will be at least as large as MAE. To see why, the Cauchy–Schwarz inequality gives $\left(\sum_{i=1}^{n}|e_i|\right)^2\leq n\sum_{i=1}^{n}e_i^2$. Dividing by $n^2$ and taking nonnegative square roots yields $\operatorname{MAE}\leq\operatorname{RMSE}$. Equality holds when all the absolute errors are equal, as they are here.

Compare with a Baseline Before Calling Performance Good

An error summary has little context without a comparison. Out-of-sample R-squared compares the model's squared prediction error on held-out cases with the squared error from a specified baseline prediction. In this example, use the training-set mean outcome, $14.5$ mm, as the baseline prediction for every held-out case. The baseline must be defined without using the held-out outcomes to choose it.

Definition: For held-out outcomes $y_i$, model predictions $\hat{y}_i$, and baseline predictions $\hat{y}_{i,\mathrm{base}}$, define out-of-sample R-squared as $$ R^2_{\mathrm{out}} =1-\frac{\sum_{i=1}^{n}(y_i-\hat{y}_i)^2} {\sum_{i=1}^{n}(y_i-\hat{y}_{i,\mathrm{base}})^2}. $$ It is positive when the model has less squared error than the stated baseline, zero when the two have equal squared error, and negative when the model has more.

Worked Example: Calculate Out-of-Sample R-Squared

For the six held-out outcomes in the table, the model's squared errors sum to $6$ mm². The training-set mean baseline predicts $14.5$ mm for each case. Its errors and squared errors are:

\begin{aligned} 10-14.5&=-4.5, &(-4.5)^2&=20.25,\\ 12-14.5&=-2.5, &(-2.5)^2&=6.25,\\ 13-14.5&=-1.5, &(-1.5)^2&=2.25,\\ 15-14.5&=0.5, &(0.5)^2&=0.25,\\ 18-14.5&=3.5, &(3.5)^2&=12.25,\\ 20-14.5&=5.5, &(5.5)^2&=30.25. \end{aligned}

The baseline squared-error total is $20.25+6.25+2.25+0.25+12.25+30.25=71.50$ mm². Therefore:

R^2_{\mathrm{out}} =1-\frac{6}{71.50} =0.9161.

For these six cases, the model's squared error is about $91.61\%$ lower than the squared error from predicting the training mean for every case. This interpretation is relative to that particular baseline and this held-out set. It is not a statement that the model explains $91.61\%$ of a causal process, nor is out-of-sample R-squared necessarily the square of a correlation. Different baseline choices can produce different values, so a report should name the benchmark.

The calculation follows directly from expressing improvement relative to the baseline squared error. If $SSE_{\mathrm{model}}$ and $SSE_{\mathrm{base}}$ are the two sums of squared errors, their difference is $SSE_{\mathrm{base}}-SSE_{\mathrm{model}}$. Dividing that improvement by $SSE_{\mathrm{base}}$ gives the fraction of baseline error removed. Rewriting it gives $1-SSE_{\mathrm{model}}/SSE_{\mathrm{base}}$, the stated definition. If the model's error exceeds the baseline's, the fraction removed is negative and so is the score.

Worked Example: Inspect Calibration and Residual Plots

Use the six same pairs of predictions and outcomes to make two plots. In a calibration plot, place predictions on the horizontal axis and observed outcomes on the vertical axis; perfect agreement lies on the reference line where observed equals predicted. The plotted points are $(11,10)$, $(11,12)$, $(14,13)$, $(14,15)$, $(17,18)$, and $(19,20)$, with coordinates in millimetres. Each point is fairly close to the agreement line. The higher two predictions fall below their observed outcomes by $1$ mm, while the other points are split around the line. The plot does not show a large departure from agreement, but six points are too few to establish that calibration is consistently good across the full range of possible cases.

For a residual plot, place predicted values horizontally and errors $e_i=y_i-\hat{y}_i$ vertically, with a horizontal reference line at zero. At prediction $11$ mm the errors are $-1$ and $1$ mm; at $14$ mm they are $-1$ and $1$ mm; at $17$ and $19$ mm each error is $1$ mm. There is no pronounced pattern across the first four points, but the two points at the higher predictions are both positive. That could be a hint of underprediction at the high end, or simply reflect having very few cases there. The plot is a prompt to describe patterns and limitations, not proof that a pattern exists in the wider population.

Worked Example: Write a Balanced Performance Summary

A concise report should name the evaluation set and baseline, give the metrics with units where applicable, and describe the plots with appropriate caution. For the hypothetical data above, a balanced summary is:

“On six held-out cases, the model had an MAE of 1.00 mm and an RMSE of 1.00 mm. Mean error was $+0.3333$ mm, indicating modest average underprediction under the convention $e_i=y_i-\hat{y}_i$, despite two overpredictions. Out-of-sample R-squared was $0.9161$ relative to a constant prediction equal to the training-set mean of $14.5$ mm. The calibration points were close to the agreement line, and the residual plot showed no strong pattern overall, although both residuals at the two largest predicted values were positive. Because the held-out set contained only six cases, these results are descriptive and provide limited evidence about performance on other cases.”

This statement avoids letting the high R-squared value stand in for every aspect of performance. It also gives the sign convention so that “underprediction” is unambiguous, identifies the benchmark behind the score, and describes the possible high-end pattern without treating two points as decisive evidence.

A Practical Reporting Sequence

A consistent sequence helps prevent arithmetic and interpretation from becoming separated. Before calculating, check that each prediction is matched to its observed outcome and that errors and metrics use the intended outcome scale. If outcomes were transformed before prediction, the scale on which errors are calculated must be stated; errors on different scales are not directly interchangeable.

Define the evaluation.
State that the cases are held out from fitting or tuning, identify the outcome and units, and define error as observed minus predicted.

Calculate the error summaries.
Use absolute errors for MAE, squared errors for RMSE, and signed errors for mean error. Check the units: MAE and RMSE use outcome units, while mean error has those same units.

Name and calculate the baseline comparison.
Use the same held-out cases for model and baseline errors, and specify how the baseline predictions were set.

Inspect both plots.
Look for departures from the calibration agreement line and patterns in residuals versus predicted values; distinguish visible features from conclusions the sample cannot support.

Write a qualified conclusion.
Combine metric values, baseline, plot observations, and limitations of the held-out sample without implying that these summaries guarantee future performance.

Common Mistakes

Reporting mean error as typical error. Opposite-signed errors can cancel. In this example the mean error is only $0.3333$ mm, while MAE is $1.00$ mm. Report a magnitude-based metric as well when the size of individual errors matters.
Forgetting the error sign convention. With $e_i=y_i-\hat{y}_i$, a positive mean error means underprediction on average. Reversing the subtraction reverses that interpretation.
Calling out-of-sample R-squared the square of a correlation. Its value depends on the model errors and the stated baseline errors. Calculate it from the definition rather than inferring it from a correlation.
Choosing a benchmark after inspecting the held-out outcomes. That makes the comparison less meaningful as an evaluation. Define the baseline independently and report what it predicts.
Over-reading a small plot. A few points may suggest a pattern but cannot establish that it will persist. Describe what is visible, give the number of cases, and qualify the conclusion.

Key takeaway: A useful prediction report combines magnitude of error (MAE and RMSE), direction of average error (mean error), performance relative to a named baseline (out-of-sample R-squared), and a cautious visual check of calibration and residual patterns.

Check Your Understanding

Use the definitions and reporting principles in this workshop to answer the questions.

If a prediction is $8$ units and the observed outcome is $10$ units, what is the error under the convention used here, and what does its sign mean?
Why can mean error be close to zero even when individual predictions are often inaccurate?
A model has held-out squared error $40$, while a specified baseline has held-out squared error $50$. Calculate out-of-sample R-squared and interpret it relative to the baseline.
What feature of a calibration plot represents perfect agreement between predictions and outcomes?
A residual plot has only two positive residuals at the largest predictions. What is a careful way to report this observation?