Why a Performance Number Needs Its Validation Context
A regression model can produce a prediction for every observation, but the average size of its prediction errors depends on which observations are evaluated and how the model was trained. A model’s errors on the same data used to fit it are not an adequate stand-in for errors on new cases: the fitting process has already used those outcomes. Predictive performance therefore needs both a metric and a description of the validation design.
For example, “the RMSE was 2.8” does not tell a reader whether this is 2.8 dollars, days, or test-score points; whether it was calculated on training data or held-out observations; or whether a particular group or time period was held out. A useful report answers those questions briefly. It also says what the estimate represents: performance on held-out cases drawn under a stated validation design, not a guarantee about every future prediction.
Metrics for Prediction Errors
Let \(y_i\) be the observed outcome for validation observation \(i\), and let \(\hat{y}_i\) be the prediction made for that observation by a model fitted without using its outcome. Define the prediction error as \(e_i=\hat{y}_i-y_i\). The sign convention helps describe over- and underprediction, but the absolute and squared error metrics below do not depend on which way the subtraction is written.
For a numeric outcome, both measures are expressed in the outcome’s units. If the outcome is waiting time in minutes, MAE and RMSE are in minutes; if it is a concentration in milligrams per litre, both are in milligrams per litre. MAE describes the average absolute error. RMSE gives relatively greater influence to large errors because errors are squared before averaging. Neither metric reveals whether predictions tend systematically to be too high or too low: absolute values and squares discard the signs.
A performance metric has meaning only alongside the observations it summarizes. A result based on ten held-out cases, for example, is not interchangeable with a result based on a much larger validation sample, even if the metric values match. The sample size does not change the units, but it helps readers judge the scope of the evaluation and how much data contributed to the reported number.
Worked Example: Calculating Validation MAE and RMSE
Suppose a hypothetical model predicts the time to complete a task, in minutes, for four held-out cases. The outcomes were not used to fit the model.
| Case | Observed time \(y_i\) (min) | Predicted time \(\hat{y}_i\) (min) | Error \(e_i=\hat{y}_i-y_i\) (min) |
|---|---|---|---|
| 1 | 10 | 12 | 2 |
| 2 | 12 | 11 | -1 |
| 3 | 15 | 14 | -1 |
| 4 | 20 | 18 | -2 |
The absolute errors are 2, 1, 1, and 2 minutes. Therefore,
The squared errors are \(4,1,1,4\) square minutes, which sum to 10. Thus,
A suitable interpretation is: “On the four held-out cases, the model’s MAE was 1.50 minutes and its RMSE was 1.58 minutes.” This describes errors on these validation cases. By itself, it does not establish that future task times will be predicted with the same accuracy.
Validation Designs and Their Meaning
A holdout validation design separates observations into a training set and a validation set. The model is fitted using the training outcomes, then used to predict outcomes in the held-out validation set. The resulting metric summarizes errors for that held-out set. Its relevance to future use depends on whether the validation observations resemble the cases for which predictions will be needed.
In k-fold cross-validation, the observations are divided into \(k\) folds. In each round, one fold is held out for validation and the model is fitted using the other \(k-1\) folds. Each observation receives a prediction from the round in which its fold was held out. Metrics can then be calculated from all these out-of-fold predictions. A clear report gives the number of folds and the number of observations evaluated, rather than merely saying “cross-validated.”
The design should reflect how predictions are intended to be used. If future predictions concern new people sampled in the same way, an appropriate holdout may consist of such people. If the actual task is to predict later measurements, an evaluation that uses later observations as validation cases may be more relevant than a random split that mixes times. In a grouped setting, holding out whole groups can address prediction for new groups. The report should state the design used; a random split is not automatically representative of every prediction task.
All steps that learn from data must be confined to the training portion in each validation round. For example, if a transformation or a rule for handling missing values is estimated from data, it must be estimated using the training data for that round and then applied to its held-out fold. Otherwise, information from validation observations has influenced the predictions being evaluated, and the resulting metric no longer has the intended interpretation as validation performance.
Worked Example: Reporting Five-Fold Cross-Validation
In a hypothetical teaching exercise, ten observations have an outcome measured in minutes. Five-fold cross-validation is used, so each fold contains two held-out observations. The table gives the sum of squared prediction errors within each fold; the model in each round was fitted on the other eight observations. These values are illustrative, not results from a real study.
| Held-out fold | Number of cases | Sum of squared errors (min²) | Fold RMSE (min) |
|---|---|---|---|
| 1 | 2 | 10 | \(\sqrt{10/2}=2.236\) |
| 2 | 2 | 20 | \(\sqrt{20/2}=3.162\) |
| 3 | 2 | 4 | \(\sqrt{4/2}=1.414\) |
| 4 | 2 | 34 | \(\sqrt{34/2}=4.123\) |
| 5 | 2 | 10 | \(\sqrt{10/2}=2.236\) |
The total squared error over all five folds is \(10+20+4+34+10=78\) min², and the total number of out-of-fold predictions is \(5(2)=10\). Pooling all ten errors gives
A report could say: “Using five-fold cross-validation on 10 observations, the pooled out-of-fold RMSE was 2.79 minutes.” This states the metric, its units, the folds and the number of evaluated observations. The small illustrative sample is not enough to suggest that the value will precisely describe performance in a broader target population.
Why Pooled and Average Fold RMSE Can Differ
There are two possible summaries when fold sizes are equal: compute one RMSE from all out-of-fold errors, or calculate an RMSE in each fold and take the arithmetic mean of those fold RMSEs. They are not generally the same. For fold \(j\), let \(n_j\) be its size and let \(\operatorname{MSE}_j\) be its mean squared error. The sum of squared errors in that fold is \(n_j\operatorname{MSE}_j\). Adding these sums across folds and dividing by the total number \(N\) of validation predictions gives the pooled mean squared error:
Taking the square root gives the pooled RMSE. Thus, the pooled result weights each fold’s mean squared error by its size before taking the square root. It is not generally the unweighted arithmetic mean of the fold RMSEs, because that would average the square roots instead.
In the example, the five fold RMSEs are 2.236, 3.162, 1.414, 4.123, and 2.236 minutes. Their unweighted mean is \(13.171/5=2.634\) minutes, whereas pooling the errors gives 2.793 minutes. These values answer slightly different summary questions. A report should name which one it provides. With unequal fold sizes, a simple mean of fold RMSEs also gives a small fold the same weight as a larger fold, while pooling uses all validation predictions.
Worked Example: Choosing MAE or RMSE for the Summary
Consider four hypothetical held-out cases with outcomes of 10 points each. Model A predicts 11, 9, 11, and 9 points, while Model B predicts 10, 10, 10, and 14 points. Both models are evaluated on the same cases, so their errors can be compared directly.
For Model A, the errors are \(1,-1,1,-1\) points. The absolute errors sum to 4 points, and the squared errors sum to 4 points squared:
For Model B, the errors are \(0,0,0,4\) points. Its absolute errors also sum to 4, but its squared errors sum to 16:
The MAEs match, but the RMSE for Model B is larger because its single four-point error contributes more after squaring. Neither summary is universally preferable: the report should make clear whether average absolute error or a metric that emphasizes large errors is useful for the prediction task. Both results must still be paired with the common validation design and the outcome units.
Writing a Complete Performance Statement
A concise performance statement usually needs four pieces: the metric, its value and units where applicable, the validation design and evaluation sample, and a short explanation of what the result estimates. Depending on the study, the number of folds or the held-out group or time period may be particularly important. The purpose is not to list every computational detail, but to give readers enough information to interpret the number.
For example: “Using a time-ordered holdout, the model’s MAE was 3.4 units on 120 observations from the later period; this summarizes errors when predicting that held-out period from data available earlier.” The final clause connects the number to the task the design represents. If a design only evaluates predictions for observations similar to those in a particular sample, the statement should not imply it establishes accuracy for substantially different settings.
A performance metric also does not tell the whole story. MAE and RMSE summarize error size, not whether predictions systematically over- or underestimate, nor whether errors are concentrated in a subgroup. Those questions require additional summaries or evaluation. Avoid claiming that one model is generally better based only on values calculated using different validation samples or designs.
Common Mistakes
- Reporting a metric without its units. “RMSE = 2.8” is incomplete when the reader does not know whether the outcome is measured in dollars, minutes, or another scale. Name the outcome units.
- Calling training error validation performance. Predictions on data used to fit the model do not constitute held-out validation predictions. State whether the metric comes from training, a holdout, or cross-validation.
- Writing “cross-validated” without describing the evaluation. Give the number of folds and the number of observations contributing to the metric; state whether fold RMSEs were averaged or errors were pooled.
- Treating a validation result as a guarantee. The metric summarizes errors under a particular evaluation design. It does not guarantee the same accuracy for future cases, especially when those cases differ from the validation observations.
- Assuming MAE and RMSE are interchangeable. They summarize errors differently. In particular, RMSE responds more strongly to large errors, so identify which metric is being reported.
Check Your Understanding
For each question, focus on both the metric and the validation context.
- A model predicts hospital waiting time in minutes. Explain why “RMSE = 5” is not a complete report, even if the arithmetic is correct.
- In one sentence, describe what a holdout validation MAE summarizes and name two details that should accompany it in a report.
- Why can the pooled RMSE from cross-validation differ from the unweighted mean of the fold RMSEs?
- Two models have the same MAE, but one has a larger RMSE. What feature of its errors could explain this difference?
- A model is intended to predict later observations, but its validation observations were randomly mixed across time with its training observations. What should a report avoid claiming about the result?