Why the Partitions Must Be Comparable
A candidate model’s validation score depends not only on its algorithm but also on which observations it was asked to predict. If two models are evaluated on different validation samples, a difference in their scores can reflect different case mix rather than different predictive performance. The remedy is to use the same training and validation partitions for every candidate model being compared.
With common partitions, each model is trained on the same observations within a resampling split and evaluated on the same held-out observations. Its predictions can then be paired with those from another model at the observation level. The comparison is more interpretable: for each validation case, the analyst can ask which model incurred the smaller loss.
The partitions are common; the fitted models are not. Each candidate algorithm is fitted separately using the training observations in each split. If there are \(K\) folds, every observation is held out once in ordinary \(K\)-fold cross-validation, and the predictions made while that observation is held out are its out-of-fold predictions. The outcome for a validation observation must not be used to fit the model that predicts it.
Building and Comparing Matched Folds
First assign every observation to a fold, then reuse that assignment for each candidate. For a given fold, each model uses the same other folds for training and the same fold for validation. If a common split is not used, the scores may no longer describe performance on comparable cases.
Record which observations belong to each validation fold. Apply that assignment unchanged to every candidate model.
For each fold, train each candidate using the same training observations. Fit any model-specific parameters using that training set.
For each validation observation, retain its outcome, fold, and prediction from every candidate. Keep the observation identifiers so that predictions cannot be accidentally paired with the wrong cases.
Calculate the chosen metric from matched outcomes and predictions. Report the direction and size of the difference, not just two disconnected scores.
For a quantitative outcome \(y_j\) and prediction \(\hat{y}_{mj}\) from model \(m\) on observation \(j\), squared error is \((y_j-\hat{y}_{mj})^2\), and absolute error is \(|y_j-\hat{y}_{mj}|\). For example, the paired squared-error difference for models A and B on case \(j\) can be defined as
Under this sign convention, \(d_j>0\) means model A has the larger squared error on that case; \(d_j<0\) means model B has the larger squared error. The mean of these differences summarizes the average paired difference in squared error. It is not itself a difference in RMSE: RMSE applies a square root after averaging squared errors.
Worked Example: Comparing RMSE on Identical Folds
A hypothetical insurer compares two models for predicting claim severity in thousands of dollars. The six validation observations are placed in two folds of three observations each. Both models use exactly these folds. For each fold, calculate the sum of squared errors (SSE), the fold RMSE, and the overall pooled RMSE. Identify the better-performing model by pooled RMSE.
Method. For a validation set of \(n\) observations, calculate \(\mathrm{RMSE}=\sqrt{\mathrm{SSE}/n}\). The values below are in thousands of dollars, so the errors and RMSE are also expressed in thousands of dollars.
| Fold | Actual values | Model A predictions | Model B predictions |
|---|---|---|---|
| 1 | 10, 20, 30 | 12, 18, 33 | 11, 22, 31 |
| 2 | 15, 25, 35 | 14, 29, 32 | 17, 24, 36 |
Fold 1, Model A. The errors are \(10-12=-2\), \(20-18=2\), and \(30-33=-3\). Their squared errors are \(4\), \(4\), and \(9\), so \(\mathrm{SSE}_{A,1}=4+4+9=17\). Thus \(\mathrm{RMSE}_{A,1}=\sqrt{17/3}=\sqrt{5.6667}\approx2.380\).
Fold 1, Model B. The errors are \(-1\), \(-2\), and \(-1\); the squared errors are \(1\), \(4\), and \(1\). Therefore, \(\mathrm{SSE}_{B,1}=6\) and \(\mathrm{RMSE}_{B,1}=\sqrt{6/3}=\sqrt{2}\approx1.414\).
Fold 2, Model A. The errors are \(1\), \(-4\), and \(3\); the squared errors are \(1\), \(16\), and \(9\). Therefore, \(\mathrm{SSE}_{A,2}=26\) and \(\mathrm{RMSE}_{A,2}=\sqrt{26/3}=\sqrt{8.6667}\approx2.944\).
Fold 2, Model B. The errors are \(-2\), \(1\), and \(-1\); the squared errors are \(4\), \(1\), and \(1\). Therefore, \(\mathrm{SSE}_{B,2}=6\) and \(\mathrm{RMSE}_{B,2}=\sqrt{6/3}=\sqrt{2}\approx1.414\).
Overall pooled results. Model A has total SSE \(17+26=43\) over six observations, giving \(\mathrm{RMSE}_A=\sqrt{43/6}=\sqrt{7.1667}\approx2.677\). Model B has total SSE \(6+6=12\), giving \(\mathrm{RMSE}_B=\sqrt{12/6}=\sqrt{2}\approx1.414\). Model B has the lower pooled RMSE by approximately \(2.677-1.414=1.263\) thousand dollars, or about $1,263. The arithmetic checks: the squared pooled RMSEs are \(43/6\) and \(12/6\), the pooled mean squared errors from the six cases.
A common mistake is to treat the arithmetic mean of fold RMSEs as the pooled RMSE. Here the mean fold RMSE for A is \((2.380+2.944)/2=2.662\), which is not the pooled RMSE of \(2.677\). The pooled calculation adds squared errors across all observations before dividing and taking the square root. Equal fold sizes make the mean fold MSE equal the pooled MSE, but taking square roots fold by fold still does not generally give the pooled RMSE.
Why Pairing Can Make Differences Easier to Read
A paired comparison focuses on differences measured under the same split conditions. The reason this can be more informative than treating scores as unrelated is visible from the variance identity below. Suppose \(A\) and \(B\) are the scores for two models on a matched fold, and the variance calculation is being used to describe a set of paired comparisons. Then
This identity follows by expanding the definition of variance:
When both models tend to score better on easier folds and worse on harder folds, their fold scores may move together, so the covariance is positive. Subtracting the paired scores can then remove some variation attributable to fold difficulty. Pairing does not guarantee a smaller variance: if the covariance is small or negative, it may not help. The formula explains a potential benefit, not a promise about every dataset.
Worked Example: Interpreting Paired Fold Differences
On four identical validation folds, two severity models have the following fold-level MAE values, in hundreds of dollars. Calculate the mean paired difference A minus B and its standard error under the simplifying assumption that the four fold differences are independent and identically distributed.
| Fold | Model A MAE | Model B MAE | A minus B |
|---|---|---|---|
| 1 | 7 | 6 | 1 |
| 2 | 5 | 5 | 0 |
| 3 | 8 | 7 | 1 |
| 4 | 6 | 5 | 1 |
Mean paired difference. The four differences are \(1,0,1,1\), so their mean is \((1+0+1+1)/4=0.75\). On these folds, A’s MAE is higher by an average of 0.75 hundreds of dollars, or $75.
Sample standard deviation of the differences. The deviations from 0.75 are \(0.25,-0.75,0.25,0.25\). Their squared deviations sum to \(0.0625+0.5625+0.0625+0.0625=0.75\). The sample variance is \(0.75/(4-1)=0.25\), so the sample standard deviation is \(\sqrt{0.25}=0.5\).
Standard error under the stated assumption. The standard error of the mean paired difference is \(0.5/\sqrt{4}=0.25\), or $25. This is a descriptive uncertainty calculation under the stated independence assumption, not a claim that ordinary cross-validation folds are independent. Their training sets overlap, so fold scores are generally dependent; do not treat this simple standard error as an automatic, formally valid cross-validation confidence interval.
For contrast, the sample variance of A’s fold scores \(7,5,8,6\) is \(5/3\), and that of B’s scores \(6,5,7,5\) is \(2.75/3=11/12\). An independence-based standard error for the difference of their separate means would be \(\sqrt{(5/3+11/12)/4}=\sqrt{31/48}\approx0.803\), larger than the paired value \(0.25\). This illustrates the potential value of pairing for these data, while the dependence caveat still applies.
Unmatched Samples Can Reverse the Story
A comparison of scores from unrelated validation samples can mix model performance with differences in case difficulty or portfolio composition. The following hypothetical example uses absolute errors for equal-sized folds. It shows why both candidates need to be evaluated on each of the same folds before their overall performance is compared.
Worked Example: Detecting a Case-Mix Confounder
A hypothetical portfolio has two equally sized validation folds. One contains lower-severity cases and the other contains higher-severity cases. The absolute errors, in hundreds of dollars, are as follows. Compute each candidate’s pooled MAE using both folds, then assess the misleading comparison that would result from evaluating A only on Fold 1 and B only on Fold 2.
| Fold | Model A absolute errors | Model B absolute errors |
|---|---|---|
| 1, four cases | 1, 2, 2, 3 | 2, 3, 3, 4 |
| 2, four cases | 7, 8, 8, 9 | 6, 7, 7, 8 |
Fold 1. A’s errors sum to \(1+2+2+3=8\), so its MAE is \(8/4=2\). B’s errors sum to \(2+3+3+4=12\), so its MAE is \(12/4=3\). The paired fold difference A minus B is \(2-3=-1\).
Fold 2. A’s errors sum to \(7+8+8+9=32\), so its MAE is \(32/4=8\). B’s errors sum to \(6+7+7+8=28\), so its MAE is \(28/4=7\). The paired fold difference is \(8-7=1\).
Matched pooled comparison. Across all eight cases, A’s total absolute error is \(8+32=40\), giving MAE \(40/8=5\). B’s total is \(12+28=40\), also giving MAE \(40/8=5\). The fold differences \(-1\) and \(1\) average to zero. Neither candidate has lower pooled MAE on these data.
Unmatched comparison. If A were reported only on Fold 1, its MAE would be 2, while B reported only on Fold 2 would have MAE 7. The apparent five-hundred-dollar advantage for A is caused by comparing different case mixes, not by a fair evaluation of the candidates. Reversing which fold is assigned to each model would change the apparent conclusion. The correction is to use the same folds for both models and compare the paired results.
Practical Checks and Limits
A shared fold assignment makes a comparison more controlled, but it does not settle every modeling decision. It does not make a chosen metric appropriate, guarantee that validation data represent future cases, or remove uncertainty from a finite sample. It also does not mean candidate predictions should be paired across different observations merely because they came from the same fold.
- Check observation alignment. Match each prediction to the correct observation identifier, outcome, and fold. A row-order mismatch can invalidate an apparently paired calculation.
- Use the same metric definition. For example, compare MAE with MAE and RMSE with RMSE, using the same units and aggregation rule.
- State how folds are aggregated. For unequal fold sizes, the pooled MAE weights each case equally and is \(\sum_k n_k\mathrm{MAE}_k/\sum_k n_k\), where \(n_k\) is the size of fold \(k\). An unweighted mean of fold MAEs instead gives each fold equal weight.
- Keep uncertainty claims modest. Reusing observations in overlapping training sets induces dependence among fold-level scores. A paired fold difference is useful for diagnosis, but a simple independent-sample standard error may not have its nominal interpretation.
- Separate model comparison from split choice. Common partitions control the comparison between candidates on those partitions. They do not prove that the observed performance difference will persist under every possible sample or future period.
For a reproducible R workflow, create the fold assignment once, retain it with the data, and use it in the resampling loop for every candidate. Save out-of-fold predictions in a long-form results table with at least an observation identifier, fold identifier, candidate name, actual outcome, and predicted value. This makes it possible to verify that every candidate was scored on the same held-out rows before calculating paired differences.
Check Your Understanding
Work through each calculation using the stated comparison rule.
- On one common fold, three actual values are \(8,12,16\). Model A predicts \(9,10,18\), and Model B predicts \(7,13,15\). Calculate the fold MAE for each and the paired difference A minus B. (Answer: MAE A \(=1.667\), MAE B \(=1.000\), difference \(=0.667\).)
- Three common folds have paired MAE differences A minus B of \(0.4,-0.2,0.3\). Calculate their mean paired difference. (Answer: \(0.1667\).)
- Two equal-sized folds each contain three observations. The candidate’s fold SSEs are 9 and 16. Calculate the pooled RMSE. (Answer: \(\sqrt{25/6}\approx2.0412\).)
- Four matched folds have A’s MAEs \(7,5,8,6\) and B’s MAEs \(6,5,7,5\). Calculate the mean paired difference and its standard error using the sample standard deviation of the differences and assuming independent fold differences. (Answer: mean difference \(=0.75\); standard error \(=0.25\).)
- Two validation folds have sizes 2 and 6, and their MAEs are 1 and 5, respectively. Calculate the pooled MAE and compare it with the unweighted mean of the two fold MAEs. (Answer: pooled MAE \(=(2(1)+6(5))/8=4\); unweighted fold mean \(=(1+5)/2=3\).)