Why Rankings Can Change Across Validation Splits
When two candidate curves have similar out-of-sample errors, a single validation split may not settle which one predicts better. A different split can place different observations in the validation set, changing the measured errors and even reversing the ranking. This is not necessarily a calculation error: it is a sign that the comparison depends, at least partly, on which observations happen to be held out.
The useful question is therefore not only which curve had the smaller error once? It is also how often, and by how much, does its advantage persist across reasonable splits? A ranking that changes repeatedly calls for a more cautious conclusion than a ranking that remains the same and shows a substantial difference.
Suppose two candidate curves, A and B, are evaluated using the same validation splits and the same error measure. On split \(j\), let their validation mean squared errors be \(MSE_{A,j}\) and \(MSE_{B,j}\). Define the paired difference as A’s error minus B’s error. This sign convention makes a positive difference favor B, since B then has the lower error.
Pairing matters: each difference compares both curves on the same held-out observations. Comparing A’s error on one split with B’s error on a different split would mix curve differences with changes in the validation data, making the comparison harder to interpret.
Summarizing the Pattern Across Splits
For \(K\) validation splits, examine the full set of paired differences \(d_1,\ldots,d_K\), not just the split with the best or worst result. Several simple summaries answer different questions:
- Sign counts: how many splits favor A, favor B, or tie? These describe ranking consistency, but ignore the sizes of the differences.
- Mean difference: \(\bar{d}=\frac{1}{K}\sum_{j=1}^{K}d_j\) summarizes the average signed difference across the evaluated splits. A positive mean favors B on average; a negative mean favors A.
- Median and range: the median describes a typical split without being as sensitive to an extreme difference as the mean can be. The range shows the span from the smallest to the largest observed difference.
- Standard deviation: the sample standard deviation of the \(d_j\) values describes how much they vary across the evaluated splits. It is a descriptive measure of split-to-split variation, not automatically a standard error for inference.
These summaries are complementary. A curve can win on most splits but lose on average if one of its losses is large. Alternatively, the mean difference can be near zero because small advantages in both directions cancel. In either case, the average alone conceals the pattern that generated it.
The mean difference also has a direct arithmetic interpretation. Because every split uses the same two curves and the same error definition, averaging the paired differences is equivalent to subtracting the average validation MSEs:
The equality follows by distributing the sum across subtraction and dividing both sums by \(K\). Thus, the mean paired difference tells which curve has the smaller average validation MSE across the recorded splits. It does not show that the same curve wins on every split.
Worked Example: A Majority of Wins but a Reversed Average
Suppose two curves predict a hypothetical outcome measured in points. Their validation MSEs across five common splits are as follows. All entries are in squared points, because each residual is squared when calculating MSE.
| Split | Curve A MSE | Curve B MSE | \(d_j=MSE_{A,j}-MSE_{B,j}\) | Lower-error curve |
|---|---|---|---|---|
| 1 | 4.00 | 4.10 | \(-0.10\) | A |
| 2 | 4.20 | 4.35 | \(-0.15\) | A |
| 3 | 3.80 | 4.00 | \(-0.20\) | A |
| 4 | 4.10 | 4.35 | \(-0.25\) | A |
| 5 | 5.40 | 4.20 | \(1.20\) | B |
Curve A has lower MSE on four of the five splits, while curve B has lower MSE on the fifth. The mean difference is
The positive mean means that curve B has the lower average validation MSE. Indeed, the average MSEs are \(21.50/5=4.30\) squared points for A and \(21.00/5=4.20\) squared points for B. The fifth split’s 1.2-squared-point difference outweighs the four smaller differences favoring A, which total \(0.70\) squared points in the opposite direction.
The median of the five differences is \(-0.15\) squared points, so the middle split favors A. The count, median, and mean tell different parts of the story: A wins more splits, but B has the lower average error because its single win is much larger. It would be misleading to report simply that A is the better curve because it won four of five splits, or that B is consistently better because its mean error is lower.
How Much Does the Ranking Vary?
A comparison is especially fragile when differences are small relative to their variation or when their signs change often. Consider both the direction of the differences and their scale. A sign count gives the frequency of wins; the mean gives the average direction and size; and a spread summary shows whether that average represents a fairly consistent pattern or a mixture of substantially different outcomes.
Worked Example: Near-Ties With Rankings in Both Directions
In another hypothetical comparison, validation MSEs for A and B across eight common splits produce the following paired differences, in squared points:
There are four negative and four positive differences, so each curve has the lower MSE on four splits. The sum of the differences is zero, giving a mean difference of \(0/8=0\) squared points. The results do not show an average advantage for either curve across these splits.
To describe the split-to-split variation, the sum of squared deviations from the mean is \(0.08^2+0.04^2+0.03^2+0.06^2+0.02^2+0.05^2+0.04^2+0.02^2=0.0174\) squared-points squared. The sample standard deviation of the differences is therefore
The standard deviation is descriptive: it indicates the amount of variation in these eight differences. The changes in sign and the zero mean suggest that neither curve has a stable edge in this set of splits. The result does not prove that the curves perform equally well in every setting; it shows that these validation comparisons do not consistently favor one of them.
An apparently stable ranking also needs careful interpretation. If a curve wins every split, that is useful evidence of ranking consistency under the chosen splitting procedure, but it does not by itself establish a large or practically important advantage. The differences could all be tiny. Conversely, a curve could win only slightly more often but have a larger mean advantage. Report the signs and the sizes rather than treating either one as a complete account.
Worked Example: A Consistent Ranking With a Modest Difference
Suppose six validation splits give the following MSEs, again in squared points:
| Split | Curve A MSE | Curve B MSE | \(d_j\) |
|---|---|---|---|
| 1 | 3.80 | 3.68 | 0.12 |
| 2 | 4.10 | 4.02 | 0.08 |
| 3 | 3.95 | 3.85 | 0.10 |
| 4 | 4.20 | 4.14 | 0.06 |
| 5 | 4.00 | 3.91 | 0.09 |
| 6 | 4.15 | 4.04 | 0.11 |
Every difference is positive, so B has lower validation MSE on all six splits. The mean difference is \((0.12+0.08+0.10+0.06+0.09+0.11)/6=0.56/6\approx0.09333\) squared points. The median is \((0.09+0.10)/2=0.095\) squared points.
The sample standard deviation is
Thus, B’s ranking is consistent across these splits, with a mean paired advantage of about \(0.09333\) squared points in validation MSE. That is a description of this comparison, not a guarantee that B will rank first under every possible sample or validation design. Whether the difference matters for the intended use requires context about the outcome and the consequences of prediction error.
What Split-to-Split Stability Can and Cannot Establish
Validation splits are usually generated from the same observed data set. Their training sets and validation sets may overlap, so the resulting differences are generally not independent replications. In particular, it is not appropriate to treat the standard deviation across a collection of overlapping splits as if it were a standard error based on independent studies, or to use it mechanically to create a confidence interval or significance test. The summaries in this tutorial describe sensitivity to the splits that were actually evaluated.
The observed pattern also depends on how the splits were constructed. A comparison made with one splitting scheme answers a question about performance under that scheme and the available data. It may change if the data are limited, if a few observations strongly affect the fitted curves, or if the split design does not reflect the way the curves will be used. The candidate curves must be fitted using the training portion of each split and evaluated on that split’s held-out observations; otherwise, the validation errors do not represent the intended out-of-sample comparison.
A careful report states the validation procedure, gives the error measure and its units, and summarizes both ranking frequency and paired differences. If the rankings vary, describe that instability directly rather than presenting a single winner as settled. If one curve wins consistently, report that consistency while still distinguishing it from evidence about how large or useful the advantage is.
Common Mistake
Check Your Understanding
Use the sign convention \(d_j=MSE_{A,j}-MSE_{B,j}\) in your answers.
- If \(d_j=-0.18\) squared points on a split, which curve has the lower validation MSE, and by how much?
- Across five splits, the differences are \(-0.02,-0.03,-0.01,-0.02,\) and \(0.20\) squared points. Which curve wins more splits? Which curve has the lower average MSE?
- Why should A and B be evaluated on the same validation split when calculating paired differences?
- What information does the standard deviation of paired differences add to a sign count?
- Why should variation across overlapping validation splits not automatically be treated as independent-replication uncertainty?