Diagnosing Variation Across Folds

Why Inspect Scores Fold by Fold?

A single cross-validation score summarizes performance, but it can conceal important differences among the held-out subsets. A model may predict well for most folds and poorly for one, for example because that fold contains outcome values rarely represented elsewhere. Or one unusual observation may contribute a large share of a fold’s error. Looking at fold-level scores and held-out outcomes helps explain what the overall summary is describing.

The purpose is diagnosis, not to replace the overall estimate with whichever fold looks most favorable. A high score in one fold is a signal to investigate the observations and their outcome coverage. It is not, by itself, evidence that the model is universally poor, that the observation is erroneous, or that the cross-validation estimate is unreliable.

Definition: In \(K\)-fold cross-validation, let \(V_k\) denote the held-out observations in fold \(k\), with \(n_k\) observations. Let \(\hat{f}_{-k}\) be the fitted prediction procedure using the observations outside \(V_k\), and let \(\hat{y}_{i,-k}\) be its prediction for held-out observation \(i\). A fold-level score is a chosen measure of prediction performance calculated using only the observed outcomes and predictions for \(i \in V_k\). Larger values indicate worse performance for error scores such as RMSE and MAE.

For a quantitative outcome, two common error scores are the root mean squared error (RMSE) and the mean absolute error (MAE). Here, \(y_i\) is the observed outcome and \(\hat{y}_{i,-k}\) is its held-out prediction. Both scores are expressed in the outcome’s units.

\operatorname{RMSE}_k = \sqrt{\frac{1}{n_k}\sum_{i\in V_k}(y_i-\hat{y}_{i,-k})^2}, \qquad \operatorname{MAE}_k = \frac{1}{n_k}\sum_{i\in V_k}|y_i-\hat{y}_{i,-k}|

A fair comparison across folds requires the same prediction procedure and the same score definition in every fold. The held-out observations must not have been used to fit the corresponding fold model. When comparing scores, also note the fold sizes: a fold based on very few observations can show substantial variation just because each individual observation has more weight.

What a Difference Between Folds Can Reveal

Fold scores can vary for several reasons. A held-out subset may contain outcomes in a range that is uncommon in the other folds. It may include a few observations that are difficult to predict, or it may represent a distinct kind of case. Differences can also arise from the particular training subsets used to fit the fold models. The score alone cannot identify which explanation applies.

Start by comparing fold sizes and scores, then examine the held-out outcome distributions. Useful summaries include the minimum and maximum outcomes, the mean or median, and the number of observations in notable ranges. For outcomes with a meaningful practical threshold, report how many held-out observations fall above or below it. The aim is to see whether some folds cover outcome values that are scarce or absent in others—not to assume that matching outcome ranges guarantees representative folds.

Worked Example: Finding a Fold With a High Score

Suppose a hypothetical five-fold evaluation uses 20 held-out observations per fold and reports these RMSE values, in outcome units: 2, 3, 3, 4, and 8. The stated cross-validation summary is the arithmetic mean of the five fold RMSEs. Calculate that summary and identify what merits investigation.

Calculate the summary. Each fold has the same size, and the specified summary averages the fold-level RMSE values:

\overline{\operatorname{RMSE}}_{\text{fold}} = \frac{2+3+3+4+8}{5} = \frac{20}{5} = 4

The mean fold RMSE is 4 outcome units. The fifth fold’s RMSE of 8 is notably larger than the other four scores, which range from 2 to 4. That gap is a reason to inspect the fifth fold’s held-out outcomes and its individual prediction errors. It is not enough to conclude that the entire evaluation is unstable or that the fifth fold should be discarded.

If the fifth fold contains outcomes at the edge of the observed range, the contrast may reflect uneven outcome coverage. If its outcomes resemble those of the other folds but a small number of cases have large errors, those cases may account for much of the score. These explanations require looking at the held-out data; neither follows from the value 8 alone.

Outcome Coverage and Case-Level Contributions

Fold assignment affects which outcomes are held out together. A fold that contains mostly low outcomes while another contains mostly high outcomes gives a different test of prediction performance in each subset. This can happen by chance, especially in a small data set. For continuous outcomes, checking ranges and distributions across folds is a useful diagnostic even when fold assignment was randomized.

A high RMSE can be driven by one or more large errors because squaring gives greater weight to large differences. A high MAE can also reflect difficult cases, but it increases in direct proportion to the absolute error rather than its square. Examining case-level contributions therefore requires knowing the score being used. An observation that has a large effect on RMSE need not affect MAE to the same degree.

Worked Example: Checking Uneven Outcome Coverage

In a hypothetical five-fold evaluation, each fold has 10 held-out observations. The outcome is measured in units, and the fold summaries are:

Fold	Held-out outcome range	Mean outcome	RMSE
1	18 to 27	22.5	2.1
2	20 to 29	24.0	2.4
3	21 to 31	26.0	2.0
4	22 to 32	27.0	2.3
5	75 to 92	83.0	7.8

The mean fold RMSE is

\frac{2.1+2.4+2.0+2.3+7.8}{5} = \frac{16.6}{5} = 3.32

Fold 5 is unusual in both its outcome coverage and its score: its outcomes lie far above the ranges in folds 1 through 4, and its RMSE is much larger. This pattern suggests a plausible diagnostic question: did the fold model have difficulty predicting outcomes in that higher range? To answer it, inspect predictions and errors within fold 5. The range and score alone do not establish why predictions were poor.

This fold assignment also leaves the evaluation with little evidence about performance on outcomes from 75 to 92 in the other held-out subsets. The overall mean of 3.32 is still the specified summary of these five folds, but it should be described alongside the coverage limitation. A single average cannot make the fold outcome distributions more alike.

How a Few Observations Can Drive a Score

A case-level check can show whether a high fold score reflects broadly elevated errors or a small number of large errors. For RMSE, the squared error contributed by a held-out observation is \((y_i-\hat{y}_{i,-k})^2\). Comparing these contributions within a fold identifies which observations contribute most to the sum of squared errors. This is a statement about their influence on that fold’s score, not about their influence on the model’s fit: they were held out when that fold model was fitted.

Worked Example: One Case Dominates a Fold’s RMSE

Consider a hypothetical fold with four held-out observations. Their observed outcomes are 10, 12, 13, and 30 units, and their corresponding predictions are 11, 11, 14, and 23 units. Calculate the MAE and RMSE, then check whether one observation contributes much of the squared error.

The absolute errors are \(1, 1, 1,\) and \(7\) units. The squared errors are \(1, 1, 1,\) and \(49\) square units. Thus:

\operatorname{MAE}_k = \frac{1+1+1+7}{4} = \frac{10}{4} = 2.5\text{ units}

\operatorname{RMSE}_k = \sqrt{\frac{1+1+1+49}{4}} = \sqrt{13} \approx 3.606\text{ units}

The observation with outcome 30 has squared error \(49\), while the three other observations together have squared errors \(1+1+1=3\). It therefore contributes most of the fold’s squared-error total. Its large error merits checking in context: perhaps it is a valid but difficult case, or perhaps there is a data or prediction issue to investigate. The calculation does not justify deleting it. Removing a valid difficult observation would change the evaluation question rather than simply repair the score.

Fold Averages and Pooled Scores

When folds have equal sizes, a common summary is the arithmetic mean of the fold scores. However, averaging fold RMSEs is not the same operation as pooling all held-out squared errors and then taking a square root. It is important to state which summary is reported, especially when explaining a difference between a fold-level pattern and an overall score.

Let \(N=\sum_{k=1}^K n_k\) be the total number of held-out observations across folds, and let \(\operatorname{MSE}_k=\operatorname{RMSE}_k^2\). Pooling the squared errors first gives:

\operatorname{RMSE}_{\text{pooled}} = \sqrt{ \frac{\sum_{k=1}^K \sum_{i\in V_k}(y_i-\hat{y}_{i,-k})^2}{N} } = \sqrt{ \frac{\sum_{k=1}^K n_k\operatorname{MSE}_k}{N} }

The second equality follows because, by definition, the sum of squared errors in fold \(k\) is \(n_k\operatorname{MSE}_k\). In contrast, the arithmetic mean of fold RMSEs is \(\frac{1}{K}\sum_{k=1}^K \operatorname{RMSE}_k\). These formulas use different operations: one averages squared errors with fold-size weights before taking a square root; the other averages the fold square roots. They need not agree. This distinction is about how the score is aggregated, not evidence that one fold is more difficult.

Fold scores are also not independent replications of a model: the training subsets overlap, and each score is calculated on a different held-out subset. Their spread is useful descriptive information, but it should not automatically be treated as a standard error or confidence interval for the overall estimate. The earlier material on uncertainty in cross-validation estimates addresses uncertainty; fold diagnosis helps describe patterns that an overall uncertainty summary may not explain.

A Practical Diagnostic Sequence

Use the same metric and evaluation protocol in every fold, then move from the overall pattern to the observations that might explain it. A disciplined sequence makes it less likely that an unusual score will be overinterpreted.

Check the score pattern.
List each fold’s score and size. Identify unusually high or low values without treating the most extreme fold as a verdict on the whole procedure.

Compare held-out outcome coverage.
Check ranges and suitable distribution summaries. Note whether a fold contains values that are scarce or absent elsewhere.

Inspect case-level errors.
For a fold with a high error score, determine whether many cases contribute or whether a few contribute disproportionately. Interpret contributions using the chosen metric.

Report both the summary and the diagnostic.
Keep the prespecified overall score, and describe important fold differences and coverage limitations without silently removing observations.

Common Mistakes

Calling the worst fold the model’s true performance. A fold score describes performance on that held-out subset under its fold model. It does not alone describe every future subset.
Calling fold-to-fold spread a confidence interval. The scores describe variation across the chosen folds; their spread is not automatically a standard error or interval for the overall estimate.
Blaming outcome coverage without checking errors. An unusual outcome range can suggest an explanation, but predictions and case-level errors are needed to assess whether it accounts for a high score.
Removing a difficult held-out case to improve the result. First determine whether the observation is invalid under the data-collection rules. A valid hard-to-predict case is part of the performance question, not an automatic exclusion.
Comparing unlike summaries. A mean of fold RMSEs and a pooled RMSE are not interchangeable. State how the reported overall score was calculated.

Key takeaway: Fold-level diagnosis explains where cross-validation performance is strong or weak and what held-out observations may contribute to that pattern. Interpret scores alongside fold sizes and outcome coverage, distinguish case-level contributions from training influence, and report the aggregation method clearly.

Check Your Understanding

Answer each question using the definitions and diagnostic principles in this tutorial.

A fold has a much higher RMSE than the others. Name two checks that could help explain the difference.
Why can a valid held-out observation with a large error strongly affect RMSE? Does that mean it influenced the model fit for that fold?
What does it suggest if one fold covers a much higher outcome range than all the others? What further evidence would you inspect before explaining its score?
Explain why the arithmetic mean of fold RMSEs need not equal the pooled RMSE.
Why should fold-to-fold variation not automatically be reported as a confidence interval for the overall cross-validation estimate?