Assigning Folds for Continuous Outcomes

Why Fold Assignment Matters

In \(K\)-fold cross-validation, each observation is held out once while a model is fitted using the observations in the other folds. The resulting prediction errors are combined to estimate how well the fitting procedure predicts cases from a target population. Fold assignment therefore affects which cases are compared with one another and, in a finite data set, how representative each held-out fold is.

With a continuous outcome, a purely shuffled assignment can put many low outcomes in one fold and many high outcomes in another. This can happen by chance, especially in a small data set or when a few values are extreme. Approximate outcome-range balancing is an alternative: it tries to put observations from across the observed outcome range into every fold. It is a way to make fold composition more even, not a way to make the folds identical or to improve the fitted model itself.

These are assignment strategies for the same cross-validation procedure. They do not change which cases are in the data set. They do change the particular training and held-out sets used in each round, so the resulting estimate can vary with the assignment.

Shuffled Random Assignment

A basic approach is to shuffle the case labels and allocate the cases in that random order to folds, cycling through the fold labels. This produces fold sizes that differ by at most one. Each observation is assigned to exactly one fold, and no case is deliberately placed according to its outcome.

When cases are independent and come from the same target population, random assignment is a natural way to form folds: before the assignment, each case has the same chance of being in any fold. It does not guarantee that every fold will look alike. In particular, random assignment cannot guarantee equal proportions of low, middle, and high outcomes in a finite sample.

Approximate Balancing Across Outcome Ranges

To balance the outcome range, first divide the observed outcomes into ordered groups, often using ranges or ranks. Then distribute observations from each group across the folds, with allocations made as evenly as possible. The groups should generally reflect their observed frequencies: if one range contains more cases than another, its per-fold allocations should usually be correspondingly larger. Randomizing the allocation within groups helps avoid a fixed assignment pattern.

The goal is broad coverage in each fold, not identical outcomes. Exact equality of fold means is neither required nor guaranteed. A rare extreme value, for example, cannot appear in every fold if there is only one such case. Approximate balancing can still prevent all the observations in a broad part of the outcome range from being concentrated in one fold.

What Balancing Does—and Does Not—Change

A useful distinction is between the target population of the evaluation and the composition of individual folds. Suppose the cases are sampled from the population whose prediction performance is of interest. A fold assignment rearranges these observed cases into training and held-out sets; it does not, by itself, replace the original population with a different one. When outcome groups are represented in each fold in proportion to their observed frequencies, balancing can reduce accidental differences in outcome-range composition while retaining the overall mix represented by the data.

The interpretation depends on how results are summarized. If the evaluation target is average prediction error for cases drawn from the original population, each case should contribute according to that target. If groups are deliberately given equal representation even though their population frequencies differ, an unweighted average across groups may instead describe a population in which those groups are equally common. To target the original population in that situation, group contributions need weights reflecting the target population frequencies.

Balancing is not a correction for a nonrepresentative sample. It cannot recover outcome ranges absent from the data, undo selection bias, or make observations independent when the data have a grouped or time-ordered structure. Nor does it guarantee less variability in every cross-validation result: prediction errors also depend on the predictors, fitted models, and which observations are in the training sets.

The sum includes only groups that have observations in fold \(k\). This restriction matters: when \(n_{hk}=0\), the group mean \(\bar{y}_{hk}\) is undefined, so its product with zero must not be included as though it were an ordinary term.

Why the Weighted Mean Identity Holds

The identity follows by adding the outcomes group by group within a fold. For each group with observations in fold \(k\), the definition of its mean gives \(\sum_{i\in(h,k)}y_i=n_{hk}\bar{y}_{hk}\). Groups with no cases contribute no outcomes and are omitted. Thus the sum of all outcomes in fold \(k\) is the sum of these group totals. Dividing by the fold size gives its mean:

Across all folds, each observation occurs exactly once. Therefore, adding the fold outcome totals recovers the total of all observed outcomes, and dividing by the total sample size recovers the full-data mean. This is an accounting identity, not evidence that fold means are equal or that prediction errors are equal.

Choosing Between the Two Approaches

Shuffled random assignment is a clear default when observations are exchangeable and the data set is large enough that each fold is likely to cover the outcome range adequately. Approximate outcome-range balancing can be useful when the sample is modest, the outcome is skewed, or a small number of observations occupy unusual parts of the range. It can make fold-level summaries less dependent on an unlucky concentration of outcomes.

Use balancing as a controlled allocation, not as a search for the assignment that produces the most favorable prediction score. One sound approach is to create outcome groups, randomize cases within each group, and allocate each group across folds as evenly as fold sizes permit. Keep the groups sufficiently broad to avoid creating many tiny groups, and check the final fold sizes and outcome summaries. Exact group boundaries are a practical choice; they do not define new scientific categories.

Outcome-based assignment should also be distinguished from using the outcome as a predictor. The outcome is used here only to organize the held-out sets. It must not be supplied to the model as an input for predicting that outcome. The observed outcome is needed to score predictions, and its use to define a balanced partition does not make it available as a predictor at deployment. However, choosing fold assignments after examining which arrangement gives a preferred model result would compromise an impartial evaluation.

Finally, random case-level assignment is not suitable for every data structure. If several records come from the same person, clinic, or other cluster, splitting those records across folds may make held-out cases too similar to training cases. For time-ordered data, randomizing across time can likewise fail to reflect prediction into the future. In such settings, assignment must respect the relevant grouping or time structure; balancing the continuous outcome does not solve that problem.

Common Mistakes

• Expecting balanced folds to have equal means. Outcome-range balancing aims for coverage, not exact equality. Small samples and extreme observations can still produce noticeably different fold means.
• Equalizing groups that are not equally common. Giving every outcome group the same number of cases in each fold can distort the target mix. Use allocations that reflect observed frequencies when the goal is to represent the original population.
• Interpreting a balanced assignment as a representative sample guarantee. It changes the arrangement of the observed cases; it cannot correct a sample that does not represent the target population.
• Including a nonexistent group mean in a calculation. If a fold has no cases from a group, its group mean is undefined. Omit that group from the weighted sum rather than writing a zero count multiplied by an undefined mean.
• Ignoring dependence in the data. Outcome balance does not justify splitting repeated measurements, clusters, or time series as if every case were independent.