Choosing the Number of Folds

Why the Number of Folds Matters

In \(K\)-fold cross-validation, the data are divided into \(K\) parts, or folds. Each fold takes a turn serving as the validation set while the model is fitted using the other folds. The resulting held-out predictions are used to summarize how well the fitting procedure predicts observations it did not use for that fit.

Choosing \(K\) changes two things directly: how much data is available to fit each model and how many model fits are needed. It also changes the sizes and overlap of the validation and training sets, which can affect how variable the cross-validation results are. There is no single value of \(K\) that is best for every data set and prediction task. The useful choice balances these considerations.

For concreteness, suppose the prediction outcome is quantitative and performance is measured by mean squared error (MSE). Write \(I_k\) for the set of observation indices in fold \(k\), and \(\hat{f}^{(-k)}(x_i)\) for the prediction at predictor value \(x_i\) from a model fitted without fold \(k\). The cross-validation MSE is the average squared prediction error across all held-out observations:

This expression weights each observation equally. When fold sizes are equal, it is also the average of the \(K\) fold-specific MSEs. When fold sizes differ, averaging the fold-specific MSEs equally would give a small fold the same weight as a large fold, so the observation-level formula is the appropriate overall average.

The interpretation depends on the data and procedure. The folds must assign each observation to one held-out fold, and a held-out observation must not be used to fit the model that predicts it. The fitting and scoring procedures should be the same across folds. The data should also be suitable for the intended prediction target: for example, a randomly sampled set of individuals is not automatically an appropriate stand-in for predictions about a different population or a different time period. The fold assignment must respect important structure in the data when such structure is part of the prediction problem.

How \(K\) Sets Training and Validation Sizes

If the folds are as equal in size as possible, each contains approximately \(n/K\) observations. A model fit in one round therefore uses approximately

observations for training, while approximately \(n/K\) observations are held out for validation. These are approximate sizes when \(n\) is not divisible by \(K\): some folds must then contain one more observation than others.

Increasing \(K\) makes each training set larger and each validation fold smaller. A larger training set can make each fitted model more like a model trained on the full data set. But each individual validation score is based on fewer observations, so it can be more sensitive to which cases happen to be in that fold. These are tendencies, not guarantees about the final cross-validation estimate.

Computational Cost and Common Choices

For one specified fitting procedure, \(K\)-fold cross-validation requires \(K\) fits: one for each held-out fold. If fitting a model takes roughly the same time for each fold, doubling \(K\) roughly doubles the time spent fitting that model. Prediction and scoring also require work, but the model fits are often the main cost. Actual time can differ if fit sizes differ substantially or the fitting algorithm behaves differently at those sizes.

Five-fold and ten-fold cross-validation are common practical choices. Five folds require fewer fits and use validation sets that are relatively large. Ten folds require twice as many fits as five folds, but use larger training sets. Neither choice automatically gives a more accurate estimate in every setting. The decision should reflect the available computing time, the sample size, and how important it is for each fitted model to use a large fraction of the data.

What Happens to Variability?

It helps to distinguish two kinds of variation. Variation among fold scores means that the errors calculated in different held-out folds are not all the same. Variation in the overall cross-validation estimate means that the final average would change if the data or the fold assignment changed. Fold scores are pieces of one cross-validation calculation; they are not independent estimates whose spread directly determines the uncertainty of the overall result.

When \(K\) is larger, each validation fold is smaller. A fold-specific score based on fewer observations can respond strongly to a few unusually easy or difficult cases. At the same time, the models use more training data, which may reduce the effect of the smaller training sample compared with a lower \(K\). The training sets also overlap substantially, so the fold results are related to one another. Consequently, there is no general rule that increasing \(K\) must increase or must decrease the variability of the overall estimate.

The sample size matters to this tradeoff. With a small data set, even a moderate \(K\) may leave each validation fold with very few observations, making individual fold scores sensitive to case composition. A smaller \(K\) gives larger validation folds but trains each model on a smaller part of the data. A larger \(K\) trains each model on more observations but demands more fitting and creates smaller validation folds. These considerations motivate inspecting fold sizes before choosing \(K\), rather than treating a conventional value as universally appropriate.

A Practical Choice

A useful choice begins with the sizes and structure of the data, then accounts for the cost of fitting. Five folds are a reasonable practical option when fitting time matters or validation folds need to be reasonably large. Ten folds may be attractive when a larger training set in each round is important and the additional fitting is affordable. If either choice produces very small validation folds, or if the data have important grouping or time structure, the fold design needs particular care.

The choice of \(K\) does not repair an unsuitable sample, an inappropriate prediction target, or a poorly chosen performance measure. It controls how the available observations are divided between training and validation in each round. Interpret the resulting cross-validation score as an estimate for the fitting procedure and evaluation setup actually used.

Common Mistake

• Wrong reasoning: “Ten-fold cross-validation uses more training data, so its estimate must be more accurate and less variable than a five-fold estimate.”
• Correction: Ten folds do use larger training sets, but they also have smaller validation folds and require more fits. Smaller folds can produce more variable fold-specific scores, and the net variability of the overall estimate is not guaranteed to move in one direction. Choose \(K\) based on the full tradeoff, not on training-set size alone.