Choosing the Number of Cross-Validation Folds

Why the Number of Folds Matters

In K-fold cross-validation, the observations are divided into \(K\) nonempty folds. The model is fitted \(K\) times; on each fit, one fold is held out for validation and the other \(K-1\) folds are used for training. The resulting losses are combined to estimate predictive performance. Choosing \(K\) changes both the data available to each fit and the amount of computation required.

A smaller \(K\) means fewer fits, but each fit uses a smaller fraction of the observations. A larger \(K\) gives each fit more training data, but requires more fits. Leave-one-out cross-validation (LOOCV) is the limiting case \(K=n\), where \(n\) is the number of observations and each validation fold contains one observation. There is no universally best value: the choice depends on the data, the model-fitting cost, and the purpose of the estimate.

Definition: In \(K\)-fold cross-validation, \(n\) observations are assigned to \(K\) nonempty validation folds. For fold \(k\), fit the model using all observations outside that fold, then evaluate its loss on the held-out observations. The \(K\) validation losses are combined into one cross-validation estimate.

Fold Sizes, Training Sizes, and the Estimate

Write \(m_k\) for the number of observations in fold \(k\), so that \(\sum_{k=1}^{K}m_k=n\). The corresponding training set has \(n-m_k\) observations. If folds are balanced and \(n\) is not divisible by \(K\), their sizes differ by at most one. In particular, some folds have \(\lfloor n/K\rfloor\) observations and the rest have \(\lceil n/K\rceil\). An arbitrary partition need not be balanced.

Let \(\hat f^{(-k)}\) denote the model fitted without fold \(k\), and let \(L(y_i,\hat f^{(-k)}(x_i))\) be the loss for observation \(i\) when it is in the held-out fold. A standard observation-weighted cross-validation estimate is the average held-out loss over all \(n\) observations:

\widehat{R}_{\mathrm{CV}} =\frac{1}{n}\sum_{k=1}^{K}\sum_{i\in F_k} L\!\left(y_i,\hat f^{(-k)}(x_i)\right)

If \(\bar L_k\) is the mean loss within fold \(k\), the same estimate is \(\sum_{k=1}^{K}(m_k/n)\bar L_k\). When all folds have equal size, this reduces to the simple mean of the \(K\) fold means. When they differ in size, averaging the fold means equally gives each fold the same weight, not each observation the same weight.

Assumptions and scope: The usual interpretation assumes observations are independent and drawn from the same target population, and that the validation loss represents the predictive task of interest. The estimate describes the fitting procedure trained on approximately \(n(1-1/K)\) observations, not exactly the final model refitted on all \(n\). If observations are dependent or have a structure that must be preserved, ordinary random-fold cross-validation may not be suitable.

How the Choice of \(K\) Changes the Tradeoff

For balanced folds, each training fit uses approximately \(n(K-1)/K\) observations, and each validation fold contains approximately \(n/K\). The number of model fits is \(K\). These relationships give a useful first comparison:

Choice of \(K\)	Approximate training size per fit	Fits	Main consideration
Small \(K\), such as 2 or 3	\(n(K-1)/K\), a relatively small fraction of \(n\)	Few	Low fitting cost, but the estimate concerns models trained on substantially less data than the final model.
Moderate \(K\), commonly 5 or 10	About 80% or 90% of \(n\)	5 or 10	A practical balance between training-set size and fitting cost in many applications.
LOOCV, \(K=n\)	\(n-1\)	\(n\)	Each fit uses nearly all observations, but the procedure can be computationally costly and its estimate can be statistically variable.

The table is a guide to the design, not a guarantee about which estimate will be closest to the future prediction error. Small-\(K\) validation evaluates models trained on fewer observations than the final model. If predictive performance improves with training-set size, this difference can make the estimate pessimistic for the final full-data fit, although the direction and size of the discrepancy depend on the learning problem.

LOOCV reduces that training-size discrepancy, but it does not make the \(n\) validation losses independent: the training sets overlap heavily. Its estimate can have high sampling variability, even though each fit uses nearly all the data. For a fixed data set, LOOCV also has no random partition to vary, but this does not mean its estimate has no statistical uncertainty. Moderate \(K\) is often a practical compromise, not a theorem that always minimizes bias or variance.

Worked Example: Weighting Unequal Fold Sizes Correctly

A hypothetical regression analysis uses 21 independent policy records and balanced five-fold validation. The five fold sizes are 4, 4, 4, 4, and 5. Their respective mean squared validation losses are 2, 3, 5, 4, and 6. Calculate the observation-weighted cross-validation estimate, and compare it with the unweighted average of fold means.

Method. Use each fold's size as its weight, because the requested estimate averages loss over observations. The training sizes are \(21-4=17\) for each of the first four fits and \(21-5=16\) for the fifth fit.

The total loss in each fold is its size multiplied by its mean loss: \(4(2)=8\), \(4(3)=12\), \(4(5)=20\), \(4(4)=16\), and \(5(6)=30\). Their sum is \(8+12+20+16+30=86\). Therefore,

\widehat{R}_{\mathrm{CV}} =\frac{86}{21} =4.095238\ldots \approx 4.0952

The unweighted mean of the five fold means is \((2+3+5+4+6)/5=20/5=4\). It differs because it gives the fold of size 5 the same weight as each fold of size 4. The observation-weighted estimate is 4.0952, rounded to four decimal places.

Check and common mistake. The fold totals sum to 86, and the fold sizes sum to 21, so the estimate is \(86/21\). Using 82 in the numerator is an arithmetic error; using 4 as the observation-weighted answer confuses an equal-fold average with an equal-observation average.

A Short Derivation of the Weighted Formula

The weighting formula follows directly from the definition of a mean. Each fold mean is \(\bar L_k=(1/m_k)\sum_{i\in F_k}L_i\), where \(L_i\) is the held-out loss for observation \(i\). Weighting that mean by its share of all observations cancels the fold denominator:

\sum_{k=1}^{K}\frac{m_k}{n}\bar L_k =\sum_{k=1}^{K}\frac{m_k}{n} \left(\frac{1}{m_k}\sum_{i\in F_k}L_i\right) =\frac{1}{n}\sum_{k=1}^{K}\sum_{i\in F_k}L_i

Every observation belongs to exactly one validation fold, so the final sum includes each held-out loss once. This proves that weighting fold means by \(m_k/n\) gives the mean loss per observation. Equal weighting of fold means is equivalent only when all \(m_k\) are equal.

Choose \(K\) From the Actual Constraints

A defensible choice starts with constraints and then considers statistical tradeoffs. Calculate the fold sizes and training sizes, estimate how many model fits are feasible, and decide whether the training fraction is adequate for the intended prediction task. A candidate \(K\) must satisfy all explicit requirements; only then is it useful to compare its likely stability with other feasible choices.

Check the data and target.
Confirm that ordinary folds are appropriate for the independent observations and that the validation loss matches the prediction goal.

Calculate the fold sizes.
For balanced folds, determine how many contain \(\lfloor n/K\rfloor\) and how many contain \(\lceil n/K\rceil\) observations.

Check training size and computation.
For each fold, the training size is \(n-m_k\); the basic number of fits is \(K\). Compare these with the minimum training size and fitting budget.

Choose among feasible values.
Prefer a practical balance: moderate \(K\) is often suitable when neither computation nor training size is an extreme constraint; use a smaller or larger value when the stated constraints justify it.

Worked Example: Meeting a Training-Size and Compute Budget

A hypothetical pricing model has \(n=120\) independent records. One fit takes 18 seconds, fits are run sequentially, and the budget is 240 seconds. The analyst requires at least 90% of the records in every training set. Compare \(K=5\), \(K=10\), and LOOCV, then select a feasible choice.

Method. For balanced folds, compare the smallest training set among folds with the requirement of \(0.90(120)=108\) records. Also multiply the number of fits by 18 seconds.

For \(K=5\), each validation fold has \(120/5=24\) records, so each training set has \(120-24=96\) records. This is \(96/120=0.80\), below the required 0.90. The fitting time is \(5(18)=90\) seconds, within budget, but the training-size requirement fails.

For \(K=10\), each fold has 12 records and each training set has \(120-12=108\) records, exactly 90% of the data. The fitting time is \(10(18)=180\) seconds, within the 240-second budget. This choice meets both requirements.

For LOOCV, there are 120 fits, each trained on 119 records. The total fitting time is \(120(18)=2{,}160\) seconds, or 36 minutes, exceeding the budget.

Decision and check. Choose \(K=10\): it is the only listed choice satisfying both constraints. Its training sets contain 108 observations and its fits take 180 seconds, leaving \(240-180=60\) seconds of budget. The common mistake is to choose LOOCV just because it uses the largest training sets; it is infeasible under the stated computation limit.

Worked Example: Selecting a Fold Count for a Small Data Set

A hypothetical study has \(n=36\) independent records. A model fit takes 7 seconds. The analyst has a 45-second budget and requires each training set to contain at least 30 records. Compare balanced \(K=5\), \(K=6\), and LOOCV.

Method. For each choice, calculate the largest validation fold, since that fold produces the smallest training set. Then check both the minimum training size and the \(7K\)-second fitting cost.

For \(K=5\), a balanced partition has one fold of 8 and four folds of 7, because \(36=5(7)+1\). The smallest training set therefore has \(36-8=28\) records, below 30. The computation takes \(5(7)=35\) seconds.

For \(K=6\), each fold has \(36/6=6\) records. Every training set has \(36-6=30\) records, meeting the minimum exactly. The computation takes \(6(7)=42\) seconds, within budget.

For LOOCV, each training set has \(36-1=35\) records, but the 36 fits take \(36(7)=252\) seconds, exceeding the budget.

Decision and check. Choose \(K=6\): its smallest training size is 30 and its total fitting time is 42 seconds. The unbalanced fold sizes for \(K=5\) are important; assuming all five folds have 7 observations would incorrectly imply 35 observations total and miss the fold of size 8.

Practical Interpretation and Exam Traps

The fold count controls the design of the validation exercise, not the intrinsic quality of a model. A lower cross-validation loss at one value of \(K\) does not by itself prove that this value is best: the estimates concern different training-set sizes and can vary with the fold assignment. Make the choice from the stated purpose and constraints rather than selecting whichever result happens to be smallest.

Do not equate LOOCV with zero variability. Each fit leaves out only one observation, but the resulting losses are dependent and the estimate can still vary across samples.
Do not assume uneven folds are balanced. Fold sizes differ by at most one only when the partition is deliberately balanced.
Do not average unequal fold means equally when the target is mean loss per observation. Use the fold-size-weighted mean.
Do not ignore training-set size. A small \(K\) may be inexpensive while evaluating models trained on much less data than the final fit.
Do not count computation as only one fit. Ordinary \(K\)-fold validation requires \(K\) fits; LOOCV requires \(n\), unless a special computational shortcut is explicitly part of the method.

The practical rule is conditional, not absolute: choose the largest fold count that is feasible only if the additional training data and resulting estimate suit the task. When compute is limited, a smaller \(K\) may be necessary; when \(n\) is modest and fitting is cheap, a moderate or larger \(K\) may be feasible. State the relevant quantities and justify the choice.

Key takeaway: Compare \(K\) using three quantities: the training observations \(n-m_k\), the number of fits \(K\), and the way fold losses are aggregated. Moderate \(K\) often balances these considerations, while LOOCV and small-\(K\) validation each have settings in which they are appropriate.

Check Your Understanding

For each problem, identify the relevant training size, fit count, or loss weighting before checking the final answer.

A hypothetical data set has 50 independent records and is split into five balanced folds. How many observations are in each validation fold and training set, and how many model fits are required? (Answer: 10 validation observations, 40 training observations per fit, and 5 fits.)
A balanced four-fold validation uses 23 observations. How many folds have 5 observations, how many have 6, and what are the corresponding training sizes? (Answer: Three folds of size 6 and one of size 5; the training sizes are three fits of 17 and one fit of 18.)
Three validation folds have sizes 3, 3, and 4, with mean losses 1, 4, and 7. Find the observation-weighted estimate. (Answer: \((3(1)+3(4)+4(7))/10=43/10=4.3\).)
A model has \(n=80\) observations, takes 12 seconds per fit, and has a 120-second sequential fitting budget. Is \(K=10\) feasible on computation alone, and how many seconds remain? (Answer: Yes; \(10(12)=120\) seconds, so 0 seconds remain.)
A balanced five-fold validation has \(n=42\) observations. What are the validation fold sizes and the smallest training-set size? (Answer: Three folds have 8 observations and two have 9; the smallest training set has \(42-9=33\) observations.)

What You'll Learn