The K-Fold Cross-Validation Estimate

What the K-Fold Estimate Measures

A prediction procedure is useful only if it performs well on observations that were not used to fit it. K-fold cross-validation estimates this performance by partitioning the observed data into $K$ folds, fitting the procedure $K$ times, and evaluating each fit on the fold left out of that fit. The estimate combines the losses from all held-out observations.

The unit being split must match the prediction target: for example, if the goal is to predict outcomes for new individuals, the held-out units should be individuals. The earlier material on choosing the unit of cross-validation addresses that design decision. Here, assume the units have been chosen appropriately and denote the observed data by $D=\{(X_i,Y_i):i=1,\ldots,n\}$, where $X_i$ denotes predictors and $Y_i$ the outcome.

Definition: A $K$-fold partition is a collection of nonempty, disjoint index sets $I_1,\ldots,I_K$ whose union is $\{1,\ldots,n\}$. Fold $k$ is the held-out set $I_k$, of size $n_k=|I_k|$. Fit the specified prediction procedure using the observations not in $I_k$, and let $\hat f^{(-k)}$ denote the resulting prediction rule. For a loss function $L(y,\hat y)$, the fold-specific mean loss is $$ \bar L_k=\frac{1}{n_k}\sum_{i\in I_k}L\bigl(Y_i,\hat f^{(-k)}(X_i)\bigr). $$ The pooled K-fold estimate is $$ \widehat R_{\mathrm{CV}}=\frac{1}{n}\sum_{k=1}^{K}\sum_{i\in I_k}L\bigl(Y_i,\hat f^{(-k)}(X_i)\bigr) =\sum_{k=1}^{K}\frac{n_k}{n}\bar L_k. $$

The loss determines what “prediction error” means. Squared error, $L(y,\hat y)=(y-\hat y)^2$, estimates error in squared outcome units. For binary classification, zero-one loss is 1 for an incorrect prediction and 0 for a correct one, so the estimate is the proportion misclassified. Other losses are possible, but the loss and prediction target should be stated when reporting the result.

The standard interpretation assumes the observations are independent draws from the target population, or are exchangeable under the intended sampling design; the fold assignment does not use held-out outcomes to construct their predictions. Each fitted procedure must be applied as it would be in practice, using only the training portion for data-dependent fitting steps. The loss must be defined for every held-out observation. Under these conditions, the estimate describes out-of-fold performance for the specified procedure and split design; it is not automatically an exact estimate of every model that might be fit to the full data.

Why the Estimate Weights Observations

The pooled formula gives each observation one loss contribution. A fold mean is a summary of its fold, so averaging fold means without weights is equivalent to giving every fold equal influence, even if the folds contain different numbers of observations. That is not generally the same as averaging the losses over all $n$ observations.

\sum_{k=1}^{K}\frac{n_k}{n}\bar L_k =\sum_{k=1}^{K}\frac{n_k}{n} \left\{\frac{1}{n_k}\sum_{i\in I_k}L\bigl(Y_i,\hat f^{(-k)}(X_i)\bigr)\right\} =\frac{1}{n}\sum_{i=1}^{n}L\bigl(Y_i,\hat f^{(-k(i))}(X_i)\bigr),

Here $k(i)$ is the unique fold containing observation $i$. The cancellation of $n_k$ shows that each observation contributes exactly $1/n$ times its held-out loss. If all folds have the same size, the weighted mean reduces to the simple average $K^{-1}\sum_k\bar L_k$. With unequal sizes, the weighted pooled estimate is the direct average over observations.

Worked Example: Unequal Fold Sizes and Squared Error

Suppose a hypothetical data set has $n=9$ observations divided into folds of sizes 4, 3, and 2. The squared prediction errors in the three held-out folds are, respectively, $1,4,0,9$; $1,0,4$; and $9,1$. Each prediction was produced by a model fitted without that observation's fold.

The first fold has total squared error $1+4+0+9=14$, so $\bar L_1=14/4=3.5$. The second has total $1+0+4=5$, so $\bar L_2=5/3\approx1.667$. The third has total $9+1=10$, so $\bar L_3=10/2=5$.

The pooled estimate is $$ \widehat R_{\mathrm{CV}}=\frac{4}{9}(3.5)+\frac{3}{9}(1.667)+\frac{2}{9}(5) =\frac{14+5+10}{9}=\frac{29}{9}\approx3.222. $$

Thus the estimated mean squared prediction error is about $3.222$ squared outcome units. The unweighted fold average would instead be $(3.5+1.667+5)/3\approx3.389$; it overweights the two-observation fold relative to the pooled observation-level average. If the outcome is measured in milligrams, for instance, the reported squared-error units are milligrams squared.

Worked Prediction-Error Calculations

Worked Example: Classification Error Across Folds

Consider a hypothetical binary-outcome prediction task with $n=14$ observations in three folds of sizes 5, 5, and 4. The fitted fold-specific rules misclassify 1, 2, and 0 held-out observations, respectively. Use zero-one loss.

The fold-specific error rates are $1/5=0.20$, $2/5=0.40$, and $0/4=0$. Weighting by fold size gives $$ \widehat R_{\mathrm{CV}}=\frac{5}{14}(0.20)+\frac{5}{14}(0.40)+\frac{4}{14}(0) =\frac{1+2+0}{14}=\frac{3}{14}\approx0.2143. $$

The estimated misclassification rate is about $21.43\%$ for this cross-validation procedure. Equivalently, three of the 14 held-out predictions were wrong. It would be incorrect to average the three fold rates equally: that calculation gives $(0.20+0.40+0)/3=0.20$, not the proportion of errors among all 14 observations.

Worked Example: What Training Size Does the Estimate Represent?

Suppose $n=12$ hypothetical observations are divided into four equal folds, each containing three observations. The fold-specific mean squared losses are $2,\ 4/3,\ 2,$ and $2$. Each fit uses the other nine observations.

Because the folds are equal, the pooled estimate is the mean of the fold means: $$ \widehat R_{\mathrm{CV}}=\frac{2+4/3+2+2}{4} =\frac{22/3}{4}=\frac{11}{6}\approx1.833. $$

The value estimates the average held-out squared loss from this procedure when it is trained on nine observations, under the split and sampling conditions used. It is not quite a direct evaluation of a single model trained on all 12 observations. More generally, for equal folds each fit uses $n(K-1)/K$ observations. This distinction matters when performance changes with training-set size: the cross-validation estimate can differ from the expected error of the final model refit using all available observations.

Why Fold Results Are Dependent

The $K$ fold means are not $K$ independent replications. Each $\hat f^{(-k)}$ is fitted using most of the data, and different fits share many training observations. For two distinct folds $j$ and $k$, their training sets are $D\setminus I_j$ and $D\setminus I_k$; for equal folds of size $n/K$, they share $n-2n/K$ observations. Shared training data can make the fitted rules, and therefore their held-out losses, statistically related.

The held-out observations differ across folds, but that alone does not make the fold losses independent: the predictions depend on overlapping training data. Consequently, the variation among the $K$ fold means is useful as a diagnostic of heterogeneity across folds, but treating those means as independent observations and dividing their standard deviation by $\sqrt K$ does not, in general, provide a justified standard error for the cross-validation estimate.

A population formulation clarifies the target. Let $\mathcal A$ be a fixed learning procedure, $D_m$ a training sample of size $m$, and $(X,Y)$ a new observation from the target population, independent of $D_m$. Its expected prediction error is $$ R_m(\mathcal A)=E_{D_m,(X,Y)} \left[L\bigl(Y,\mathcal A(D_m)(X)\bigr)\right]. $$ With equal folds, K-fold cross-validation applies $\mathcal A$ to training sets of size $m=n(K-1)/K$ and averages the resulting held-out losses. The computed estimate is random because both the observed data and, when randomized, the partition can vary. It is not an exact calculation of $R_m(\mathcal A)$ for a particular data set.

Justification: If each training set has $m$ observations and each held-out observation is an independent draw from the same target population as those training observations, then conditional on a fitted rule, its held-out loss has the target-population interpretation for that rule. Averaging across held-out observations and folds estimates the corresponding procedure-level risk. However, the fitted rules share training data, so this argument does not make the fold losses independent or imply that their sample variation is an ordinary independent-replicate standard error.

Choosing and Checking a K-Fold Estimate

For roughly equal folds, common choices include $K=5$ and $K=10$, but neither is universally optimal. Increasing $K$ makes each training set closer in size to the full data set, while requiring more fits. When $K$ is large, training sets overlap extensively, so fold results remain dependent; a large $K$ does not turn the folds into independent test studies. The suitable choice depends on computation, sample size, and the prediction target.

A single holdout split is a simpler alternative: fit once and evaluate on one held-out set. It requires less computation but the result can depend strongly on that particular split, and only the holdout observations contribute direct evaluation losses. K-fold cross-validation uses each observation for held-out evaluation once and often makes more efficient use of a modest data set. Leave-one-out cross-validation is the special case $K=n$; it trains $n$ models on $n-1$ observations each, which can be computationally expensive and still yields strongly overlapping training sets.

Before interpreting a result, check that the folds form a complete, nonoverlapping partition; every observation has one held-out prediction and one loss; and the reported estimate is the pooled mean loss, with fold-size weights when necessary. Large differences between fold means can flag instability or differing difficulty across folds. They do not by themselves identify the cause, and they should not be converted into a confidence interval by assuming independent folds.

A further limitation is that the estimate applies to the prediction procedure and split design actually evaluated. The data-dependent steps used to produce predictions must not use the held-out observations. In addition, if a procedure is changed after inspecting the same cross-validation results, the reported minimum error can be optimistic as an estimate of the selected procedure's future performance. K-fold averaging does not remove that selection effect.

Common Mistakes

Averaging unequal fold means equally. This gives small folds as much weight as large folds. Use $\sum_k(n_k/n)\bar L_k$, or pool all held-out losses directly.
Calling the folds independent test samples. Their models share training observations, so their losses are generally dependent. Do not use an independent-sample standard-error formula without an appropriate justification.
Claiming the estimate is the exact error of the final full-data model. With equal folds, each model is trained on $n(K-1)/K$, not $n$, observations. State the training-size interpretation.
Reporting an error rate without its loss or units. Say whether the result is mean squared error, misclassification proportion, or another loss, and include the relevant units or percentage.

Key takeaway: K-fold cross-validation averages one held-out loss per observation. Weight fold means by their sizes when folds are unequal, interpret the result for the procedure trained on the fold complements, and remember that overlapping training sets make fold losses dependent.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions.

Three folds have sizes 2, 4, and 4, with mean losses 1, 3, and 2. What is the pooled cross-validation estimate?
With $n=20$ and $K=5$ equal folds, how many observations are used to train each fold-specific model?
Why is the average of fold means not generally the correct pooled estimate when fold sizes differ?
For $K=4$ equal folds, how many observations are shared by the training sets for two distinct folds?
A researcher treats the four fold means as four independent observations and uses their standard deviation divided by 2 as a standard error. What is the main concern?

Solutions. (1) Weight by fold size: $(2/10)(1)+(4/10)(3)+(4/10)(2)=0.2+1.2+0.8=2.2$. (2) Each fit uses $20(5-1)/5=16$ observations. (3) An unweighted average gives every fold equal influence, whereas the pooled estimate gives each of the ten observations equal influence; weighting by $n_k/n$ produces that observation-level average. (4) Each training set omits one fold of five observations. Two such training sets share the other three folds, or $20-2(5)=10$ observations. (5) The fold means are generally dependent because the fitted models use overlapping training data, so the independent-replicate standard-error calculation is not justified merely by having four folds.