K-Fold Cross-Validation Mechanics

How K-Fold Cross-Validation Works

K-fold cross-validation uses the observed data repeatedly, but keeps each observation out of the model fit used to predict it. First, divide the \(n\) observations into \(K\) non-overlapping folds. Then fit \(K\) models: for iteration \(k\), use every fold except fold \(k\) for training and use fold \(k\) only for evaluation. The resulting held-out predictions can be combined to assess the model’s predictive performance.

The mechanics are simple, but small implementation errors can undermine the evaluation. An observation omitted from every held-out set, included in its own training data, or counted twice in the final metric changes what the result measures. A reliable workflow makes the fold assignment, training indices, held-out predictions, and metric aggregation explicit.

For ordinary K-fold cross-validation, \(K\) is an integer of at least 2 and no larger than the number of observations. When \(n\) is not divisible by \(K\), fold sizes should differ by at most one: some folds have \(\lfloor n/K \rfloor\) observations and the rest have \(\lceil n/K \rceil\). The assignment method must reflect the prediction problem. In particular, ordinary random folds are not automatically appropriate when observations have a time order or repeated-entity dependence; the split must preserve the structure relevant to the intended prediction task.

Creating Folds and Building the R Loop

An observation’s fold assignment is an integer from 1 to \(K\). In R, the following approach randomly assigns observations while keeping fold sizes as even as possible: set \(K\) to the chosen number of folds, run set.seed(410) for reproducibility, and create the assignment with fold_id <- sample(rep(seq_len(K), length.out = nrow(dat))). The seed is an example; the important point is to record it when reproducibility matters.

For fold \(k\), the held-out row indices are those for which fold_id == k; the training indices are those for which fold_id != k. A compact implementation for a regression model is: create pred_oof <- rep(NA_real_, nrow(dat)); then, for each \(k\) from 1 to \(K\), set test_idx <- which(fold_id == k) and train_idx <- which(fold_id != k), fit lm(y ~ x, data = dat[train_idx, ]), and write predict(fit, newdata = dat[test_idx, ]) into pred_oof[test_idx]. The vector pred_oof is indexed in the original row order, making it straightforward to compare every prediction with its observed outcome.

The code is a template, not a complete prescription for every modelling task: replace the formula and fitting method as needed. If a model requires data-dependent preprocessing, estimate that preprocessing using the training rows within each iteration, then apply it to the held-out rows. Applying a transformation learned from all rows before assigning folds can allow held-out information to affect the fit.

Why Each Observation Is Held Out Once

The key property follows from the partition. Each observation belongs to exactly one fold, say fold \(j\). It is excluded from the training rows when iteration \(j\) is fitted, so it receives a held-out prediction in that iteration. In every other iteration, it belongs to the training set. Because the folds are disjoint and cover all observations, it cannot belong to a second held-out fold.

This also explains the out-of-fold vector’s structure: after all \(K\) iterations, each of its \(n\) positions should contain one prediction. A simple check in R is all(!is.na(pred_oof)), assuming the prediction method itself returns no missing values. If that check fails, inspect the fold assignment and indexing before calculating a score.

Here \(F_k\) denotes the set of row indices in fold \(k\). The first condition says that the folds cover the full data set; the second says they do not overlap. Together, they establish that every observation is held out exactly once.

Combining Regression Errors

For a quantitative response, a common metric is the root mean squared error (RMSE). Let \(y_i\) be the observed response and \(\hat{y}_i^{\text{OOF}}\) its held-out prediction. The pooled mean squared error (MSE) averages squared errors over all \(n\) observations, and the pooled RMSE is the square root of that average. The prediction for observation \(i\) must come from the iteration in which \(i\) was held out.

If fold \(k\) has \(n_k\) held-out observations and fold MSE \(\operatorname{MSE}_k\), the pooled MSE is a size-weighted average of the fold MSEs. This follows by splitting the sum of squared errors into its fold-specific sums: fold \(k\) contributes \(n_k\operatorname{MSE}_k\), so the total squared error is \(\sum_k n_k\operatorname{MSE}_k\). Divide by the total number of observations, \(n=\sum_k n_k\), to obtain the result.

When all folds have the same size, this is also the unweighted mean of the fold MSEs. When fold sizes differ, an unweighted average gives every fold equal influence even though the folds contain different numbers of observations. Also, the pooled RMSE is the square root of the pooled MSE; it is generally not the mean of the fold RMSEs.

Combining Classification Predictions

For classification, retain the held-out predicted probability for each observation, not just a fold-level score. If a class prediction is needed, apply a stated threshold to those probabilities after they have been generated. For example, a threshold rule may classify an observation as positive when its held-out predicted probability is at least 0.5. The pooled confusion matrix then counts outcomes and predictions across all held-out observations.

Implementation Checks and Common Traps

Before interpreting a cross-validation score, verify the mechanics. The following checks catch many avoidable errors:

• One assignment per row: every observation must have a fold ID from 1 through \(K\).
• One held-out prediction per row: the out-of-fold prediction vector should have no unfilled entries.
• No held-out rows in their own fit: the training indices in iteration \(k\) must exclude every row in fold \(k\).
• Metric units and aggregation: calculate the metric on held-out results, and account for fold sizes when combining fold-level averages.
• Repeat the complete fitting procedure: any preprocessing estimated from data must be estimated within each training iteration and then applied to that iteration’s held-out fold.

A particularly common exam trap is to report the unweighted mean of fold RMSEs as if it were the pooled RMSE. Even with equal fold sizes, the mean of square roots is generally not the square root of the mean. For example, in the preceding regression example, the pooled MSE is \(329/9\), so the pooled RMSE is \(\sqrt{329/9}\). Averaging the fold RMSEs instead performs a different calculation. State which quantity is being reported and show how it was formed.

K-fold cross-validation is a procedure for generating held-out results, not a guarantee that any chosen fold design is suitable. The model, preprocessing, fold assignment, and performance measure must match the prediction task. The central mechanical rule remains: fit on \(K-1\) folds, predict the one excluded fold, and combine the held-out results without losing track of which observation each prediction belongs to. This workflow is aligned with the SOA Exam PA (Predictive Analytics) syllabus.