Hand-Calculating Five-Fold Cross-Validation

What a Five-Fold Calculation Represents

Cross-validation estimates how a prediction procedure performs on observations not used to fit the model that predicts them. In five-fold cross-validation, the observations are divided into five groups, called folds. Each fold takes a turn being held out, while the procedure is fitted using the observations in the other four folds. Predictions for the held-out observations are then compared with their observed outcomes.

A hand calculation usually starts after those predictions have been supplied. The main tasks are to keep each observation in the correct fold, calculate a score for each fold, and combine the scores without giving the wrong weight to folds of different sizes. The result summarizes held-out performance across the five fits; it is not the training error from fitting once to all the observations.

Definition: In five-fold cross-validation, a dataset is partitioned into five nonoverlapping folds. For each fold, the prediction procedure is fitted using observations outside that fold and evaluated on observations inside it. Each observation is held out exactly once. A five-fold cross-validation estimate is the chosen average prediction loss across all held-out predictions.

Set Up the Calculation

Suppose observation \(i\) has observed response \(y_i\), and let \(\hat{y}_i^{(-k)}\) be its prediction when it belongs to fold \(k\). The superscript indicates that the fit used to make the prediction excludes fold \(k\). For a hand calculation, the fold membership and these out-of-fold predictions should be given or established before calculating the scores.

We will use squared prediction error for the examples. The squared error for observation \(i\) is \((y_i-\hat{y}_i^{(-k)})^2\). The mean squared error (MSE) for fold \(k\) is the average squared error among the observations in that fold. If fold \(k\) contains \(n_k\) observations, its MSE is:

\operatorname{MSE}_k = \frac{1}{n_k} \sum_{i\in F_k} \left(y_i-\hat{y}_i^{(-k)}\right)^2

Here, \(F_k\) denotes the set of observation indices assigned to fold \(k\), and \(n_k\) is the number of observations in that fold. The five-fold estimate pools the held-out squared errors:

\operatorname{CV}_{5} = \frac{1}{n} \sum_{k=1}^{5} \sum_{i\in F_k} \left(y_i-\hat{y}_i^{(-k)}\right)^2 = \frac{\sum_{k=1}^{5} n_k\operatorname{MSE}_k}{n}

The total sample size is \(n=\sum_{k=1}^{5}n_k\). When all five folds have the same size, this pooled score equals the ordinary average of the five fold MSEs. When fold sizes differ, it is a size-weighted average: a fold with more held-out observations contributes more individual squared errors.

Worked Example: Five Equal-Sized Folds

Consider ten hypothetical observations with outcomes measured in points. They have been assigned to five folds of two observations each. The table gives the supplied out-of-fold predictions. For each row, calculate the squared error; then calculate the fold MSE and the overall five-fold estimate.

Fold	Observed outcome	Out-of-fold prediction	Squared error
1	10	11	\((10-11)^2=1\)
1	14	12	\((14-12)^2=4\)
2	8	7	\((8-7)^2=1\)
2	16	18	\((16-18)^2=4\)
3	12	10	\((12-10)^2=4\)
3	20	19	\((20-19)^2=1\)
4	15	13	\((15-13)^2=4\)
4	9	10	\((9-10)^2=1\)
5	18	20	\((18-20)^2=4\)
5	11	8	\((11-8)^2=9\)

Each fold contains two observations. The fold MSEs are:

\operatorname{MSE}_1=\frac{1+4}{2}=2.5,\quad \operatorname{MSE}_2=\frac{1+4}{2}=2.5,\quad \operatorname{MSE}_3=\frac{4+1}{2}=2.5

\operatorname{MSE}_4=\frac{4+1}{2}=2.5,\qquad \operatorname{MSE}_5=\frac{4+9}{2}=6.5

Pooling gives a total squared error of \(1+4+1+4+4+1+4+1+4+9=33\). Thus the five-fold estimate is \(33/10=3.3\) points squared. Because all folds have equal size, averaging the fold MSEs gives the same answer: \((2.5+2.5+2.5+2.5+6.5)/5=3.3\).

Why the Fold Sizes Matter

When the sample size is not divisible by five, folds commonly differ in size by one observation. That does not change the rule that each observation contributes once to the pooled score. It does mean that a simple average of the five fold MSEs may no longer give the same result. The following calculation makes the distinction visible.

Worked Example: Combining Unequal Fold Sizes

Suppose eleven hypothetical observations are partitioned into five folds of sizes three, two, two, two, and two. All outcomes and predictions below are in points. Calculate each fold MSE, then combine them by pooling the eleven squared errors.

Fold	Observed outcomes	Predictions	Squared errors	Fold MSE
1 (3 observations)	10, 12, 15	10, 11, 17	0, 1, 4	\(5/3=1.667\)
2 (2 observations)	8, 16	7, 19	1, 9	\(10/2=5\)
3 (2 observations)	12, 20	10, 20	4, 0	\(4/2=2\)
4 (2 observations)	15, 9	18, 10	9, 1	\(10/2=5\)
5 (2 observations)	18, 11	16, 13	4, 4	\(8/2=4\)

The pooled total is \(5+10+4+10+8=37\), so the estimate is \(37/11=3.364\) points squared. Equivalently, weight each fold MSE by its size:

\operatorname{CV}_{5} = \frac{3(1.667)+2(5)+2(2)+2(5)+2(4)}{11} \approx 3.364

An unweighted average of these five fold MSEs is approximately \((1.667+5+2+5+4)/5=3.533\), which is not the pooled estimate. It gives the three-observation fold the same influence as each two-observation fold, even though the fold MSEs summarize different numbers of held-out errors.

A Short Justification of the Pooling Rule

The pooled estimate averages losses over observations, not over fold labels. Since each observation belongs to exactly one fold, adding the fold sums adds every held-out squared error exactly once. The fold sum can also be written as its fold size multiplied by its fold mean:

\sum_{k=1}^{5}\sum_{i\in F_k} \left(y_i-\hat{y}_i^{(-k)}\right)^2 = \sum_{k=1}^{5} n_k\operatorname{MSE}_k

There are \(n\) observations altogether, so dividing both sides by \(n=\sum_{k=1}^{5}n_k\) gives the weighted formula. This explains why equal fold sizes permit an ordinary mean of fold MSEs, while unequal fold sizes require weighting.

Worked Example: Five Folds With One Observation Each

A dataset can be very small. For five hypothetical observations, a five-fold partition has one observation per fold. Suppose a response is measured in points and the supplied out-of-fold predictions are as follows.

Fold	Observed outcome	Prediction	Squared error
1	40	39	\((40-39)^2=1\)
2	52	54	\((52-54)^2=4\)
3	48	45	\((48-45)^2=9\)
4	61	63	\((61-63)^2=4\)
5	55	52	\((55-52)^2=9\)

Each fold has one held-out observation, so its fold MSE equals that observation's squared error. The pooled estimate is:

\operatorname{CV}_{5} = \frac{1+4+9+4+9}{5} = \frac{27}{5} = 5.4\text{ points}^2

This value summarizes the squared errors of these five held-out predictions. It does not say that every prediction misses by 5.4 points: the score is in squared points, and it averages squared, rather than signed, differences.

Conditions for a Meaningful Hand Calculation

The arithmetic is only useful if the supplied predictions correspond to a valid five-fold evaluation. Check that the following conditions are met:

The five folds form a partition: no observation appears in two folds, and none is omitted.
Each held-out prediction was produced by fitting the stated prediction procedure without using that observation's fold.
The procedure is applied consistently across folds, including any preprocessing or model-selection steps that are part of it.
The loss being averaged is specified in advance and calculated the same way for every held-out observation.
The observations and fold assignment are suitable for the prediction question. For example, when observations are dependent, a random partition may not reflect the intended prediction setting.

Five-fold cross-validation does not require a normal distribution of outcomes to perform this arithmetic. Its usefulness instead depends on how well the data and fold construction represent the intended prediction task. With five folds, each fit uses the other four folds, so the estimate concerns a procedure trained on roughly four-fifths of the available observations. It is therefore an estimate for that training size, not a direct score for a final model refitted on all \(n\) observations.

Common Mistakes and Why This Calculation Matters

Averaging unequal fold scores without weighting. The mistaken reasoning is that five folds mean each fold should count equally. The correction is to pool the observation-level losses, or weight each fold MSE by its size before dividing by \(n\).
Using predictions from a fit that included the held-out observations. Such predictions are not out-of-fold predictions, so their errors do not represent the intended cross-validation calculation.
Leaving out a fold's observations from the final average. Each observation must contribute its held-out loss once; verify that the fold sizes sum to \(n\).
Calling the cross-validation estimate the error of the final full-data fit. The estimate aggregates predictions from five fits trained on subsets. It is a validation summary for the specified procedure, not a guarantee about the particular refitted model.
Forgetting the score's units. With squared error and a response measured in points, the reported MSE is in points squared, not points.

Hand calculation is valuable because it exposes the bookkeeping behind a software-reported cross-validation score. A fold table makes it possible to check that every observation was evaluated exactly once, that predictions were held out properly, and that the overall score weights observations rather than fold names.

Check Your Understanding

Use the definitions and calculations above to answer these questions.

In five-fold cross-validation, which observations are used to fit the prediction procedure when fold 3 is being evaluated?
A fold contains three squared errors: 1, 4, and 4. What is its fold MSE?
Why can the ordinary average of five fold MSEs differ from the pooled cross-validation estimate?
Five folds have sizes 2, 2, 2, 2, and 3. What should the sum of their sizes equal?
If the outcome is measured in kilograms, what are the units of the cross-validation MSE based on squared error?