Cross-Validation With a Small Delivery-Time Dataset

Why Evaluate Predictions Across Different Splits?

A regression model can fit the observations used to estimate it well and still predict new delivery times poorly. When a dataset is small, setting aside a single test sample may be wasteful, and the resulting error estimate can depend strongly on which observations happen to be held out. Cross-validation addresses this by repeatedly dividing the available data into training and validation portions. Each validation portion tests predictions from a model fitted without those observations.

This tutorial uses a deliberately tiny, hypothetical delivery dataset to compare a mean-only prediction model with a linear regression model. The calculations show both the average prediction error and its variation across splits. The small size also makes an important limitation visible: cross-validation can organize an evaluation, but it cannot make a handful of observations representative of every future delivery.

Definition: In \(K\)-fold cross-validation, the \(n\) observations are divided into \(K\) nonoverlapping folds of roughly equal size. For each fold \(k\), fit the model using all observations outside that fold, then calculate prediction error on fold \(k\). In repeated \(K\)-fold cross-validation, this complete partition-and-evaluate process is repeated using different fold assignments. Each repeat gives every observation a turn in a validation fold.

A Small Delivery-Time Dataset

Suppose a hypothetical courier records the distance and delivery time for four deliveries. The observations are not real study data; they are chosen to make the calculations transparent.

Delivery	Distance (km)	Time (minutes)
1	1	12
2	2	14
3	3	17
4	4	19

We compare two candidate prediction rules. The mean-only model predicts the average delivery time in its training data for every new delivery. The linear model fits delivery time as a straight-line function of distance using the training data. In each fold, both models must be fitted again using that fold's training data only.

Definition: For a validation set with \(m\) observations, the mean squared error (MSE) is the average of the squared differences between observed and predicted outcomes:

\mathrm{MSE}=\frac{1}{m}\sum_{i=1}^{m}(y_i-\hat{y}_i)^2.

Here \(y_i\) is an observed delivery time and \(\hat{y}_i\) is its prediction. The units of MSE are minutes squared; a smaller MSE indicates smaller squared prediction errors on the evaluated observations.

Worked Example: Evaluating One Validation Fold

For the first split, hold out deliveries 1 and 2 for validation, and train on deliveries 3 and 4. Fit the linear model using only the training deliveries, \((x,y)=(3,17)\) and \((4,19)\). The slope is \((19-17)/(4-3)=2\) minutes per kilometre, and the intercept is \(17-2(3)=11\) minutes. Thus the fitted line is \(\hat{y}=11+2x\).

For validation distances of 1 km and 2 km, the predictions are \(11+2(1)=13\) minutes and \(11+2(2)=15\) minutes. The observed times are 12 and 14 minutes, so the residuals \(y_i-\hat{y}_i\) are \(-1\) and \(-1\) minute. Therefore,

\mathrm{MSE}_{\text{linear, fold 1}} =\frac{(12-13)^2+(14-15)^2}{2} =\frac{1+1}{2} =1\ \text{minute}^2.

For the mean-only model, the training mean is \((17+19)/2=18\) minutes, so its two predictions are both 18. Its validation MSE is \(\big((12-18)^2+(14-18)^2\big)/2=(36+16)/2=26\ \text{minutes}^2\). This is a comparison on this fold only, not yet a summary across the dataset.

Repeated Cross-Validation, Worked Through

With four observations, use two folds of two observations each. One repeat assigns deliveries 1 and 2 to one fold and deliveries 3 and 4 to the other. A second repeat uses a different partition: deliveries 1 and 3 form one fold, and deliveries 2 and 4 form the other. Each repeat covers all four deliveries in validation exactly once.

Worked Example: Comparing Models Across Two Repeats

In the first repeat, the fold evaluated in the previous example has linear-model MSE \(1\ \text{minute}^2\) and mean-only MSE \(26\ \text{minutes}^2\). For the other fold, train the linear model on deliveries 1 and 2. The points \((1,12)\) and \((2,14)\) give the line \(\hat{y}=10+2x\). It predicts 16 and 18 minutes for distances 3 and 4, compared with observed times 17 and 19. Its fold MSE is \(\big((17-16)^2+(19-18)^2\big)/2=1\ \text{minute}^2\). The mean-only model predicts the training mean, 13 minutes, for both; its MSE is \(\big((17-13)^2+(19-13)^2\big)/2=26\ \text{minutes}^2\). Thus the repeat-level MSEs, averaging the two equally sized validation folds, are \(1\) for the linear model and \(26\) for the mean-only model.

In the second repeat, first hold out deliveries 1 and 3 and train on deliveries 2 and 4. Those training points are \((2,14)\) and \((4,19)\), giving slope \((19-14)/(4-2)=2.5\) and intercept \(14-2.5(2)=9\). The predictions for the held-out distances 1 and 3 are 11.5 and 16.5 minutes. The errors are \(0.5\) and \(0.5\) minutes in magnitude, so the fold MSE is \((0.5^2+0.5^2)/2=0.25\ \text{minute}^2\).

The mean-only prediction in that fold is \((14+19)/2=16.5\) minutes. Its validation errors are \(-4.5\) and \(0.5\) minutes, giving MSE \(((-4.5)^2+0.5^2)/2=(20.25+0.25)/2=10.25\ \text{minutes}^2\). For the other fold, train the linear model on deliveries 1 and 3: the line through \((1,12)\) and \((3,17)\) is \(\hat{y}=9.5+2.5x\). It predicts 14.5 and 19.5 minutes at distances 2 and 4, each \(0.5\) minute above the observed time, so its MSE is \(0.25\ \text{minute}^2\). The training mean is \((12+17)/2=14.5\); the mean-only errors are \(0.5\) and \(4.5\) minutes, giving MSE \(10.25\ \text{minutes}^2\).

The repeat-level score is the mean of its two fold MSEs. For the second repeat, these scores are \(0.25\ \text{minute}^2\) for the linear model and \(10.25\ \text{minute}^2\) for the mean-only model. Averaging the two repeat-level scores gives:

\overline{\mathrm{MSE}}_{\text{linear}} =\frac{1+0.25}{2}=0.625\ \text{minute}^2, \qquad \overline{\mathrm{MSE}}_{\text{mean}} =\frac{26+10.25}{2}=18.125\ \text{minute}^2.

In this illustration, the linear model has the lower average cross-validated MSE. But the scores also vary across repeats: the linear-model scores are \(1\) and \(0.25\), while the mean-only scores are \(26\) and \(10.25\), all in minutes squared. With only two repeats, these values are a description of these splits, not a reliable estimate of the full range of possible error.

What the Cross-Validation Score Represents

For each repeat, average the validation losses across all held-out observations. When folds are equally sized, this is the same as averaging their MSEs. When fold sizes differ, a pooled observation-level MSE weights each fold according to its number of validation observations. A repeat-level score is therefore based on predictions made for every observation once during that repeat. The overall repeated-CV score is the average of the repeat-level scores.

The calculation estimates prediction error for models trained on only part of the available dataset: in \(K\)-fold cross-validation, each fit uses about \((K-1)/K\) of the observations. The final model fitted to all available data has a different training size. Cross-validation is useful for comparing candidate methods under a common splitting plan, but its score is not a guarantee of future error and is not automatically an unbiased estimate for every setting.

The interpretation depends on how the data were collected and how predictions will be used. The rows should represent the kind of new deliveries the model is meant to predict, and the split should reflect that prediction task. For example, if future prediction concerns later time periods, a random split that mixes earlier and later deliveries may not represent that task. Likewise, closely related or repeated observations should not be separated across training and validation if doing so would make the validation task unrealistically easy.

Preprocessing Must Stay Inside Each Fold

A model pipeline may include steps before regression is fitted, such as filling in missing predictor values, scaling predictors, or selecting predictors. These steps learn quantities from data. To evaluate the full procedure fairly, estimate those quantities using the training fold only, then apply the learned transformation to that fold's validation data. Repeating this sequence separately for each fold prevents the held-out observations from influencing model preparation.

Worked Example: Keeping Scaling Within the Training Fold

Suppose a training fold has observed delivery distances of 2, 4, and 6 km, and the validation fold contains a delivery at 100 km. To standardize distance, calculate the training mean: \((2+4+6)/3=4\) km. Use that training mean (and the standard deviation calculated from those same three training values) to transform both training and validation distances. The validation value is transformed using the training quantities; it does not help determine them.

If instead the mean were calculated from all four distances, it would be \((2+4+6+100)/4=28\) km. The held-out delivery would then influence a transformation used to fit the model. The same principle applies to estimating an imputation value or choosing predictors: fit the step on the training fold, then apply it to validation data. In software, the safest approach is to place learned preprocessing steps and the regression model together in a pipeline that is refitted for every training fold.

This fold-specific rule applies even if a preprocessing step seems routine. Using the full dataset to choose predictors based on their association with delivery time is especially problematic: the validation outcomes would have helped determine what the model uses. The resulting validation errors would no longer evaluate a procedure that was kept separate from those outcomes. Decide the modeling and preprocessing procedure within the training data, and use the validation fold only to evaluate its predictions.

Why the Small Sample Matters

A four-observation example is helpful for hand calculations, but its model comparisons are highly fragile. In two-fold cross-validation, each linear regression fit here uses just two points, so a single unusual delivery could substantially change the fitted line. Repeating the splits reveals that the estimated error depends on how the observations are grouped; it does not create new independent deliveries or remove that instability.

For an actual small dataset, report the number of observations, the cross-validation design, the error measure with its units, and variation across repeats or splits. Avoid reporting only the lowest observed score. If the data are very limited, present model rankings as uncertain and keep the model simple enough for the available information. A final performance claim should match the evaluation design and the intended use.

Common mistake: “I standardized the predictor and selected the useful variables once using all observations, then cross-validated the regression. Since the validation outcomes were not used to fit the final regression coefficients, the errors are fair.” The correction is that model evaluation includes every data-dependent step, not just coefficient fitting. Refit scaling, imputation, and outcome-guided selection inside each training fold; otherwise information from held-out data can influence the predictions being evaluated.

Key takeaway: Repeated cross-validation compares prediction procedures across multiple fold assignments. Report the average error and how it varies across splits, and keep all learned preprocessing inside each training fold. A low score on a very small dataset is evidence about those data and splits, not a promise about future deliveries.

Check Your Understanding

Answer these questions using the definitions and calculations above.

In \(K\)-fold cross-validation, how are training and validation observations chosen for a particular fold?
Why does the second repeat in the delivery example provide useful information even though it contains no new deliveries?
For a validation set of three deliveries, how would you calculate its MSE from the three observed and predicted times?
Why should a mean used to fill missing predictor values be estimated separately within each training fold?
What limitation should be mentioned when interpreting a repeated-CV score from only four observations?