Why Sample Size Changes the Meaning of Cross-Validation
Cross-validation reuses a limited data set in a structured way: observations take turns being held out for validation, while a model is fitted on the others. When the sample is large, each validation fold may contain many observations and each training fold may still resemble the full data set. With a small sample, a single observation can change what a fold contains, how much data are available for fitting, and even the fitted model itself. A cross-validation score can therefore be much more sensitive to the particular fold assignment.
The issue is not that cross-validation becomes unusable whenever \(n\) is small. Rather, its result needs to be read in light of the sample size and the split: it estimates performance for models fitted on smaller training sets, and it may vary substantially across folds. A precise-looking average does not remove that limitation.
For this procedure to provide a useful estimate, the data and split must match the prediction task. The observations should represent the population to which predictions will be applied, and the validation observations must not influence the fitted model or any preprocessing used to make predictions. If observations are clustered, repeated measures on the same person, or ordered in time, a random split of individual rows can put closely related observations on both sides and give an over-optimistic assessment. The split should respect the independent unit or the prediction setting. These design requirements matter at any sample size, but a small sample leaves fewer opportunities to create suitable folds.
Fold Sizes and the Available Training Data
If \(n\) is not divisible by \(K\), fold sizes cannot all be equal. A balanced division gives folds of either \(\lfloor n/K \rfloor\) or \(\lceil n/K \rceil\) observations, with sizes adding to \(n\). A fold of size \(v\) leaves \(n-v\) observations for fitting. Thus, the cross-validation model is generally trained on fewer than all \(n\) observations. This is especially consequential when \(n\) is small.
Worked Example: Seven Observations in Three Folds
Suppose a hypothetical data set has \(n=7\) independent observations, and a researcher uses \(K=3\) folds. How many observations can be in each validation fold, and how many are available for fitting on each turn?
The average fold size is \(n/K=7/3=2.333\), so balanced folds must have sizes 2, 2, and 3. The corresponding training sizes are \(7-2=5\), \(7-2=5\), and \(7-3=4\). Thus, one of the three fitted models uses only four observations, while the other two use five.
This calculation does not show that the validation estimate is necessarily poor. It does show what the estimate is based on: predictions from models fitted on four or five observations, not a model fitted on all seven. If a predictor’s range or an important subgroup is represented by only one observation, leaving that observation out can produce a particularly different training set. For this hypothetical data set, the fold assignment should be inspected for such omissions rather than treated as a neutral bookkeeping choice.
Small training folds can affect the fitted model as well as the validation score. In regression, a change in a few training observations can move the fitted coefficients, alter predictions, or make a fit impossible—for example, if the chosen procedure needs more distinct predictor information than a training fold contains. The direction and size of such changes depend on the data and model; small \(n\) does not guarantee instability, but it makes the consequences of individual observations harder to average away.
A Small-Sample Example of Fitted-Model Instability
The following calculation uses simple least-squares regression to make the effect visible. The four points are hypothetical measurements, not results from a real study. For a training set with distinct \(x\) values, the fitted line has slope \(\hat b_1=\sum (x_i-\bar{x})(y_i-\bar{y})/\sum (x_i-\bar{x})^2\) and intercept \(\hat b_0=\bar{y}-\hat b_1\bar{x}\). The held-out squared prediction error is \((y-\hat y)^2\), measured in squared outcome units.
Worked Example: Leave-One-Out Validation With Four Points
Consider the hypothetical points \((0,1)\), \((1,2)\), \((2,3)\), and \((10,0)\). In leave-one-out cross-validation, each point is held out once and the line is fitted to the other three. Calculate each prediction error and their mean.
When \((10,0)\) is held out, the other three points lie exactly on \(y=1+x\). The fitted line predicts \(\hat y=11\) at \(x=10\), so the squared error is \((0-11)^2=121\).
When \((0,1)\) is held out, the training points are \((1,2)\), \((2,3)\), and \((10,0)\). For them, \(\sum x_i=13\), \(\sum y_i=5\), \(\sum x_i y_i=8\), and \(\sum x_i^2=105\). The slope and intercept are
At \(x=0\), the prediction is \(421/146\). The held-out squared error is \((1-421/146)^2=(-275/146)^2\approx 3.548\).
When \((1,2)\) is held out, the training points are \((0,1)\), \((2,3)\), and \((10,0)\). Here \(\sum x_i=12\), \(\sum y_i=4\), \(\sum x_i y_i=6\), and \(\sum x_i^2=104\). Therefore
The prediction at \(x=1\) is \(43/21-5/28=157/84\). The squared error is \((2-157/84)^2=(11/84)^2=121/7056\approx 0.01715\).
Finally, when \((2,3)\) is held out, the training points are \((0,1)\), \((1,2)\), and \((10,0)\). Their sums are \(\sum x_i=11\), \(\sum y_i=3\), \(\sum x_i y_i=2\), and \(\sum x_i^2=101\). This gives
The prediction at \(x=2\) is \(281/182-54/182=227/182\), so the squared error is \((3-227/182)^2=(319/182)^2\approx 3.072\). The leave-one-out mean squared prediction error is therefore
The large error of 121 comes from holding out the point at \(x=10\): the other three training points support a line that predicts 11 there, although its observed outcome is 0. The calculation also illustrates how each training fit uses only three observations, and how the fitted line depends on which one is omitted. The mean is a valid summary of these four held-out squared errors, but four observations cannot make it a broadly stable estimate of performance in every setting. The points are independent by construction here; in a real application, the sampling and split would also need to match the intended prediction task.
Reading Variation Across Folds
Variation across fold scores is a warning sign to investigate, not a diagnosis by itself. Large differences may reflect a few hard-to-predict validation observations, different fold composition, or models that change noticeably when training data change. With small validation folds, one observation can have a large effect on that fold’s score. Also, fold scores are not independent replicates: their training sets overlap, so a simple calculation treating them as independent measurements can overstate how much information the folds provide.
Worked Example: A High-Variation Three-Fold Score
Suppose a hypothetical six-observation data set is split into three folds of two observations each. The outcome is measured in points. The squared errors in the three validation folds are, respectively, \((0,2)\), \((1,1)\), and \((0,32)\) points squared. Find each fold’s mean squared error, the overall mean squared error, and the sample standard deviation of the three fold scores.
The fold mean squared errors are \((0+2)/2=1\), \((1+1)/2=1\), and \((0+32)/2=16\) points squared. Because all folds have equal size, their mean is also the average over all six held-out errors: \((1+1+16)/3=6\) points squared. Equivalently, the six errors sum to \(0+2+1+1+0+32=36\), and \(36/6=6\).
The fold-score sample standard deviation is \[ s=\sqrt{\frac{(1-6)^2+(1-6)^2+(16-6)^2}{3-1}} =\sqrt{\frac{150}{2}} =\sqrt{75} \approx 8.660 \] points squared. The third fold’s score is much higher than the other two, so the average masks substantial variation across these validation folds. This standard deviation describes the observed fold-score spread; because the training sets overlap and there are only three folds, it should not automatically be treated as a standard error or as a confidence interval for future prediction error. The specific held-out cases and the fits that produced their predictions should be examined.
When a Split-Based Estimate Is Fragile
A split-based estimate deserves particular caution when the sample is small relative to the variety of situations where predictions must work. Warning signs include folds with very different predictor ranges, a rare but important category appearing in only one fold, a large change in fitted coefficients when an observation is omitted, or a score dominated by one or two validation cases. A fold average can still be computed in each situation; the concern is how much evidence that average provides about future performance.
Cross-validation also does not make the final fitted model identical to the models evaluated within the folds. Each fold model is fitted on a subset, while a procedure refitted on all available data uses \(n\) observations. The cross-validation estimate is most directly informative about the procedure trained on fold-sized samples. With a small data set, that distinction can matter: training on all \(n\) may improve a model, but cross-validation does not establish the size of that improvement.
Common Mistakes
- Assuming more folds solve a small-sample problem. More folds can leave more observations in each training set, but do not create new independent information. In leave-one-out validation, each validation fold contains just one observation, so an individual case can have a strong effect.
- Reading a fold average as a stable score without checking the folds. A mean can hide one unusually poor fold. Inspect the fold sizes, composition, and scores, especially when each validation fold is small.
- Treating fold scores as independent replicates. Their training sets overlap. Fold-to-fold spread is useful descriptively, but it does not by itself justify an ordinary independent-sample standard error.
- Assuming cross-validation evaluates a model trained on all observations. Each fold fit omits its validation fold. State that the estimate comes from fits trained on fewer than \(n\) observations.
- Letting related observations cross the split. If the prediction target is a new person, household, or later time period, the split must prevent information from related observations from leaking across training and validation.
Check Your Understanding
Use the ideas in this tutorial to answer the following questions.
- A data set has 11 independent observations and is divided into four balanced folds. What are the possible validation-fold sizes and corresponding training sizes?
- Why can a leave-one-out score vary substantially when the sample contains one unusual observation?
- In the three-fold example, why is the mean fold score also the pooled mean squared error? Would that equality necessarily hold if fold sizes differed and fold scores were averaged equally?
- What does a large spread among fold scores suggest, and why is it not automatically a standard error for future prediction error?
- A data set contains several measurements per person, but the goal is to predict outcomes for new people. Why could randomly splitting individual rows give an unsuitable validation estimate?