Why Small Samples Require a Validation Plan
With limited data, each observation has competing uses. It can help fit the model, help choose tuning settings, or help assess predictive performance. A single holdout set gives a direct assessment on observations not used for fitting or tuning, but it can leave very little data for both tasks. Cross-validation uses the observations more efficiently, but the score can be optimistic if the same cross-validation results are used both to select a model and to report its performance.
There is no universally best split for a small sample. The appropriate plan depends on the number of independent observations, the amount of tuning, the prediction task, and how much uncertainty in the performance estimate is tolerable. A carefully limited analysis may be more credible than a complicated search whose apparent precision exceeds what the data support.
For the basic calculations below, suppose there are \(n\) independent observations drawn from the population to which predictions will be applied, and a loss function \(L(y,\hat{y})\) has been selected in advance. For example, squared-error loss is \(L(y,\hat{y})=(y-\hat{y})^2\). A validation score is an average of losses on held-out observations; it is not itself a guarantee that future performance will equal that value. The calculations assume that the split respects the data-generating structure. If records are clustered, repeated for the same person, or ordered in time, randomly splitting individual rows may fail to create an appropriate assessment.
What a Fold Plan Gives Up and Gains
In \(K\)-fold cross-validation, the observations are divided into \(K\) folds. Each fold is held out in turn, and the model for that round is fitted using the other \(K-1\) folds. If \(n\) is divisible by \(K\), each validation fold has \(n/K\) observations and each fit uses \(n(K-1)/K\). Thus, increasing \(K\) makes each fit use more data, but makes each individual validation fold smaller. With nondivisible \(n\), fold sizes differ by at most one if the folds are balanced.
The cross-validation score is usually formed by pooling the held-out losses, so each observation contributes once when each fold is used as validation. If folds have unequal sizes, pooling gives the observation-weighted average. Averaging fold averages with equal weight instead can give a slightly different answer when the fold sizes differ.
Here \(F_k\) is the set of observations in fold \(k\), and \(\hat{f}_{-k}\) is the fitted model that excludes that fold. The expression is a useful estimate of the performance of a fitting procedure trained on about \(n(K-1)/K\) observations. It does not exactly estimate the performance of the final model refitted on all \(n\) observations, and the difference can matter when the sample is very small or model performance changes substantially with training size.
Holdout, Cross-Validation, and Nested Assessment
A single holdout is easy to explain: set aside a test set, use the remaining observations for fitting and tuning, and evaluate once on the test set. Its weakness in a small sample is that the test score may depend heavily on which few observations happened to be held out, while the model has less data available for fitting. Repeatedly trying different splits and reporting the most favorable result does not solve this problem; it uses the assessment results to make a selection.
K-fold cross-validation is often a practical alternative when reserving a sizeable test set would be too costly. However, if a candidate setting is chosen because it has the lowest cross-validation loss, that same minimum is affected by selection optimism. Nested cross-validation separates the two tasks: inner folds choose the model or tuning setting using the outer-training observations, and the selected procedure is evaluated on the outer fold that was excluded from that choice. This costs computation and leaves each outer fit with fewer observations, so its value must be weighed against the sample size.
A small sample also limits how much tuning is sensible. A long, flexible search can select a setting that happens to suit a few observations. Limiting the candidate set based on substantive reasoning, using a prespecified loss, and avoiding unnecessary model comparisons can improve the credibility of the analysis. The aim is not to minimize the number of models at any cost; it is to match the search complexity to the information in the data.
Why the Cross-Validation Score Has Its Target
The fold-size formula follows directly from the construction. When each of \(K\) equally sized folds is used for validation, the other \(K-1\) folds are used for fitting. Each fold contains \(n/K\) observations, so the number in a training set is
For the performance score, each observation belongs to exactly one validation fold. Its loss is therefore computed from a fit that excluded that observation. Summing the validation losses across folds collects one loss for each of the \(n\) observations; dividing by \(n\) gives the observation-level average. Under independent, representative sampling and a split that does not leak information, this estimates the predictive loss of the fitting-and-selection procedure at the training size used in each fold. It does not make the \(K\) fold scores independent: their training sets overlap, and fold results can be correlated.
Worked Example: Choosing Nested Folds for a Limited Tuning Study
A hypothetical insurer has 30 independent policy records and wants to compare three prespecified elastic-net settings. The analyst needs an estimate of the performance of the procedure that chooses among those settings. Consider five outer folds, each containing six records. Within each outer-training set, use four-fold cross-validation to select one of the three settings.
Step 1: Calculate the outer sample sizes. Each outer fit has \(30-6=24\) records. Those 24 records are used for the inner selection, and the six outer-fold records remain untouched until assessment.
Step 2: Calculate the inner sample sizes. A four-fold split of 24 records has \(24/4=6\) validation records per inner fold. Each inner fit uses \(24-6=18\) records. The three settings are compared using these inner validation results; the outer fold is not used to choose between them.
Step 3: Aggregate the outer assessment. Suppose the selected procedure has outer-fold mean losses of 0.18, 0.22, 0.20, 0.26, and 0.24, each based on six records. The total held-out loss, expressed as six times each fold mean, is \(1.08+1.32+1.20+1.56+1.44=6.60\). There are 30 outer predictions, so the pooled outer estimate is \(6.60/30=0.22\).
Conclusion and check. The estimated loss for the tuning procedure is 0.22, based on five outer folds. The calculation checks because the five fold means are equal-sized, and their average is \((0.18+0.22+0.20+0.26+0.24)/5=1.10/5=0.22\). This design is more informative about the selection procedure than reporting the smallest inner-CV loss. A common mistake is to treat the inner minimum as an independent performance estimate; it is not, because those same inner results selected the setting.
Repeated Validation Does Not Create New Observations
Repeating cross-validation with different fold assignments can show how much a result varies with the random partition. This is useful when a single fold assignment seems unusually favorable or unfavorable. But repeated scores are not independent replications of the study: the same observations appear repeatedly, and the training sets overlap. Repetition can reveal split sensitivity; it cannot turn a small sample into a large one.
Likewise, nested cross-validation is not automatically feasible for every small data set. In a very small outer-training set, the inner folds may contain too few observations to support stable tuning. The analyst may need to reduce the number of candidate settings, use fewer folds, or state that the data do not support a reliable estimate of the full tuning procedure. A numerical score should not conceal that limitation.
Worked Example: Repeated Cross-Validation With Twelve Observations
A hypothetical health-benefit study has 12 independent cases. The analyst compares a small, fixed set of candidate models using repeated three-fold cross-validation. Each fold has four cases, so every fit uses \(12-4=8\) cases. Across four repeat assignments, the three fold mean losses are as follows: repeat 1, 0.30, 0.20, 0.25; repeat 2, 0.28, 0.22, 0.26; repeat 3, 0.24, 0.27, 0.21; repeat 4, 0.29, 0.23, 0.25.
Step 1: Average each repeat. The first repeat has mean \((0.30+0.20+0.25)/3=0.75/3=0.25\). The second has \((0.28+0.22+0.26)/3=0.76/3=0.2533\) to four decimals. The third has \((0.24+0.27+0.21)/3=0.72/3=0.24\). The fourth has \((0.29+0.23+0.25)/3=0.77/3=0.2567\) to four decimals.
Step 2: Calculate the overall mean. The sum of all 12 fold means is \(0.75+0.76+0.72+0.77=3.00\). The average is \(3.00/12=0.25\). Since each fold has the same size, this also equals the mean of the four repeat averages: \((0.25+0.2533+0.24+0.2567)/4=1.00/4=0.25\), using unrounded values.
Conclusion and check. The repeated-CV score is 0.25, and the repeat means range from 0.24 to 0.2567, showing modest variation across these assignments. This is not evidence from 48 independent cases: there are still only 12 cases, reused in each repeat. A common mistake is to divide by 48 and describe the repetitions as 48 independent validation observations. The correction is to report the repeated score as a sensitivity summary and retain the original sample size when discussing evidence.
Preserve the Independent Unit
The number of rows is not always the number of independent observations. For example, several records may belong to one policyholder, clinic, or employer. If related records are split between training and validation, the model may benefit from information about that same unit during fitting. The resulting score may answer an easier question than predicting for a new independent unit.
Worked Example: Five-Fold Validation by Policyholder
A hypothetical portfolio study contains four claim records from each of five policyholders, for 20 rows total. The intended use is prediction for a new policyholder. The analyst assigns each entire policyholder to one of five folds, so each outer validation fold contains four rows from one policyholder.
Step 1: Count the independent units. There are five policyholders, not 20 independent policyholders. Each validation fold holds out one policyholder, and each corresponding fit uses the other four policyholders, or \(4\times4=16\) rows.
Step 2: Compute the pooled score. Suppose the mean losses for the five held-out policyholders are 0.10, 0.30, 0.20, 0.40, and 0.25. Each contributes four rows. The total loss across the held-out rows is \(4(0.10+0.30+0.20+0.40+0.25)=4(1.25)=5.00\). Dividing by 20 rows gives a pooled loss of \(5.00/20=0.25\), equal to the mean of the five policyholder means, \(1.25/5=0.25\).
Conclusion and check. The row-level pooled score is 0.25, but the assessment is based on only five held-out policyholders. Grouping all rows from each policyholder prevents direct leakage across folds and matches the stated prediction target. Randomly splitting 20 rows could put records from one policyholder in both training and validation, overstating performance for new policyholders. The score should not be interpreted as if 20 independent units had been assessed.
Practical Choices and Common Traps
A useful plan begins by identifying the target of prediction and the independent unit. Next, determine whether a final test set can be large enough to be meaningful without starving the fitting and tuning stages. If not, use cross-validation with a clearly stated target and protect any outer assessment from tuning decisions. Keep preprocessing inside each training split: quantities such as centering and scaling parameters must be learned from the training observations and then applied to the held-out observations. Otherwise, information from validation data can leak into fitting.
For very small samples, avoid presenting a single score as precise. Report the fold design, training size per fit, number of independent units, and variability across folds or repeats where appropriate. A high-variance estimate may still be the best available estimate, but that uncertainty belongs in the conclusion. If the sample cannot support the desired tuning and assessment separately, say so rather than treating reuse of the data as independence.
| Plan | Strength for limited data | Main limitation |
|---|---|---|
| Single holdout | Simple, direct assessment on data not used for model choices | Consumes observations and can be highly split-dependent |
| K-fold cross-validation | Each observation is assessed once while most observations are used for each fit | Using the same results for tuning and reporting can be optimistic |
| Nested cross-validation | Outer assessment evaluates the tuning procedure, not just the selected setting | Computationally and statistically demanding; outer training samples are smaller |
| Repeated cross-validation | Shows sensitivity to fold assignment | Repeats reuse observations and are not independent studies |
- Trap: choosing the largest possible \(K\) automatically. Larger \(K\) increases training size per fit, but leaves smaller validation folds and does not remove tuning optimism.
- Trap: counting records instead of independent units. Related rows should generally remain together when the intended prediction is for a new person or group.
- Trap: calling a cross-validation score a test-set result. The score is an estimate based on fold-specific fits, and its interpretation depends on whether those results influenced model selection.
- Trap: treating the nested estimate as the exact performance of the final refit. Outer models are trained on less than the full data. After tuning, refitting on all available observations is appropriate for deployment, but the outer estimate still describes the validation procedure at its outer training size.
Check Your Understanding
For each plan, calculate the requested quantities and check whether the validation design matches the stated goal.
- A study has 35 independent observations and uses five balanced folds. How many observations are in each validation fold and each training fit? (Answer: 7 validation observations and 28 training observations per fit.)
- A sample of 18 independent observations is divided into four balanced folds. The fold sizes are 5, 5, 4, and 4. How many observations are in the corresponding training fits, in the same order, and what is their average size? (Answer: 13, 13, 14, and 14; average training size 13.5.)
- Three equal-sized validation folds have mean losses 0.16, 0.22, and 0.28. What is the pooled cross-validation score? If the lowest of several candidate scores was selected using these folds, should that minimum also be described as an independent test estimate? (Answer: 0.22; no, the same results influenced selection.)
- A prediction study has 24 rows, with three rows for each of eight independent clinics. The target is prediction for a new clinic. In eight-fold validation that holds out one clinic at a time, how many rows are held out and how many rows are used to fit each model? (Answer: 3 held-out rows and 21 training rows per fit; the independent-unit count is eight clinics.)
- In nested five-fold validation with 40 independent observations, each outer fold has eight observations. Within each outer-training set, use four-fold inner validation. How many observations are in each outer fit, each inner validation fold, and each inner fit? (Answer: 32 outer-training observations; 8 inner-validation observations; 24 observations per inner fit.)