Why Small Samples Make Splitting Difficult
A model needs observations to learn patterns, but evaluating it also requires observations that were not used to fit it. With a large sample, it may be practical to reserve data for both model selection and a final evaluation. With a small sample, every reservation has a cost: the model is fitted on less data, while each evaluation set may be too small to give a stable performance estimate.
There is no universally correct percentage for a training, validation, and test split. A split that is reasonable for several thousand observations may leave only a handful in each evaluation set when there are 40 observations. The decision depends on the total sample size, the prediction task, the loss being measured, and whether an independent final evaluation is essential.
The distinction between validation and test data matters even when the sample is small. If several candidates are compared on the same validation data, the selected candidate tends to be the one that benefited most from random features of that particular validation set. Its validation score is therefore not a neutral final estimate. A test set used only after selection can provide a separate check, but a very small test set may yield a highly uncertain score.
How Evaluation Size Affects Precision
Suppose a fixed model is evaluated on \(m\) independent observations from the target population, and each observation has loss \(L_i\). The estimated mean loss is \(\bar{L}=m^{-1}\sum_{i=1}^{m}L_i\). If the losses have finite variance \(\sigma_L^2\) and are independent with the same distribution, then \(\operatorname{Var}(\bar{L})=\sigma_L^2/m\). Thus, increasing the evaluation size reduces the variance of the estimated mean loss. This formula concerns evaluation uncertainty for a fixed model; it does not include the extra uncertainty from fitting or selecting the model.
where \(s_L\) is the sample standard deviation of the observed losses. The estimated standard error is not a guarantee that the evaluation is representative; it quantifies sampling variability under the stated assumptions.
For classification accuracy, an equivalent view is the error indicator \(E_i\), equal to 1 for a misclassification and 0 otherwise. If the model is fixed and the test outcomes are independent Bernoulli observations with common error probability \(p\), then the number of errors \(K\) has a binomial distribution with parameters \(m\) and \(p\). The observed error rate is \(\hat p=K/m\), with estimated standard error \(\sqrt{\hat p(1-\hat p)/m}\). This approximation can be particularly unstable for a small test set, especially if \(K\) is near 0 or \(m\).
Worked Example: How Much Can a Small Test Set Tell You?
A hypothetical insurer has 80 independent policy records set aside for model development and evaluation. A fixed classifier is evaluated on a random test set. Assume each test record has an independent Bernoulli error indicator with a common, unknown error probability \(p\). Compare the uncertainty from test sets of 10 and 20 records if the observed error rate is 0.20 in each case. Use the plug-in standard error and a normal-approximation interval of estimate plus or minus 1.96 standard errors.
Method. For \(m\) test records and observed error rate \(\hat p\), calculate \(\widehat{\operatorname{SE}}=\sqrt{\hat p(1-\hat p)/m}\). The normal interval is only an illustration of the uncertainty; with these small samples it can extend outside the possible range from 0 to 1.
Test set of 10. An observed rate of 0.20 means \(K=10(0.20)=2\) errors. The estimated standard error is \(\sqrt{0.20(0.80)/10}=\sqrt{0.016}=0.1265\), rounded to four decimals. The interval endpoints are \(0.20-1.96(0.1265)=-0.0479\) and \(0.20+1.96(0.1265)=0.4479\), rounded to four decimals.
Test set of 20. An observed rate of 0.20 means \(K=20(0.20)=4\) errors. The estimated standard error is \(\sqrt{0.20(0.80)/20}=\sqrt{0.008}=0.0894\), rounded to four decimals. The interval endpoints are \(0.20-1.96(0.0894)=0.0248\) and \(0.20+1.96(0.0894)=0.3752\), rounded to four decimals.
Interpretation and check. The larger test set has a smaller estimated standard error, as expected from the \(1/\sqrt{m}\) relationship. Both intervals are wide, and the 10-record interval includes an impossible negative error rate. That is a warning about the normal approximation and the limited information in a tiny sample, not evidence that an error rate can be negative. The common mistake is to report 2 errors out of 10 as a precise estimate of future performance; the arithmetic gives 20%, but the evaluation remains uncertain.
Holdout Splits and Cross-Validation
A single holdout split is easy to explain: fit on one part and evaluate on another. Its weakness in a small sample is that the result depends heavily on which observations happened to be held out. Reserving a validation set as well as a test set divides the sample still further. A small validation set may rank candidates unreliably; a small test set may give a noisy final estimate; and the smaller training set may produce a model that differs from the one that would be fitted using all available data.
Cross-validation reuses the sample for assessment in a structured way. In \(K\)-fold cross-validation, the observations are partitioned into \(K\) folds. For each fold, fit the candidate model using the other \(K-1\) folds and calculate its loss on the held-out fold. Combine the held-out losses to form a cross-validation score. Each observation is held out once, but it is included in the training data for the other fits.
For equal-sized folds, the overall mean loss is the average of the fold mean losses. For unequal fold sizes, the overall mean should weight each fold mean by its number of held-out observations. When data are scarce, the choice of \(K\) also involves a tradeoff: larger \(K\) makes each training fit use a larger fraction of the sample, but produces smaller held-out folds; smaller \(K\) gives larger held-out folds but fits on less data. Neither choice removes sampling uncertainty or model-selection optimism.
Worked Example: Comparing Candidates With Three Folds
A hypothetical pricing team has 24 independent records and compares two fixed regression procedures using three-fold cross-validation. Each fold contains 8 records, so each fit uses 16 records. The reported mean squared errors (in squared response units) for candidate A are 16, 25, and 9 across the three held-out folds; those for candidate B are 20, 16, and 16. Calculate each overall cross-validation score and identify the lower-scoring candidate.
Method. Because the three held-out folds have the same size, the overall mean squared error is the unweighted average of their fold mean squared errors.
Candidate A. Its score is \((16+25+9)/3=50/3=16.6667\), rounded to four decimals. The total of the three fold losses expressed as fold means is 50.
Candidate B. Its score is \((20+16+16)/3=52/3=17.3333\), rounded to four decimals. Its corresponding total is 52.
Candidate A has the lower cross-validation mean squared error: \(16.6667<17.3333\), a difference of \(17.3333-16.6667=0.6666\) using the rounded scores (the unrounded difference is \(2/3\)). Thus A is preferred by this criterion. The calculation checks against the pooled total: each fold has 8 observations, so averaging three equally sized fold means gives the same result as averaging all 24 held-out squared errors. The common mistake is to select A and then present its cross-validation score as an independent test result; the scores were used for selection, and only three fold results provide limited evidence about the performance difference.
Choosing a Strategy When the Sample Is Limited
The decision should begin with the purpose of evaluation rather than a conventional split percentage. If there is enough data to reserve a final test set of meaningful size, it can be kept untouched while training and validation choices are made with the remaining data. If reserving both validation and test sets would leave each too small, cross-validation can be a more useful way to compare candidates. The final report should then describe the cross-validation result as a model-selection estimate, not as a separate test-set result.
If an independent final evaluation is a strict requirement, a small sample may simply be inadequate for both robust model development and a precise final estimate. The honest response is to state that limitation, seek additional data if feasible, or use a simpler prespecified model-development plan that reduces the number of choices. A small test set should not be made to sound more conclusive by reporting many decimal places.
Use the number of genuinely independent units available for the intended evaluation, not merely the number of rows if rows are repeated measurements on the same unit.
Separate the goal of choosing a model from the goal of reporting final performance. A validation result used for selection has a different role from an untouched test result.
For each proposed split, note the number used for fitting and the number used for each evaluation. Check whether a small evaluation size makes the reported metric too uncertain for the decision.
Report the evaluation design, sample size, and whether the score was used for model selection. Do not imply that cross-validation or a small test set eliminates uncertainty.
A useful supporting calculation is the approximate standard error of the metric at the proposed evaluation size, when its assumptions are appropriate. For a mean loss, use the sample standard deviation of losses divided by the square root of the evaluation size. For a binary error rate, use the Bernoulli calculation shown above, while recognizing that more careful intervals may be preferable for very small counts. These calculations can reveal when two close scores do not justify a strong claim that one candidate is better.
Worked Example: A Holdout Result Versus Five-Fold Assessment
A hypothetical safety program has 30 independent observations and evaluates a fixed classifier. In one holdout design, 24 observations are used for fitting and 6 for evaluation; 1 of the 6 held-out observations is misclassified. In an alternative five-fold assessment, each fit uses 24 observations and the numbers of errors in the five held-out folds are 0, 1, 1, 2, and 0. Assume the held-out observations are distinct and each fold has 6 observations. Calculate both error rates and decide what the comparison can support.
Holdout calculation. The holdout error rate is \(1/6=0.1667\), or 16.67% to two decimal places. Its plug-in standard error under the independent Bernoulli model is \(\sqrt{(1/6)(5/6)/6}=\sqrt{5/216}=0.1521\), rounded to four decimals.
Five-fold calculation. The total number of held-out errors is \(0+1+1+2+0=4\). Each of the 30 observations is held out once, so the pooled cross-validation error rate is \(4/30=0.1333\), or 13.33% to two decimal places. Equivalently, the five fold error rates are \(0/6\), \(1/6\), \(1/6\), \(2/6\), and \(0/6\); their equal-weight average is \((0+1/6+1/6+2/6+0)/5=4/30\).
The two estimates differ by \(0.1667-0.1333=0.0334\) using rounded rates (the unrounded difference is \(1/30\)). This small difference does not establish that the classifier's true error rate changed. The holdout and cross-validation fits are not identical evaluation procedures, and cross-validation folds are not independent replications. The common mistake is to treat the five fold results as five independent test samples and claim a precise comparison. The calculations support reporting the observed rates and their designs, not an unsupported certainty about which estimate is closer to future performance.
Common Traps and Practical Conclusions
- Applying a standard percentage mechanically. A 20% test allocation may be adequate in one study and far too small in another. Calculate actual split counts before deciding.
- Ignoring the training cost. A larger test set can improve evaluation precision while leaving fewer observations to fit the model. The fitted model may then differ from the model eventually trained on all available data.
- Using the validation score as the final score. Once it has helped choose a candidate, it is part of the selection process. A favorable score may partly reflect chance variation.
- Assuming cross-validation removes uncertainty. It makes more efficient use of a limited sample for assessment, but does not add information or guarantee stable rankings.
- Overinterpreting small score differences. A lower observed loss is a numerical fact about the evaluation data. It is not automatically strong evidence of better performance on future observations.
Check Your Understanding
For each problem, identify the split tradeoff and check the numerical result before interpreting it.
- A fixed classifier is tested on 12 independent cases and makes 3 errors. Under an independent Bernoulli error model, calculate the observed error rate and its plug-in standard error. What should be said about the precision of this result? (Answer: Error rate \(3/12=0.2500\); standard error \(\sqrt{0.25(0.75)/12}=0.1250\), so the estimate is imprecise.)
- A regression model has cross-validation fold mean squared errors of 4, 10, and 7 on three equal-sized folds. Calculate the overall cross-validation score. If this score was used to select the model, is it an independent final test score? (Answer: \((4+10+7)/3=7\); no, it was used in model selection.)
- A sample has 50 independent observations. A proposed design uses 35 for training, 10 for validation, and 5 for testing. State the percentage in each set and identify one concern about the design. (Answer: 70%, 20%, and 10%, respectively; the test estimate based on only 5 observations may be highly uncertain.)
- With 40 observations, four-fold cross-validation gives fold sizes of 10 and errors of 1, 0, 2, and 1. Calculate the pooled error rate. Each fit uses how many observations? (Answer: \(4/40=0.1000\); each fit uses \(40-10=30\) observations.)
- A team compares 12 candidates using one small validation set, selects the one with the lowest validation loss, and plans to report that loss as final performance. Name the problem and give a more defensible evaluation option if no additional data are available. (Answer: The selected validation score is subject to selection optimism; use a prespecified cross-validation strategy for model comparison and report it as a selection estimate, not as an independent test result.)