Why Small Data Make Validation Design Difficult
When data are plentiful, a holdout set can provide a straightforward evaluation while leaving many observations for fitting. With limited data, reserving even a modest fraction can make the training sample too small, while using nearly all the data for fitting may leave an unstable evaluation. Cross-validation reuses the available observations more efficiently, but it does not make a small data set large or remove all uncertainty.
The choice depends on the purpose of validation. Are you comparing candidate models, choosing settings, or estimating how well a modeling procedure will perform on new cases? These goals compete for the same observations. A design should provide enough data for a useful fit and enough relevant validation cases to make the comparisons meaningful. This topic is aligned with the SOA Exam PA (Predictive Analytics) syllabus.
A holdout design is easy to describe, but its result can depend heavily on which observations happen to be assigned to the validation set. Cross-validation gives each observation a validation role and usually fits on a larger fraction of the data than a single holdout fit. However, each cross-validation model is fitted on less than the full data set, and the fold results are not independent repetitions. Earlier material on data leakage, split stability, stratified folds, grouped folds, and time-series cross-validation remains relevant: a small-data design must still respect the structure of the observations.
Count the Data Available for Each Job
Let \(n\) be the number of observations. If a holdout set contains \(n_v\) observations, then \(n-n_v\) observations remain for fitting. If the holdout fraction is \(r\), the approximate counts are \(rn\) for validation and \((1-r)n\) for fitting. In practice, counts must be integers, and the actual allocation should be stated.
For \(k\)-fold cross-validation with equally sized folds, each validation fold contains about \(n/k\) observations, and each model is fitted on about \(n(k-1)/k\). When \(n\) is not divisible by \(k\), fold sizes differ by one or more observations. The exact fold sizes matter when the data set is very small.
This count addresses only one constraint: whether each fit has enough data. The validation fold must also contain enough information to assess the chosen metric. For a rare outcome, a fold with no positive cases cannot provide a fold-specific sensitivity. Likewise, a validation set with very few cases can produce a highly variable error rate even when the overall sample size appears adequate.
Why Each Observation Contributes One Cross-Validated Prediction
Suppose the \(n\) observations are assigned to \(k\) non-overlapping folds. For each fold \(j\), fit a model without that fold, then calculate a loss for each observation in fold \(j\). Every observation belongs to exactly one fold, so it receives exactly one prediction from a model that did not use that observation for fitting.
Here, \(j(i)\) is the fold containing observation \(i\), \(y_i\) is its observed outcome, \(x_i\) is its predictor information, and \(\widehat{f}^{(-j(i))}\) is the model fitted without fold \(j(i)\). The formula averages \(n\) out-of-fold losses, rather than averaging only one holdout set's losses.
The training-size calculation follows directly from the partition. If all folds contain \(n/k\) observations, then a fit omits \(n/k\) and uses \(n-n/k=n(k-1)/k\). This explains why increasing \(k\) generally gives each fit more observations, while making each validation fold smaller. It does not prove that a larger \(k\) always gives a more dependable estimate: small folds, repeated model selection, and variation in which cases are grouped together still matter.
Cross-validation estimates performance for a procedure trained on roughly \(n(k-1)/k\) observations. It is not exactly an evaluation of one final model trained on all \(n\) observations. For small data sets, the difference in training size may be consequential, especially if model performance changes sharply with sample size.
Worked Examples
Worked Example: Comparing a Holdout With Five-Fold Cross-Validation
A hypothetical insurer has 72 independent policy records and wants to compare predictive models using mean squared error. The team considers a holdout of 15 records or five-fold cross-validation. Calculate the fitting and validation counts, then interpret the trade-off. Assume the outcome and predictors are available for all 72 records, and no time or group restriction prevents random folds.
Method. Count the records used for fitting and validation under each design before choosing. The common mistake is to assume that a holdout and cross-validation provide the same number of model-fitting observations and the same amount of evaluation information.
Holdout calculation. The fitting set contains \(72-15=57\) records. One model is fitted on 57 records and evaluated on 15. If that evaluation produces 3 squared-error losses above a prespecified error threshold in a binary illustration, the corresponding rate would be \(3/15=0.20\); the important point here is that this summary rests on 15 validation cases.
Five-fold calculation. Since \(72/5=14.4\), the fold sizes must be integers: three folds can contain 14 records and two folds can contain 15, because \(3(14)+2(15)=72\). The corresponding fitting sizes are 58 for each 14-record validation fold and 57 for each 15-record validation fold. Thus all five models fit on either 57 or 58 records, close to the holdout fit size of 57.
Across the five folds, each of the 72 records receives one out-of-fold prediction. If the pooled cross-validated error rate in a binary illustration is \(14/72\), then \(14/72=0.1944\) to four decimal places. This is close to \(3/15=0.20\), but the two figures are not a controlled comparison: the designs use different validation observations and, in cross-validation, different fitted models.
Decision. Five-fold cross-validation is a reasonable choice when the team wants to compare models while retaining nearly all the records in each fit and using every record for validation once. The holdout is simpler, but its evaluation is based on only 15 cases. Neither design makes the result precise by itself; the error metric, case mix, and variability of results also matter.
Check. The fold counts sum to \(3(14)+2(15)=72\), and each fitting count is 72 minus its validation-fold size. The method avoids the mistake of treating “five folds” as five independent full-size evaluations.
Worked Example: Rare Outcomes and Fold Design
A hypothetical health insurer has 40 claims records, including 4 cases of a rare outcome and 36 cases without it. The analyst wants to assess sensitivity, defined as the proportion of actual positive cases predicted positive. Compare a random 10-record holdout with stratified four-fold cross-validation. Assume records are independent and each fold can be assigned the stated class counts.
Method. Check how many positive cases enter each validation portion, because sensitivity requires actual positive cases in the evaluation data. A common mistake is to choose the fold count using only the total sample size and ignore the outcome count.
Holdout calculation. If the holdout is stratified to contain one positive and nine negative records, then the fitting set has \(40-10=30\) records: 3 positive and 27 negative. Sensitivity on the holdout can only be 0 or 1, depending on whether that single positive case is detected. One case therefore determines the entire estimate.
Four-fold calculation. Each validation fold has \(40/4=10\) records. A stratified allocation can place 1 positive and 9 negative cases in each fold. Each fit then uses \(40-10=30\) records, including 3 positive and 27 negative cases. If the four held-out positive cases receive predictions of which three are positive, pooled sensitivity is \(3/4=0.75\).
The pooled result has four positive cases contributing to the sensitivity calculation, instead of one in the holdout. But its possible values still change in increments of \(1/4=0.25\), so it remains coarse. Moreover, each fold's fitted model uses only 3 positive cases. Cross-validation has improved the use of the scarce positive cases for evaluation; it has not removed the limitation on learning their patterns.
Decision. Four-fold stratified cross-validation is more informative than a single 10-record holdout for this sensitivity assessment, provided the stated allocation is feasible. The analyst should report the small number of positive cases and avoid describing \(0.75\) as a highly precise estimate.
Check. The folds contain \(4(1)=4\) positive and \(4(9)=36\) negative cases. Each training set contains 3 positive and 27 negative cases. The method avoids evaluating sensitivity on a fold with no positive cases.
Worked Example: Fitting Minimums and Model Selection
A hypothetical agricultural insurer has 90 independent farm records. Its modeling team plans to compare six candidate specifications, and preliminary modeling indicates that each fit needs at least 68 records to be useful. The team is considering a 25% holdout or five-fold cross-validation. Choose a design for comparing candidates, and state a limitation if the same validation results are used to select a candidate and summarize its performance.
Method. First check the number of records in each fit against the minimum. Then consider whether the validation data are adequate for comparing six candidates. The common mistake is to focus on evaluation size while overlooking that the resulting training set may fail the stated fitting requirement.
Holdout calculation. A 25% holdout is \(0.25(90)=22.5\) records. An integer allocation could use 22 validation records and 68 fitting records, or 23 validation records and 67 fitting records. The first allocation meets the 68-record minimum exactly; the second does not. Thus a 25% holdout is only feasible under the first allocation, leaving a small validation set for comparing six candidates.
Five-fold calculation. Since \(90/5=18\), each validation fold has 18 records and each fit uses \(90-18=72\) records. All five fits exceed the minimum by \(72-68=4\) records. Each candidate can receive out-of-fold predictions for all 90 records, so the comparison uses more validation cases than the single holdout.
Decision and limitation. Five-fold cross-validation is the more practical design for comparing the six candidates while meeting the fitting minimum. However, if the team chooses the candidate because it had the best cross-validation result, that same selected result may be optimistic: the selection favors candidates that happened to score well on those folds. Cross-validation supplies more validation predictions, but it does not make repeated selection on those results harmless. If a separate, reliable performance assessment is required, the design must account for that additional goal rather than treating the selected score as an independent evaluation.
Check. The holdout allocation of 22 validation and 68 fitting records sums to 90 and meets the minimum; the 23/67 allocation does not. Five-fold counts are \(5(18)=90\), with \(72\) training records per fit. The method avoids choosing a split that silently violates the minimum.
Decision Rules for Limited Data
A practical design begins with constraints, not a customary percentage. The following sequence makes the trade-offs explicit.
Identify whether the immediate task is candidate comparison, model selection, or performance assessment. A single set of results may not serve all purposes without qualification.
Use the problem context to identify a minimum useful training size, if one is known. Calculate the exact number of records each candidate design leaves for fitting.
For each metric, verify that the validation cases include the necessary outcomes and meaningful variation. Rare classes can make a seemingly reasonable split uninformative.
Repeatedly choosing among candidates based on the same validation results can create optimism. Also preserve groups or time order where the data-generating structure requires it; random folds are not automatically appropriate.
Report the sample counts and design, and explain what the estimate can support. Small-data results should not be presented with a level of precision the validation sample cannot justify.
A holdout is attractive when a single, untouched validation sample is important and enough data remain for both fitting and evaluation. Cross-validation is attractive when a single holdout would discard too much fitting information or produce an evaluation based on very few cases. When data are exceptionally scarce, neither method guarantees a dependable estimate; the most defensible choice is the one whose limitations are explicit and whose partitions reflect the intended use.
| Consideration | Holdout design | k-fold cross-validation |
|---|---|---|
| Fitting data | One fit uses \(n-n_v\) records | Each fit uses about \(n(k-1)/k\) records |
| Validation use | Only the holdout records are evaluated | Each record is evaluated once out of fold |
| Main small-data concern | A small or unrepresentative holdout can be unstable | Each fit is smaller than the full-data fit; fold results are related |
| Rare outcomes | May contain too few positive cases | Stratification can distribute positives, but cannot create more |
| Model selection | Repeated use can encourage overfitting to the holdout | Repeated selection can also make the best reported score optimistic |
Common Mistakes
- Choosing a split percentage by habit. Calculate the actual fitting and validation counts, then check them against the task's needs.
- Assuming cross-validation removes small-sample uncertainty. It reuses observations, but the number of rare outcomes and the underlying information remain limited.
- Ignoring the metric's requirements. A fold without positive cases cannot yield a fold-specific sensitivity, and a tiny holdout may give a misleadingly coarse result.
- Treating fold scores as independent replicates. The training sets overlap, so the fold results are related; a simple standard error calculation based on independent folds is not automatically justified.
- Using the best validation score as if model selection had not occurred. Selecting the best among many candidates based on the same scores can make the selected score optimistic.
- Violating data structure to increase fold counts. If records are grouped or time-ordered, a random partition can give an evaluation that does not match the intended prediction setting.
Check Your Understanding
For each problem, calculate the relevant counts before deciding what the design can support.
- A data set has 65 independent observations. In five-fold cross-validation, how many observations are in each validation fold and how many are used for fitting in each fold? (Answer: 13 validation and 52 fitting observations per fold.)
- A data set has 48 records, including 6 positive cases, and uses four stratified folds of equal size. How many records are in each validation fold, and how many positive cases can be assigned to each fold if they are distributed as evenly as possible? (Answer: 12 records per fold; two folds can have 2 positive cases and two folds can have 1 positive case.)
- A team has 100 observations and compares a 20% holdout with five-fold cross-validation. Give the fitting and validation counts for each design. (Answer: Holdout: 80 fitting and 20 validation; five-fold: 80 fitting and 20 validation per fold, with all 100 observations receiving an out-of-fold prediction.)
- A data set has 84 observations, and each fit must use at least 68. Compare three-fold and seven-fold cross-validation. Which meets the requirement, and how many validation records does it use per fold? (Answer: Three-fold uses 56 for fitting and fails the requirement; seven-fold uses 72 for fitting and 12 for validation per fold, so seven-fold meets it.)
- A hypothetical claims data set has 72 independent records, including 8 positive outcomes. The analyst proposes a stratified 15-record holdout or four-fold cross-validation. How many records are used for fitting in each design, and how many positives can each four-fold validation group contain if positives are distributed evenly? (Answer: The holdout leaves 57 fitting records; four-fold uses 54 fitting records per fold, with 2 positive cases in each validation fold.)