Why Holdout Sample Size Is a Tradeoff
A holdout evaluation divides available observations into separate groups: the training sample is used to fit the model, and the test sample is reserved to evaluate predictions on observations not used for fitting. The training group affects what model can be estimated; the test group affects how much information the evaluation contains. Increasing one group necessarily decreases the other when the total dataset is fixed.
This makes the choice of sample sizes consequential. A small training sample may produce a model that is unstable or cannot be fit reliably. A small test sample may produce an evaluation that changes substantially depending on which observations happened to be held out. A split percentage is therefore a design choice, not a universal rule.
The model and its intended use matter. A model with many candidate predictors, complicated fitting requirements, or rare outcome patterns may need more training data than a simple model. Meanwhile, a noisy performance measure or a desire for a narrow uncertainty range may require more test observations. The size needed for training depends on the modeling task; the size needed for testing depends on the evaluation goal and the variability of the measured performance.
What a Larger Test Sample Adds
Suppose a fitted model is evaluated on \(n_{\text{test}}\) independent test observations drawn from the population relevant to the intended predictions. For each test observation, calculate a loss \(L_i\), such as squared prediction error. The mean test loss is the average of these losses. Conditional on the already-fitted model, if the test losses have finite variance and the test observations are independent and representative, the standard error of their mean is estimated by
where \(s_L\) is the sample standard deviation of the test losses. This relationship explains the basic precision tradeoff: with similar variability in losses, quadrupling the test size halves the standard error. It does not make the estimate more accurate in every possible sense, and it does not correct bias caused by an unrepresentative test sample or an evaluation that differs from the intended use.
For classification accuracy, each test outcome can be represented as correct or incorrect. If the true probability of a correct prediction for the fixed fitted model is \(p\), the observed accuracy is a sample proportion. Its approximate standard error is \(\sqrt{p(1-p)/n_{\text{test}}}\). Since \(p(1-p)\leq 0.25\), the largest such standard error is approximately \(0.5/\sqrt{n_{\text{test}}}\). This is a planning guide, not a guarantee of an exact confidence interval; the approximation can be poor with very few test cases or very few errors.
Worked Example: Comparing Test-Set Precision
A hypothetical dataset contains 150 independent cases for a binary classification task. Consider two possible allocations: 80% for training and 20% for testing, or 70% for training and 30% for testing. Compare their approximate worst-case standard errors for observed accuracy.
With an 80/20 split, the test sample has \(0.20(150)=30\) cases. Using \(p=0.5\), which gives the largest value of \(p(1-p)\),
With a 70/30 split, the test sample has \(0.30(150)=45\) cases, giving
The larger test group yields a smaller approximate standard error, while leaving 105 rather than 120 cases for training. These calculations do not establish which split is better: that also depends on whether 105 training cases are enough to fit the intended model, and whether either test sample represents the intended population. The figures are planning approximations under the stated sampling assumptions.
Choosing an Allocation in Practice
A defensible choice begins by considering both sides of the tradeoff. First ask what the modeling procedure needs to fit a plausible model. Then ask how much test information is needed for the performance measure to be useful. If one side has a clear minimum requirement, the other side must fit within the remaining data. The final choice should respect the data structure and the purpose of the evaluation.
For example, a prediction task involving a rare outcome may need enough test cases of that outcome for the evaluation to be informative. A random split can leave very few such cases even when the overall test proportion looks reasonable. Likewise, a model that cannot be fitted reliably with a small training group does not become adequate just because the test group is large. A proposed percentage should be translated into actual case counts, then checked against the task.
Specify the population and prediction setting the test results are meant to represent, and choose a performance measure that suits that goal.
Consider whether the remaining training cases can support the planned model and whether important outcome groups or predictor patterns will be represented adequately.
Translate the test proportion into a number of cases. Consider the variability of the chosen performance measure and whether the test set can provide a useful evaluation.
Record the actual group sizes and recognize that a holdout estimate evaluates a model fitted on the training group, not a model fitted on all \(N\) cases.
Worked Example: Planning for an Accuracy Precision Goal
A hypothetical team wants an approximate 95% margin of error of no more than 0.10 for a test-set accuracy estimate. Assume the test observations are independent and representative, and use the normal approximation for a sample proportion. Because the accuracy is not known in advance, use \(p=0.5\) for a conservative planning calculation.
The approximate margin of error is \(1.96\sqrt{p(1-p)/n_{\text{test}}}\). Requiring this to be at most 0.10 gives
Divide by 1.96 and square both sides:
Since the test size must be a whole number, this calculation calls for at least 97 test cases. If the team plans to reserve 20% of the dataset for testing, the total sample size would need to satisfy \(0.20N\geq97\), or \(N\geq485\). This is an approximate planning result, not a promise that every realized interval will have the desired width. It also addresses only test precision: the remaining training cases must still be sufficient for the model.
Small Datasets and Competing Needs
With a small dataset, even a seemingly standard allocation can leave both groups inadequate. The training group may be too small for the planned model, while the test group may be too small for a stable evaluation. Reporting a percentage without the counts can hide this problem: a 20% test fraction means 10 cases when \(N=50\), but 200 cases when \(N=1{,}000\).
When the available sample cannot support both a useful fit and a precise holdout estimate, there is no arithmetic trick that removes the limitation. One may need to collect more data, simplify the modeling goal, or use a different evaluation design suited to limited data. Whatever design is chosen, the limitation should be reported rather than implying that a small test set provides a precise estimate.
Worked Example: A 50-Case Dataset
Suppose a hypothetical dataset has 50 cases. A proposed 80/20 split gives \(0.80(50)=40\) training cases and \(0.20(50)=10\) test cases. For binary accuracy, the largest approximate standard error is
The corresponding approximate 95% margin of error at the worst-case accuracy is \(1.96(0.158)=0.310\), or about 31 percentage points. For instance, an observed accuracy of 0.70 would not justify describing performance as known precisely from these 10 test cases. The approximation itself is also rough at such a small sample size.
The 40 training cases may also be insufficient for a complicated model; whether they are adequate depends on the predictors, outcome, and fitting method. Thus this split has two concerns: little information for evaluation and a potentially restricted training sample. A larger test fraction would improve the count available for evaluation but further reduce training data. The appropriate response is to assess the model and evaluation goal honestly, not to assume that 80/20 is automatically suitable.
A Short Justification for the Precision Rule
The standard-error relationship follows from the variance of a sample mean. If test losses \(L_1,\ldots,L_m\) are independent draws with common variance \(\sigma_L^2\), then the mean loss is \(\bar{L}_{\text{test}}=(L_1+\cdots+L_m)/m\). Independence makes the variance of the sum equal to the sum of the variances, so
Taking the square root gives \(\sigma_L/\sqrt{m}\), and replacing the unknown \(\sigma_L\) with the sample standard deviation \(s_L\) gives the estimated standard error. The argument assumes independent test observations with the same loss distribution for the fixed fitted model. Dependence, sampling bias, or a mismatch between the test sample and intended prediction population can make this calculation misleading.
Common Mistakes
- Treating a familiar percentage as a rule. An 80/20 split is not automatically appropriate. Check the actual training and test counts and the requirements of the analysis.
- Maximizing the test set without considering model fitting. More test cases can improve evaluation precision, but they leave fewer cases to fit the model.
- Assuming a large test proportion ensures a representative test set. Size alone does not correct selection bias or guarantee coverage of rare but important cases.
- Calling a test estimate the performance of a model trained on all available data. The holdout result evaluates the model fitted on the training sample. A model refitted using additional data is a different fitted model.
- Reading an approximate precision calculation as a guarantee. Standard-error and margin-of-error calculations rely on sampling assumptions and approximations; they do not ensure a particular realized result.
Check Your Understanding
Answer each question using the allocation and precision ideas from this tutorial.
- A dataset has 240 cases. How many cases go to training and testing under a 75/25 split? What tradeoff does the larger training group imply?
- Why does increasing the test sample from 25 to 100 cases reduce the approximate standard error of a sample mean of losses by half, if loss variability stays similar?
- For a binary accuracy estimate with 64 independent test cases, calculate the approximate worst-case standard error using \(p=0.5\).
- Explain why a 30% test allocation may still yield an uninformative evaluation for an outcome that is very rare.
- A holdout evaluation is based on a model fitted using 80% of the available observations. Which fitted model does the test result evaluate, and why is that not automatically the model trained on all observations?