Choosing Training And Holdout Proportions

Why Split Proportions Matter

A holdout design must reserve enough observations for each distinct task. The training set supports model fitting; the validation set supports decisions such as selecting among candidate models; and the test set provides a final evaluation on observations that did not guide those decisions. A split that devotes too much data to one task can weaken the others.

There is no universally best allocation such as 70% training, 15% validation, and 15% test. The suitable proportions depend on the total sample size, the complexity of the model, how much selection is planned, and how precise the final performance estimate needs to be. The aim is not to memorize a ratio, but to justify a design in terms of these needs. This is aligned with the SOA Exam PA (Predictive Analytics) syllabus.

Definition: The training set is used to fit the model. The validation set is used to make modeling or tuning choices. The test set is reserved for estimating the performance of the selected modeling procedure after those choices are complete. The three sets contain distinct observations, and the test results should not be used to revise the model and then still be presented as an independent final evaluation.

Start With the Tasks, Then Choose the Counts

A useful allocation begins with the decisions the analysis requires. Ask how many observations are needed to fit a credible model, how many are needed to compare or tune candidate approaches, and how large a test set is needed for a useful performance estimate. Proportions are then calculated from the resulting counts; they are not goals in themselves.

Training needs depend on the modeling problem. A model with many parameters or a flexible learning algorithm may need more observations to estimate its structure reliably. A small training set can produce unstable fitted models, even if the validation and test sets are large. Conversely, setting aside nearly all the data for training can leave too few observations to compare candidates or to assess final performance meaningfully.

Validation needs also depend on the task. If only a small number of models will be compared, a modest validation set may be adequate. If many modeling choices will be assessed using the same validation results, those results can become increasingly tailored to that set; a large validation set does not remove the need to reserve a genuinely independent test set. The number of observations available and the anticipated complexity of selection both matter.

The test set should be planned for the purpose of the final estimate. For a simple performance measure, a larger test set generally reduces sampling variability. But a test set cannot repair poor model fitting, and its estimate is only useful if the test observations represent the population or future cases to which the result is meant to apply. Proportions alone do not establish that representativeness.

Worked Example: Converting a Conventional Split to Counts

A hypothetical portfolio study has 2,400 independent records available for a random holdout design. The analyst proposes 70% for training, 15% for validation, and 15% for testing. Find the number of records in each set, and explain what the allocation does and does not establish.

Method. Multiply the total number of records by each proposed fraction:

n_{\text{train}}=2{,}400(0.70)=1{,}680,\qquad n_{\text{valid}}=2{,}400(0.15)=360,\qquad n_{\text{test}}=2{,}400(0.15)=360.

Check. The counts sum to \(1{,}680+360+360=2{,}400\). Their proportions are \(1{,}680/2{,}400=0.70\) and \(360/2{,}400=0.15\) for each holdout set.

The proposed design gives the model 1,680 records for fitting, 360 for selection, and 360 for final evaluation. Whether that is suitable still depends on model complexity, selection needs, and desired precision. The arithmetic verifies the split; it does not prove that the split is statistically adequate.

Common mistake avoided: Treating a familiar ratio as a guarantee of adequacy. A 70/15/15 allocation is a starting proposal, not a rule that overrides the analysis requirements.

Integer Counts and Limited Data

A proportion may imply a non-integer number of observations. In practice, allocation must use whole records. Round the proposed counts in a way that preserves the total, then report the actual counts and, where useful, the resulting proportions. Avoid rounding each set independently if that makes the allocated counts add to more or less than the available total.

For a small data set, it is often more useful to specify minimum counts for the tasks than to insist on standard percentages. If minimum requirements leave little flexibility, the data may not support a separate set for every role at the desired level. That is a design constraint to report, not a reason to claim that a very small holdout gives a precise evaluation.

Worked Example: Allocating a Small Data Set Around Minimums

A hypothetical claims-screening project has 360 records. The analyst judges that at least 250 are needed for fitting, at least 50 for validation, and at least 60 for a final test. Allocate the records using those minimums and give the proportions.

Method. The three minimum counts use the full available data:

$$ 250+50+60=360. $$

Therefore the allocation is 250 training records, 50 validation records, and 60 test records. The corresponding proportions are:

\frac{250}{360}=0.6944,\qquad \frac{50}{360}=0.1389,\qquad \frac{60}{360}=0.1667.

As percentages, these are approximately 69.44%, 13.89%, and 16.67%. The rounded percentages sum to 100.00%. The count check is \(250+50+60=360\), so no record is left unassigned or counted twice.

This allocation meets the stated minimums exactly, leaving no flexibility. If the model-fitting requirement increased to 270 while the two holdout minimums stayed at 50 and 60, the total required would be \(270+50+60=380\), which exceeds the available 360 records by 20. A different design or additional data would then be needed; simply changing the percentages cannot meet incompatible count requirements.

Common mistake avoided: Reporting only rounded proportions and overlooking the count constraint. Here, the counts show immediately that every record is assigned and that the minima consume the full sample.

Relating Test-Set Size to Evaluation Precision

Holdout size can be connected to the uncertainty of a performance estimate. For example, suppose each test observation produces a binary indicator equal to 1 if a prediction is correct and 0 otherwise. Assume the test observations are independent and have a common probability \(p\) of a correct prediction. The number correct among \(n\) test observations then has a binomial distribution, and the estimated accuracy \(\hat p\), the sample proportion correct, has approximate standard error \(\sqrt{\hat p(1-\hat p)/n}\).

For planning, a preliminary value of \(p\) can be used in place of \(\hat p\). A normal-approximation margin of error for a two-sided 95% interval is approximately \(1.96\sqrt{p(1-p)/n}\). This calculation is a planning aid, not a guarantee: the actual estimate, independence, and the adequacy of the normal approximation matter. The assumptions are especially important when observations are dependent or the outcome is not a binary indicator.

Justification: Let \(I_j\) be 1 if test observation \(j\) is classified correctly and 0 otherwise. Under the stated independent, common-probability assumption, \(E[I_j]=p\) and \(\operatorname{Var}(I_j)=p(1-p)\). Since \(\hat p=(I_1+\cdots+I_n)/n\), independence gives \(\operatorname{Var}(\hat p)=n p(1-p)/n^2=p(1-p)/n\). Taking the square root gives the standard error. The factor \(1/\sqrt n\) explains why increasing the test count improves precision, but with diminishing returns.

Worked Example: Planning a Test Set for an Accuracy Estimate

A hypothetical model is expected to achieve 80% accuracy. The analyst wants the approximate 95% margin of error for test accuracy to be no more than 0.04. A separate validation requirement calls for 300 records, and 3,000 records are available in total. Find a test-set size using the planning approximation, then allocate the remaining records.

Method. Set the approximate margin \(1.96\sqrt{p(1-p)/n}\) to 0.04, using \(p=0.80\), and solve for \(n\):

1.96\sqrt{\frac{0.80(0.20)}{n}}\leq 0.04 \quad\Longrightarrow\quad n\geq \frac{1.96^2(0.80)(0.20)}{0.04^2}.

Compute each part: \(1.96^2=3.8416\), \(0.80(0.20)=0.16\), and \(0.04^2=0.0016\). Thus

n\geq\frac{3.8416(0.16)}{0.0016} =\frac{0.614656}{0.0016} =384.16.

The test count must be a whole number, so round up to 385. This is the smallest integer meeting the inequality. With 3,000 total records and 300 assigned to validation, the training count is

n_{\text{train}}=3{,}000-300-385=2{,}315.

The resulting proportions are \(2{,}315/3{,}000=0.7717\), \(300/3{,}000=0.1000\), and \(385/3{,}000=0.1283\), or approximately 77.17%, 10.00%, and 12.83%. The count check is \(2{,}315+300+385=3{,}000\).

At the planning value \(p=0.80\), the standard error for \(n=385\) is \(\sqrt{0.16/385}\), approximately 0.0204, giving a 95% margin of \(1.96(0.0204)\), approximately 0.0400. This checks the target to the stated precision. It is an approximate planning result conditional on the stated assumptions, not a guaranteed interval width for every realized test result.

Common mistake avoided: Rounding 384.16 down to 384. That would not satisfy the calculated minimum. When a sample-size inequality gives a non-integer lower bound, round up.

What a Proportion Cannot Tell You

A split fraction does not measure model quality, ensure representative samples, or establish that performance will remain stable in use. It also does not capture every source of uncertainty in a fitted modeling procedure. In particular, a performance estimate from one random partition can vary with the particular observations assigned to each set.

Keep the roles separate when interpreting results. Training performance describes fit on data used to estimate the model and is not a substitute for independent evaluation. Validation performance guides choices and is therefore part of the selection process. Test performance is most informative when the selected procedure is evaluated on the test data only after those choices have been made. If the test results prompt further revisions, the test set has become part of the development process, and a fresh independent evaluation would be needed for a new final estimate.

A practical allocation can be described as a sequence of decisions:

Set the roles.
State what fitting, validation, and final testing must accomplish in the particular analysis.

Set count requirements.
Choose plausible training, validation, and test counts based on model needs, the number of choices to assess, and evaluation precision.

Check the total and proportions.
Confirm that the integer counts sum to the available sample and calculate their actual proportions.

State limitations.
Explain if a limited sample forces trade-offs or makes the final evaluation less precise than desired.

Common Allocation Traps

Assuming one conventional ratio is always correct. Proportions should reflect the data volume and analysis tasks; a memorized ratio is not a justification.
Allocating by percentage without checking counts. Integer rounding can leave records unassigned or make minimum requirements impossible.
Making the test set too small because training is important. A larger training set may help fit the model, but an extremely small test set can yield a noisy final estimate.
Expecting a larger test set to solve every problem. Test precision does not correct an inadequate training sample, excessive reuse of validation results, or a mismatch between the test observations and the intended use population.
Using the test results to keep tuning. Once the test set guides model changes, it no longer serves as an untouched final evaluation of those changes.

Key takeaway: Choose counts by balancing three needs: enough training observations to fit the model, enough validation observations to support planned choices, and enough test observations to make the final performance estimate useful. Then verify the total, report the actual proportions, and state the assumptions and limitations behind the allocation.

Check Your Understanding

For each problem, calculate the requested counts or precision and check the arithmetic before comparing with the answer.

A hypothetical study has 1,800 records and proposes 75% training, 15% validation, and 10% test. Give the count for each set and check that the counts sum to the total. (Answer: 1,350 training, 270 validation, and 180 test; total 1,800.)
A data set has 950 records and a proposed 70%/15%/15% allocation. Use the largest-remainder approach: first take the integer parts of the three exact counts, then assign any remaining records to the sets with the largest fractional remainders. Give the final counts. (Answer: 665 training, 143 validation, and 142 test.)
For a binary test-set measure, assume independent observations and a planning value \(p=0.70\). What test count is required for an approximate 95% margin of error no greater than 0.05, using 1.96 as the normal critical value? Round up. (Answer: 323 test observations.)
A hypothetical project has 1,200 records. It requires at least 800 training, 150 validation, and 200 test records. Allocate the 50 records left after meeting these minimums to training, and report the final counts and proportions. (Answer: 850 training, 150 validation, and 200 test; approximately 70.83%, 12.50%, and 16.67%, respectively.)
A model has estimated test accuracy \(\hat p=0.75\) on 400 independent test observations. Using the plug-in standard error \(\sqrt{\hat p(1-\hat p)/n}\), find the approximate 95% margin of error with critical value 1.96. (Answer: standard error approximately 0.0217; margin of error approximately 0.0424, or 4.24 percentage points.)

What You'll Learn