Choosing the Training-Validation Split Fraction

Why the Split Fraction Is a Statistical Choice

A holdout split assigns $m$ of $n$ observations to training and $v=n-m$ to validation. Increasing $m$ can improve the fitted candidates by giving the learning procedure more data. Increasing $v$ can make validation risks more precise and selection among candidates less vulnerable to noise. These goals compete: at fixed $n$, allocating one more observation to training removes one from validation.

There is no universally optimal training fraction. The answer depends on how quickly training risk improves with sample size, how noisy the validation loss is, how many candidates are compared, and what criterion is being optimized. This tutorial makes those dependencies explicit using finite-sample calculations. The formulas are guides under stated assumptions, not rules that every problem should use the same way.

Definition: Let $Z_1,\ldots,Z_n$ be iid observations from a target population $P$, and let $\mathcal A_m$ be a learning procedure trained on $m$ observations. A split has training size $m$ and validation size $v=n-m$. For a loss function $L(f,Z)$, the target risk of a fitted candidate $f$, conditional on the training data, is $$ R(f)=E[L(f,Z)\mid D_{\mathrm{tr}}], $$ where $Z$ is a new observation from $P$, independent of the training data $D_{\mathrm{tr}}$. The validation estimate is $$ \widehat R_{\mathrm{val}}(f)=\frac{1}{v}\sum_{i=1}^{v}L(f,Z_i^{\mathrm{val}}). $$ The split fraction for training is $m/n$; the validation fraction is $v/n=1-m/n$.

The iid target-population assumption matters. Conditional on the training data, validation observations must be independent of that data and sampled from the population defining $R(f)$. Independence alone is not enough: if validation observations come from a different population, their average need not estimate the target risk. For a random split of iid data, the training and validation subsets have the required independence and representativeness.

What Validation Size Controls

Suppose first that a candidate $f$ is fixed after training and the loss is bounded, $0\leq L(f,Z)\leq 1$. Conditional on the training data, the validation losses are iid with mean $R(f)$. Therefore, $$ E[\widehat R_{\mathrm{val}}(f)\mid D_{\mathrm{tr}}]=R(f), \qquad \operatorname{Var}(\widehat R_{\mathrm{val}}(f)\mid D_{\mathrm{tr}}) =\frac{\operatorname{Var}(L(f,Z)\mid D_{\mathrm{tr}})}{v} \leq \frac{1}{4v}. $$ The variance decreases in proportion to $1/v$, while the typical size of an error bound decreases in proportion to $1/\sqrt v$. These are different summaries of uncertainty, and they lead to different allocation formulas.

Worked Example: Validation Precision for a Fixed Candidate

Suppose a trained classifier has a fixed conditional error probability $R(f)=0.20$, and validation observations are iid from the target population. With $v=25$, the validation error indicator is Bernoulli with variance $0.20(0.80)=0.16$. Thus the validation error rate has conditional variance $0.16/25=0.0064$ and standard deviation $0.08$.

If the validation set is increased to $v=100$, its variance becomes $0.16/100=0.0016$, and its standard deviation becomes $0.04$. Quadrupling the validation size halves the standard deviation; it does not reduce it by a factor of four. This calculation concerns a fixed candidate, not the extra uncertainty caused by choosing the best among many candidates.

When several candidates are compared, validation noise can make the selected candidate appear better than it is. A finite-class bound shows how this cost depends on both $v$ and the number of candidates.

Theorem: Suppose $M$ candidates are fitted using the training data, their losses lie in $[0,1]$, and the validation observations are iid from the target population and independent of training. Let $\hat f$ minimize validation risk among these candidates. For any $0<\delta<1$, with conditional probability at least $1-\delta$ given the training data, $$ R(\hat f)\leq \min_{1\leq j\leq M}R(f_j) +2\sqrt{\frac{\log(2M/\delta)}{2v}}. $$

Proof. Conditional on the training data, each candidate is fixed. Hoeffding's inequality for a mean of $v$ iid variables in $[0,1]$ gives $$ P\left(\left|\widehat R_{\mathrm{val}}(f_j)-R(f_j)\right|>\epsilon\mid D_{\mathrm{tr}}\right) \leq 2e^{-2v\epsilon^2}. $$ Taking a union bound over $M$ candidates, the probability that any candidate violates this deviation bound is at most $2M e^{-2v\epsilon^2}$. Set $\epsilon=\sqrt{\log(2M/\delta)/(2v)}$; the probability is then at most $\delta$. On the event that all deviations are at most $\epsilon$, let $j_*$ be a candidate of minimum risk. Since $\hat f$ minimizes validation risk, $$ R(\hat f) \leq \widehat R_{\mathrm{val}}(\hat f)+\epsilon \leq \widehat R_{\mathrm{val}}(f_{j_*})+\epsilon \leq R(f_{j_*})+2\epsilon. $$ This proves the result. $\square$

A Split Criterion and Its Minimizer

A simple finite-sample planning criterion combines a training-risk term that decreases with $m$ and a validation-estimation term that decreases with $v$. For example, suppose the relevant excess training risk is approximated or bounded by $A/m$, while the mean squared error of a validation estimate contributes $B/v$, for positive constants $A$ and $B$. The resulting criterion is $$ J(m)=\frac{A}{m}+\frac{B}{n-m},\qquad 0<m<n. $$ This is a model for the tradeoff, not a universal risk identity. In particular, a high-probability selection bound has a $1/\sqrt v$ term rather than a $1/v$ term, as shown below.

Result: For positive $A,B$, the continuous minimizer of $J(m)=A/m+B/(n-m)$ is $$ m^*=\frac{n\sqrt A}{\sqrt A+\sqrt B}, \qquad v^*=\frac{n\sqrt B}{\sqrt A+\sqrt B}. $$ For integer split sizes, evaluate the feasible integers nearest $m^*$ (and respect any minimum training or validation size required by the procedure).

Derivation. Differentiating gives $$ J'(m)=-\frac{A}{m^2}+\frac{B}{(n-m)^2}. $$ At an interior stationary point, both denominators are positive, so taking positive square roots yields $\sqrt A/m=\sqrt B/(n-m)$. Rearranging gives $m\sqrt B=(n-m)\sqrt A$, hence the displayed $m^*$ and $v^*=n-m^*$. Moreover, $$ J''(m)=\frac{2A}{m^3}+\frac{2B}{(n-m)^3}>0, $$ so $J$ is strictly convex and this stationary point is its unique global minimum. For integer $m$, strict convexity means the integer minimum is attained among the feasible integers immediately below and above the continuous minimizer. $\square$

Worked Example: An Inverse-Size Split Criterion

Suppose $n=100$, with planning criterion $J(m)=4/m+9/(100-m)$. Here $A=4$ and $B=9$, so the continuous optimum is $$ m^*=\frac{100\sqrt{4}}{\sqrt{4}+\sqrt{9}} =\frac{100(2)}{2+3}=40, \qquad v^*=60. $$ The criterion at this split is $J(40)=4/40+9/60=0.10+0.15=0.25$.

For comparison, $J(39)=4/39+9/61\approx0.2501$ and $J(41)=4/41+9/59\approx0.2501$. The continuous optimum is already an integer, and the strict-convexity result confirms it is the integer minimum. In this criterion, validation receives the larger share because its coefficient $B=9$ exceeds the training coefficient $A=4$.

Using a Selection Bound to Guide Allocation

The finite-class theorem gives a different criterion when the goal is a high-probability guarantee for the selected candidate. Suppose the minimum risk among candidates trained on $m$ observations is bounded by $R_\infty+A/m$, where $R_\infty$ is a baseline and $A>0$. Combining that assumption with the theorem gives, with probability at least $1-\delta$, $$ R(\hat f)\leq R_\infty+\frac{A}{m} +2\sqrt{\frac{\log(2M/\delta)}{2(n-m)}}. $$ This expression makes the competing effects visible: training risk is penalized when $m$ is small, while validation-based selection uncertainty is penalized when $n-m$ is small. Its constants and its $1/\sqrt v$ dependence differ from the mean-squared-error criterion above.

Worked Example: Allocating Data Under a Selection Bound

Take $n=200$, $M=20$, $\delta=0.05$, and assume the training-risk term is bounded by $A/m$ with $A=20$. Since $2M/\delta=800$, the bound, apart from $R_\infty$, is $$ G(m)=\frac{20}{m}+\frac{\sqrt{2\log(800)}}{\sqrt{200-m}} \approx \frac{20}{m}+\frac{3.656}{\sqrt{200-m}}. $$ The validation term here includes the factor $2$ in the theorem.

The derivative is $G'(m)=-20/m^2+(3.656/2)(200-m)^{-3/2}$, and the second derivative is positive: $$ G''(m)=\frac{40}{m^3}+\frac{3(3.656)}{4(200-m)^{5/2}}>0. $$ Thus there is a unique continuous minimizer. Evaluating around the derivative's zero gives an optimum near $m=103$. The integer comparisons are $G(100)\approx0.5656$, $G(103)\approx0.5654$, and $G(104)\approx0.5655$ (slightly larger than at 103). The bound therefore favors a training fraction near $103/200=0.515$, rather than a fixed conventional fraction. This is a planning calculation under the assumed $20/m$ training-risk behavior, not a guarantee that the same split is best for another learner or loss.

Training Risk and Validation Noise Can Interact

The two terms in a split criterion need not be independent or have fixed coefficients. Even the variance of a validation loss estimate can depend on how well training went. A simple normal example makes that dependence explicit.

Worked Example: Validation Variance After Estimating a Mean

Let training observations and future observations be iid $N(\mu,\sigma^2)$, and let the fitted predictor be the training sample mean $\hat\mu$ from $m$ observations. Conditional on training, write $d=\hat\mu-\mu$. For a validation observation $Y$, the squared prediction loss is $(Y-\hat\mu)^2$. Since $Y-\hat\mu$ is normal with mean $-d$ and variance $\sigma^2$, its squared value has conditional mean $\sigma^2+d^2$ and variance $2\sigma^4+4\sigma^2d^2$.

Consequently, the validation average over $v$ iid target-population observations has conditional variance $(2\sigma^4+4\sigma^2d^2)/v$. Since $\hat\mu-\mu\sim N(0,\sigma^2/m)$, $E[d^2]=\sigma^2/m$, and averaging over training gives $$ E\!\left[\operatorname{Var}(\widehat R_{\mathrm{val}}\mid D_{\mathrm{tr}})\right] =\frac{2\sigma^4+4\sigma^2(\sigma^2/m)}{v} =\frac{2\sigma^4(1+2/m)}{v}. $$ Thus the validation-estimation variance decreases with $v$, but its average also depends on $m$ through the fitted predictor. Treating the training and validation contributions as fixed, independent constants can miss such interactions.

Practical Interpretation and Common Pitfalls

A useful split decision begins by stating the quantity being optimized. If it is expected squared error of a validation estimate, a $1/v$ term may be appropriate. If it is a high-probability uniform deviation over $M$ bounded candidates, a term proportional to $\sqrt{\log(M/\delta)/v}$ is natural. These are not interchangeable. The training term also requires a credible description of how risk changes with $m$; the form $A/m$ is an assumption or approximation, not a general law of learning.

Pitfall: using independence without representativeness. Independent validation losses do not automatically estimate target-population risk. If the target loss mean is $0.5$ but validation observations come from a different population with loss always $0$, their average estimates $0$, not $0.5$. The correction is to require validation observations sampled from the population defining the risk, as well as independence from training.
Pitfall: optimizing validation precision alone. Taking nearly all observations for validation may make its average precise, but leaves too little data to fit strong candidates. The training-risk term must also be considered.
Pitfall: treating the one-candidate variance formula as a selection guarantee. Choosing the smallest of several noisy validation estimates introduces a multiple-comparison effect. A finite-class bound includes $\log M$; a fixed-candidate calculation does not.
Pitfall: interpreting a calculated fraction as universal. The optimizer changes with the training-risk curve, loss variability, candidate count, confidence level, and constraints on feasible sample sizes. A split fraction is a consequence of a criterion and assumptions, not a constant of statistical learning.

Key takeaway: Choose a split by balancing how training performance changes with $m$ against how validation uncertainty changes with $v=n-m$. Use a criterion matched to the goal, verify that validation data represent the target population, and treat the resulting fraction as problem-specific.

Check Your Understanding

Use the definitions and calculations above to answer the following.

Under what two conditions does a validation average estimate a candidate's conditional target risk?
In the criterion $A/m+B/(n-m)$, how does increasing $B$ affect the optimal validation allocation?
Why does a uniform validation bound for $M$ candidates contain a $\log M$ term, whereas the fixed-candidate variance calculation does not?
If validation uncertainty is represented by a standard-deviation-scale term proportional to $1/\sqrt v$, why would optimizing it differ from optimizing a mean-squared-error term proportional to $1/v$?
Give one reason that the split optimizing a theoretical bound might not optimize the actual prediction risk in a particular application.