Stratified Holdout Validation for Rare Classes

Why Stratify a Holdout?

In a classification problem, an ordinary random holdout can contain few or no observations from a rare class. That makes its estimated error for the class unstable or undefined. Stratifying the split fixes the number of validation observations from each class, ensuring that each selected class is represented. But fixing class counts changes the calculation: the validation set’s class proportions need not match the population proportions. To estimate population classification risk, the class-specific validation errors must be combined with suitable population weights.

This tutorial focuses on the effect of fixed validation counts on the target and variability of estimated classification loss. The central calculations first treat the fitted classifier as fixed, then extend them to a classifier trained on the complementary training set.

Target Risk and the Stratified Estimator

Let \(Y\) take values in \(\{1,\ldots,K\}\), with \(P(Y=k)=\pi_k>0\). For a fixed classifier \(g\), define its class-\(k\) error rate and its population classification risk as follows.

Suppose the validation set contains exactly \(m_k\geq 1\) observations from class \(k\), with total size \(M=\sum_{k=1}^K m_k\). Let \(\widehat R_k\) be the fraction misclassified among those \(m_k\) observations. The class proportions in this validation set are \(m_k/M\), but those are design choices rather than necessarily the population proportions \(\pi_k\). Thus the unweighted overall fraction of validation errors estimates the risk under the validation-set class mix. A population-risk estimate instead uses the population weights:

If the intended target is macro-averaged risk, use \(K^{-1}\sum_k\widehat R_k\). If the target is risk under a different deployment class distribution, use that distribution’s weights. Stratification itself does not determine which target is appropriate.

Unbiasedness and Variance with Fixed Class Counts

For a fixed classifier, suppose observations in the validation sample are independent draws from the class-conditional feature distribution within each stratum. Define \(Z_{ki}=1\{g(X_{ki})\ne k\}\) for validation observation \(i\) in class \(k\). Then \(Z_{ki}\) is Bernoulli with success probability \(R_k(g)\), and \(\widehat R_k=m_k^{-1}\sum_{i=1}^{m_k}Z_{ki}\).

Proof. Within class \(k\), the \(m_k\) error indicators are independent Bernoulli variables. Therefore \(E[\widehat R_k]=R_k(g)\) and \(\operatorname{Var}(\widehat R_k)=R_k(g)\{1-R_k(g)\}/m_k\). The validation observations from distinct strata are independent under the stated sampling scheme, so the class-specific estimates are independent. By linearity of expectation,

Independence also makes the variance of the weighted sum the sum of the weighted variances:

This proves both claims. \(\square\)

The result describes validation-sample variability for a fixed classifier. When the classifier is trained on the complementary training data, the same calculation applies conditionally on that training data, provided the validation observations are independent of it and have the stated class-conditional distributions.

Training Randomness and the Validation Target

In practice, \(g\) is usually fitted using the training portion of the split. Write \(D\) for that training data and \(g_D\) for the resulting classifier. Conditional on \(D\), define \(R_k(D)=P(g_D(X)\ne k\mid Y=k,D)\), where the probability is for a new class-\(k\) observation. If validation observations are independent of \(D\), the preceding result gives

Thus the estimate is conditionally unbiased for the population risk of the fitted classifier. Unconditionally, its variability includes both the validation-sample noise and variation in the fitted classifier’s risk across possible training sets:

Derivation. Write \(A=\widehat R_{\pi}\). Since \(E[A\mid D]=R(D)\), add and subtract this conditional mean: \(A-E[A]=(A-E[A\mid D])+(E[A\mid D]-E[A])\). On squaring and taking expectations, the cross term is zero, because \(E[A-E[A\mid D]\mid D]=0\). The first squared term has expectation \(E[\operatorname{Var}(A\mid D)]\); the second has expectation \(\operatorname{Var}(E[A\mid D])=\operatorname{Var}(R(D))\). This proves the identity. \(\square\)

The training sample size and its class composition matter too: stratifying a split may change how many examples of each class are available for fitting. The conditional target is the risk of the model trained on that particular training data, not automatically the risk of a model trained on a larger data set or with a different class balance.

Allocation and Rare-Class Trade-Offs

When the total validation size \(M\) is fixed, the variance formula makes the allocation trade-off explicit. Set \(a_k=\pi_k^2R_k(1-R_k)\). For positive \(a_k\), assigning more observations to class \(k\) reduces that class’s variance contribution \(a_k/m_k\). In a continuous allocation with no lower-bound requirement on any \(m_k\), the variance-minimizing allocation is proportional to \(\sqrt{a_k}=\pi_k\sqrt{R_k(1-R_k)}\).

To see this, assume first that every \(a_k>0\) and \(m_k>0\), with \(\sum_km_k=M\). The Cauchy–Schwarz inequality gives

Therefore the variance is at least \((\sum_k\sqrt{a_k})^2/M\). Equality in Cauchy–Schwarz requires \(\sqrt{a_k/m_k}\) to be proportional to \(\sqrt{m_k}\), or equivalently \(m_k\propto\sqrt{a_k}\). This establishes the stated optimum when all \(a_k\) are positive. In practice, counts must be integers, and a minimum count may be required for reporting or operational reasons, so the continuous allocation is a guide rather than necessarily an exactly attainable design.

If a positive-weight class has \(R_k\in\{0,1\}\), then \(a_k=0\). The variance lower bound remains valid, but allocating counts proportional to \(\sqrt{a_k}\) would give that class zero observations. If every class must receive a positive count, this unconstrained allocation is infeasible. With integer counts and a minimum of one per class, assign the required minimum to zero-variance classes and allocate the remaining counts among positive-\(a_k\) classes according to the variance criterion. If all \(a_k=0\), the classification errors are deterministic within every class under the fixed-classifier model, and the variance is zero for any positive allocation.

Common Mistakes and Practical Interpretation

• Using the validation proportions as risk weights automatically. The raw error fraction estimates risk under the validation mix. For population risk, use \(\pi_k\); for macro risk, use equal class weights.
• Assuming stratification removes uncertainty. It guarantees class representation, but the class-specific error rates still vary. Their variance depends on both the class risks and the fixed counts.
• Interpreting an empty class stratum as zero error. If an ordinary random holdout contains no examples from a class, its error rate for that class is unobserved, not zero. For a rare class with prevalence \(\pi\), the probability of no examples in an unstratified validation sample of size \(M\) is \((1-\pi)^M\). For example, at \(\pi=0.01\) and \(M=100\), this probability is \(0.99^{100}\), approximately \(0.366\).
• Ignoring the training split. A stratified holdout estimates the risk of the classifier produced by its training data. Changing class counts in training can change the fitted classifier as well as the validation estimator’s variability.

Stratification is especially useful when rare classes need reliable representation, but it does not make the holdout automatically representative for every performance target. State the target weights, report class-specific results when they matter, and distinguish uncertainty from the validation sample from variability due to fitting the model.