Stratified Folds for Imbalanced Outcomes

Why Stratify Folds?

In binary genomic prediction, the outcome of interest may be rare: for example, a small fraction of individuals may have a particular molecular subtype or clinical event. In ordinary \(K\)-fold cross-validation, observations are divided into \(K\) validation folds and each fold is used once to evaluate a model trained on the other folds. If few cases are present, a random division can leave some folds with very few cases or none at all. Metrics that rely on both outcome classes may then be unstable or undefined.

Stratified folds address this allocation problem by distributing cases and controls separately across folds. The purpose is to make the validation sets more comparable in their observed outcome composition. Stratification does not create additional cases, guarantee representative data in every other respect, or change the population to which the prediction target refers.

Let \(N\) be the total number of observations, \(N_1\) the number of cases, and \(N_0=N-N_1\) the number of controls. The observed case proportion is \(\hat{p}=N_1/N\). If \(N_1\) and \(N_0\) are both at least \(K\), each fold can contain at least one case and one control. If a class has fewer than \(K\) observations, that guarantee is impossible. Fold sizes may differ by one when \(N\) is not divisible by \(K\); stratification balances class counts as closely as the integer totals allow.

Constructing the Folds

A standard construction first shuffles the cases and divides them into \(K\) groups with sizes differing by at most one. It repeats this process for controls. The case group and control group with the same fold index are combined. Randomizing within each class avoids making the folds depend on the order of the data, while fixing a random seed makes the allocation reproducible.

The training set for fold \(k\) is the complement of validation fold \(k\): it contains all observations assigned to the other \(K-1\) folds. Each observation is therefore used for validation once and for training in \(K-1\) fits. The stratification rule concerns fold membership; it does not alter the observed outcomes or the model’s prediction target.

Why Random Folds Can Be Unstable

A random fold of fixed size \(m\) drawn without replacement from a data set with \(N_1\) cases and \(N_0\) controls has a hypergeometric number of cases. In particular, the probability that it contains no cases is the probability that all \(m\) selected observations come from the controls:

This calculation makes the problem concrete: when the class is rare, a seemingly substantial fold can still have a meaningful chance of containing no cases. Stratification prevents an empty-case fold when there are at least \(K\) cases, but random allocation does not.

Justification and Interpretation

The key benefit is a reduction in variation due solely to accidental allocation of rare outcomes. Conditional on the observed class totals, stratification controls the number of cases and controls in each validation set. For example, if each of five folds has one case, a fold-level sensitivity is based on one case in every fold rather than being undefined in some folds and based on several cases in others.

This is a design choice for estimating performance on the same target population represented by the data, assuming observations are exchangeable draws from that target and the folds are otherwise formed without using information that would compromise evaluation. The total number of cases and controls is unchanged; no observation is reweighted or duplicated. Under this setup, validation observations still represent the observed data source, and the overall prediction target is not redefined by stratification. The folds are conditionally balanced given the observed outcomes, rather than independent random samples from the population.

This distinction matters for metrics. Sensitivity, specificity, and area under the receiver operating characteristic curve are class-conditional summaries; their estimates can be erratic when a fold contains very few members of a class, and the area under the curve is undefined if a fold contains only one class. Accuracy and positive predictive value depend on class prevalence as well as model behavior. Stratified folds preserve the sample's case proportion more consistently, but they do not make that proportion equal to the prevalence in some different intended population.

If the target population has a different prevalence from the study sample, preserving the sample proportion does not solve that mismatch. In particular, a positive predictive value computed from a validation sample reflects that sample's case prevalence. It should not automatically be interpreted as the positive predictive value in a population with a different prevalence. The target population and performance measure must be stated independently of the fold construction.

Assumptions, Diagnostics, and Limits

Stratified folds are most useful when the observed outcome is binary, cases are scarce enough that random allocation could produce very different fold compositions, and the chosen number of folds is compatible with the class counts. The observations must also be assigned at the appropriate unit for the prediction target. Stratifying individual records does not by itself ensure that a validation design respects other dependence or sampling structures; the split unit must match the scientific question.

Before fitting models, inspect the number of cases, controls, and total observations in every fold. Confirm that each observation appears in exactly one validation fold and that the folds cover the full data set. Check whether every fold has the classes required by the chosen metric. If \(N_1<K\) or \(N_0<K\), reduce \(K\) when feasible, or use an evaluation plan that acknowledges that not every fold can contain both classes. Do not report an undefined fold-level metric as though it were zero.

Stratification has limits. With very few cases, fold estimates remain noisy even when cases are evenly allocated. The training sets also overlap, so fold-specific results are not independent replications. Stratification controls one source of split-to-split variation; it does not guarantee that folds are similar in predictors, clinical characteristics, or other factors. Nor does it remedy an inappropriate prediction target, biased sampling, or a mismatch between the observed and intended deployment populations.

Compared with ordinary random \(K\)-fold assignment, stratified assignment gives more controlled class counts and often makes rare-class metrics more interpretable across folds. Ordinary random assignment is simpler and may be adequate when both classes are abundant relative to \(K\). In either approach, the overall cross-validation estimate should be formed and interpreted consistently with the metric. For example, pooled out-of-fold predictions yield one overall sensitivity by dividing the total true positives by the total number of cases; this is not necessarily equal to an unweighted average of fold-specific sensitivities when fold case counts differ.

Common Mistake

• Wrong reasoning: “Every stratified fold has the same case proportion, so the performance estimate must represent any population with that outcome.”
• Correction: Stratification balances the observed cases and controls across validation folds. It neither changes the study sample's overall prevalence nor guarantees that this prevalence matches a different target population.
• Another diagnostic error: Treating an empty-case fold's sensitivity as zero confuses “no cases available to evaluate sensitivity” with “the model failed on every case.” The metric is undefined for that fold.