Stratified Splits for Unequal Class Frequencies

Why Stratify a Classification Split?

In a classification problem, the outcome has a finite set of classes, such as claim/no claim or renewal/nonrenewal. If one class is much less common than another, a simple random holdout can contain a noticeably different class mix from the original data. It may even contain no records from a rare class. A model trained or assessed on such a partition may not serve the intended purpose: the training data may provide little information about a class, or the holdout may not permit a meaningful assessment of predictions for it.

A stratified split addresses this by allocating records to the holdout separately within each class. The goal is generally to keep each partition’s class proportions close to the full data’s proportions while still meeting a specified total holdout size. Stratification controls the class counts; it does not guarantee that a small class has enough observations for stable model fitting or evaluation.

Definition: A stratified split divides a classification data set into training and holdout partitions by sampling separately within each outcome class. For a data set with $K$ classes, let $n_k$ be the number of records in class $k$, $N=\sum_{k=1}^K n_k$ the total number of records, and $H$ the required holdout size. The proportional quota for class $k$ is $q_k=Hn_k/N$. Since records are indivisible, the quotas must be converted to integer allocations whose sum is exactly $H$.

Turning Proportional Quotas into Record Counts

A common allocation method is the largest-remainder method. First assign each class the floor of its quota. Then distribute the unassigned holdout places to the classes with the largest fractional remainders. The result has the requested total size and is as close as this rule allows to proportional allocation. A tie-breaking rule should be stated in advance, because equal remainders can occur. In the examples below, ties go first to the class with more records; if still tied, use a fixed class-label order.

Calculate each quota.
For each class, multiply its share of all records by the holdout size: $q_k=Hn_k/N$.

Take the floor.
Set the initial holdout allocation to $\lfloor q_k\rfloor$. Add these integer allocations to find how many places remain.

Assign remaining places.
Give one additional place at a time to classes with the largest fractional remainders, applying the predetermined tie rule.

Verify both partitions.
Subtract each holdout allocation from its class count to get the training count. Confirm totals, class representation, and the practical adequacy of the counts.

Worked Example: Allocating a 25% Holdout with a Tie

A hypothetical classification data set contains 120 records: 102 in class A and 18 in class B. A holdout of 25%, or 30 records, is required. Use the largest-remainder method, breaking a tied remainder in favor of the larger class. Find the class counts in the holdout and training partitions.

Method. Calculate proportional quotas using $H=30$ and $N=120$:

q_A=30\left(\frac{102}{120}\right)=25.5,\qquad q_B=30\left(\frac{18}{120}\right)=4.5.

The floors allocate 25 records from class A and 4 from class B, for $25+4=29$ places. One place remains. Both fractional remainders are 0.5, so the specified tie rule gives the remaining place to class A. The holdout therefore has 26 records from A and 4 from B.

Subtract these from the original class counts. Training has $102-26=76$ class A records and $18-4=14$ class B records. The checks are $26+4=30$ holdout records and $76+14=90$ training records, with $30+90=120$ overall. The class totals also check: $26+76=102$ and $4+14=18$.

Common mistake avoided: The calculation $120-26=94$ gives the total number of training records only if 26 were the entire holdout. Here 26 is the class A holdout count; the full holdout has 30 records, so training has 90 records.

Why Proportional Allocation Works—and What It Cannot Do

The proportional quota follows directly from the goal of matching class proportions. The original share of class $k$ is $n_k/N$. If the holdout had exactly that same share and contained $H$ records, its class $k$ count would have to be $H(n_k/N)$, which is $q_k$. Since this value may not be an integer, rounding is necessary; largest remainder keeps the total fixed while assigning the leftover records to the classes whose quotas were rounded down by the greatest fractions.

For the training partition, the class $k$ count is $n_k-h_k$, where $h_k$ is its integer holdout allocation. If the quota could be used without rounding, this count would be

n_k-q_k=n_k-H\frac{n_k}{N} =n_k\left(1-\frac{H}{N}\right).

Thus the idealized training count has the original class share as well. Integer allocation means the actual proportions may differ slightly, especially for small classes or small partitions. Stratification makes the split’s class mix deliberate; it does not make the mix mathematically identical in every finite sample.

Worked Example: Checking Whether a Rare Class Is Represented

A hypothetical data set has 200 records: 196 in class A and 4 in class B. The required holdout size is 40. Apply largest-remainder allocation and assess class representation in both partitions.

The quotas are $q_A=40(196/200)=39.2$ and $q_B=40(4/200)=0.8$. Taking floors gives 39 class A and 0 class B, leaving one place. Class B has the larger remainder, 0.8 rather than 0.2, so the holdout allocation is 39 from A and 1 from B.

The training counts are $196-39=157$ from A and $4-1=3$ from B. Totals check: the holdout has $39+1=40$, training has $157+3=160$, and $40+160=200$. Both classes are present in each partition, but the holdout contains just one class B record and training has only three. This is representation, not evidence that class-specific model performance can be estimated reliably.

Method and mistake avoided: The largest-remainder rule gives a reproducible integer allocation while honoring the fixed holdout size. A tempting alternative is to round each quota independently and stop; that can produce a total different from 40. Always check the sum and the class counts after rounding.

A useful usability check is more demanding than asking whether a class count is nonzero. Ask whether the training partition contains enough examples of each class for the planned fitting process, and whether the holdout contains enough examples to support the intended assessment. There is no universal minimum: it depends on the modeling method, the intended conclusions, and how rare outcomes are. If counts are too small, possible responses include collecting more data or reconsidering the evaluation design. Do not silently change the holdout’s class mix merely to make results appear more stable.

Worked Example: Building and Verifying a Stratified Split in R

A hypothetical data set has 150 independent records, with 132 in outcome class 0 and 18 in class 1. The analyst requires a 30-record holdout. Construct the allocation, create a stratified sample in R, and check the resulting counts.

The quotas are $30(132/150)=26.4$ for class 0 and $30(18/150)=3.6$ for class 1. Floors give 26 and 3, leaving one place. Class 1 has the larger fractional remainder, so the holdout allocation is 26 class 0 and 4 class 1. Training therefore has $132-26=106$ class 0 and $18-4=14$ class 1. The holdout total is $26+4=30$, the training total is $106+14=120$, and $30+120=150$.

Assume the outcome is stored in a column named outcome and has no missing values. The following R code calculates the largest-remainder allocation, samples within each class, and checks the partitions. It breaks tied remainders by larger class count and then by class label.

y <- droplevels(factor(dat$outcome))
strata <- split(seq_len(nrow(dat)), y, drop = TRUE)
counts <- lengths(strata)
H <- 30
quota <- H * counts / sum(counts)
allocation <- floor(quota)
left <- H - sum(allocation)
ord <- order(-(quota - allocation), -counts, names(counts))
if (left > 0) allocation[ord[seq_len(left)]] <- allocation[ord[seq_len(left)]] + 1L
test_idx <- unlist(Map(function(idx, m) idx[sample.int(length(idx), m)], strata, as.list(allocation)), use.names = FALSE)
train_idx <- setdiff(seq_len(nrow(dat)), test_idx)
table(y[test_idx])
table(y[train_idx])

The two tables should show holdout counts of 26 and 4, and training counts of 106 and 14, respectively. Using sample.int to select positions within each class also avoids an R edge case: sample(x, size) can treat a length-one numeric x as a sequence from 1 to x, rather than as a single record identifier. This code assumes an ordinary row-level random split is appropriate; it should not be used when the data structure requires a different partitioning constraint.

Common Traps and Practical Checks

Rounding each quota independently. For 10 records split into classes of 7 and 3, a holdout of 5 gives quotas 3.5 and 1.5. Rounding both up gives 4 and 2, totaling 6 rather than 5. Use an integer allocation rule and verify that its sum equals the required size.
Confusing a class allocation with the whole partition size. In the first example, 26 is the class A holdout count, not the total holdout count. Add the allocations across classes before calculating the training total.
Assuming stratification guarantees a useful minority-class sample. A class represented by one holdout record is technically present, but conclusions based on that record will be highly limited.
Ignoring absent classes or empty factor levels. Determine which classes actually occur in the data before calculating shares. A class with no observed records cannot be represented by a split.
Leaving the tie rule implicit. Different reasonable tie rules can assign the final record to different classes. State the rule and use it consistently so that the allocation can be reproduced.

Key takeaway: A sound stratified split has the requested total size, allocates records within each class using a stated integer rule, and is checked separately in training and holdout. Confirm not only that each class appears, but also that the resulting counts are adequate for the intended modeling and evaluation.

Check Your Understanding

For each problem, calculate integer holdout counts, derive training counts, and check whether the stated class representation is adequate for the described purpose.

A hypothetical data set has 240 records: 216 in class A and 24 in class B. A 15% holdout is required. Use largest remainder and report holdout and training counts by class. (Answer: Holdout: 32 A and 4 B; training: 184 A and 20 B.)
A hypothetical portfolio classification data set has 80 records: 73 in class 0 and 7 in class 1. Select a 16-record holdout. Use largest remainder and verify both partition totals. (Answer: Holdout: 15 class 0 and 1 class 1; training: 58 class 0 and 6 class 1; totals 16 and 64.)
A data set has 50 records: 47 in class A and 3 in class B. A 10-record holdout is planned. Find the allocation and explain one limitation of the resulting holdout for assessing class B. (Answer: Holdout: 9 A and 1 B; training: 38 A and 2 B. The holdout has only one class B record, so class-specific assessment is very limited.)
A hypothetical data set contains 600 records: 570 in class A and 30 in class B. A holdout of 90 is required. The two quotas have equal fractional remainders; apply the stated tie rule that favors the larger class. (Answer: Quotas 85.5 and 4.5; holdout: 86 A and 4 B; training: 484 A and 26 B.)
A small data set has 30 records: 28 in class A and 2 in class B. A 6-record holdout is required. Use largest remainder, then state whether the allocation represents every class in the holdout. (Answer: Quotas 5.6 and 0.4; holdout: 6 A and 0 B; training: 22 A and 2 B. Class B is absent from the holdout, so proportional rounding does not provide representation of every class there.)

What You'll Learn