Why Stratify Classification Folds?
In \(K\)-fold cross-validation, the data are divided into \(K\) parts. Each part is used once as an assessment fold while the other \(K-1\) parts form the corresponding training set. For classification, a purely random allocation can put too many or too few observations from a rare class into a particular fold. If one assessment fold contains no positive outcomes, for example, some class-specific measures cannot be calculated for that fold.
Stratification addresses this issue by allocating observations separately within each outcome class. Its goal is not to make every fold identical, which is often impossible with finite counts. It is to make the class counts across folds as even as the data permit, while keeping each observation in exactly one assessment fold. This is aligned with the SOA Exam PA (Predictive Analytics) syllabus.
How Class Counts Determine the Folds
Let \(n_c\) be the number of observations in class \(c\), and let \(K\) be the chosen number of folds. Divide \(n_c\) by \(K\):
A balanced allocation gives \(q_c+1\) observations of class \(c\) to \(r_c\) folds and \(q_c\) observations to each of the other \(K-r_c\) folds. Thus, for any one class, the largest and smallest fold counts differ by at most one. If \(n_c<K\), then \(q_c=0\): exactly \(n_c\) folds can receive one observation of that class, and the remaining folds must receive none.
This count rule follows directly from division with remainder. If every fold received at least \(q_c+1\) observations, the total would be at least \(K(q_c+1)>n_c\). If any fold received fewer than \(q_c\), keeping the other folds at or below \(q_c+1\) would require another fold to exceed \(q_c+1\) in order to reach \(n_c\). The most even possible counts are therefore \(q_c\) and \(q_c+1\), with exactly \(r_c\) folds receiving the extra observation.
Apply this allocation separately to every class. When choosing which folds receive the remainder observations, an implementation can also consider current fold sizes so that the total number of observations per fold stays reasonably balanced. The class counts are the essential requirement; equal total fold sizes are desirable but may not always be achievable alongside an especially sparse class.
Record the number of observations in each class, including the rarest class. Check for missing outcome labels before assigning folds.
For each class, calculate its quotient and remainder on division by \(K\). Assign the remainder observations one per fold before giving any fold a second extra observation.
For each assessment fold, use all remaining observations as the training data for that fold.
Confirm that each observation was assigned once, then tabulate class counts in every assessment fold and its corresponding analysis set.
For binary classification, if a class has at least \(K\) observations, the balanced allocation can put at least one of that class in every assessment fold. If it has fewer than \(K\), some assessment folds necessarily omit it. A separate condition applies to the analysis sets: to ensure every analysis set contains at least one observation of each class, each class must have at least two observations. Even that condition does not guarantee a stable model; one or two events may be too little information for dependable fitting.
Worked Examples
Worked Example: Five Folds With a Moderately Uncommon Class
A hypothetical claim review data set has 23 records: 6 claims are classified as requiring further review and 17 are not. Create a five-fold assignment, check its class balance and total sizes, and determine whether every fold can support an assessment-set AUC calculation.
Method. Allocate each class separately using quotient and remainder. The class counts are \(n_{\text{review}}=6\) and \(n_{\text{not}}=17\), with \(K=5\).
For the review class, \(6=5(1)+1\). Therefore, one fold receives 2 review claims and the other four receive 1 each. For the not-review class, \(17=5(3)+2\). Therefore, two folds receive 4 not-review claims and three receive 3 each.
One valid arrangement is:
| Assessment fold | Review claims | Not-review claims | Total records | |
|---|---|---|---|---|
| 1 | 2 | 4 | 6 | |
| 2 | 1 | 4 | 5 | |
| 3 | 1 | 3 | 4 | |
| 4 | 1 | 3 | 4 | |
| 5 | 1 | 3 | 4 | |
| Total | 6 | 17 | 23 |
The class counts are as balanced as possible: review counts are 2 or 1, and not-review counts are 4 or 3. The total fold sizes are 6, 5, 4, 4, and 4. They are not all equal because class-wise balance and total-size balance do not always coincide. Every assessment fold contains both classes, so an AUC can be calculated for each fold, assuming the model produces scores for all records.
Check. The row totals sum to \(6+5+4+4+4=23\). The review column sums to \(2+1+1+1+1=6\), and the other column sums to \(4+4+3+3+3=17\). No observation is missing from the allocation. This method avoids the mistake of requiring identical fold sizes or identical class proportions when integer counts make that impossible.
Worked Example: Three Events Across Five Folds
A hypothetical fraud-screening sample contains 21 transactions, of which 3 are confirmed cases and 18 are not. A team proposes five-fold cross-validation. Determine the class counts that stratification can achieve and identify a limitation for fold-level AUC.
Method. Apply the quotient-and-remainder allocation to both classes. For confirmed cases, \(3=5(0)+3\), so three folds get 1 case and two folds get 0. For non-cases, \(18=5(3)+3\), so three folds get 4 non-cases and two folds get 3.
One assignment is:
| Assessment fold | Confirmed cases | Non-cases | Total records |
|---|---|---|---|
| 1 | 1 | 4 | 5 |
| 2 | 1 | 3 | 4 |
| 3 | 1 | 3 | 4 |
| 4 | 0 | 4 | 4 |
| 5 | 0 | 4 | 4 |
| Total | 3 | 18 | 21 |
The fold sizes sum to \(5+4+4+4+4=21\), and the class totals are 3 and 18. Three folds contain both classes. In folds 4 and 5, AUC is undefined because the assessment outcomes contain no confirmed cases; AUC compares the model's ranking of positive and negative observations, and there is no positive observation to compare.
Reducing to \(K=3\) would give one confirmed case and six non-cases in every assessment fold. That would allow fold-level AUC calculations, but each fold would still contain only one positive assessment observation, making the fold-level results very sensitive to that observation. A smaller \(K\) changes the distribution across folds; it does not create additional events.
Check. The analysis set for each fold contains the other 18 records, including 2 confirmed cases and 16 non-cases. Thus every analysis set has both classes, even though two assessment folds do not. This method avoids treating “stratified” as meaning “every fold must contain every class,” which is impossible when a class count is smaller than \(K\).
Worked Example: One Positive Outcome Cannot Support Four-Fold Fitting
A hypothetical pilot data set has 12 applications: 1 is accepted and 11 are declined. An analyst creates four stratified folds and claims that stratification makes the validation design suitable for fitting and assessing a binary classifier. Audit that claim.
Method. Allocate the accepted applications first. Since \(1=4(0)+1\), one assessment fold contains the single accepted application and the other three contain none. For the 11 declined applications, \(11=4(2)+3\), so three folds contain 3 declines and one contains 2.
A possible assignment is:
| Assessment fold | Accepted | Declined | Total | Accepted in analysis set |
|---|---|---|---|---|
| 1 | 1 | 3 | 4 | 0 |
| 2 | 0 | 3 | 3 | 1 |
| 3 | 0 | 3 | 3 | 1 |
| 4 | 0 | 2 | 2 | 1 |
| Total | 1 | 11 | 12 |
In fold 1, the analysis set has 0 accepted applications and 8 declined applications. A usual binary classifier cannot learn a distinction between the two outcome classes from that training set because only one class is present. In folds 2, 3, and 4, the assessment sets have no accepted applications, so their fold-level AUC is undefined. The single accepted record cannot be both held out and available for fitting in the same fold.
Check. The accepted counts sum to 1 and the declined counts sum to 11. The table shows that stratification cannot make every analysis and assessment set contain both classes when there is only one observation in a class. The common mistake is to assume that stratification repairs inadequate event counts; it can only distribute the observed counts.
Auditing Folds and Interpreting Their Results
A fold assignment should be checked, not inferred from the method used to create it. For each assessment fold, count the observations by class and calculate the fold's class proportions. Then check the corresponding analysis set. This catches assignment errors as well as unavoidable limitations caused by rare outcomes.
- Coverage: every record appears in exactly one assessment fold.
- Class balance: within each class, fold counts differ by no more than one if the allocation is intended to be maximally balanced.
- Training feasibility: each analysis set has the classes needed by the chosen fitting procedure.
- Metric feasibility: each assessment set has the classes needed for any fold-level metric being reported. For example, AUC requires both positive and negative outcomes; sensitivity requires at least one positive outcome.
A metric can be undefined in a fold even when the overall out-of-fold predictions contain both classes. Do not silently replace an undefined fold metric with zero: zero is a numerical result and has a different meaning. State how the limitation is handled, or choose an evaluation summary that is appropriate to the available observations. The choice of metric and how it is summarized should be made with the evaluation question in mind.
Stratification preserves only the representation of the outcome classes. It does not ensure that other variables are balanced, make a dependent data set independent, or correct a biased sample. If observations have a grouping or time structure, a class-balanced random assignment may break the structure relevant to the intended prediction task. The partition should match that task; stratification is not a substitute for a suitable validation design.
Likewise, stratification does not permit information from an assessment fold to be used when fitting the model or preprocessing steps. For each fold, the model and any learned preprocessing must be fitted using its analysis set, then applied to its assessment set. The labels may guide the fold allocation, but they must not be used to train the model for the fold in which those observations are being assessed.
Check Your Understanding
For each problem, determine the class allocation or the validation limitation requested.
- A data set has 31 records, including 7 positive outcomes, and uses 4 folds. What positive counts can the four assessment folds receive? If the 24 negative outcomes are also allocated as evenly as possible, what are the resulting fold sizes? (Answer: Positive counts are 2, 2, 2, and 1; negative counts are 6 in each fold; fold sizes are 8, 8, 8, and 7.)
- A sample has 2 positive outcomes and uses 5 folds. How many folds must have no positive outcome, and in how many folds is fold-level AUC therefore undefined? (Answer: Three folds must have no positive outcome; AUC is undefined in at least those three folds.)
- A sample has 26 records: 5 positive and 21 negative outcomes. Under 4-fold stratification, give one possible set of positive counts, negative counts, and total fold sizes. (Answer: Positive counts 2, 1, 1, 1; negative counts 6, 5, 5, 5; total sizes 8, 6, 6, 6.)
- A sample has 1 positive and 15 negative outcomes and uses 3 folds. What happens to the analysis set for the fold holding out the positive, and what happens to fold-level AUC in the other two folds? (Answer: The analysis set for the positive-containing fold has no positive outcomes; AUC is undefined in each of the other two assessment folds because each has no positive outcome.)
- A sample has 12 positive and 48 negative outcomes and uses 6 folds. If both classes are allocated evenly, how many of each class are in each fold, and what is the positive proportion in each fold? (Answer: Each fold has 2 positive and 8 negative outcomes, for 10 records; the positive proportion is \(2/10=0.2\).)