Why the Composition of a Split Matters
A random partition does not guarantee that each subset has the same composition as the full data set. In classification, a validation set may contain unusually few or unusually many members of a rare class. Its measured performance can then be unstable or misleading, especially for class-specific measures such as sensitivity and precision.
The right split depends on the question. If future records are expected to resemble a representative sample from the same population, a random split can be appropriate. If it is important to evaluate performance across each class, a stratified split can preserve class proportions. If future predictions concern a different time period, group, or population, the split should reflect that target setting rather than merely reproduce the observed data’s proportions.
How Much Can a Random Split Vary?
Suppose a finite data set contains \(N\) records, of which \(M\) belong to a class of interest. A simple random subset of size \(n\), selected without replacement, has a class count \(K\) with a hypergeometric distribution. Its expected count is \(n(M/N)\). The count varies because the subset may include a different mix of classes than the full data set.
The factor \((N-n)/(N-1)\) is the finite-population correction. When a large fraction of the data is sampled, the count varies less than it would under independent sampling, because taking one record leaves fewer records of that type available for the rest of the subset.
The variance formula follows by representing the count as a sum of indicators. Let \(I_j\) equal 1 if the \(j\)th sampled record belongs to the class and 0 otherwise, so \(K=\sum_{j=1}^{n}I_j\). Write \(p=M/N\). Each indicator has variance \(p(1-p)\). For two distinct sampled positions, the indicators have covariance \(-p(1-p)/(N-1)\): after a class member is selected, the remaining fraction of class members is slightly lower. Therefore,
This describes the variation in a random subset, not a guarantee that any one subset will be close to its expected count. In particular, the expected count need not be an integer, and a rare class can have a small expected count even when the overall sample size looks substantial.
Worked Example: A Rare Class in a Random Validation Set
A hypothetical claims data set has 200 records, including 20 records with a specified claim outcome. A simple random validation set of 40 records is drawn without replacement. Find the expected number and standard deviation of records with that outcome. Then interpret a validation set containing only one such record.
Method. Use the hypergeometric mean and variance because the records are selected without replacement from a finite data set. The class proportion is \(p=20/200=0.10\), with \(N=200\) and \(n=40\).
The expected class count is
The variance is
Thus the standard deviation is
A validation set with one such record is possible even though the expected count is four. With only one positive-class record, a single correct or incorrect prediction changes the observed sensitivity by 1. The count alone does not establish that the split was faulty; it signals that class-specific performance estimates may be imprecise.
Check and common mistake. The finite-population factor is \(160/199\), not 1, because selection is without replacement. The variance is about 2.8945 and its square root is about 1.7013. A common mistake is to treat the expected count of four as a guaranteed count in every random split.
What Stratification Controls
In a class-stratified split, records are partitioned by the observed outcome class, and random assignment is performed within each class. If the full data set has \(M\) positive and \(N-M\) negative records, a validation fraction \(r\) suggests target counts \(rM\) and \(r(N-M)\). When these are integers, the validation set can preserve the full data’s class proportion exactly. When they are not integers, a rounding rule is needed, so the proportions may differ slightly.
Stratification reduces random variation in class counts across subsets. It does not guarantee that a subset is representative with respect to other variables, such as claim size, calendar period, or customer group. Nor does it correct a biased or unrepresentative source data set. It is a design choice for partitioning the records available, not a remedy for how those records were collected.
Worked Example: Allocating a Stratified Split
A hypothetical classification data set contains 240 records: 36 positive outcomes and 204 negative outcomes. A 75% training and 25% validation split is required. Allocate each class to the two subsets using exact stratification, then calculate the expected positive count and standard deviation for a random validation set of the same total size.
Method. For the stratified allocation, multiply each class count by the subset fraction. The validation set has \(0.25(240)=60\) records.
Its positive count is \(0.25(36)=9\), and its negative count is \(0.25(204)=51\). Therefore, the training set receives \(36-9=27\) positive and \(204-51=153\) negative records, for a total of \(180\). The validation proportion of positives is \(9/60=0.15\), matching the original proportion \(36/240=0.15\).
For comparison, in a simple random validation subset of 60 records, the expected positive count is
Its variance and standard deviation are
Stratification fixes the validation positive count at 9, while a random split has expected count 9 but can vary around it. The training set’s corresponding class counts are also fixed by this allocation.
Check and common mistake. The two validation class counts sum to \(9+51=60\), and the training counts sum to \(27+153=180\); overall, they recover all 36 positives and 204 negatives. A common mistake is to apply 25% to the total only, then assume the positive count must be exactly the expected value under an unstratified random split.
Class Proportions and Performance Measures
The class proportion in a validation set affects the interpretation of some performance measures. Accuracy is the fraction of records classified correctly. Sensitivity is the fraction of actual positives classified as positive, and specificity is the fraction of actual negatives classified as negative. Precision is the fraction of predicted positives that are actual positives. Precision, and generally accuracy, can change when the prevalence of positives changes, even if sensitivity and specificity stay the same.
This matters when stratification deliberately gives the validation set a class proportion unlike the target population. For example, a validation set with equal numbers of positive and negative records may be convenient for checking performance by class, but its raw accuracy and precision do not automatically describe performance at a much lower deployment prevalence. Those measures must be interpreted using the target prevalence and the relevant class-specific rates.
Worked Example: Adjusting Interpretation to Target Prevalence
A hypothetical validation study uses a case-enriched sample with 100 positive and 900 negative records. At a chosen classification threshold, the estimated sensitivity is 0.80 and the specificity is 0.90. A model will instead be used in a population with a positive prevalence of 0.02. Calculate the accuracy and precision implied by the validation prevalence and by the target prevalence. Also find the expected target-population counts in 10,000 records.
Method. For prevalence \(\pi\), the expected proportions of true positives, false negatives, false positives, and true negatives are \(\pi\) times sensitivity, \(\pi\) times one minus sensitivity, \(1-\pi\) times one minus specificity, and \(1-\pi\) times specificity, respectively.
The validation prevalence is \(100/1000=0.10\). Its implied accuracy is
Its implied precision is
At the target prevalence of 0.02, the implied accuracy is
The implied target precision is
For 10,000 target-population records, there are \(10{,}000(0.02)=200\) actual positives and 9,800 actual negatives. The expected true positives are \(200(0.80)=160\), and false negatives are \(200-160=40\). The expected true negatives are \(9{,}800(0.90)=8{,}820\), and false positives are \(9{,}800-8{,}820=980\). Thus, expected accuracy is \((160+8{,}820)/10{,}000=0.898\), and expected precision is \(160/(160+980)=160/1{,}140\approx0.1404\).
Check and common mistake. The target confusion-matrix counts total \(160+40+8{,}820+980=10{,}000\). Precision is about 0.1404, not the validation value 0.4706: the target population has far fewer positives, so false positives form a larger share of predicted positives. A common mistake is to report a case-enriched validation set’s precision as if it applied unchanged at deployment prevalence.
Choose the Split for the Intended Use
A split should support the performance claim being made. A random partition of a representative, independent sample is often suitable when future observations are expected to come from the same population and process. Class stratification can be useful when class counts are small or when each class needs to be represented in both training and validation subsets. It makes class-specific comparisons less sensitive to accidental class-count variation, but it does not by itself make the overall evaluation representative.
If the intended use involves a later time period, a new organization, or distinct individuals, a random record-level split may not reflect that use. The validation design should mimic the relevant target setting: for example, evaluating later-period predictions may call for holding out later observations, while evaluating performance on new groups may call for holding out groups. The purpose is not to force every subset to look alike, but to make the evaluation answer the intended question.
When class proportions are deliberately changed in a validation set, state that choice and distinguish class-specific rates from prevalence-sensitive measures. If deployment prevalence is known or can be credibly specified, use it when interpreting accuracy and precision. If it is not known, do not imply that a single precision or accuracy value from a class-enriched split necessarily transfers to every population.
Check Your Understanding
For each problem, identify what the split design does and does not establish.
- A finite data set has 500 records, of which 25 are positive. A random subset of 100 is selected without replacement. Find the expected positive count, variance, and standard deviation. (Answer: \(E[K]=5\); variance \(=100(0.05)(0.95)(400/499)=3.8076\); standard deviation \(\approx1.9513\).)
- A data set has 2,000 records, including 240 positive outcomes. Allocate a 25% validation set with exact class stratification. Give the positive and negative counts in validation and training. (Answer: validation: 60 positive and 440 negative; training: 180 positive and 1,320 negative.)
- At a target prevalence of 0.04, a classifier has sensitivity 0.85 and specificity 0.95. Find implied accuracy and precision. (Answer: accuracy \(=0.04(0.85)+0.96(0.95)=0.9460\); precision \(=0.034/(0.034+0.048)\approx0.4146\).)
- A data set has 1,000 records with 30 positives. A random validation subset contains 200 records sampled without replacement. Find the expected positive count and its standard deviation. (Answer: \(E[K]=6\); variance \(=200(0.03)(0.97)(800/999)\approx4.6607\); standard deviation \(\approx2.1589\).)
- A model will be used for future claims from a different calendar period. The current records have been randomly divided into training and validation sets, with class proportions preserved in both. Does that alone establish future-period performance? Name one design feature that would better match the target. (Answer: No. Preserving class proportions does not establish performance in a later period; holding out observations from the later period would better match that target.)