Why Imbalance Changes the Modeling Workflow
In an imbalanced classification problem, one outcome is much less common than the other. For example, a hypothetical insurer might want to flag a small fraction of policies that will generate a specified type of claim. A model that predicts “no flagged claim” for every policy can have high accuracy when flagged claims are rare, yet be useless for finding them. The modeling question is not simply how to make the classes equally common. It is how to learn a useful classifier and assess it using data that represent the population where predictions will be used.
Class weighting and resampling are two ways to make the less common class matter more during model fitting. They change the training procedure, not the real-world frequency of outcomes. Validation and test data should therefore retain their natural class prevalence. Applying resampling before creating folds can put duplicates, or information derived from the same observations, on both sides of a validation split and produce an overly optimistic assessment.
Weighting and Resampling
With class weighting, the observations remain in the training data, but errors on one class can contribute more to the fitting objective. For a binary classifier with predicted class probabilities \(\hat p_i\), a weighted log-loss is
where \(w_1\) and \(w_0\) are the weights for positive and negative observations. The fitting method may use a different objective, but the general idea is the same: errors on a class with a larger weight have a larger influence on fitting. A commonly used starting choice for \(K\) classes is \(w_k=n/(K n_k)\), where \(n_k\) is the number of training observations in class \(k\). This gives each class the same total weight, though it is not automatically the best choice for every prediction objective.
Random undersampling removes observations from the more common class in the training data. Random oversampling draws additional observations from the less common class, often by duplicating existing cases. Both can make a fitting data set more balanced. They also have limitations: undersampling discards some available majority-class information, while duplicated cases in oversampling do not create genuinely new information. A method that synthesizes new minority-class observations must likewise be fitted using training data only.
| Approach | What changes during fitting? | Key concern |
|---|---|---|
| Class weighting | The contribution of each observation to the fitting objective | Weights affect the fitted model and may affect probability calibration |
| Undersampling | The number of majority-class observations used for fitting | Discarded observations may contain useful variation |
| Oversampling | The number of minority-class observations used for fitting | Duplicated cases can encourage overfitting |
Worked Example: Balance a Training Fold Without Changing Its Test Fold
A hypothetical portfolio data set has 40 positive cases among 1,000 observations. A stratified five-fold split places 8 positive and 192 negative cases in each validation fold. For one fold, find the class counts in the training portion, the counts after random undersampling to equal class counts, and the prevalence in the untouched validation fold. Explain why undersampling the full data set before splitting would be unsafe.
Method. Split first, then resample only the training portion. Across the four training folds, the counts are \(4(8)=32\) positive and \(4(192)=768\) negative, for \(32+768=800\) observations. The training prevalence before resampling is \(32/800=0.04\), or 4%.
To undersample to equal class counts, retain all 32 positive training cases and randomly select 32 of the 768 negative training cases. The resampled training set then contains \(32+32=64\) observations, with prevalence \(32/64=0.50\). The validation fold remains at 8 positive and 192 negative cases, so its prevalence is \(8/200=0.04\), or 4%.
If undersampling happened before the fold split, the selected cases would be chosen using the entire data set, including observations that should have been held out. More generally, resampling before splitting can place copies of one observation in both training and validation data. Either problem can make validation performance look better than performance on new cases. Common mistake avoided: balance the training fold, not the complete data set; validate against the unaltered fold.
Resampling Must Be Inside Each Training Fold
Cross-validation repeatedly separates observations into a training fold and a validation fold. For an imbalanced classifier, each repetition must follow the same order: split the data, apply resampling to that repetition’s training fold, fit the model, and predict the untouched validation fold. Stratification can help preserve the approximate class proportions in the folds, especially when positive cases are few. It does not replace the rule that resampling occurs only after each split.
Use the outcome labels to form folds, stratifying when appropriate so each fold has usable representation of each class.
Estimate any data-dependent preprocessing from that training fold, then apply class weighting or resampling there. Do not use validation-fold observations to decide which cases to duplicate, remove, or synthesize.
Fit the classifier on the prepared training fold. Predict the validation observations without resampling them.
Combine fold results using a metric suited to the prediction goal. If choices such as weights, sampling ratios, or decision thresholds are tuned, make those choices using training-side validation evidence rather than the final test set.
The same boundary applies when selecting among multiple modeling choices: every validation result used for selection must come from a process that keeps that validation data out of fitting and resampling. After choices are fixed, fit the selected procedure using the available development data, applying any resampling only to that development data. Evaluate it once on a separate test set that retains the natural prevalence.
Why Natural Prevalence Matters for Evaluation
The number of false positives depends on how many negative cases are present; the number of false negatives depends on how many positive cases are present. Consequently, accuracy and precision can change substantially with class prevalence. A test set artificially balanced by oversampling or undersampling does not directly describe performance in a population with a different prevalence. Measures such as sensitivity, \(TP/(TP+FN)\), and specificity, \(TN/(TN+FP)\), describe class-conditional performance, but they do not by themselves state the fraction of flagged cases that are truly positive.
Precision, also called positive predictive value, is \(TP/(TP+FP)\). If positive cases are uncommon, even a classifier with reasonable sensitivity and specificity can generate many false positives relative to true positives. Thus a natural-prevalence holdout provides a direct check on precision and other quantities that depend on the class proportions. The selected metric should reflect the decision: missing a positive case, investigating a false alarm, or balancing both may have different consequences.
Worked Example: Compare Weighted and Resampled Fits at Natural Prevalence
Two hypothetical classifiers are evaluated on the same untouched test set of 500 policies. The test set contains 25 positive cases and 475 negative cases, matching the population prevalence of 5%. Model A, fitted with class weights, has sensitivity 0.80 and specificity 0.96. Model B, fitted after training-fold oversampling, has sensitivity 0.60 and specificity 0.99. Compute the confusion counts, precision, and accuracy for each. Which model has greater recall, and which has greater precision?
Method. Apply each sensitivity to the 25 positive test cases, and each specificity to the 475 negative test cases. Since the test counts are natural and fixed, these calculations evaluate both models on the same population mix.
Model A. The true positives are \(25(0.80)=20\), so false negatives are \(25-20=5\). The true negatives are \(475(0.96)=456\), so false positives are \(475-456=19\). Precision is \(20/(20+19)=20/39=0.5128\) to four decimals. Accuracy is \((20+456)/500=476/500=0.952\), or 95.2%.
Model B. The true positives are \(25(0.60)=15\), so false negatives are \(25-15=10\). The true negatives are \(475(0.99)=470.25\), and false positives are \(475(0.01)=4.75\). These are expected counts implied by the stated rates, not literal fractional people. Precision is \(15/(15+4.75)=15/19.75=0.7595\) to four decimals. Accuracy is \((15+470.25)/500=485.25/500=0.9705\), or 97.05%.
Model A has greater recall (sensitivity), 0.80 versus 0.60. Model B has greater precision, about 0.7595 versus 0.5128, and greater accuracy in this example. The preferred model depends on the costs and purpose of the decision; accuracy alone does not settle that choice. Common mistake avoided: do not calculate precision using the artificial prevalence of a resampled training set. Use the natural-prevalence test cases.
Resampling and Probability Calibration
A classifier’s predicted probability may be affected by how the training cases were weighted or sampled. A useful correction can be derived for a particular setting: sampling depends only on the class, and within each class the sampled observations have the same predictor distribution as the population. Let \(\pi\) be the population prevalence and \(\pi_s\) the prevalence in the sampled training data. Let \(q(x)\) be the probability of a positive outcome for predictor value \(x\) under the sampled distribution, and \(p(x)\) the corresponding population probability.
By Bayes’ rule, the posterior odds equal the class-conditional likelihood ratio multiplied by the prior odds. Under class-only sampling, the class-conditional distributions of \(x\) are unchanged, so that likelihood ratio is the same in the sample and population. Dividing the sampled odds by the sampled prior odds therefore leaves the likelihood ratio; multiplying by the population prior odds gives the population posterior odds:
This correction is not guaranteed to fix predictions from arbitrary class weighting or more complex synthetic sampling. It relies on the stated class-only sampling condition and on \(q(x)\) representing a calibrated probability under the sampled distribution. When accurate population probabilities matter, calibration should be assessed using validation data with natural prevalence, without fitting that calibration on the final test set.
Worked Example: Correct a Probability After Case-Control Sampling
A hypothetical training sample is created by random sampling within outcome class and has 50% positive cases. A model calibrated to that sample assigns \(q(x)=0.80\) to a particular policy. In the target population, positive prevalence is 2%. Assuming class-only sampling and unchanged predictor distributions within each class, calculate the corrected population probability. Then determine whether the corrected probability exceeds a decision threshold of 10%.
Method. Convert the sampled probability to odds, multiply by the ratio of population prior odds to sampled prior odds, then convert back to probability. The sampled odds are \(0.80/(1-0.80)=4\). The population prior odds are \(0.02/0.98=1/49\), while the sampled prior odds are \(0.50/0.50=1\). Therefore the corrected population odds are \(4(1/49)=4/49\).
Convert these odds to probability: \(p(x)=(4/49)/(1+4/49)=4/53=0.07547\ldots\), or 0.0755 to four decimals. Since \(0.0755<0.10\), the corrected probability does not exceed the decision threshold. Common mistake avoided: treating the sample probability 0.80 as a population probability ignores that the sample’s positive prevalence is far higher than the target population’s.
Practical Checks
Before reporting an imbalanced-classification result, verify the data flow and the interpretation of the metric. In particular:
- Record the natural prevalence in the development, validation, and test data before resampling.
- Apply sampling, synthetic-data generation, and data-dependent preprocessing separately within each training fold.
- Keep validation and test observations at their natural class proportions.
- Report the selected metric in context; include confusion counts or class-specific rates where useful.
- Use a separate natural-prevalence assessment for claims about population precision or probability calibration.
Resampling can change the training signal available to a classifier, but it cannot create information about positive cases that the data do not contain. Very few positive cases can also make fold results variable. Report uncertainty and avoid interpreting a single favorable split as conclusive evidence. These checks are aligned with the SOA Exam SRM (Statistics for Risk Modeling) syllabus: the central task is to distinguish a change in fitting from a change in the population being evaluated.
Check Your Understanding
Work each problem using the stated folds and prevalence; do not resample validation observations.
- A data set has 25 positive and 475 negative observations. In five stratified folds, find the training-fold counts when one fold is held out. If negatives are undersampled to match the positive count, how many observations remain in that training set? (Answer: 20 positive and 380 negative before undersampling; 20 positive and 20 negative afterward, 40 total.)
- A training fold has 24 positive and 576 negative observations. Using \(w_k=n/(2n_k)\), calculate the positive and negative class weights and their ratio. (Answer: positive weight 12.5; negative weight \(600/1152=0.5208\) to four decimals; positive-to-negative weight ratio 24.)
- A natural-prevalence test set has 500 cases, including 25 positives. A classifier has sensitivity 0.80 and specificity 0.96. Find its true positives, false negatives, true negatives, false positives, and precision. (Answer: TP 20, FN 5, TN 456, FP 19; precision \(20/39=0.5128\) to four decimals.)
- Under class-only sampling, a calibrated sampled-data probability is \(q(x)=0.75\), sampled prevalence is 0.50, and population prevalence is 0.05. Find the corrected population probability using the odds correction. (Answer: sampled odds \(=3\); corrected odds \(=3(0.05/0.95)=3/19\); probability \(=3/22=0.1364\) to four decimals.)
- A colleague duplicates positive observations across the full data set and then creates cross-validation folds. State the main evaluation problem and the correct order of operations. (Answer: duplicates can appear in both training and validation folds, making validation optimistic; split into folds first, oversample only each training fold, and leave its validation fold unchanged.)