Class Weights And Sampling For Imbalanced Outcomes

Why Imbalanced Outcomes Need Deliberate Treatment

In a classification problem, an outcome is imbalanced when one class is much less frequent than another. For example, a hypothetical insurer might want to identify a rare type of claim requiring additional review. A forest that predicts the common class for nearly every case could achieve high overall accuracy while detecting very few of the rare cases. That may be unacceptable if missing a rare case is costly.

Class weights and class-specific sampling change how the forest learns from the classes. Neither method, by itself, decides which predicted cases should receive an action. That decision depends on a threshold and on the consequences of false positives and false negatives. Keep three questions separate: how is the forest fitted, what does its score mean, and what threshold will be used?

Definition: Let $Y$ be a binary outcome, with $Y=1$ denoting the class of interest and $Y=0$ the other class. The prevalence is $P(Y=1)$. Class weighting assigns different contributions to observations from the two classes during model fitting. Class-specific sampling selects observations from the classes at different rates when constructing training samples. Either can make errors or observations from the rare class more influential, but they do so in different ways.

Class Weights Change Fitting Priorities

A weighted fitting procedure gives an observation from class $y$ a weight $w_y$. For example, a weighted loss has the form $\sum_i w_{y_i} L_i$, where $L_i$ is the loss contributed by observation $i$. In a classification forest, class weights may also affect split selection or class decisions, depending on the implementation and its settings. They do not change the observed number of cases in either class.

A useful starting point is the relative cost of the two kinds of classification error. Suppose predicting 1 when $Y=0$ has cost $C_{FP}$, and predicting 0 when $Y=1$ has cost $C_{FN}$. Assume correct decisions have zero cost, and let $p=P(Y=1\mid X=x)$ be a calibrated probability for a case with predictors $x$. Predicting 1 has conditional expected cost $C_{FP}(1-p)$; predicting 0 has conditional expected cost $C_{FN}p$. Therefore, predicting 1 is less costly when the first quantity is no greater than the second.

C_{FP}(1-p) \le C_{FN}p \quad\Longleftrightarrow\quad p \ge \frac{C_{FP}}{C_{FP}+C_{FN}}.

This is a decision rule for a calibrated probability under the stated cost assumptions, not a guarantee about how a particular forest's class-weight setting behaves. In an idealized probability model fitted by minimizing weighted log loss, weights $w_1,w_0$ produce a weighted probability $q$ satisfying $q=w_1p/[w_1p+w_0(1-p)]$. A cutoff of $q=0.5$ then corresponds to $p\ge w_0/(w_0+w_1)$. This explains why a larger positive-class weight can favor positive predictions. A forest's split rules, probability estimates, and implementation details can prevent this ideal relationship from holding exactly.

Worked Example: Set Priorities From Error Costs

A hypothetical review team screens 10,000 claims. In the evaluation population, 200 claims are in the rare class and 9,800 are not. A missed rare-class claim costs $180, while an unnecessary review costs $20. Compare two candidate classification rules by their total error cost, and identify a corresponding relative class-weight choice.

Method. First calculate the cost-based probability threshold. For the stated costs, $C_{FN}=180$ and $C_{FP}=20$, so the threshold is $20/(20+180)=0.10$, assuming calibrated population probabilities and zero cost for correct decisions. The relative weight suggested by these error costs is $w_1:w_0=180:20=9:1$. This is a fitting priority, not proof that a forest's reported scores are calibrated.

Candidate A. Its sensitivity is 0.70 and specificity is 0.90. Of 200 positive cases, it detects $0.70(200)=140$, leaving $200-140=60$ false negatives. Of 9,800 negative cases, it correctly classifies $0.90(9,800)=8,820$, leaving $9,800-8,820=980$ false positives. Its total cost is $60(180)+980(20)=10,800+19,600=\$30,400$.

Candidate B. Its sensitivity is 0.85 and specificity is 0.85. It detects $0.85(200)=170$, leaving $200-170=30$ false negatives. It correctly classifies $0.85(9,800)=8,330$, leaving $9,800-8,330=1,470$ false positives. Its total cost is $30(180)+1,470(20)=5,400+29,400=\$34,800$.

Candidate B finds 30 more rare cases, but its additional false positives make its total cost $4,400 higher under the stated costs. Candidate A is preferred by this cost calculation. The comparison checks that sensitivity alone is not enough to select a model. The common mistake is to choose B solely because it has higher sensitivity, without accounting for false-positive cost.

Class-Specific Sampling Changes the Training Mix

Class-specific sampling changes which observations are used to fit a tree or forest. A common approach is to retain or draw different fractions from each class. For example, a training set with many negative cases and few positive cases could be subsampled so that a tree sees more balanced class counts. This changes the class proportions in the sampled data; it is not the same as assigning a larger weight to each positive observation while keeping the same observations.

Let $s_1$ and $s_0$ be the probabilities that a positive and a negative observation, respectively, are selected. Assume selection depends on the class only, not on predictor values within a class. Write $p=P(Y=1\mid X=x)$ for the population probability and $q=P(Y=1\mid X=x,\text{ selected})$ for the probability in the sampled data. By Bayes' rule, the odds in the sample equal the population odds multiplied by the relative selection rate:

\frac{q}{1-q} = \frac{p}{1-p}\frac{s_1}{s_0}, \qquad \frac{p}{1-p} = \frac{q}{1-q}\frac{s_0}{s_1}.

The first identity follows because the selected positive density at $x$ is proportional to $s_1P(X=x\mid Y=1)P(Y=1)$, while the selected negative density is proportional to $s_0P(X=x\mid Y=0)P(Y=0)$. Dividing these densities cancels the common normalization and gives the odds relation. It applies when class-conditional predictor distributions are preserved by sampling. It does not guarantee that a forest's estimated score $q$ is an accurate sampled-population probability, so any correction based on it is only as reliable as that estimate and the sampling assumptions.

Worked Example: Correct for Different Selection Rates

A hypothetical training population has prevalence $P(Y=1)=0.02$. Positive observations are selected with probability $s_1=0.80$, and negative observations with probability $s_0=0.10$. A fitted model gives a sampled-data probability $q=0.60$ for one case. Calculate the sampled prevalence and the population probability implied by the odds correction.

Sampled prevalence. Out of a population proportion of 1, the selected positive proportion is $0.02(0.80)=0.016$, and the selected negative proportion is $0.98(0.10)=0.098$. The total selected proportion is $0.016+0.098=0.114$. Thus the sampled prevalence is $0.016/0.114=0.1404$, to four decimals.

Correct the case's score. The sampled odds for the case are $q/(1-q)=0.60/0.40=1.5$. The relative selection rate is $s_1/s_0=0.80/0.10=8$. Therefore the population odds are $1.5(0.10/0.80)=1.5/8=0.1875$. Converting odds to probability gives $p=0.1875/(1+0.1875)=0.1579$, to four decimals.

The result is a population probability of about 0.1579 under the stated assumptions, not 0.60. The common mistake is to multiply the sampled odds by $s_1/s_0=8$ when correcting them. The correction uses the inverse ratio $s_0/s_1$, because the positive class was selected more often and is consequently overrepresented in the sample.

Keep Scores, Thresholds, and Validation Distinct

Class weights or sampling can change the fitted forest and its scores. A score that ranks cases well need not be a well-calibrated probability of the outcome in the intended population. In particular, sampling that changes class proportions can shift probability estimates. A class-weighted fit may also prioritize class separation rather than reproduce the population probabilities. Do not assume that a score of 0.20 means a 20% population risk merely because the model output is labelled a probability.

A decision threshold is a separate choice. It maps a score to an action: predict class 1 when the score is at least the chosen threshold. A threshold of 0.5 is not automatically appropriate for an imbalanced outcome, and class weighting does not automatically select the operationally best threshold. Choose and assess the threshold using validation data that reflect the intended population, together with the relevant costs or performance requirements. If the threshold is selected using validation results, keep a separate test set for a final assessment when one is available.

Worked Example: Compare Thresholds Using Validation Costs

A hypothetical validation set contains 1,000 cases: 40 positives and 960 negatives. A forest assigns scores to cases. At a cutoff of 0.20, the predicted-positive group contains 30 positives and 120 negatives. At a cutoff of 0.10, it contains 36 positives and 300 negatives. Let a false negative cost $50 and a false positive cost $10. Compare the two cutoffs using total validation cost.

Method. For each cutoff, obtain false negatives by subtracting detected positives from the 40 actual positives, and count all negative cases included in the predicted-positive group as false positives. Then multiply each error count by its cost and add.

Cutoff 0.20. There are $40-30=10$ false negatives and 120 false positives. The total cost is $10(50)+120(10)=500+1,200=\$1,700$.

Cutoff 0.10. There are $40-36=4$ false negatives and 300 false positives. The total cost is $4(50)+300(10)=200+3,000=\$3,200$.

Among these two candidate cutoffs, 0.20 has lower validation cost by $\$3,200-\$1,700=\$1,500$. This calculation does not establish that 0.20 is the best possible cutoff; other thresholds should be considered if operationally appropriate. Nor does it establish that the score is calibrated. The cost-based probability threshold would be $10/(10+50)=0.1667$ if the scores were calibrated population probabilities and the cost assumptions applied case by case. The common mistake is to treat that theoretical threshold as a substitute for validating the actual forest scores and candidate operating points.

Practical Choices in a Classification Forest

In R, the ranger package provides class-weight and class-specific sampling controls for classification forests. For class weights, check the outcome factor's level order and make the weight vector correspond to those classes. For class-specific sampling, check which class each fraction refers to and whether the chosen sampling scheme is compatible with the intended tree sample size. Exact behavior can depend on the package version and settings, so verify the function documentation rather than assuming that every implementation uses weights in the same way.

For a fair comparison, keep the training and validation cases, predictors, and other forest controls fixed while changing the weighting or sampling choice. Record the random seed when a procedure uses random sampling. Evaluate more than overall accuracy: report class-specific error information, such as sensitivity and specificity, and evaluate an operational threshold using the intended error costs or other decision criteria. If probability estimates will be used as risks, assess calibration on data representative of the target population.

A balanced training sample can help a learner pay attention to rare cases, but it does not make the real-world prevalence balanced. Likewise, assigning high weight to a rare class does not create additional independent information about that class. A small number of distinct positive observations may still limit how reliably the forest learns patterns that generalize to new cases.

Key takeaway: Class weighting changes the influence of classes during fitting; class-specific sampling changes which class observations enter the training samples. Both can alter forest predictions, but neither replaces choosing a decision threshold for the intended costs and validating performance at the intended population prevalence.

Check Your Understanding

Work through each problem, keeping the sampling assumptions and the decision threshold separate.

A hypothetical screening model has false-negative cost $90 and false-positive cost $10. Correct decisions have zero cost. What is the calibrated-probability decision threshold, and what relative positive-to-negative class-weight ratio reflects those costs? (Answer: Threshold $0.1000$; weight ratio $9:1$.)
A population has 5% positives. A class-specific sample selects positives with probability 0.60 and negatives with probability 0.20. What is the positive prevalence in the selected sample? (Answer: $0.03/(0.03+0.19)=0.1364$, or 13.64%.)
Positive cases are selected at rate 0.50 and negative cases at rate 0.25. A model estimates sampled-data probability $q=0.75$ for a case. Under class-only sampling and an accurate sampled probability estimate, what population probability follows from the odds correction? (Answer: Sampled odds $=3$; population odds $=3(0.25/0.50)=1.5$; population probability $=1.5/2.5=0.6000$.)
A training set contains 25 positives and 475 negatives. With no replacement, you want to retain all positives and select an equal number of negatives for a tree sample. What fraction of negatives should be selected? (Answer: $25/475=1/19=0.0526$, or about 5.26%.)
On a representative validation set, candidate A has 3 false negatives and 40 false positives; candidate B has 1 false negative and 100 false positives. A false negative costs $100 and a false positive costs $5. Which candidate has lower total error cost, and by how much? (Answer: A costs $3(100)+40(5)=\$500$; B costs $1(100)+100(5)=\$600$. Candidate A costs $100 less.)

What You'll Learn