Class Weighting For Imbalanced Boosted Classification

Why Weight Classes?

A binary classification data set may contain many more observations from one class than the other. For example, a hypothetical insurer might have few claims that require manual review compared with routine claims. A model that predicts the majority class for every case could have high accuracy while failing to identify the cases of greatest interest. Class weighting gives selected observations more influence during model fitting.

Weighting does not create additional minority-class observations, and it does not guarantee better predictions. It changes the optimization objective: errors on a class with a larger weight contribute more to the loss and therefore can exert more influence on the boosting updates. Whether that change is useful must be checked against the intended decision and held-out validation data.

Definition: In binary classification, let $y_i$ be the observed class for case $i$, with $y_i\in\{0,1\}$, and let $p_i$ be the model's predicted probability that $y_i=1$. A weighted binary cross-entropy loss is $$ L_w=-\sum_{i=1}^{n}w_i\left[y_i\log(p_i)+(1-y_i)\log(1-p_i)\right], $$ where $w_i>0$ is the observation weight. Class weighting assigns a common weight to each observation in a class, such as $w_1$ for class 1 and $w_0$ for class 0.

When all weights equal 1, this is the usual binary cross-entropy loss. Multiplying every weight by the same positive constant does not change the minimizer of the summed loss, though it can change the numerical scale of the loss and may affect implementation-specific regularization or stopping behavior. For comparability, many workflows choose weights whose average over the training observations is 1.

What Weighting Changes in Boosting

Gradient boosting builds an additive model in stages. At each stage, the current predictions determine the loss gradients; a new tree is fitted to help reduce that loss. An observation with a larger weight contributes more to the weighted loss and, correspondingly, has a larger influence on the loss derivatives used to fit subsequent trees. Thus, weighting can change which splits are attractive and how the fitted scores move.

For binary logistic loss, write $f_i$ for the model's score on the log-odds scale and $p_i=1/(1+\exp(-f_i))$. The unweighted derivative of the loss for case $i$ with respect to $f_i$ is $p_i-y_i$; its second derivative is $p_i(1-p_i)$. With observation weight $w_i$, these become $w_i(p_i-y_i)$ and $w_i p_i(1-p_i)$. The weight therefore affects the fitting calculations, not just a final classification threshold.

Justification: For one observation, the negative log-likelihood is $-y_i\log(p_i)-(1-y_i)\log(1-p_i)$. Since $p_i$ is the logistic transform of $f_i$, differentiating with respect to $f_i$ gives $p_i-y_i$. Multiplication of the observation's loss by $w_i$ multiplies both derivatives by $w_i$, because $w_i$ is fixed with respect to the model score.

A standard starting point for inverse-frequency class weights is to assign each class half of the total weight. If the training set has $n_1$ class-1 cases, $n_0$ class-0 cases, and $N=n_1+n_0$, use $w_1=N/(2n_1)$ and $w_0=N/(2n_0)$. Then the total weight contributed by each class is $N/2$, and the average observation weight is 1. This is a convention, not a universal optimum: the weights should reflect the modeling objective and be assessed using validation data.

Worked Example: Computing Weighted Loss Contributions

A two-case illustration has one positive case with observed outcome $y=1$ and predicted probability $p=0.2$, and one negative case with $y=0$ and $p=0.8$. Suppose the positive class has weight 5 and the negative class has weight $5/9$. Compute each weighted loss contribution and the mean weighted loss, using the sum of weights as the denominator.

Method. Apply binary cross-entropy separately to each case, multiply each loss by its observation weight, then divide the sum of contributions by the sum of weights. This distinction matters: the weighted mean is not the unweighted mean of the two losses.

For the positive case, the loss is $-\log(0.2)=1.6094379\ldots$. Its weighted contribution, calculated from the unrounded logarithm, is $5[-\log(0.2)]=8.0471895\ldots$, which rounds to 8.0472.

For the negative case, the loss is $-\log(1-0.8)=-\log(0.2)=1.6094379\ldots$. This is not the loss $-\log(0.8)$: for a negative case the relevant probability is $1-p$. Its weighted contribution is $(5/9)[-\log(0.2)]=0.8941322\ldots$, which rounds to 0.8941.

The total weighted contribution is $8.0471895\ldots+0.8941322\ldots=8.9413217\ldots$. The sum of weights is $5+5/9=50/9$. Therefore the mean weighted loss is $8.9413217\ldots/(50/9)=1.6094379\ldots$, or 1.6094 to four decimal places.

Check and trap. Both cases assign probability 0.2 to the class that actually occurred, so their unweighted losses are equal. The mean weighted loss must therefore equal that common loss, regardless of the unequal weights. Using $-\log(0.8)$ for the negative case would misread the event whose probability enters the loss.

Choosing and Applying Class Weights

Inverse-frequency weights are a transparent baseline when the main concern is that the majority class dominates fitting. They do not encode the financial cost of each error unless the chosen weights have been deliberately linked to those costs. They also do not change the observed class counts: a training set with 20 positive and 180 negative cases still contains 20 and 180 cases after weighting.

In R with xgboost, weights can be attached to the training data matrix. The following outline assumes that the predictor data and binary labels have already been prepared, and that validation cases were kept separate. Calculate the weights using the training labels only:

R outline: Let n1 be the number of training labels equal to 1, n0 the number equal to 0, and N their sum. Set w to ifelse(y_train == 1, N/(2*n1), N/(2*n0)). Create dtrain_w with xgb.DMatrix(X_train, label = y_train, weight = w), and create dtrain_u from the same predictors and labels but without the weight argument. Fit each model with the same training cases, predictors, objective, and chosen model settings; predict on the same validation predictors.

This is a controlled comparison only if the validation observations do not influence weight calculation or fitting. For a fair evaluation of predictive probabilities, calculate validation log loss without class weights unless the stated target explicitly calls for a weighted evaluation. Report class-specific measures as well when identifying the minority class is important.

Worked Example: Inverse-Frequency Weights for a Training Set

A hypothetical training sample for a binary review flag contains 20 positive cases and 180 negative cases. Find inverse-frequency weights that give each class half the total training weight, verify their average, and state how the weights are used in a weighted fit.

Method. The total sample size is $N=20+180=200$. The class-1 weight is $w_1=N/(2n_1)=200/(2\cdot20)=5$. The class-0 weight is $w_0=N/(2n_0)=200/(2\cdot180)=5/9$, or approximately 0.5556.

The total weight on class 1 is $20\cdot5=100$. The total weight on class 0 is $180\cdot(5/9)=100$. The overall weight is $100+100=200$, so the average observation weight is $200/200=1$. These checks confirm that the classes have equal total weight and that the weights are normalized to average 1.

In the weighted fit, each positive training case receives weight 5 and each negative training case receives weight $5/9$. The model is still fitted to the original 200 cases. The weights change their relative influence on the fitting loss; they do not replicate cases or alter the labels.

Check and trap. Giving every positive case weight 1 and every negative case weight $1/9$ would also make the class totals equal, but the average observation weight would be $20/200+(180/200)(1/9)=0.15$, not 1. A common scaling may leave an idealized loss minimizer unchanged, but it needlessly changes the loss scale and can interact with implementation settings. The normalized weights above avoid that ambiguity.

Interpreting Weighted Probabilities

Class weighting can shift fitted probabilities even when the predictors contain no information. To see why, suppose every case has the same predicted probability $q$, and the training prevalence of class 1 is $\pi=n_1/N$. The weighted cross-entropy per unit of total weight is minimized at

q_w=\frac{w_1\pi}{w_1\pi+w_0(1-\pi)}.

This follows by differentiating the expected weighted loss $-w_1\pi\log(q)-w_0(1-\pi)\log(1-q)$ with respect to $q$, setting the derivative to zero, and solving $w_1\pi/q=w_0(1-\pi)/(1-q)$. The result shows that the weighted model's probability scale need not match the population prevalence scale. Class weighting is therefore not automatically a calibration correction.

Worked Example: A Weighted Intercept-Only Classifier

Suppose the training prevalence of the positive class is $\pi=0.10$. Use class weights $w_1=5$ and $w_0=5/9$, as in the normalized inverse-frequency scheme for this prevalence. Find the constant probability minimizing the weighted loss, and compare it with the unweighted optimum.

Method. With a constant prediction, the weighted loss per observation is $-w_1\pi\log(q)-w_0(1-\pi)\log(1-q)$. The minimizing probability is the weighted-prevalence expression above. Substituting gives

q_w=\frac{5(0.10)}{5(0.10)+(5/9)(0.90)} =\frac{0.5}{0.5+0.5} =0.5.

Without class weighting, $w_1=w_0=1$, so the optimum is $q=\pi=0.10$. Weighting has moved the constant fitted probability from 0.10 to 0.50 because it makes the total positive and negative contributions equal.

Check and trap. The value 0.50 is the optimum for this weighted objective, not evidence that half of the population is positive. Interpreting it as an unweighted population probability would confuse the training objective with the target probability scale. If calibrated population probabilities are required, assess calibration on appropriately representative, unweighted validation data and use a suitable calibration procedure if justified.

Compare Models on Validation Data

A weighted model should be compared with an unweighted model using the same data split and otherwise comparable fitting choices. Evaluate both on the same validation cases. At a specified threshold, a confusion matrix separates true positives, false positives, true negatives, and false negatives. Accuracy alone can conceal poor minority-class detection, so consider class-specific measures such as recall and precision alongside specificity or the overall error cost relevant to the application.

The threshold is a decision rule applied to predicted probabilities; class weights are part of the fitting objective. These are different tools. A model trained without class weights can sometimes meet a recall target by using a different threshold, while weighting may change the ranking or probability estimates themselves. Comparisons should state the threshold and the validation criterion.

Worked Example: Comparing Weighted and Unweighted Validation Results

A validation set has eight cases, of which two are positive. At a threshold of 0.50, the unweighted model predicts no positives. The weighted model predicts three positives: two are truly positive and one is negative. Compare the two models using accuracy, positive-class recall, precision, and specificity.

Method. Count each confusion-matrix entry from the stated predictions. For the unweighted model, $TP=0$, $FP=0$, $FN=2$, and $TN=6$. Thus accuracy is $(TP+TN)/8=6/8=0.75$, recall is $TP/(TP+FN)=0/2=0$, precision is undefined because $TP+FP=0$, and specificity is $TN/(TN+FP)=6/6=1$.

For the weighted model, $TP=2$, $FP=1$, $FN=0$, and $TN=5$. Accuracy is $(2+5)/8=7/8=0.875$. Recall is $2/(2+0)=1$. Precision is $2/(2+1)=2/3=0.6667$, and specificity is $5/(5+1)=5/6=0.8333$. Probabilities and proportions are rounded to four decimal places.

Interpretation. In this hypothetical validation result, the weighted model identifies both positive cases but also flags one negative case. It improves recall and accuracy here, while reducing specificity. That does not prove weighting will improve future performance; this small illustration only demonstrates how to interpret a comparison.

Check and trap. Reporting only specificity would favor the unweighted model, and reporting only recall would favor the weighted model. Neither measure alone establishes which model is preferable. The choice depends on the application’s error priorities and should be supported by validation results. Precision for the unweighted model is undefined, not zero, because it made no positive predictions.

Practical Checks and Common Traps

Before treating weighting as an improvement, verify how it affects the full modeling and evaluation process. In particular, weight calculation belongs to the training data; validation results should reflect the target population and decision. If the validation sample was deliberately case-control sampled or otherwise differs from that target, its metrics may need appropriate adjustment, which must be stated rather than assumed.

Do not confuse weighting with changing the threshold. Weights change the fitted objective; a threshold changes how fitted scores are converted to class decisions.
Do not report accuracy alone for a rare class. A high accuracy can coexist with zero positive predictions.
Do not assume a weighted score is calibrated. Class weighting can shift the probability scale, as the intercept-only calculation demonstrates.
Do not use validation outcomes to choose the weights and then report the same results as an unbiased final assessment. Such tuning makes the reported validation performance optimistic.
Do not assume inverse-frequency weights are always best. They are a baseline; application-specific error costs and validation performance matter.

Key takeaway: Class weighting changes the contribution of each class to the boosting loss and its fitting updates. Fit weighted and unweighted models under comparable conditions, keep weight selection separate from validation outcomes, and compare validation measures that reflect the actual decision objective.

Check Your Understanding

Work each problem before checking the answer at its end.

A training set has 30 positive and 270 negative cases. Using weights that give each class half the total weight and make the overall average weight 1, find the class weights. (Answer: positive weight 5; negative weight $5/9$, or approximately 0.5556.)
A positive case has predicted probability 0.4 and weight 3. Calculate its weighted binary cross-entropy contribution to four decimals. (Answer: $3[-\log(0.4)]=2.7489$.)
A negative case has predicted probability 0.25 and weight 2. Calculate its weighted loss contribution to four decimals. (Answer: $2[-\log(0.75)]=0.5754$.)
A validation set has 12 positive and 108 negative cases. At a chosen threshold, a model has $TP=9$, $FN=3$, $FP=18$, and $TN=90$. Find recall, precision, and accuracy to four decimals. (Answer: recall $9/12=0.7500$; precision $9/27=0.3333$; accuracy $99/120=0.8250$.)
The positive prevalence is 0.05 and the class weights are $w_1=10$ and $w_0=10/19$. Find the constant probability minimizing the weighted loss. (Answer: $q_w=[10(0.05)]/[10(0.05)+(10/19)(0.95)]=0.5$.)

What You'll Learn