Handling Imbalanced Renewal Outcomes

Why Imbalanced Renewal Outcomes Need Careful Handling

A renewal model can predict well for the common outcome and still be ineffective for the outcome that matters operationally. For example, if most policies renew, a model that labels every policy as a renewal may have high overall accuracy while detecting none of the nonrenewals. The issue is not simply that one class is smaller: it is that a model, metric, or decision rule can hide weak performance on that class.

Start with a clearly defined prediction unit and outcome window. Here, let \(Y=1\) mean that a policy renews at its eligible renewal date, and let \(Y=0\) mean that it does not. The renewal prevalence in the target portfolio is the proportion of eligible policies with \(Y=1\). The model’s estimated renewal probability for policy \(j\) is \(p_j=P(Y_j=1\mid X_j)\), where \(X_j\) denotes predictors available at the time the prediction is made.

The relevant prevalence is the prevalence in the population for which predictions will be used. A training sample altered by oversampling or undersampling can have a different prevalence. Confusing that altered sample with the target portfolio can lead to miscalibrated probabilities and incorrect renewal forecasts.

Diagnose the Imbalance Before Choosing a Method

Report the number and proportion of each outcome, both overall and for important cohorts such as renewal period or product segment. Check that counts use the intended prediction unit: a policy-level outcome should not be counted as if each repeated record were an independent policy. Also verify that the renewal label and observation window are consistently defined.

Accuracy alone can be misleading. If renewal is common, predicting renewal for everyone may produce high accuracy but zero ability to identify nonrenewals. A confusion matrix makes this visible. For a nonrenewal detection task, treat nonrenewal as the positive class: a true positive is a correctly identified nonrenewal, and a false negative is a nonrenewal predicted as renewal. Recall for nonrenewal is the proportion of actual nonrenewals identified. Precision is the proportion of policies labeled nonrenewal that actually do not renew.

Class Weighting and Probability Correction

Class weighting changes the contribution of observations to the fitting objective. For example, an analyst may assign greater weight to nonrenewals so that errors on the less frequent class have more influence. Weighting can help a model learn a useful distinction, but the resulting fitted probabilities generally correspond to the weighted objective, not directly to the natural portfolio prevalence.

A useful adjustment follows when the only change is a constant class weight: let the renewal class \(Y=1\) have weight \(w_1\), and the nonrenewal class \(Y=0\) have weight \(w_0\). Under correct model specification and a weighted likelihood that changes only these class contributions, the weighted conditional renewal probability \(q\) and natural probability \(p\) satisfy the odds relation below. This relation is exact for the population optimum under those assumptions; finite samples, model misspecification, regularization, or weights that vary within a class can make a simple correction imperfect.

The reason is that, conditional on predictors, the weighted contributions associated with renewal and nonrenewal are proportional to \(w_1p\) and \(w_0(1-p)\), respectively. Their ratio is \(w_1p/[w_0(1-p)]\), which is the weighted renewal odds. To return to natural odds, multiply by \(w_0/w_1\), then convert odds to probability using \(p=O/(1+O)\).

Weighting is a fitting choice, not a substitute for checking probability quality on data representative of the intended portfolio. It can change slopes as well as the intercept in finite or misspecified models. If weighting is used to improve identification of nonrenewals, assess both the operational classification performance and whether probabilities remain suitable for forecasting.

Resampling and the Natural Prevalence

Resampling changes which outcomes, or how many copies of outcomes, appear in the training data. Oversampling may repeat or otherwise increase the representation of a less frequent class; undersampling may remove observations from the more frequent class. Neither operation changes the underlying outcome prevalence in the target portfolio. If selection into the resampled data depends only on outcome class, a prevalence correction can be derived from the class-specific inclusion rates.

Let \(s_1\) and \(s_0\) be the probabilities that a renewal and a nonrenewal, respectively, are included in the resampled data. If \(q\) is the renewal probability in that data and sampling is independent of predictors within each outcome class, then sampled renewal odds equal \((s_1/s_0)\) times natural renewal odds. Consequently, natural odds are sampled odds multiplied by \(s_0/s_1\). The condition about sampling within classes matters: if selection also depends on policy characteristics, a single prevalence adjustment may not recover policy-level probabilities.

Resampling and class weighting can have similar effects on class contributions, but their implementation and assumptions differ. In either case, evaluate on validation data that reflect the target population rather than on a deliberately balanced training sample. Any correction should be documented, and probability checks should be made by cohort and renewal period as appropriate.

Threshold Selection Is Not Probability Estimation

A threshold converts a probability into a class label. For example, a policy may be labeled “likely nonrenewal” when its estimated nonrenewal probability exceeds a chosen cutoff. Changing that cutoff changes the number of policies labeled in each class, as well as measures such as recall and precision. It does not, by itself, change the underlying probabilities \(p_j\).

There is no universally correct threshold. A cutoff may be selected to meet a staffing or outreach capacity, to achieve a required recall, or to reflect the consequences of different decisions. The choice should be made using validation data and a stated operational objective. A threshold chosen for classification should not be applied to replace probabilities when the task is to forecast a portfolio total.

Choosing and Validating an Approach

Use the method that addresses the actual problem. If the model fails to learn useful distinctions because the less frequent class has little influence on the fitting objective, weighting or resampling may be worth testing. If the model’s probability estimates are useful but an operational team needs a manageable list of policies, threshold selection may be the relevant decision. These choices can be compared, but they answer different questions.

A common exam and practice trap is to balance the training data, obtain a more even class split, and then report the model’s raw outputs as natural portfolio probabilities. Another is to lower a threshold and claim the model’s probability estimates improved. The first ignores the change in class prevalence; the second confuses classification performance with probability estimation. Report the target prevalence, the adjustment method, and the intended use of the output.