Regularized Logistic Models With Rare Outcomes

Why Rare Outcomes Need Special Attention

When a binary outcome is uncommon, a seemingly routine split into validation folds can put very different numbers of events into different folds. One fold might contain several events while another contains none. That imbalance matters: a single event can have a large effect on a probability-based score, and the information available to estimate event-related patterns can be concentrated in only a few observations.

For a regularized logistic model, the penalty does not remove this data limitation. It can shrink estimates and help control model complexity, but it cannot create event information that is absent from a training fold. Consequently, a selected value of \(\lambda\), a cross-validation score, or a selected predictor set may depend noticeably on how the few events were allocated. The aim here is to diagnose and explain that sensitivity, not to replace the broader procedures for tuning penalties or assessing a final model.

Probability Loss and the Contribution of an Event

A useful way to understand the impact of a sparse fold is to examine the Bernoulli negative log-likelihood, often called log loss. For an observed outcome \(y_i\) and a predicted event probability \(\hat p_i\), where \(0<\hat p_i<1\), the contribution is

An event contributes \(-\log(\hat p_i)\); a non-event contributes \(-\log(1-\hat p_i)\). A confident but wrong prediction can therefore be costly. If an event receives a very small predicted probability, its loss is large; likewise, a non-event assigned a probability close to one receives a large loss. Log loss uses the predicted probabilities, not only classifications at a threshold.

For binary data, the Bernoulli likelihood is the product of the probabilities assigned to the observed outcomes. Taking the negative logarithm turns that product into a sum:

This is why one poorly predicted event can materially affect a fold’s total loss. The average log loss is the sum of the individual contributions divided by the number of observations. Binomial deviance is conventionally twice the negative log-likelihood relative to the saturated model; for individual binary observations, this is twice the corresponding log-loss contribution. Keep the chosen metric and its scaling consistent when comparing results.

Fold Counts: What to Inspect Before Tuning

For \(N\) observations, \(E\) events, and \(K\) folds, the overall prevalence is \(E/N\). If folds have approximately equal sizes and events were allocated randomly, the expected number of events in a fold of size \(n_k\) is \(n_k(E/N)\). This expected count is a planning aid, not a guarantee that each fold will contain that many events.

The following binomial approximation illustrates the chance of a fold with no events when observations are allocated independently with event probability \(p\):

When the data contain a fixed total number of events and are randomly partitioned without replacement, the exact probability follows a hypergeometric calculation rather than this binomial approximation. The approximation is useful for quick diagnosis, but state which calculation is being used. Whatever the assignment method, inspect the actual counts in every fold; the expected count alone does not establish that the folds are adequate.

A fold with no events can still yield probability-based losses for its non-events if predictions are finite, but it supplies no validation evidence about how well the model ranks or predicts events in that fold. Metrics that require both classes, such as a fold-specific ROC AUC, cannot be calculated when a fold contains only one class. A training fold with no events is a separate and more serious warning: the model cannot learn event patterns from that training subset, and fitting may be unstable or fail. Check the model fit and its warnings rather than treating a returned prediction as proof of an adequate fit.

Assigning Folds and Inspecting Predictions in R

When the data structure permits ordinary random resampling, stratifying fold assignment by outcome can spread scarce events more evenly. Stratification is not a cure for a small event count: if there are fewer events than folds, it is impossible to put an event in every validation fold. Nor should an analyst break a genuine time, group, or cluster structure merely to make event counts look balanced. The resampling design must respect how observations were generated.

For independent observations, a base-R sketch for assigning events and non-events across five folds is: set.seed(2026); K = 5; foldid = integer(length(y)); for (g in 0:1) { idx = sample(which(y == g)); foldid[idx] = rep(seq_len(K), length.out = length(idx)) }. Here, y contains 0/1 outcomes. Inspect the cross-tabulation table(foldid, y) before fitting. The assignment spreads each class as evenly as its count permits, although total fold sizes can differ slightly.

A corresponding tuning call is cv.glmnet(x, y, family = "binomial", alpha = 0.5, foldid = foldid, type.measure = "deviance"), assuming x is a suitable numeric predictor matrix and the data are prepared appropriately. Examine the event count and total size for each fold alongside the cross-validation curve. If a fold is too small or contains no events, explain that limitation when interpreting the curve; do not assume that a smooth-looking curve removes it.

To inspect out-of-fold probability predictions at a prespecified penalty, fit each training portion and predict the held-out portion. In each iteration, use training rows only for the model fit, and put predictions back in the original observation order. Any preprocessing step that estimates quantities from data, such as centering or imputation, must likewise be estimated using only that iteration’s training observations. This prevents held-out outcomes or predictor information from leaking into the training process. If the penalty was chosen using the same folds, these predictions are useful for examining the tuning process but should not be described as an independent final performance assessment.

Review the predictions together with the fold counts. Check whether event predictions are finite and whether the few events receive plausible probabilities, and calculate the chosen loss from the observed outcomes and predicted probabilities. A threshold-based accuracy figure can be uninformative when nearly all observations are non-events: predicting “non-event” for everyone may look accurate while missing every event. This is one reason to match the evaluation measure to the question being asked.

Why Selected Predictors Can Change

In penalized logistic regression, the fitted coefficients reflect both the observed data and the penalty. When events are scarce, a small change in which observations enter a training fold can change which event-predictor patterns the model sees. Those changes can affect coefficient estimates and which predictors remain active at a chosen penalty. Correlated predictors can add another reason that the identity of selected variables changes across fits.

Selection instability should be reported precisely. A predictor selected in many resamples may be a useful candidate for further study, but its selection frequency is not a probability that the predictor is truly relevant. Conversely, a predictor omitted in some fits is not automatically irrelevant. Frequencies describe the behavior of this modeling procedure under these resamples, with this sample and these modeling choices.

Common Exam Traps and Practical Conclusions

If the event count is too small to support the chosen number of folds, consider fewer folds or repeated stratified resampling where the data structure allows it. Fewer folds may increase the training-set size but also change the validation design; repeated assignments reveal variability but do not create new independent events. If no defensible resampling plan can provide informative event validation, say so explicitly. A numerical tuning result should not be presented as more precise than its event-level evidence allows.