Why Class Imbalance Changes the Interpretation
When an event is uncommon, a classifier can appear accurate simply by predicting that the event never occurs. That result may be useless if the purpose is to identify events, even though its percentage of correct classifications is high. Logistic regression adds another distinction: it estimates probabilities, while a classification rule converts those probabilities into labels. The estimated probability and the assigned label answer different questions.
For example, a hypothetical insurer might model whether a submitted claim requires a particular type of specialist review. If only a small share of claims require it, the overall event rate is low. A probability of 0.04 can still be a meaningful estimate of risk, even if a chosen classification threshold assigns the claim to the “no review” class.
A standard logistic model specifies the conditional event probability \(p_i=P(Y_i=1\mid x_i)\), where \(x_i\) denotes the predictors. Conditional on the predictors, the response is Bernoulli: \(Y_i\mid x_i\sim\operatorname{Bernoulli}(p_i)\). The model has the form
The model estimates a probability between 0 and 1. To classify an observation, a separate rule is needed. At a selected threshold \(c\), predict class 1 if \(\hat p_i\ge c\), and predict class 0 otherwise. A threshold of 0.50 is a convention, not a requirement of logistic regression. The value of \(c\) affects the classifications, but does not change the definition of the fitted probability \(\hat p_i\).
Prevalence, Baselines, and Accuracy
Prevalence describes how common the event is in a specified population or sample. It is not the same as a model’s sensitivity, and it does not by itself describe how well the model separates the two classes. An all-negative classifier predicts class 0 for every observation. If the event prevalence is \(\pi\), its accuracy is \(1-\pi\), because it correctly classifies every non-event and misses every event.
Worked Example: High Accuracy With No Event Detection
A hypothetical review study contains 1,000 claims. Each claim’s event status is modeled as Bernoulli, with the observed data containing 20 events and 980 non-events. Consider a classifier that predicts no event for every claim. Calculate prevalence, accuracy, and sensitivity. Explain what the accuracy does and does not show.
Method. First calculate the observed prevalence. Then construct the counts implied by predicting class 0 for all 1,000 claims.
Prevalence. There are 20 events out of 1,000 claims, so \(20/1{,}000=0.0200\), or 2.00%.
Classification counts. The classifier produces 980 true negatives, 20 false negatives, no true positives, and no false positives. Its accuracy is \((980+0)/1{,}000=0.9800\), or 98.00%.
Sensitivity. Sensitivity is the fraction of actual events classified as events. Here it is \(0/(0+20)=0\). The classifier detects none of the events.
The 98% accuracy reflects the large number of non-events, not successful event detection. The method avoids the common mistake of treating a high accuracy percentage as proof that a classifier is useful for every purpose. Accuracy alone does not establish whether the model identifies the event class.
The confusion matrix makes the distinction explicit. A true positive (TP) is an event classified as an event; a false negative (FN) is an event classified as a non-event. A true negative (TN) is a non-event classified as a non-event; a false positive (FP) is a non-event classified as an event.
| Metric | Formula | Interpretation |
|---|---|---|
| Accuracy | \((TP+TN)/(TP+FN+FP+TN)\) | Fraction of all observations classified correctly |
| Sensitivity | \(TP/(TP+FN)\) | Fraction of actual events identified |
| Specificity | \(TN/(TN+FP)\) | Fraction of actual non-events identified |
| Positive predictive value | \(TP/(TP+FP)\) | Fraction of predicted events that are actual events |
These metrics answer different questions. In particular, positive predictive value depends on event prevalence as well as on how the classifier performs among events and non-events. A low prevalence can mean that even a classifier with useful sensitivity and specificity produces many false positives relative to true positives.
Worked Example: Reading Metrics Under Low Prevalence
In a hypothetical validation set of 2,000 accounts, the event status is Bernoulli and 80 accounts have the event. At a specified threshold, the classifier gives 52 true positives, 28 false negatives, 192 false positives, and 1,728 true negatives. Calculate the prevalence, accuracy, sensitivity, specificity, and positive predictive value. Compare the accuracy with that of an all-negative classifier.
Method. Use the observed event and non-event totals as denominators for prevalence, sensitivity, and specificity. Use all observations for accuracy and all predicted positives for positive predictive value.
There are \(52+28=80\) actual events and \(192+1{,}728=1{,}920\) actual non-events. Thus prevalence is \(80/2{,}000=0.0400\), or 4.00%.
Accuracy is \((52+1{,}728)/2{,}000=1{,}780/2{,}000=0.8900\), or 89.00%.
Sensitivity is \(52/(52+28)=52/80=0.6500\), or 65.00%. Specificity is \(1{,}728/(1{,}728+192)=1{,}728/1{,}920=0.9000\), or 90.00%. Positive predictive value is \(52/(52+192)=52/244=0.2131\), or 21.31% to four decimals.
An all-negative classifier would have accuracy \(1{,}920/2{,}000=0.9600\), or 96.00%, but sensitivity 0. The specified classifier has lower accuracy but identifies 65% of the events. The calculations show why accuracy should be interpreted alongside the event-detection objective and other relevant metrics. The mistake to avoid is concluding that the 96%-accurate classifier must be preferable without considering that it misses every event.
How Prevalence Relates to the Logistic Baseline
In an intercept-only logistic model, every observation has the same event probability \(p\). If the responses are independent Bernoulli observations with a common probability, the maximum-likelihood estimate of \(p\) is the observed event fraction. Equivalently, the fitted intercept is the logit of that fraction, provided the data contain at least one event and at least one non-event.
The result follows directly from the likelihood. It describes the intercept-only model; with predictors, the fitted intercept alone is generally not the overall sample prevalence. The presence of predictors means the fitted probability varies across observations.
For independent Bernoulli responses with a common probability \(p\), the log-likelihood is \(\ell(p)=\sum_{i=1}^n[y_i\log(p)+(1-y_i)\log(1-p)]\).
The score is \(d\ell/dp=\sum_i y_i/p-\sum_i(1-y_i)/(1-p)=m/p-(n-m)/(1-p)\).
Setting it to zero gives \(m(1-p)=(n-m)p\), so \(m= np\) and \(\hat p=m/n\). The log-likelihood is concave for \(0<p<1\), since \(d^2\ell/dp^2=-m/p^2-(n-m)/(1-p)^2<0\) when \(0<m<n\); thus this solution is the maximum.
The intercept-only model has \(\log\{p/(1-p)\}=\beta_0\), so substituting \(\hat p=m/n\) gives \(\hat\beta_0=\log\{(m/n)/(1-m/n)\}\).
Worked Example: A Low Baseline Probability and an Odds Ratio
A hypothetical logistic model predicts whether a small equipment fault occurs during a scheduled inspection. Assume the baseline event probability for predictor value \(x=0\) is 0.020, and the odds ratio for changing from \(x=0\) to \(x=1\) is 3. Calculate the probability at \(x=1\), the probability-point increase, and the classification at threshold \(c=0.05\) for both predictor values. Treat the event as Bernoulli conditional on \(x\).
Method. Convert the baseline probability to odds, multiply the odds by the odds ratio, and convert the resulting odds back to a probability. An odds ratio multiplies odds; it does not generally multiply probability.
At \(x=0\), the baseline odds are \(0.020/(1-0.020)=0.020/0.980=1/49\). An odds ratio of 3 gives odds at \(x=1\) of \(3(1/49)=3/49\).
If the odds are \(o\), the probability is \(o/(1+o)\). Therefore the probability at \(x=1\) is \((3/49)/(1+3/49)=3/52=0.05769\), or 5.77% to two decimal places. The probability-point increase is \(0.05769-0.02000=0.03769\), or 3.77 percentage points to two decimal places.
At threshold 0.05, the \(x=0\) probability 0.020 is below the threshold, so it is classified as 0. The \(x=1\) probability 0.05769 is at least 0.05, so it is classified as 1. These classifications do not make either individual outcome certain; they are labels assigned by the stated rule.
The method avoids the common error of multiplying 0.020 by 3 and reporting 0.060 as the probability. That calculation multiplies probability rather than odds. The correct probability is 0.05769.
Interpreting Probabilities and Class Labels
A fitted probability is an estimate of the event chance under the model and its predictors. A class label is the result of applying a decision threshold to that estimate. For example, a fitted probability of 0.058 does not mean the event is “58% of a class,” nor does it guarantee that the event will occur. It means the model assigns that probability to the event, conditional on the stated predictors and model.
A threshold can be selected for a particular application, but no threshold makes probability and outcome interchangeable. A low threshold will generally classify more observations as events; a high threshold will generally classify fewer. The resulting counts, and therefore accuracy, sensitivity, specificity, and positive predictive value, can change even though the fitted probabilities have not changed.
There is a further prevalence distinction: the event rate in a study sample need not match the event rate in the population where decisions will be made. If the populations differ, metrics calculated in the study sample may not carry over unchanged. In particular, positive predictive value can change when prevalence changes, even when sensitivity and specificity remain the same. A classification summary should therefore be interpreted with its population and threshold in view.
Common Mistakes to Avoid
- Equating accuracy with usefulness. With low event prevalence, predicting every case as a non-event can give high accuracy and zero sensitivity.
- Calling an event probability a class label. A probability such as 0.04 is not automatically a prediction of class 0 or class 1; the threshold must be specified.
- Multiplying probability by an odds ratio. Logistic coefficients and odds ratios act on odds. Convert the adjusted odds back to probability before interpreting the effect on the probability scale.
- Using the wrong denominator. Sensitivity divides by actual events; specificity divides by actual non-events; positive predictive value divides by predicted events.
- Treating the intercept as prevalence in every logistic model. The intercept-only result applies when there are no predictors. With predictors, the fitted probabilities vary across observations.
Check Your Understanding
For each problem, identify the relevant denominator before calculating a metric.
- A hypothetical test sample has 500 observations, including 15 events. What is its event prevalence, and what accuracy does an all-negative classifier achieve? (Answer: Prevalence \(=15/500=0.0300\), or 3.00%; all-negative accuracy \(=485/500=0.9700\), or 97.00%.)
- A classifier has \(TP=24\), \(FN=16\), \(FP=90\), and \(TN=870\) in a hypothetical set of 1,000 observations. Calculate accuracy, sensitivity, specificity, and positive predictive value. (Answer: Accuracy \(=894/1{,}000=0.8940\); sensitivity \(=24/40=0.6000\); specificity \(=870/960=0.9063\); positive predictive value \(=24/114=0.2105\).)
- For a hypothetical event, the baseline probability is 0.010 and a predictor contrast has odds ratio 4. Calculate the probability after the contrast. At threshold 0.03, what class is assigned? (Answer: Baseline odds \(=0.01/0.99=1/99\); adjusted odds \(=4/99\); probability \(=(4/99)/(1+4/99)=4/103=0.0388\) to four decimals; assigned class 1.)
- In 1,000 hypothetical observations, there are 20 events. At a chosen threshold, a classifier produces \(TP=18\), \(FN=2\), \(FP=12\), and \(TN=968\). Calculate sensitivity, specificity, positive predictive value, and accuracy. (Answer: Sensitivity \(=18/20=0.9000\); specificity \(=968/980=0.9878\); positive predictive value \(=18/30=0.6000\); accuracy \(=986/1{,}000=0.9860\).)
- An intercept-only logistic model is fitted to 250 independent Bernoulli observations, of which 10 are events. Find the fitted event probability and intercept. (Answer: \(\hat p=10/250=0.0400\); \(\hat\beta_0=\log(0.04/0.96)=\log(1/24)\approx-3.1781\).)