Why Rare Events Need More Than One Performance Summary
When the event of interest is rare, a classifier can appear accurate simply by predicting the common class almost every time. For example, if only one unit in a hundred has a failure, a classifier that predicts “no failure” for every unit is 99% accurate but identifies none of the failures. Accuracy alone therefore does not describe whether a model is useful for finding rare events.
A more informative assessment separates the model’s ranking of cases from the threshold used to trigger action. ROC summaries describe how true-positive and false-positive rates change across thresholds. Precision-recall summaries emphasize how many of the flagged cases are actually positive. The practical choice of threshold must also respect an operational constraint, such as a maximum number of cases that staff can review.
The prevalence, denoted by \(\pi\), is the proportion of cases that are positive in the population being assessed. If the population has \(N\) cases, the expected numbers of positives and negatives are \(N\pi\) and \(N(1-\pi)\), respectively. In a rare-event setting, even a modest false-positive rate can produce many false positives because it is applied to the much larger negative class.
Ranking Summaries and Threshold Summaries
A classifier assigns each case a score, with larger scores typically indicating stronger evidence for the positive class. A threshold \(t\) classifies a case as positive when its score is at least \(t\). Lowering \(t\) generally increases or preserves both recall and the false-positive rate: more cases are flagged, including some additional positives and negatives.
The ROC curve plots the false-positive rate on the horizontal axis and recall on the vertical axis as the threshold varies. Its area, commonly called ROC AUC, summarizes ranking over thresholds. AUC can be computed from a sequence of ROC points using trapezoids, with the curve conventionally extending from \((0,0)\) to \((1,1)\). Precision-recall summaries instead plot precision against recall. Because the positive class is rare, they make the proportion of flagged cases that are truly positive especially visible.
Worked Example: Ranking Rare Equipment Failures
In a hypothetical set of 10,000 refrigeration-unit monitoring periods, 100 periods contain a failure and 9,900 do not. A scoring model gives the following cumulative counts at four thresholds. The rows show all cases with scores at least as high as the stated cutoff.
| Threshold | TP | FP | Recall | False-positive rate | Precision | Cases flagged |
|---|---|---|---|---|---|---|
| At least 0.90 | 40 | 200 | 0.40 | 0.0202 | 0.1667 | 240 |
| At least 0.70 | 70 | 800 | 0.70 | 0.0808 | 0.0805 | 870 |
| At least 0.40 | 90 | 2,500 | 0.90 | 0.2525 | 0.0347 | 2,590 |
| At least 0.20 | 100 | 6,000 | 1.00 | 0.6061 | 0.0164 | 6,100 |
Step 1: Check the denominators. There are 100 positive periods and 9,900 negative periods. At threshold 0.70, recall is \(70/100=0.70\), while the false-positive rate is \(800/9{,}900=0.080808\ldots\). Precision is \(70/(70+800)=70/870=0.080459\ldots\). The flagged total is TP plus FP, not the total number of periods.
Step 2: Calculate ROC AUC. Include the endpoints \((0,0)\) and \((1,1)\). The successive ROC points are \((0.020202\ldots,0.40)\), \((0.080808\ldots,0.70)\), \((0.252525\ldots,0.90)\), and \((0.606061\ldots,1.00)\). For adjacent points \((x_1,y_1)\) and \((x_2,y_2)\), the trapezoid area is \((x_2-x_1)(y_1+y_2)/2\). Thus the five areas, including both endpoint segments, are:
Using the exact rates \(200/9{,}900\), \(800/9{,}900\), \(2{,}500/9{,}900\), and \(6{,}000/9{,}900\) gives an area of approximately 0.904545. The displayed rounded ROC coordinates are for readability; retaining the underlying fractions avoids small discrepancies in the final digits.
Step 3: Calculate non-interpolated average precision. Recall rises by 0.40, then 0.30, then 0.20, and finally 0.10. The precision at those steps is \(40/240\), \(70/870\), \(90/2{,}590\), and \(100/6{,}100\), respectively. Therefore:
This calculation uses the stated non-interpolated convention: each recall increment is weighted by the precision achieved at the threshold that produces it. A trapezoidal area under a precision-recall curve is a different summary and can give a different value.
Step 4: Apply the operational limit. If the inspection team can review at most 900 periods, threshold 0.70 flags 870 and meets the limit. The next lower threshold, 0.40, flags 2,590 and exceeds it. Of the listed thresholds, 0.70 is the lowest threshold that stays within capacity, and it captures 70 of the 100 failures.
Check: At threshold 0.70, the confusion counts are TP = 70, FP = 800, FN = 30, and TN = 9,100. They sum to 10,000; recall is \(70/(70+30)=0.70\), and the false-positive rate is \(800/(800+9{,}100)=800/9{,}900\), consistent with the table.
Why Precision Changes With Prevalence
At a fixed threshold, recall describes the proportion of positives caught, and the false-positive rate describes the proportion of negatives flagged. Neither calculation directly includes the population prevalence. Precision, however, asks what fraction of the flagged cases are positive. It depends on how many positives and negatives were available to be flagged in the first place.
Among a population of size \(N\), the expected number of true positives is \(N\pi \times TPR\).
The expected number of false positives is \(N(1-\pi)\times FPR\).
Precision is true positives divided by all positive predictions, so substitute the two expected counts and cancel \(N\).
This gives the standard prevalence relationship:
The formula is a direct consequence of the definition of precision: the numerator is the proportion of the population that is both positive and flagged, and the denominator adds the proportion that is positive and flagged to the proportion that is negative and flagged. It applies when the prevalence and class-conditional rates refer to the population and threshold being assessed. If those rates change between settings, this calculation does not by itself predict the new precision.
Worked Example: Reassessing Fraud Flags After a Prevalence Shift
A hypothetical transaction-screening model has recall 0.70 and a false-positive rate of \(800/9{,}900\) at a chosen threshold. In its original assessment population, prevalence was 1%. Estimate precision if the model is applied in a setting where prevalence is 2%, assuming the class-conditional score behavior—and therefore recall and false-positive rate—remains unchanged.
Method: use the prevalence formula. The new prevalence is \(\pi=0.02\), \(TPR=0.70\), and \(FPR=800/9{,}900=0.080808\ldots\). Substitute these values:
The estimated precision is 0.1502, or about 15.02%. For comparison, at 1% prevalence the same rates give:
Thus precision rises from about 8.05% to 15.02% when prevalence increases, even though recall and the false-positive rate are held fixed. The approximation depends on the stated stability assumption; it is not enough to know that the threshold itself has not changed.
Common mistake avoided: carrying the old precision into the new setting. Precision is not an intrinsic constant of a model or threshold. It must be recalculated when prevalence changes.
Selecting a Threshold Under an Action Constraint
A ranking metric does not decide how many cases to investigate. That decision depends on what actions are feasible and what performance requirement matters. A capacity constraint might specify a maximum number of predicted positives; a screening rule might instead require recall to meet a minimum. If both conditions are stated, the threshold must satisfy both. Compare the candidate thresholds on the same target population and apply the constraint to the count or rate it actually names.
Worked Example: Meeting a Recall Requirement Without Exceeding Review Capacity
A hypothetical insurer ranks submitted property claims for a specialist review. At three candidate thresholds, its validation counts are as follows. The review unit can handle no more than 900 claims, and the insurer requires recall of at least 0.75. Choose a threshold that meets both conditions.
| Threshold | TP | FP | FN | Recall | Claims flagged |
|---|---|---|---|---|---|
| At least 0.85 | 55 | 245 | 45 | 0.55 | 300 |
| At least 0.65 | 78 | 702 | 22 | 0.78 | 780 |
| At least 0.45 | 92 | 1,308 | 8 | 0.92 | 1,400 |
Step 1: Verify recall. At the middle threshold, there are \(78+22=100\) positive claims, so recall is \(78/100=0.78\). The high threshold has recall \(55/(55+45)=0.55\); the low threshold has recall \(92/(92+8)=0.92\).
Step 2: Check the review limit. The number of claims flagged is TP + FP. The three totals are \(55+245=300\), \(78+702=780\), and \(92+1{,}308=1{,}400\). The low threshold exceeds the 900-claim capacity.
Step 3: Select among feasible thresholds. The high threshold fits capacity but fails the minimum recall requirement. The middle threshold fits capacity and has recall 0.78, which is at least 0.75. Select the threshold of 0.65.
Check: The selected threshold flags 780 claims, leaving 120 spaces below the capacity limit, and misses \(22\) of \(100\) positive claims, giving recall \(1-22/100=0.78\). Both constraints are satisfied.
Common mistake avoided: choosing the threshold with the highest recall without checking the action limit. The threshold of 0.45 has the highest recall, but it requires 1,400 reviews and is infeasible under the stated capacity.
Common Traps in Rare-Event Ranking
- Using the wrong denominator. Recall divides TP by all actual positives, \(TP+FN\). Precision divides TP by all predicted positives, \(TP+FP\). The false-positive rate divides FP by all actual negatives, \(FP+TN\).
- Interpreting high ROC performance as high precision. ROC rates use class-specific denominators. With very low prevalence, false positives can still substantially outnumber true positives.
- Mixing thresholds or populations. Compare performance at the same threshold, or state how thresholds are matched. When prevalence shifts, recompute precision using the new prevalence if the class-conditional rates are assumed stable.
- Ignoring ties or indivisible groups. If multiple cases share a score, a threshold may flag all of them together. A capacity target may therefore be impossible to meet exactly using a score threshold alone.
- Rounding before summing ROC areas. Retain full precision in trapezoid calculations and round the final result. For example, full-precision areas \(0.0048214286\), \(0.0391836735\), and \(0.7991071429\) sum to approximately \(0.8431122449\), which rounds to \(0.8431\) to four decimal places—not \(0.843110\) at six decimal places.
Check Your Understanding
For each problem, identify the relevant denominator or constraint before calculating.
- A hypothetical monitoring system scores 5,000 machine-days, of which 25 have a fault. At one threshold it flags 20 fault days and 495 non-fault days. Find recall, precision, and prevalence. (Answer: Recall = 0.8000; precision = 0.0388; prevalence = 0.0050.)
- A transaction model has recall 0.80 and false-positive rate 0.04. In a new setting, prevalence is 3%. Assuming the class-conditional rates are unchanged, calculate precision. (Answer: \(0.024/(0.024+0.0388)\approx 0.3822\).)
- A ROC curve has successive points \((0,0)\), \((0.05,0.50)\), \((0.25,0.80)\), and \((1,1)\). Calculate trapezoidal AUC. (Answer: \(0.0125+0.1300+0.6750=0.8175\).)
- A review system has candidate thresholds with (recall, cases flagged) of (0.55, 300), (0.78, 780), and (0.92, 1,400). It must achieve recall of at least 0.75 and can review at most 900 cases. Which candidate should be selected? (Answer: Select the middle candidate: recall 0.78 with 780 cases flagged.)
- At a threshold, a population has 60 true positives, 240 false positives, 40 false negatives, and 4,660 true negatives. Calculate recall, precision, and false-positive rate. (Answer: Recall = \(60/100=0.6000\); precision = \(60/300=0.2000\); false-positive rate = \(240/4{,}900\approx 0.0490\).)