Comparing Classifiers at Matched Sensitivity

Why Compare at Matched Sensitivity?

A classifier is often used by choosing a score threshold: cases at or above the threshold are flagged positive, while cases below it are classified negative. A comparison at a common threshold can be misleading because different classifiers may produce scores on different scales. Comparing at matched sensitivity instead asks: if both classifiers identify the same proportion of actual positives, which one generates fewer false alarms and how many cases does each flag?

This comparison is useful when a decision-maker wants to hold the fraction of missed positives fixed, then examine the consequences for the negative cases and the volume of flagged cases. It is aligned with the SOA Exam PA (Predictive Analytics) syllabus.

Definition: For a specified threshold, a true positive (TP) is an actual positive classified positive; a false negative (FN) is an actual positive classified negative; a false positive (FP) is an actual negative classified positive; and a true negative (TN) is an actual negative classified negative. Sensitivity, also called the true positive rate, is \(TP/(TP+FN)\). The false positive rate is \(FP/(FP+TN)\). Precision, also called the positive predictive value, is \(TP/(TP+FP)\). The flagged-case workload is \(TP+FP\), or its fraction of the total cohort when expressed as a rate.

Two classifiers are compared at matched sensitivity when their chosen operating thresholds produce the same sensitivity on the comparison population. Their numerical thresholds need not be equal. For example, a sensitivity of 0.90 means that 90% of the actual positives are flagged and 10% are missed. It says nothing by itself about how many negatives are flagged.

When scores are thresholded, a higher threshold will generally flag fewer cases. A threshold can be selected for each classifier to reach a target sensitivity, using validation data. For a fair comparison, calculate both classifiers' metrics on the same labeled population and use the same definition of a positive outcome. The chosen thresholds and their resulting metrics should then be reported explicitly.

What Changes When Sensitivity Is Held Fixed?

At a fixed sensitivity, the false negative rate is fixed too: \(1-\text{sensitivity}\). The false positive rate can still differ between classifiers. The classifier with the lower false positive rate flags fewer actual negatives, which will usually also mean fewer total cases flagged. Its precision will generally be higher on the same population because it has fewer false positives for the same number of true positives.

The prevalence of the positive outcome matters. Let \(\pi\) be the proportion of actual positives in the population, let \(s\) be sensitivity, and let \(r\) be the false positive rate. For a cohort of size \(N\), the expected or proportional counts are \(N\pi s\) true positives and \(N(1-\pi)r\) false positives. Therefore, precision and the fraction flagged are:

\text{Precision} = \frac{\pi s}{\pi s+(1-\pi)r}, \qquad \text{Flagged fraction} = \pi s+(1-\pi)r.

These formulas describe the population proportions implied by the rates. With observed finite data, counts must be whole numbers, and the calculated metrics use those observed counts. In particular, precision is undefined if no cases are flagged, because \(TP+FP=0\).

Why the precision formula holds: Among \(N\) cases, there are \(N\pi\) actual positives. Sensitivity \(s\) means \(N\pi s\) of them are true positives. The remaining \(N(1-\pi)\) cases are actual negatives, of which a fraction \(r\) are false positives, giving \(N(1-\pi)r\). Precision is true positives divided by all flagged cases. Substituting these two counts and canceling \(N\) gives the formula above.

Worked Comparisons

Worked Example: Fewer False Positives at the Same Sensitivity

A hypothetical screening program evaluates two classifiers on the same 1,000 people. There are 100 actual positives and 900 actual negatives. At threshold 0.42, Classifier A has 90 true positives and 180 false positives. At threshold 0.67, Classifier B has 90 true positives and 90 false positives. Compare sensitivity, false positive rate, precision, and flagged workload.

Method. Construct the confusion counts for each classifier, then calculate each metric from its definition. The thresholds differ, but both are assessed on the same cohort.

For both classifiers, \(TP=90\) and \(FN=100-90=10\). Thus each has sensitivity \(90/(90+10)=0.9000\). The sensitivity is matched.

For Classifier A, the number of true negatives is \(TN=900-180=720\). Its false positive rate is \(180/(180+720)=180/900=0.2000\). Its precision is \(90/(90+180)=90/270=0.3333\). It flags \(90+180=270\) people, or \(270/1{,}000=0.2700\) of the cohort.

For Classifier B, \(TN=900-90=810\). Its false positive rate is \(90/(90+810)=90/900=0.1000\). Its precision is \(90/(90+90)=90/180=0.5000\). It flags \(90+90=180\) people, or \(180/1{,}000=0.1800\) of the cohort.

Check. Each classifier identifies 90 of the 100 actual positives, so both have sensitivity 0.9000. Classifier B produces 90 rather than 180 false positives, and its flagged count is 90 lower. Its precision is consequently higher. The common mistake avoided is comparing the thresholds 0.42 and 0.67 as though one were inherently better; the relevant comparison is their performance at the matched sensitivity.

Worked Example: Same Operating Rates, Different Prevalence

A hypothetical classifier is operated at a threshold with sensitivity 0.80 and false positive rate 0.10. Compare its precision and workload in two separate cohorts of 5,000 people: Cohort L has 2% prevalence, while Cohort H has 20% prevalence.

Method. Calculate the confusion counts from each cohort's prevalence, then use the observed counts to find precision and workload. The sensitivity and false positive rate are the same in both settings.

In Cohort L, the actual positive count is \(5{,}000(0.02)=100\), leaving 4,900 actual negatives. The true positives are \(100(0.80)=80\), so \(FN=100-80=20\). The false positives are \(4{,}900(0.10)=490\), so \(TN=4{,}900-490=4{,}410\). Precision is \(80/(80+490)=80/570=0.1404\) to four decimals. The classifier flags \(80+490=570\) people, or \(570/5{,}000=0.1140\) of the cohort.

In Cohort H, the actual positive count is \(5{,}000(0.20)=1{,}000\), leaving 4,000 actual negatives. The true positives are \(1{,}000(0.80)=800\), so \(FN=200\). The false positives are \(4{,}000(0.10)=400\), so \(TN=4{,}000-400=3{,}600\). Precision is \(800/(800+400)=800/1{,}200=0.6667\) to four decimals. The flagged workload is 1,200 people, or \(1{,}200/5{,}000=0.2400\) of the cohort.

Check. In each cohort, \(TP/(TP+FN)=0.80\) and \(FP/(FP+TN)=0.10\). Yet precision rises from 0.1404 to 0.6667 when prevalence rises. The common mistake avoided is assuming that matched sensitivity alone determines precision; the number of actual positives relative to negatives also matters.

Worked Example: Choosing Thresholds From Score Groups

A validation sample contains 20 actual positives and 80 actual negatives. For Classifier A, the score groups are: at least 0.8, containing 12 positives and 10 negatives; 0.7 to below 0.8, containing 4 positives and 10 negatives; 0.5 to below 0.7, containing 3 positives and 20 negatives; and below 0.5, containing 1 positive and 40 negatives. For Classifier B, the groups are: at least 0.9, containing 10 positives and 5 negatives; 0.6 to below 0.9, containing 6 positives and 7 negatives; 0.4 to below 0.6, containing 3 positives and 18 negatives; and below 0.4, containing 1 positive and 50 negatives. Compare the classifiers at sensitivity 0.80.

Method. Add score groups from the highest scores downward until 16 of the 20 positives are included, then calculate the negative counts flagged at those thresholds.

For Classifier A, the group at least 0.8 contains 12 positives, which gives sensitivity \(12/20=0.60\). Including the next group, at least 0.7, gives \(12+4=16\) true positives. Thus \(FN=20-16=4\) and sensitivity is \(16/20=0.80\). The number of false positives is \(10+10=20\), so \(TN=80-20=60\). The false positive rate is \(20/80=0.2500\), precision is \(16/(16+20)=16/36=0.4444\), and workload is 36 cases.

For Classifier B, the group at least 0.9 contains 10 positives, giving sensitivity \(10/20=0.50\). Including the next group, at least 0.6, gives \(10+6=16\) true positives. Thus \(FN=4\) and sensitivity is \(16/20=0.80\). The false positive count is \(5+7=12\), so \(TN=80-12=68\). The false positive rate is \(12/80=0.1500\), precision is \(16/(16+12)=16/28=0.5714\), and workload is 28 cases.

Check. Both operating points identify 16 of the 20 positives. Classifier B has 8 fewer false positives and flags 8 fewer cases, with higher precision. The common mistake avoided is using the same numerical threshold for both classifiers. The score scales differ, so each threshold was selected by the required sensitivity instead.

Thresholds, Ties, and Interpretation

The matched-sensitivity comparison can also be understood as comparing points at the same vertical coordinate on a receiver operating characteristic (ROC) plot: sensitivity is the vertical axis, and false positive rate is the horizontal axis. At a fixed sensitivity, the point farther left has the lower false positive rate. If the classifiers are evaluated on the same population, it will also have fewer false positives at that sensitivity.

With a finite sample, scores may be tied. A threshold rule that includes all cases tied at a score can cause sensitivity to jump over a desired target. For example, if sensitivity increases from 0.75 to 0.85 when a tied score group is included, a deterministic threshold may not achieve exactly 0.80. Do not silently report 0.80 as observed sensitivity. Instead, report the attainable operating points, or state an explicit randomized rule if a fraction of tied cases is assigned positive. Interpolating between ROC points is not the same as applying an ordinary deterministic threshold.

A lower false positive rate at matched sensitivity is evidence of fewer false alarms on the evaluated population, not a guarantee that every flagged case is correct. Precision can be low when positives are rare, even if sensitivity is high. Workload should therefore be reported in a form that fits the decision: as a count for a specified cohort or as a fraction of cases. If the population size or prevalence changes, counts and precision may change even when sensitivity and false positive rate stay fixed.

Key takeaway: State the target sensitivity, the threshold used for each classifier, and the resulting false positive rate, precision, and flagged workload. Matched sensitivity holds the positive-case detection rate constant; it does not hold false alarms, precision, or workload constant.

Check Your Understanding

For each problem, compare the stated operating points using the relevant counts and rates. Give probabilities and rates to four decimals where appropriate.

A cohort has 600 people, of whom 120 are actual positives. A classifier has sensitivity 0.75 and false positive rate 0.10. Find \(TP\), \(FN\), \(FP\), \(TN\), precision, and the number flagged. (Answer: \(TP=90\), \(FN=30\), \(FP=48\), \(TN=432\), precision \(=0.6522\), flagged \(=138\).)
A hypothetical population of 2,500 has prevalence 0.04. At a selected threshold, sensitivity is 0.85 and false positive rate is 0.15. Calculate precision and the flagged fraction. (Answer: \(TP=85\), \(FN=15\), \(FP=360\), \(TN=2{,}040\), precision \(=0.1910\), flagged fraction \(=0.1780\).)
On the same cohort of 2,000 people, prevalence is 0.05. Classifier A and Classifier B both have sensitivity 0.90; their false positive rates are 0.08 and 0.04, respectively. Find each classifier's false positives, precision, and workload, and identify which has fewer flags. (Answer: A has \(FP=152\), precision \(=0.3719\), and workload \(=242\); B has \(FP=76\), precision \(=0.5422\), and workload \(=166\); B has fewer flags.)
A validation sample has 30 positives and 70 negatives. At selected thresholds, Classifier A has \(TP=24\), \(FP=21\), and Classifier B has \(TP=24\), \(FP=14\). Verify matched sensitivity and compare precision and workload. (Answer: Both have sensitivity \(=0.8000\); A has precision \(=0.5333\) and workload \(=45\); B has precision \(=0.6316\) and workload \(=38\).)
A population of 10,000 is assessed at sensitivity 0.95 and false positive rate 0.05 in two settings. One has prevalence 0.01 and the other 0.10. Calculate precision and flagged count in each setting. (Answer: At prevalence 0.01, precision \(=0.1610\) and flagged count \(=590\); at prevalence 0.10, precision \(=0.6786\) and flagged count \(=1{,}400\).)

What You'll Learn