Why an Operating Threshold Requires More Than One Metric
A screening score is useful only after a decision rule turns it into an action. For a chosen threshold, people scoring at or above the threshold may be referred for further assessment, while those scoring below it are not. Raising the threshold will usually reduce false-positive referrals but can also miss more people who have the condition. Lowering it can improve case detection while creating more false alarms.
When these errors have unequal consequences, neither accuracy nor any single classification metric is enough to choose a threshold. A defensible recommendation should show the confusion-matrix counts, translate errors into costs using stated assumptions, and check any policy or clinical priorities. The calculations below use hypothetical screening programs; their counts and costs are illustrative, not industry data.
These metrics answer different questions. Sensitivity is the fraction of observed cases detected; specificity is the fraction of observed non-cases left unflagged; positive predictive value is the fraction of flagged people who are cases. A threshold comparison must use the same outcome definition, cohort, and score convention throughout. In this tutorial, a score exactly equal to the threshold is classified as positive.
From Error Counts to Error Costs
Suppose each false negative has cost \(C_{FN}\) and each false positive has cost \(C_{FP}\), both stated in dollars per error. Assume these costs are constant across individuals, correct classifications have no incremental error cost, and there are no other threshold-dependent costs. For a fixed observed cohort, total error cost is the number of each type of error multiplied by its unit cost.
If the cohort contains \(N\) people, dividing total cost by \(N\) gives the observed error cost per screened person. This is an average over this cohort, not automatically a forecast for a different population. It depends on the cohort's prevalence as well as the threshold's sensitivity and specificity.
To see why this expression holds, let the observed positive fraction be \((TP+FN)/N\). Among those observed cases, the fraction missed is \(FN/(TP+FN)=1-\text{sensitivity}\); multiplying these quantities gives \(FN/N\). Similarly, the observed non-case fraction is \((TN+FP)/N\), and the fraction incorrectly flagged is \(FP/(TN+FP)=1-\text{specificity}\), giving \(FP/N\). Weighting these two error fractions by their respective costs produces the expression above. It does not apply unchanged if the error costs vary by person, if correct classifications have relevant costs, or if the cohort prevalence is not appropriate for the population where the screening rule will be used.
Worked Examples
In each example, first construct the confusion matrix at each threshold. Then calculate the relevant metrics and costs from those counts. Score bands below are mutually exclusive, so their counts can be added to find the results for any stated threshold.
Worked Example: Comparing Three Thresholds in a Screening Cohort
A hypothetical screening program evaluates 1,000 adults. There are 100 observed cases and 900 observed non-cases. The score-band counts are shown below. A false negative is assigned a cost of $800, and a false positive a cost of $120. Compare thresholds 0.80, 0.50, and 0.25 using sensitivity, specificity, positive predictive value, and total error cost.
Method. For a threshold, add all score bands at or above it. For example, at 0.50 the flagged groups are “at least 0.80” and “0.50 to below 0.80.” The table gives the cumulative flagged counts directly.
| Threshold | TP | FP |
|---|---|---|
| 0.80 | 30 | 20 |
| 0.50 | 70 | 110 |
| 0.25 | 90 | 360 |
At threshold 0.80, \(FN=100-30=70\) and \(TN=900-20=880\). Thus sensitivity is \(30/100=0.3000\), specificity is \(880/900=0.9778\), and positive predictive value is \(30/(30+20)=0.6000\). Total cost is \(70($800)+20($120)=$56,000+$2,400=$58,400\).
At threshold 0.50, \(FN=100-70=30\) and \(TN=900-110=790\). Sensitivity is \(70/100=0.7000\), specificity is \(790/900=0.8778\), and positive predictive value is \(70/(70+110)=70/180=0.3889\). Total cost is \(30($800)+110($120)=$24,000+$13,200=$37,200\).
At threshold 0.25, \(FN=100-90=10\) and \(TN=900-360=540\). Sensitivity is \(90/100=0.9000\), specificity is \(540/900=0.6000\), and positive predictive value is \(90/(90+360)=0.2000\). Total cost is \(10($800)+360($120)=$8,000+$43,200=$51,200\).
The observed minimum cost among these three options is at threshold 0.50: $37,200, or \(37,200/1,000=$37.20\) per screened adult. This does not mean threshold 0.50 is best for every purpose; the conclusion follows from the specified costs and cohort. The common mistake avoided here is treating the largest sensitivity as automatically best. Threshold 0.25 detects more cases, but its additional false positives make its total cost higher under these assumptions.
Check: For each row, \(TP+FN=100\) and \(FP+TN=900\). The calculated counts therefore reconcile with the cohort totals.
Worked Example: When the Lower Threshold Has the Lowest Cost
A hypothetical program screens 500 applicants, of whom 100 are observed cases and 400 are non-cases. At thresholds 0.75, 0.45, and 0.25, the cumulative flagged counts \((TP,FP)\) are respectively \((50,10)\), \((85,70)\), and \((95,180)\). Each false negative costs $2,000 and each false positive costs $150. Find the total and per-person error cost at each threshold, then identify the least-cost option.
Method. Obtain the missed cases by subtracting TP from 100, and obtain unflagged non-cases by subtracting FP from 400. Apply the unit error costs to FN and FP; correct decisions do not add cost under the stated model.
At 0.75, \(FN=100-50=50\) and \(TN=400-10=390\). Sensitivity is \(50/100=0.5000\), and specificity is \(390/400=0.9750\). Total cost is \(50($2,000)+10($150)=$100,000+$1,500=$101,500\). Dividing by 500 gives $203.00 per applicant.
At 0.45, \(FN=100-85=15\) and \(TN=400-70=330\). Sensitivity is \(85/100=0.8500\), and specificity is \(330/400=0.8250\). Total cost is \(15($2,000)+70($150)=$30,000+$10,500=$40,500\). The per-person cost is \(40,500/500=$81.00\).
At 0.25, \(FN=100-95=5\) and \(TN=400-180=220\). Sensitivity is \(95/100=0.9500\), and specificity is \(220/400=0.5500\). Total cost is \(5($2,000)+180($150)=$10,000+$27,000=$37,000\). The per-person cost is \(37,000/500=$74.00\).
Threshold 0.25 has the lowest observed cost, $37,000, which is $3,500 less than the cost at 0.45. Its extra false positives cost \(110($150)=$16,500\), while its reduction of 10 false negatives saves \(10($2,000)=$20,000\); the net saving is $3,500. This incremental comparison makes the result transparent. A common mistake is to compare only the change in false positives and overlook the larger saving from avoided false negatives.
Check: At threshold 0.25, \(TP+FN=95+5=100\) and \(FP+TN=180+220=400\). Also, $74.00 multiplied by 500 equals the total cost of $37,000.
Worked Example: Choosing a Threshold Under a Sensitivity Requirement
A hypothetical screening service reviews 800 people: 80 are observed cases and 720 are non-cases. A policy requires sensitivity of at least 0.85. Candidate thresholds have cumulative counts \((TP,FP)\) as follows: at 0.80, \((32,18)\); at 0.50, \((60,90)\); and at 0.30, \((72,270)\). The error costs are $1,500 per false negative and $100 per false positive. Which candidate is eligible under the policy, and what is its cost?
Method. First test the sensitivity constraint, rather than selecting the cheapest option without regard to policy. Then report the costs so the trade-off is clear. If multiple candidates satisfy the constraint, compare costs among those eligible candidates.
At 0.80, \(FN=80-32=48\), so sensitivity is \(32/80=0.4000\). Its specificity is \((720-18)/720=702/720=0.9750\), and its cost is \(48($1,500)+18($100)=$72,000+$1,800=$73,800\).
At 0.50, \(FN=80-60=20\), so sensitivity is \(60/80=0.7500\). Its specificity is \((720-90)/720=630/720=0.8750\), and its cost is \(20($1,500)+90($100)=$30,000+$9,000=$39,000\).
At 0.30, \(FN=80-72=8\), so sensitivity is \(72/80=0.9000\). Its specificity is \((720-270)/720=450/720=0.6250\), and its cost is \(8($1,500)+270($100)=$12,000+$27,000=$39,000\).
Only threshold 0.30 meets the required sensitivity of at least 0.85, so it is the selected operating point, with total cost $39,000. Although threshold 0.50 has the same cost, its sensitivity of 0.7500 fails the stated requirement. The common mistake avoided is choosing a cost tie-breaker before applying the eligibility condition. If a policy constraint is binding, an option outside the permitted set is not a valid recommendation.
Check: At 0.30, \(72+8=80\) observed cases and \(270+450=720\) observed non-cases. Its sensitivity exceeds the minimum by \(0.9000-0.8500=0.0500\).
Making the Recommendation Defensible
The calculations support a recommendation only relative to the assumptions and priorities stated. A sound short explanation should identify the selected threshold, cite its important metrics and error cost, and explain why the alternatives are not preferred. If a minimum sensitivity or specificity is mandatory, apply that constraint before minimizing cost. If there is no hard constraint, compare total costs under the specified error-cost model.
Keep the following qualifications in view:
- Costs must match the decision. A cost per error should represent the consequences relevant to the screening decision. If follow-up tests, staff capacity, or treatment costs vary with the threshold, include them if the task supplies them; the simple formula above omits them.
- Prevalence matters. The same sensitivity and specificity can lead to different numbers of false positives and false negatives in cohorts with different case prevalence. Applying observed costs to another population requires an appropriate prevalence and performance assumptions.
- Use counts before rounding metrics. Rounding sensitivity or specificity early can distort a cost comparison, especially when thresholds have similar costs. Calculate costs from the integer confusion-matrix counts.
- A computed minimum is not automatically a policy decision. A decision-maker may prioritize a required detection rate or other stated objective over the lowest modeled cost. Make that priority explicit rather than presenting the minimum-cost threshold as universally optimal.
These steps are aligned with the SOA Exam PA (Predictive Analytics) syllabus: show how threshold-specific classification results connect to a decision, and state the assumptions that make the comparison meaningful.
Check Your Understanding
For each hypothetical problem, use the stated threshold convention and costs. Answers are provided so you can check the calculation.
- A cohort has 50 cases and 150 non-cases. At one threshold, \(TP=42\), \(FP=18\), \(FN=8\), and \(TN=132\). Calculate sensitivity, specificity, and positive predictive value. (Answer: sensitivity \(=42/50=0.8400\); specificity \(=132/150=0.8800\); positive predictive value \(=42/60=0.7000\).)
- In a cohort of 200 people, \(TP=36\), \(FP=24\), \(FN=14\), and \(TN=126\). If a false negative costs $500 and a false positive costs $40, calculate total and per-person error cost. (Answer: total \(=14($500)+24($40)=$7,000+$960=$7,960\); per person \(=$7,960/200=$39.80\).)
- A screening cohort has 40 cases and 360 non-cases. Threshold A produces \(FN=8, FP=45\), and threshold B produces \(FN=4, FP=90\). If a false negative costs $400 and a false positive costs $30, which threshold has lower cost, and by how much? (Answer: A costs \(8($400)+45($30)=$4,550\); B costs \(4($400)+90($30)=$4,300\); B is lower by $250.)
- There are 100 cases and 400 non-cases. Three progressively lower thresholds produce TP counts of 45, 85, and 95, with FP counts of 20, 100, and 220, respectively. A policy requires sensitivity of at least 0.90. If a false negative costs $1,000 and a false positive costs $100, identify the eligible threshold and its cost. (Answer: only the third threshold qualifies, with sensitivity \(95/100=0.9500\); \(FN=5\), so cost \(=5($1,000)+220($100)=$27,000\).)
- A cohort has 80 cases and 920 non-cases. A higher threshold gives \(TP=64\), \(FP=46\); a lower threshold gives \(TP=72\), \(FP=184\). A false negative costs $750 and a false positive costs $50. Which threshold has lower total cost, and what is the difference? (Answer: higher-threshold cost \(=16($750)+46($50)=$14,300\); lower-threshold cost \(=8($750)+184($50)=$15,200\); the higher threshold is lower by $900.)