Prioritizing Sensitivity or Specificity

Why the Priority Depends on the Decision

A classification model sorts cases into positive and negative groups. The positive class is the event the model is intended to detect: for example, a claim needing review, a machine fault, or a health condition. A false negative is an actual positive case classified as negative; a false positive is an actual negative case classified as positive. Choosing whether to prioritize sensitivity or specificity therefore starts with the consequences of those two errors, not with a general preference for one metric.

If missing a true case is especially harmful, the decision process may prioritize finding as many true cases as possible. If unnecessary interventions are particularly costly or disruptive, it may prioritize correctly leaving true negative cases unflagged. These priorities often conflict: a policy that flags more cases may catch more true positives, but it may also create more false alarms. The appropriate balance depends on the purpose and constraints of the task.

Definition: Sensitivity is the proportion of actual positive cases classified as positive. It is also called the true positive rate. Specificity is the proportion of actual negative cases classified as negative, also called the true negative rate. If TP, FN, TN, and FP denote the numbers of true positives, false negatives, true negatives, and false positives, respectively, then

\text{Sensitivity}=\frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FN}}, \qquad \text{Specificity}=\frac{\mathrm{TN}}{\mathrm{TN}+\mathrm{FP}}.

The denominators are different. Sensitivity is calculated among actual positive cases; specificity is calculated among actual negative cases. Neither metric, by itself, tells how many cases will be sent for action, and neither assigns a dollar value to errors. They describe distinct aspects of classification performance.

From Error Consequences to a Metric Priority

Before choosing a priority, define the positive class and the action attached to each classification. Then ask what happens when the model misses an actual positive and what happens when it flags an actual negative. The consequences may include financial cost, health or safety risk, customer burden, staff workload, or legal and ethical concerns. They should be stated in the context of the decision, not inferred from the metric names.

A task that emphasizes sensitivity accepts particular concern about false negatives. A task that emphasizes specificity accepts particular concern about false positives. This does not mean the other metric is irrelevant: a screening process that catches almost every positive case may still be impractical if it flags nearly all negative cases, and a process that seldom raises false alarms may fail if it misses too many important cases.

Decision principle: Prioritize sensitivity when the consequences of missing actual positive cases dominate, subject to acceptable false-alarm burden. Prioritize specificity when unnecessary actions on actual negative cases dominate, subject to acceptable missed-case risk. If both consequences matter, compare the policies using an explicit decision criterion rather than declaring one metric universally best.

A cost calculation can make that comparison explicit. Suppose each false negative has cost $C_{\mathrm{FN}}$, each false positive has cost $C_{\mathrm{FP}}$, and correct classifications have no incremental cost in the comparison. If the positive prevalence is $\pi$, then the expected error cost per case under a policy with sensitivity $s$ and specificity $c$ is

C_{\mathrm{FN}}\pi(1-s)+C_{\mathrm{FP}}(1-\pi)(1-c).

This expression assumes that the stated error costs apply to each relevant case, that the prevalence describes the population where the policy will be used, and that there are no other policy costs to include. If correct classifications, follow-up actions, or capacity constraints also have material costs, they must be added to the decision analysis. The formula does not make the costs objective facts: they must be estimated or agreed for the particular use.

Why Expected Error Cost Has This Form

The formula follows by counting the two kinds of error separately. Among a population of $N$ cases, approximately $N\pi$ are actual positives. Sensitivity $s$ means the fraction of those cases correctly detected is $s$, so the fraction missed is $1-s$. The expected number of false negatives is therefore $N\pi(1-s)$.

The remaining $N(1-\pi)$ cases are actual negatives. Specificity $c$ means the fraction correctly left unflagged is $c$, so the false-positive fraction is $1-c$. Thus the expected number of false positives is $N(1-\pi)(1-c)$. Multiplying each error count by its per-error cost and adding gives total expected error cost. Dividing by $N$ gives the per-case expression above.

\begin{aligned} \text{Expected total error cost} &= C_{\mathrm{FN}}N\pi(1-s) + C_{\mathrm{FP}}N(1-\pi)(1-c),\\ \text{Expected cost per case} &= C_{\mathrm{FN}}\pi(1-s) + C_{\mathrm{FP}}(1-\pi)(1-c). \end{aligned}

The derivation also explains why the two metrics cannot be compared without context. A one-percentage-point change in sensitivity applies to the positive cases, while a one-percentage-point change in specificity applies to the negative cases. The number of cases in each group and the consequences of each error both matter.

Worked Comparisons

Worked Example: Fraud Screening With Unequal Error Costs

A hypothetical insurer evaluates two policies on the same 2,000 accounts. There are 100 fraudulent accounts and 1,900 nonfraudulent accounts. Policy A identifies 90 fraudulent accounts and flags 380 nonfraudulent accounts. Policy B identifies 70 fraudulent accounts and flags 95 nonfraudulent accounts. Suppose a missed fraudulent account costs $500 on average and an unnecessary investigation costs $40. Which policy has higher sensitivity, which has higher specificity, and which has lower total error cost?

Method. Build the confusion counts from the problem, calculate each rate with its correct denominator, and then compare costs. This avoids treating the larger raw count of negative cases as the sensitivity denominator.

For Policy A, TP $=90$, so FN $=100-90=10$. FP $=380$, so TN $=1{,}900-380=1{,}520$. Its rates are

\text{Sensitivity}_A=\frac{90}{90+10}=0.90, \qquad \text{Specificity}_A=\frac{1{,}520}{1{,}520+380}=0.80.

For Policy B, TP $=70$, so FN $=100-70=30$. FP $=95$, so TN $=1{,}900-95=1{,}805$. Its rates are

\text{Sensitivity}_B=\frac{70}{70+30}=0.70, \qquad \text{Specificity}_B=\frac{1{,}805}{1{,}805+95}=0.95.

Policy A has higher sensitivity, while Policy B has higher specificity. Policy A’s error cost is $10(\$500)+380(\$40)=\$5{,}000+\$15{,}200=\$20{,}200$. Policy B’s error cost is $30(\$500)+95(\$40)=\$15{,}000+\$3{,}800=\$18{,}800$. Under the stated costs, Policy B has the lower total error cost by $\$1{,}400$, despite its lower sensitivity.

Check. Each policy’s false negatives plus true positives equals 100, and false positives plus true negatives equals 1,900. The comparison uses error counts from the same population and cost assumptions. A common mistake would be to choose A solely because catching more fraud sounds desirable; the calculation shows why the false-alarm consequences must also be considered.

Worked Example: When Missing a Positive Case Dominates

A hypothetical screening program evaluates 10,000 people, of whom 200 have a condition requiring prompt assessment and 9,800 do not. Policy H has 95% sensitivity and 90% specificity. Policy S has 80% sensitivity and 98% specificity. Suppose a missed case has an estimated consequence cost of $10,000 and an unnecessary assessment costs $100. Compare the error costs and state which policy is preferred under these assumptions.

Method. Convert each rate into counts within its actual-case group, then apply the stated costs. This avoids comparing percentages without accounting for the different numbers of positive and negative cases.

For Policy H, TP $=0.95(200)=190$, so FN $=200-190=10$. TN $=0.90(9{,}800)=8{,}820$, so FP $=9{,}800-8{,}820=980$. Its total error cost is

10(\$10{,}000)+980(\$100) =\$100{,}000+\$98{,}000 =\$198{,}000.

For Policy S, TP $=0.80(200)=160$, so FN $=200-160=40$. TN $=0.98(9{,}800)=9{,}604$, so FP $=9{,}800-9{,}604=196$. Its total error cost is

40(\$10{,}000)+196(\$100) =\$400{,}000+\$19{,}600 =\$419{,}600.

Policy H is preferred under the stated assumptions, with an error cost $\$221{,}600$ lower than Policy S. It has more false positives, but it avoids 30 additional false negatives, and each missed case carries a much larger assumed cost. This supports emphasizing sensitivity in this scenario; it does not establish that sensitivity should always be prioritized in screening.

Check. The positive-group counts total 200 for each policy, and the negative-group counts total 9,800. A common mistake would be to focus on Policy S’s 98% specificity without considering that its 40 missed cases are costly under the assumptions.

Worked Example: A Required Sensitivity Floor

A hypothetical safety program assesses 1,000 commercial boilers. Of these, 120 are unsafe and 880 are safe. Policy C detects 114 unsafe boilers and incorrectly flags 176 safe boilers. Policy D detects 102 unsafe boilers and incorrectly flags 44 safe boilers. Program rules require sensitivity of at least 95%. If a missed unsafe boiler has an estimated consequence cost of $5,000 and an unnecessary inspection costs $150, assess both policies.

Method. First check the mandatory sensitivity requirement, then compare error costs for any policy that satisfies it. This avoids choosing a low-cost policy that violates a non-negotiable safety condition.

For Policy C, FN $=120-114=6$, and TN $=880-176=704$. Therefore,

\text{Sensitivity}_C=\frac{114}{120}=0.95, \qquad \text{Specificity}_C=\frac{704}{880}=0.80.

Policy C meets the required sensitivity exactly. Its error cost is $6(\$5{,}000)+176(\$150)=\$30{,}000+\$26{,}400=\$56{,}400$.

For Policy D, FN $=120-102=18$, and TN $=880-44=836$. Therefore,

\text{Sensitivity}_D=\frac{102}{120}=0.85, \qquad \text{Specificity}_D=\frac{836}{880}=0.95.

Policy D fails the required sensitivity. Its error cost is $18(\$5{,}000)+44(\$150)=\$90{,}000+\$6{,}600=\$96{,}600$. Policy C is both the only policy meeting the stated requirement and the lower-cost policy. In a different scenario, a policy could satisfy a mandatory floor yet have a higher estimated cost; a hard requirement and a cost preference are distinct decision rules.

Check. The denominators are 120 unsafe boilers for sensitivity and 880 safe boilers for specificity. A common mistake would be to prioritize D merely because its specificity is higher, ignoring the explicit safety requirement.

Common Traps and Practical Checks

Using the wrong denominator. Sensitivity is calculated among actual positives, not all cases. Specificity is calculated among actual negatives, not all cases.
Treating a metric as a decision rule. A high sensitivity or specificity does not by itself show that a policy is best. The importance and number of each kind of error matter.
Confusing priority with exclusivity. Prioritizing sensitivity does not mean ignoring false alarms. The false-positive burden still needs to be acceptable for the action to be workable.
Ignoring a required floor. If policy or safety rules require a minimum sensitivity or specificity, check that requirement before choosing among policies on another basis.
Using costs from the wrong context. An error cost depends on the action and setting. If costs, prevalence, or consequences differ in deployment from the evaluation data, the comparison may not transfer.

In practice, a classification decision may use a threshold to change how many cases are flagged. Changing that threshold can alter both sensitivity and specificity, so the resulting policies should be evaluated against the same population, action, and decision assumptions. The goal is not to maximize one metric in isolation; it is to meet the task’s requirements while managing the consequences of both kinds of error.

This approach is aligned with the SOA Exam PA (Predictive Analytics) syllabus: define the event and its consequences, interpret the metrics using their correct denominators, and justify the priority in the context of the decision.

Check Your Understanding

For each hypothetical decision, identify the requested metric or compare the policies using the stated information.

A device flags 72 of 90 actual faults and correctly leaves 760 of 800 nonfaulty devices unflagged. Calculate sensitivity and specificity. (Answer: Sensitivity $=72/90=0.8000$; specificity $=760/800=0.9500$.)
A screening task has 50 actual positives. Policy A has sensitivity 0.92; Policy B has sensitivity 0.84. Which policy misses fewer actual positives, and by how many? (Answer: A misses 4 and B misses 8, so A misses 4 fewer.)
A hypothetical review sample contains 100 actual positive claims and 900 actual negative claims. A policy has sensitivity 0.90 and specificity 0.95. Find its false-negative and false-positive counts. (Answer: FN $=10$; FP $=45$.)
In a population of 1,000 cases, 100 are positive and 900 are negative. Policy A has sensitivity 0.90 and specificity 0.80; Policy B has sensitivity 0.80 and specificity 0.95. A false negative costs $300 and a false positive costs $20. Which policy has lower error cost? (Answer: A costs $10(\$300)+180(\$20)=\$6{,}600$; B costs $20(\$300)+45(\$20)=\$6{,}900$; Policy A is lower by $300.)
A safety policy requires sensitivity of at least 0.97. Policy X has sensitivity 0.96 and specificity 0.99; Policy Y has sensitivity 0.98 and specificity 0.90. Which policy meets the stated requirement, and can specificity alone justify choosing X? (Answer: Only Y meets the requirement; X’s higher specificity does not override the required sensitivity floor.)

What You'll Learn