Why the Measure Must Match the Decision
A classification model assigns a score to each case, and a decision rule converts that score into a predicted class. For a threshold \(t\), a common rule predicts positive when the score is at least \(t\), and negative otherwise. Once the threshold is fixed, the resulting class labels can be compared with the observed outcomes. The resulting measures answer different questions: how many actual positives were found, how many actual negatives were correctly rejected, how many predicted positives were truly positive, and what fraction of all classifications were correct.
These distinctions matter in practice. A team investigating flagged insurance claims may ask what fraction of flagged claims are genuinely problematic; a screening process may ask what fraction of all problematic cases were flagged. Those are different questions, so they call for different measures. A metric is useful when its numerator and denominator correspond to the operational question—not merely because it is familiar or easy to calculate.
| Predicted positive | Predicted negative | |
|---|---|---|
| Actual positive | TP | FN |
| Actual negative | FP | TN |
Four Measures, Four Denominators
Each measure is a ratio. Before calculating it, identify the group represented by its denominator. This is the simplest way to avoid swapping measures or interpreting a correct calculation incorrectly.
Sensitivity is also called the true positive rate; specificity is also called the true negative rate. Sensitivity asks, “Of the actual positive cases, what proportion did the rule find?” Specificity asks, “Of the actual negative cases, what proportion did the rule correctly classify as negative?” Precision asks, “Of the cases the rule flagged positive, what proportion were actually positive?” Accuracy asks, “Of all cases, what proportion received the correct class?”
The denominators make the distinction concrete. Sensitivity and specificity condition on the actual class. Precision conditions on the predicted class. Accuracy does not condition on either class; it counts correct predictions across the whole evaluation set. Thus, a report should name the measure and explain its denominator in context, especially when a stakeholder may otherwise read “90% correct” as a statement about a different group.
Worked Calculations and Operational Interpretation
Worked Example: Selecting Measures for a Claims Review Queue
A hypothetical insurer tests a fixed classification threshold on 200 previously resolved claims. Twenty claims were actually positive for a review-worthy issue, and 180 were actual negatives. The rule flags 38 claims: 18 of those are actual positives, and 20 are actual negatives. Calculate the four measures and identify which one answers the question, “What share of flagged claims truly had the issue?”
Method. Complete the confusion matrix first. Of the 20 actual positives, 18 were flagged, so \(TP=18\) and \(FN=20-18=2\). Of the 180 actual negatives, 20 were flagged, so \(FP=20\) and \(TN=180-20=160\). The row totals are 20 and 180; the column totals are 38 predicted positives and \(2+160=162\) predicted negatives. These sum to 200.
Sensitivity: \(18/(18+2)=18/20=0.9000\). The rule found 90.00% of actual issue-positive claims.
Specificity: \(160/(160+20)=160/180=0.8889\). The rule correctly classified 88.89% of actual negatives.
Precision: \(18/(18+20)=18/38=0.4737\). Of the 38 flagged claims, 47.37% were actually positive. Precision answers the question in the prompt.
Accuracy: \((18+160)/200=178/200=0.8900\). Thus 89.00% of all classifications were correct.
Check and common mistake. The four cells total \(18+2+20+160=200\), consistent with the evaluation-set size. A common mistake is to call \(18/20\) precision because it describes “positive” cases. Its denominator is actual positives, so it is sensitivity. Precision uses the 38 predicted positives as its denominator.
Worked Example: Matching a Measure to a Screening Question
A hypothetical equipment-monitoring team evaluates a threshold for identifying components that will require urgent repair. Among 300 components, 30 actually required urgent repair and 270 did not. The rule flags 48 components, including 24 that required repair. Calculate sensitivity, specificity, precision, and accuracy. Then identify the measure for the question, “What proportion of components that needed urgent repair were flagged?”
Method. The flagged group contains 24 actual positives, so \(TP=24\) and \(FP=48-24=24\). There were \(30-24=6\) actual positives not flagged, so \(FN=6\). There were \(270-24=246\) actual negatives not flagged, so \(TN=246\). The counts check: \(24+6+24+246=300\).
Sensitivity: \(24/(24+6)=24/30=0.8000\). This is the required measure: 80.00% of components needing urgent repair were flagged.
Specificity: \(246/(246+24)=246/270=0.9111\).
Precision: \(24/(24+24)=24/48=0.5000\). Half of the flagged components actually needed urgent repair.
Accuracy: \((24+246)/300=270/300=0.9000\).
Check and common mistake. The sensitivity denominator is the 30 actual repair cases, whereas the precision denominator is the 48 flagged components. Using \(24/48\) for the prompt would answer the different question of what share of flags were correct. The figures can be checked against the known totals: 30 actual positives and 270 actual negatives.
Why Accuracy Is a Weighted Average
When both actual classes occur in the evaluation set, accuracy can be expressed using sensitivity and specificity. Let \(n\) be the total number of cases, \(n_+\) the number of actual positives, and \(n_-\) the number of actual negatives. Define the observed positive prevalence as \(\pi=n_+/n\), so \(1-\pi=n_-/n\). For this identity, both classes must be present: \(0<\pi<1\).
Derivation. The number classified correctly is \(TP+TN\). Since sensitivity is \(TP/n_+\), we have \(TP=n_+\times\text{Sensitivity}\). Similarly, since specificity is \(TN/n_-\), we have \(TN=n_-\times\text{Specificity}\). Substituting these into the definition of accuracy gives
The derivation works because the two groups partition the evaluation set, and each group contributes its share of correctly classified cases. The weights are the groups’ proportions, not equal weights unless the two groups happen to be equal in size. For the claims example, \(\pi=20/200=0.1\). The identity gives \(0.1(0.9000)+0.9(0.8889)=0.0900+0.8000=0.8900\), matching the direct accuracy calculation.
The condition \(0<\pi<1\) is essential. If there are no actual positives, sensitivity is undefined; if there are no actual negatives, specificity is undefined. Accuracy itself remains defined whenever the evaluation set is nonempty, but the weighted-average expression cannot be used as stated when either class is absent.
Worked Example: Handling an All-Negative Prediction Rule
A hypothetical maintenance rule is evaluated on 100 components: 12 actually required repair and 88 did not. At the selected threshold the rule predicts every component negative. Calculate the measures that are defined, identify any undefined measure, and explain what the accuracy does and does not tell the team.
Method. No cases are predicted positive, so \(TP=0\), \(FP=0\), \(FN=12\), and \(TN=88\). The counts add to \(100\).
Sensitivity: \(0/(0+12)=0/12=0.0000\). None of the actual repair cases were found.
Specificity: \(88/(88+0)=88/88=1.0000\). Every actual non-repair case was correctly classified.
Precision: \(0/(0+0)=0/0\), which is undefined. There are no predicted positives from which to calculate a proportion of flags that are correct. It is not correct to label precision as zero or as 100%.
Accuracy: \((0+88)/100=0.8800\). The rule is correct on 88% of this evaluation set, entirely because it correctly predicts the 88 actual negatives.
Check and common mistake. The observed prevalence is \(12/100=0.12\), and both actual classes occur, so the weighted identity applies: \(0.12(0)+0.88(1)=0.88\), agreeing with direct accuracy. The trap is to infer from 88% accuracy that the rule is effective at finding repairs; its sensitivity is zero.
Choose the Measure by Stating the Question
A practical way to select a measure is to translate the operational question into its population and outcome. If the question begins with “Of the actual positives,” sensitivity is the relevant fraction. If it begins with “Of the actual negatives,” specificity is the relevant fraction. If it begins with “Of the cases we flagged,” precision is the relevant fraction. If it asks for the proportion of all cases classified correctly, use accuracy.
State what counts as a positive classification, including the threshold that creates the predicted labels.
Is the question about actual positives, actual negatives, predicted positives, or all evaluated cases?
Use the appropriate confusion-matrix denominator, then report the proportion in the original operational context.
A metric does not by itself determine whether a model or decision rule is acceptable. It describes one aspect of performance at the chosen threshold. The operational definition of success, the consequences of errors, and any stated decision requirements determine which aspect should be emphasized. This is why a report should give the measure’s name, value, denominator in context, and the threshold or rule that produced the classifications.
Common Traps
- Swapping sensitivity and precision. Both involve true positives, but sensitivity divides by actual positives \((TP+FN)\), while precision divides by predicted positives \((TP+FP)\).
- Using accuracy as a substitute for every other measure. Accuracy is the fraction correct overall. It does not separately describe how well actual positives or actual negatives were classified.
- Reporting an undefined rate as zero. Sensitivity is undefined if there are no actual positives; specificity is undefined if there are no actual negatives; precision is undefined if there are no predicted positives. A zero numerator with a positive denominator gives a rate of zero; a zero denominator does not.
- Using the accuracy identity when a class is absent. The formula using \(\pi\), sensitivity, and specificity requires \(0<\pi<1\). If either actual class is missing, one of the component rates is undefined, even though accuracy may still be calculable.
- Reporting a measure without its question. “Precision is 50%” is less informative than “half of the flagged components actually required repair.”
Check Your Understanding
For each problem, identify the relevant confusion-matrix counts before calculating. Round proportions to four decimal places where needed.
- A hypothetical claims screen has \(TP=27\), \(FN=3\), \(FP=18\), and \(TN=112\). Calculate sensitivity, specificity, precision, and accuracy. (Answer: Sensitivity \(=0.9000\), specificity \(=0.8615\), precision \(=0.6000\), accuracy \(=0.8688\).)
- A component failure classifier is tested on 500 components. It correctly flags 32 of 40 components that failed and incorrectly flags 40 of the 460 that did not fail. Calculate the four measures. (Answer: Sensitivity \(=0.8000\), specificity \(=0.9130\), precision \(=0.4444\), accuracy \(=0.9040\).)
- For a hypothetical screening test, \(TP=45\), \(FN=30\), \(FP=15\), and \(TN=210\). A manager asks what fraction of cases flagged positive were truly positive. Which measure answers the question, and what is its value? Also calculate accuracy. (Answer: Precision \(=45/(45+15)=0.7500\); accuracy \(=(45+210)/300=0.8500\).)
- An evaluation set contains 40 actual positive cases and no actual negatives. The rule predicts 34 positive and 6 negative. Which of sensitivity, specificity, precision, and accuracy are defined, and what are their values when defined? (Answer: Sensitivity \(=34/40=0.8500\), precision \(=34/34=1.0000\), and accuracy \(=34/40=0.8500\) are defined; specificity is undefined because there are no actual negatives.)
- Two thresholds are evaluated on the same 200 cases, including 20 actual positives and 180 actual negatives. Threshold A has \(TP=18\), \(FN=2\), \(FP=54\), and \(TN=126\); threshold B has \(TP=14\), \(FN=6\), \(FP=18\), and \(TN=162\). If the stated requirement is to find at least 85% of actual positives, which threshold meets it? Give both sensitivities and explain why accuracy alone does not answer the requirement. (Answer: A has sensitivity \(18/20=0.9000\) and meets the requirement; B has sensitivity \(14/20=0.7000\) and does not. Their accuracies are \(144/200=0.7200\) and \(176/200=0.8800\), respectively, but accuracy is not the stated requirement.)