What a Confusion Matrix Reports
A classification score does not, by itself, say how many cases would be flagged or how those flags would compare with actual outcomes. A confusion matrix makes this concrete for one specified classification threshold: it cross-tabulates the observed outcome against the model’s threshold-based classification. For a decision-maker, it connects a rule such as “flag scores at or above 0.40” to counts of correct and incorrect classifications.
The positive class must be named in context. For example, a hypothetical insurer might define “positive” as a policy that will have a claim in the next year. A negative classification then means the model did not flag that policy at the chosen threshold; it does not prove that no claim will occur. The matrix describes results on a defined, labeled evaluation set, not certainty about future cases.
The matrix is easiest to read when actual outcomes label the rows and model classifications label the columns. State that orientation and the event represented by “positive” so that a reader does not have to infer either one.
| Actual outcome | Classified positive | Classified negative | Actual total |
|---|---|---|---|
| Positive | True positive (TP) | False negative (FN) | TP + FN |
| Negative | False positive (FP) | True negative (TN) | FP + TN |
| Classified total | TP + FP | FN + TN | N |
Four Rates That Give the Counts Meaning
The matrix contains counts; the rates summarize different conditional proportions. Each rate has a specific denominator. Reporting the name, formula, and denominator helps prevent a seemingly plausible but incorrect interpretation.
Sensitivity answers, “Of the cases that were actually positive, what fraction did the model flag?” Precision answers a different question: “Of the cases the model flagged, what fraction were actually positive?” Specificity describes the actual negatives, while prevalence describes the actual outcome mix before looking at the classifications. In a written report, include the relevant counts alongside the rates; “sensitivity was 75%” is more informative when paired with, for example, “450 of 600 actual positives were flagged.”
Accuracy, the fraction of all classifications that are correct, is sometimes useful as a supplementary description: \((TP+TN)/N\). It does not replace the four rates above. In particular, accuracy can conceal a large number of missed positives when positive outcomes are uncommon.
Why Precision Depends on Prevalence
A common reporting surprise is that a model can have a respectable sensitivity and specificity but still produce many false alarms relative to its true positive flags. The reason is that precision is calculated among the cases classified positive. When actual positives are relatively uncommon, even a modest false-positive rate can generate a substantial number of false positives.
The relationship can be derived by imagining a population of \(N\) cases with prevalence \(p\), sensitivity \(s\), and specificity \(c\). The actual-positive count is \(Np\), so the expected true-positive count under these rates is \(Nps\). The actual-negative count is \(N(1-p)\), and the false-positive rate is \(1-c\), so the expected false-positive count is \(N(1-p)(1-c)\). Substituting these quantities into the definition of precision gives:
The factor \(N\) cancels, leaving a relationship determined by prevalence and the two conditional rates. This is a rate-based identity when the rates and prevalence refer to the same population and threshold; for a finite validation sample, the observed counts and rates may differ slightly from underlying population values. It is not a reason to substitute rates from a different population without justification.
Worked Examples
Worked Example: Reporting a Claim-Flagging Rule
A hypothetical insurer evaluates a rule that flags a policy when its model score is at least 0.40. In a validation set of 5,000 policies, 12% had a claim during the defined outcome period. At this threshold, sensitivity is 75% and specificity is 90%. Construct the matrix, calculate precision and accuracy, and give a concise interpretation.
Method. First calculate the actual-positive and actual-negative totals from prevalence, then apply sensitivity to actual positives and specificity to actual negatives. This uses each rate’s correct denominator.
There are \(5{,}000(0.12)=600\) actual positives. Therefore actual negatives number \(5{,}000-600=4{,}400\). The true positives are \(600(0.75)=450\), and the false negatives are \(600-450=150\). The true negatives are \(4{,}400(0.90)=3{,}960\), and the false positives are \(4{,}400-3{,}960=440\).
The count check is \(450+150+3{,}960+440=5{,}000\). There are \(450+440=890\) positive classifications. Thus precision is \(450/890=0.5056\) to four decimals. Accuracy is \((450+3{,}960)/5{,}000=4{,}410/5{,}000=0.8820\). The matrix therefore has 450 TP, 440 FP, 3,960 TN, and 150 FN.
Interpretation. At the 0.40 threshold, the rule flags 890 policies. Of those, 450 had a claim in the validation period, giving precision of about 50.56%. The rule identifies 450 of 600 actual positives, giving sensitivity of 75%, while correctly leaving 3,960 of 4,400 actual negatives unflagged, giving specificity of 90%. The accuracy of 88.20% is not a substitute for those details.
Common mistake avoided. Applying sensitivity to all 5,000 policies would give 3,750 true positives, which is impossible because only 600 cases were actual positives. Sensitivity’s denominator is the actual-positive group, not the full sample.
Worked Example: A Low-Prevalence Screening Set
A hypothetical service team tests a score threshold of 0.65 on 1,200 accounts. The outcome of interest occurred for 5% of accounts. At this threshold, the model has sensitivity 60% and specificity 90%. Calculate all four rates from the resulting counts and explain what a positive flag means in this set.
Method. Convert prevalence to actual counts, then calculate the threshold outcomes within the positive and negative groups. Do not infer that high specificity guarantees high precision.
The actual-positive count is \(1{,}200(0.05)=60\), leaving \(1{,}200-60=1{,}140\) actual negatives. True positives are \(60(0.60)=36\), so false negatives are \(60-36=24\). True negatives are \(1{,}140(0.90)=1{,}026\), so false positives are \(1{,}140-1{,}026=114\).
The total is \(36+24+1{,}026+114=1{,}200\). Sensitivity is \(36/(36+24)=36/60=0.6000\). Specificity is \(1{,}026/(1{,}026+114)=1{,}026/1{,}140=0.9000\). Precision is \(36/(36+114)=36/150=0.2400\). Prevalence is \(60/1{,}200=0.0500\).
Interpretation. The threshold flags 150 accounts, of which 36 were actual positives in the evaluated period. Thus 24.00% of flags corresponded to an actual positive outcome in this set. The other 114 flags were false positives. This is a threshold-specific description of this validation set, not a promise that exactly 24% of future flags will be positive.
Common mistake avoided. Reporting 90% specificity as though it meant 90% of flagged accounts were positive reverses the conditioning. Specificity is calculated among actual negatives; precision is calculated among positive classifications.
Worked Example: Applying the Threshold to Individual Scores
A hypothetical equipment-maintenance model assigns scores to 20 machines. A machine is flagged when its score is at least 0.70. Four machines were actually due for maintenance: their scores were 0.92, 0.78, 0.63, and 0.35. Among the 16 machines not due for maintenance, three scores were 0.88, 0.73, and 0.70; the other 13 scores were below 0.70. Construct the matrix and calculate sensitivity, specificity, precision, and prevalence.
Method. Apply the stated inclusive threshold to each group, counting a score exactly equal to 0.70 as a positive classification.
Among the four actual positives, 0.92 and 0.78 meet the threshold. Hence \(TP=2\), and the two scores below threshold give \(FN=2\). Among the 16 actual negatives, 0.88, 0.73, and 0.70 meet the threshold. Hence \(FP=3\), and the 13 scores below threshold give \(TN=13\). The count check is \(2+2+3+13=20\).
Sensitivity is \(2/(2+2)=2/4=0.5000\). Specificity is \(13/(13+3)=13/16=0.8125\). Precision is \(2/(2+3)=2/5=0.4000\). Prevalence is \((2+2)/20=4/20=0.2000\). There are five positive classifications, and two of those five machines were actually due for maintenance.
Interpretation. The threshold flags five machines: two actual positives and three actual negatives. It misses two of the four machines that were due for maintenance. Including the score equal to 0.70 is essential because the rule says “at least 0.70,” not “greater than 0.70.”
Common mistake avoided. Excluding the score of 0.70 would incorrectly reduce the false-positive count to two and change both specificity and precision. A report should state the threshold convention whenever equality could affect classification.
Presenting a Threshold-Specific Result Clearly
A decision-ready explanation should identify the positive outcome, evaluation population, threshold and comparison rule, counts, and rates. A compact report can then distinguish what the model finds from what remains unflagged. This structure also lets a reader audit the arithmetic instead of relying on a percentage without its denominator.
Specify what counts as positive, the period or event definition, and which labeled cases were evaluated.
Give the numerical threshold and say whether equality is included. Avoid describing a score as a probability unless that interpretation is warranted.
Confirm that \(TP+FP+TN+FN=N\), and make clear which matrix dimension represents actual outcome and which represents classification.
Report sensitivity among actual positives, specificity among actual negatives, precision among positive classifications, and prevalence among all evaluated cases.
At a fixed threshold, sensitivity and specificity describe performance conditional on the actual class, while precision reflects the mix of actual classes in the evaluated set as well as those two rates. Therefore, include the observed prevalence and avoid presenting precision as an invariant property of a model. AUC, by contrast, summarizes ranking performance across thresholds; it does not tell a reader how many cases this particular threshold flags.
A metric may also be undefined. If there are no actual positives, sensitivity has denominator zero; if there are no actual negatives, specificity has denominator zero. If the rule makes no positive classifications, precision has denominator zero. In these cases, report the relevant count and state “not defined” rather than assigning a value of zero or silently omitting the issue. If the evaluation set is small, raw counts are especially important because a change of one case can materially alter a rate.
Common Reporting Traps
- Reversing a denominator. Sensitivity is \(TP/(TP+FN)\); precision is \(TP/(TP+FP)\). They answer different questions.
- Reporting only accuracy. A large number of correctly classified negatives can make accuracy look high even when many actual positives are missed.
- Leaving out prevalence. The outcome mix gives essential context for interpreting how many positive flags are true positives.
- Omitting the threshold convention. The cutoff determines the classifications, including how a score equal to the cutoff is handled.
- Describing a flag as a confirmed outcome. A positive classification is a model decision at the stated threshold; it is not itself evidence that the event has occurred.
Check Your Understanding
For each problem, calculate from the stated evaluation set and apply the stated threshold convention. Use four decimal places for rates.
- A hypothetical fraud-screening rule flags an account when its score is at least 0.55. On 1,000 labeled accounts, \(TP=84\), \(FN=16\), \(FP=36\), and \(TN=864\). Calculate sensitivity, specificity, precision, prevalence, and accuracy. (Answer: sensitivity \(=0.8400\), specificity \(=0.9600\), precision \(=0.7000\), prevalence \(=0.1000\), accuracy \(=0.9480\).)
- A model’s threshold produces 25 true positives, 75 false negatives, 50 false positives, and 850 true negatives. How many cases are flagged, and what fraction of those flags are true positives? (Answer: 75 cases are flagged; precision \(=25/75=0.3333\).)
- In 400 evaluated cases, actual prevalence is 10%. At the chosen threshold, sensitivity is 75% and specificity is 95%. Calculate all four matrix counts and precision. (Answer: \(TP=30\), \(FN=10\), \(TN=342\), \(FP=18\); precision \(=30/48=0.6250\).)
- A binary classification rule is applied to 80 cases. It produces \(TP=0\), \(FP=0\), \(FN=12\), and \(TN=68\). Calculate sensitivity, specificity, precision, and prevalence, and identify any undefined rate. (Answer: sensitivity \(=0.0000\), specificity \(=1.0000\), prevalence \(=0.1500\); precision is undefined because there are no positive classifications.)
- A validation set has 50 actual positives and 450 actual negatives. A threshold has sensitivity 80% and specificity 92%. Calculate TP, FN, TN, FP, the number of positive classifications, and precision. (Answer: \(TP=40\), \(FN=10\), \(TN=414\), \(FP=36\); 76 positive classifications; precision \(=40/76=0.5263\).)