Reporting Classification Performance and Thresholds

Start With the Decisions Made at a Threshold

A classification model often assigns each case a score, and an operational rule turns that score into an action. For example, a review team might investigate a case when its score reaches a selected threshold. A useful report must therefore say more than “the model performed well”: it should identify which cases were correctly flagged, which were missed, and which were flagged unnecessarily.

The counts depend on both the threshold and the cases used for evaluation. For a fair comparison of two thresholds, hold the evaluation cases and their observed outcomes fixed. Also state which direction triggers a positive prediction; below, a case is predicted positive when its score is at least the threshold.

Definition: For a binary outcome, an actual positive is a case whose observed outcome is positive, and an actual negative is a case whose observed outcome is negative. At a stated threshold, a true positive (TP) is an actual positive predicted positive; a false negative (FN) is an actual positive predicted negative; a false positive (FP) is an actual negative predicted positive; and a true negative (TN) is an actual negative predicted negative. A false positive and a false negative are errors; the two true counts are correct decisions.

These definitions describe a fixed set of evaluated cases, not a probability model. They require that each case has a known observed class and a model score, and that the threshold rule is applied consistently. The counts themselves do not require a distributional assumption. To use them as evidence about future cases, the validation data and evaluation process must be relevant to the intended population and use.

Build the Confusion Matrix Before Calculating Rates

The confusion matrix puts actual outcomes in rows and threshold-based predictions in columns. Its row totals are the observed class counts; its column totals are the numbers predicted into each class. Check these totals before calculating any metric.

	Predicted positive	Predicted negative	Actual total
Actual positive	TP	FN	TP + FN
Actual negative	FP	TN	FP + TN
Predicted total	TP + FP	FN + TN	N

The four commonly reported rates answer different questions. Sensitivity asks what fraction of actual positives were found. Specificity asks what fraction of actual negatives were correctly left unflagged. Precision asks what fraction of flagged cases were actually positive. Accuracy asks what fraction of all cases were classified correctly.

\text{Sensitivity}=\frac{TP}{TP+FN},\qquad \text{Specificity}=\frac{TN}{TN+FP}

\text{Precision}=\frac{TP}{TP+FP},\qquad \text{Accuracy}=\frac{TP+TN}{N}

Each denominator corresponds to the question being answered. Sensitivity is calculated among actual positives, specificity among actual negatives, precision among predicted positives, and accuracy among all evaluated cases. If a denominator is zero, its rate is undefined; do not report it as zero without explanation.

Why these formulas hold: Sensitivity is the share of actual positives classified correctly. The actual-positive row contains $TP+FN$ cases, of which $TP$ were classified correctly, so the share is $TP/(TP+FN)$. Precision instead starts with the predicted-positive column, which contains $TP+FP$ cases; $TP$ of these really are positive, giving $TP/(TP+FP)$. The other two formulas follow in the same way from the actual-negative row and the full set of cases.

Worked Example: Reporting One Threshold Clearly

A hypothetical equipment-monitoring team evaluates a classifier on 200 completed inspections. Forty inspections had an observed fault and 160 did not. At the chosen threshold, the model flags 30 of the fault inspections and 24 inspections without a fault. Construct the matrix and report sensitivity, specificity, precision, and accuracy. Round rates to four decimal places.

Method and counts. The given counts are $TP=30$ and $FP=24$. Since 40 inspections actually had a fault, $FN=40-30=10$. Since 160 did not have a fault, $TN=160-24=136$. The check is $30+10=40$, $24+136=160$, and $30+10+24+136=200$.

Sensitivity. The model flags $30$ of the $40$ actual faults: $30/40=0.7500$.

Specificity. It correctly leaves $136$ of the $160$ non-fault inspections unflagged: $136/160=0.8500$.

Precision. It flags $30+24=54$ inspections, of which $30$ had a fault: $30/54=0.5556$.

Accuracy. It classifies $30+136=166$ inspections correctly: $166/200=0.8300$.

A concise report would say that the threshold identified 30 of 40 observed faults and correctly left 136 of 160 non-fault inspections unflagged. Of the 54 inspections it flagged, 24 had no observed fault. This gives decision-makers both the overall rates and the concrete workload and miss counts. The method avoids describing accuracy alone as if it showed how many faults were missed.

Explain What Changes When the Threshold Moves

Suppose scores and evaluated cases remain fixed, and a lower threshold is used with the rule “flag if score is at least the threshold.” Every case flagged at the higher threshold is still flagged at the lower one, and some additional cases may be flagged. Therefore TP cannot decrease and FP cannot decrease; FN cannot increase and TN cannot increase. If the model is refit, the cases change, or the score definition changes, this nesting need not hold. State the comparison conditions rather than assuming them.

A lower threshold can find more actual positives, reducing missed positives, but it can also flag more actual negatives. That can increase review workload or other costs. A higher threshold often has the opposite trade-off. These are changes in the classifications; whether the new trade-off is preferable depends on the consequences of each type of error and the operational capacity.

Worked Example: Comparing Two Thresholds on the Same Cases

A hypothetical screening team evaluates two thresholds on the same 300 cases. Sixty cases are actually positive and 240 are actually negative. At threshold A, $TP=42$ and $FP=48$. At the lower threshold B, $TP=51$ and $FP=78$. For each threshold, calculate sensitivity, precision, and accuracy, then compare the error counts. Round rates to four decimal places.

Threshold A. The false-negative count is $FN=60-42=18$. The true-negative count is $TN=240-48=192$. The check is $42+18=60$, $48+192=240$, and the four counts sum to 300. Sensitivity is $42/60=0.7000$. There are $42+48=90$ predicted positives, so precision is $42/90=0.4667$. Accuracy is $(42+192)/300=234/300=0.7800$.

Threshold B. Here $FN=60-51=9$, and $TN=240-78=162$. The checks are $51+9=60$, $78+162=240$, and the counts sum to 300. Sensitivity is $51/60=0.8500$. There are $51+78=129$ predicted positives, so precision is $51/129=0.3953$. Accuracy is $(51+162)/300=213/300=0.7100$.

Threshold B finds 9 more actual positives, so the false-negative count falls from 18 to 9. It also produces 30 more false positives, increasing that count from 48 to 78. Sensitivity rises, while precision and accuracy fall. The lower threshold is not simply “better” or “worse”: it exchanges more false alarms for fewer missed positives. The comparison avoids the mistake of inferring accuracy from sensitivity or from the number of correct positive predictions alone.

Connect the Counts to the Decision

A rate is easier to interpret when paired with its numerator, denominator, and operational meaning. “Sensitivity was 85%” is more informative when accompanied by “the threshold found 51 of 60 observed positives and missed 9.” Likewise, precision should not be confused with sensitivity: a model can find most actual positives while many of its alerts are false positives.

Accuracy also needs context. When one class is much more common than the other, a high accuracy rate can coexist with poor performance on the less common class. Even when class counts are balanced, accuracy does not distinguish the consequences of the two types of error. A report can include accuracy, but should not use it as the sole description of classification results.

Worked Example: Threshold Trade-Off With Different Error Costs

A hypothetical review operation compares two thresholds on 500 cases: 50 are actually positive and 450 are actually negative. At a higher threshold, $TP=30$ and $FP=18$; at a lower threshold, $TP=40$ and $FP=45$. For planning only, suppose each false negative has an assigned cost of $400 and each false positive has an assigned cost of $40. Calculate the error counts and total assigned error cost for each threshold, then explain the trade-off.

Higher threshold. $FN=50-30=20$, and $TN=450-18=432$. Its assigned error cost is $20\times\$400+18\times\$40=\$8{,}000+\$720=\$8{,}720$.

Lower threshold. $FN=50-40=10$, and $TN=450-45=405$. Its assigned error cost is $10\times\$400+45\times\$40=\$4{,}000+\$1{,}800=\$5{,}800$.

Moving to the lower threshold finds 10 additional positives and reduces false negatives by 10, but it adds 27 false positives. Under the stated assigned costs, the total falls by $\$8{,}720-\$5{,}800=\$2{,}920$. This arithmetic is not proof that the lower threshold is operationally optimal: the assigned costs may omit capacity constraints or other consequences. The report should label them as assumptions and keep the false-positive and false-negative counts visible.

Common Reporting Traps

Using the wrong denominator. Precision uses $TP+FP$, not all actual positives. Sensitivity uses $TP+FN$, not all predicted positives.
Calculating true negatives from the total case count. In the 300-case comparison, threshold A has 240 actual negatives, so $TN=240-48=192$. The incorrect calculation $300-48=252$ subtracts false positives from all cases, mixing the positive and negative rows. It would falsely give accuracy $(42+252)/300=0.9800$; the correct accuracy is $(42+192)/300=0.7800$.
Reporting percentages without counts. Counts let the reader see the scale of missed cases and false alarms and help check the denominator.
Calling one threshold universally superior. A threshold changes the balance of errors. State the decision consequences and constraints relevant to the intended action.
Comparing thresholds on different bases. A threshold comparison is clearest when it uses the same cases, observed outcomes, and score definition. Otherwise, differences may not be caused by the threshold alone.

Key takeaway: Name the threshold rule, show the four confusion-matrix counts, and report rates with their denominators or corresponding counts. When comparing thresholds, describe both the added correct decisions and the added errors, then explain the operational trade-off without declaring a winner unless the decision criteria support it.

Check Your Understanding

For each problem, treat the supplied counts as results on the stated fixed evaluation set. Round rates to four decimal places where requested.

A hypothetical quality check has 30 actual positives and 90 actual negatives. At a threshold, $TP=21$ and $FP=18$. Find $FN$, $TN$, sensitivity, precision, and accuracy. (Answer: $FN=9$, $TN=72$; sensitivity $=0.7000$, precision $=0.5385$, accuracy $=0.7750$.)
On 200 cases, 50 are positive and 150 are negative. Threshold X gives $TP=35$, $FP=30$; lower threshold Y gives $TP=42$, $FP=45$. For each, find sensitivity, precision, accuracy, and the two error counts. (Answer: X: sensitivity $0.7000$, precision $0.5385$, accuracy $0.7750$, $FN=15$, $FP=30$; Y: sensitivity $0.8400$, precision $0.4828$, accuracy $0.7350$, $FN=8$, $FP=45$.)
A hypothetical review set has 80 actual positives and 320 actual negatives. A higher threshold gives $TP=60$, $FP=64$; a lower threshold gives $TP=68$, $FP=96$. If each false negative is assigned a cost of $250 and each false positive $25, calculate each threshold’s total assigned error cost. (Answer: Higher threshold: $FN=20$, cost $=\$6{,}600$; lower threshold: $FN=12$, cost $=\$5{,}400$.)
With the same 40 actual positives and 160 actual negatives, a current threshold has $TP=28$, $FP=32$, while a lower threshold has $TP=36$, $FP=50$. How many additional true positives and false positives result, and how do FN and TN change? (Answer: 8 additional true positives and 18 additional false positives; FN falls from 12 to 4, and TN falls from 128 to 110.)
A hypothetical classifier is evaluated on 250 cases, of which 25 are positive and 225 negative. At its selected threshold, $TP=20$, $FP=45$. Calculate sensitivity, specificity, precision, accuracy, and the false-negative count. (Answer: sensitivity $=0.8000$, specificity $=0.8000$, precision $=0.3077$, accuracy $=0.8000$, $FN=5$.)

What You'll Learn