Comparing Classifiers at a Common Threshold

Why Compare Classifiers at a Common Threshold?

A classifier often produces a score for each case, and a threshold turns that score into a predicted class. Comparing two classifiers at the same stated threshold is a direct way to compare the decisions each model makes under that rule. The comparison is useful only if the score definitions, threshold convention, evaluation cases and actual outcomes are all clear.

A common threshold does not guarantee that the classifiers make the same number of positive predictions, achieve the same sensitivity, or operate under equivalent business conditions. Their scores may be on different scales, or one model's scores may be calibrated differently. The calculations therefore answer a specific question: what did each classifier do on these cases when its score was compared with this cutoff?

Definition: Let \(s_j\) be classifier \(j\)'s score for a case, and let \(c\) be the stated threshold. Using the convention that a score equal to the threshold is positive, the predicted class is positive when \(s_j \ge c\), and negative when \(s_j < c\). A confusion matrix counts predicted and actual classes: true positives (TP), false positives (FP), false negatives (FN), and true negatives (TN).

The positive class must be identified before calculating metrics. For example, a screening analysis might define a positive case as a person who has the condition, while a claims analysis might define it as a claim that will exceed a specified amount. Reversing which outcome is called positive changes the meanings of sensitivity, specificity and precision.

Confusion Matrices and Threshold-Dependent Metrics

For a fixed evaluation set, the actual positive cases are split into TP and FN; the actual negative cases are split into FP and TN. In the matrix below, rows are actual outcomes and columns are predictions.

	Predicted positive	Predicted negative
Actual positive	TP	FN
Actual negative	FP	TN

The row totals describe the actual class counts: \(TP+FN\) actual positives and \(FP+TN\) actual negatives. The column totals describe the predictions: \(TP+FP\) predicted positives and \(FN+TN\) predicted negatives. The following metrics are commonly compared at a chosen threshold.

\begin{aligned} \text{Sensitivity} &= \frac{TP}{TP+FN}, & \text{Specificity} &= \frac{TN}{TN+FP},\\ \text{Precision} &= \frac{TP}{TP+FP}, & \text{Accuracy} &= \frac{TP+TN}{TP+FP+FN+TN},\\ F_1 &= \frac{2TP}{2TP+FP+FN}. \end{aligned}

Sensitivity is the fraction of actual positives predicted positive; specificity is the fraction of actual negatives predicted negative. Precision is the fraction of predicted positives that are actually positive. Accuracy is the fraction of all cases classified correctly. The \(F_1\) score is the harmonic mean of precision and sensitivity when both are defined; its displayed count formula is convenient for direct calculation.

Derivation: Sensitivity conditions on the actual-positive group. That group contains \(TP+FN\) cases, of which \(TP\) are predicted positive, so sensitivity is \(TP/(TP+FN)\). Precision instead conditions on the predicted-positive group, which contains \(TP+FP\) cases, of which \(TP\) are truly positive. Thus precision is \(TP/(TP+FP)\). The denominators differ because the questions differ: “Among actual positives, how many were found?” versus “Among predicted positives, how many were correct?”

This distinction is important when comparing models. A classifier can have higher sensitivity but lower precision if it finds more actual positives while also flagging more actual negatives. Accuracy alone can conceal this trade-off, particularly when actual outcomes are unevenly distributed. No single metric should be treated as a complete comparison unless it reflects the decision objective.

Worked Comparisons at a Stated Threshold

Worked Example: Comparing Two Models on the Same Cases

A hypothetical insurer evaluates two classifiers on the same 100 claims, using a threshold of \(0.60\) for each. There are 40 claims in the actual-positive class and 60 in the actual-negative class. At this threshold, Model A predicts 50 claims positive, including 32 actual positives. Model B predicts 40 claims positive, including 30 actual positives. Construct both confusion matrices and compare sensitivity, specificity, precision and accuracy.

Method. For each model, subtract the true positives from the actual-positive total to get FN. Subtract the true positives from predicted positives to get FP. The remaining actual negatives are TN.

For Model A, \(FN=40-32=8\). Its predicted-positive total is 50, so \(FP=50-32=18\). Then \(TN=60-18=42\). The matrix counts are \(TP=32\), \(FN=8\), \(FP=18\) and \(TN=42\). They check: \(32+8=40\) actual positives, \(18+42=60\) actual negatives, and all counts sum to 100.

Model A's sensitivity is \(32/(32+8)=32/40=0.8000\). Its specificity is \(42/(42+18)=42/60=0.7000\). Its precision is \(32/(32+18)=32/50=0.6400\). Its accuracy is \((32+42)/100=74/100=0.7400\).

For Model B, \(FN=40-30=10\). Its predicted-positive total is 40, so \(FP=40-30=10\). Then \(TN=60-10=50\). Its counts check: \(30+10=40\), \(10+50=60\), and the total is 100.

Model B's sensitivity is \(30/40=0.7500\). Its specificity is \(50/60=0.8333\). Its precision is \(30/(30+10)=0.7500\). Its accuracy is \((30+50)/100=0.8000\).

At this common threshold, Model A has higher sensitivity, while Model B has higher specificity, precision and accuracy. This is a trade-off in the observed classifications, not evidence that one model is best for every purpose. Common mistake avoided: do not call the model with more true positives the unqualified winner; Model A also generates more false positives.

Worked Example: Counting Scores Equal to the Threshold

A hypothetical screening team has 12 cases: 6 actual positives and 6 actual negatives. At threshold \(0.50\), a score equal to \(0.50\) counts as predicted positive. For Model C, among actual positives, 3 scores exceed \(0.50\) and 2 equal it; among actual negatives, 1 exceeds it and 1 equals it. For Model D, the corresponding counts are 4 exceed and none equal among actual positives, and 1 exceeds and none equal among actual negatives. Calculate the confusion matrices and compare sensitivity and precision.

Method. Include both scores greater than the threshold and scores equal to it in the predicted-positive count. For Model C, \(TP=3+2=5\), so \(FN=6-5=1\). Its \(FP=1+1=2\), so \(TN=6-2=4\). Its matrix is \(TP=5\), \(FN=1\), \(FP=2\), \(TN=4\), which sums to 12.

Model C's sensitivity is \(5/6=0.8333\). Its precision is \(5/(5+2)=5/7=0.7143\).

For Model D, \(TP=4+0=4\), \(FN=6-4=2\), \(FP=1+0=1\), and \(TN=6-1=5\). These counts also sum to 12. Its sensitivity is \(4/6=0.6667\), and its precision is \(4/(4+1)=0.8000\).

Model C identifies a larger fraction of actual positives, but Model D has a higher fraction of correct cases among its positive predictions. Common mistake avoided: treating a score exactly equal to the cutoff as negative would contradict the stated rule \(s_j \ge c\) and would change the counts.

Same Numeric Threshold, Different Operating Conditions

A threshold is applied to a score, and the score's meaning matters. If two models produce calibrated probabilities for the same outcome in the same population, a threshold such as \(0.40\) has a common probability interpretation: predict positive when the model-estimated probability is at least 40%. Calibration means that, among cases assigned a probability near a given value, the observed event frequency is approximately that value. Even then, the two models can assign different scores to individual cases and produce different confusion matrices.

If one model produces probabilities and another produces an uncalibrated index, applying \(0.40\) to both does not impose the same probability-based decision rule. A common numeric cutoff is still a valid calculation if both score rules are specified, but it should not be described as an equivalent operating condition. Further, the score scales may differ even when both outputs lie between zero and one.

Worked Example: A Shared Cutoff on Differently Calibrated Scores

A hypothetical service evaluates two models on the same 30 cases: 10 actual positives and 20 actual negatives. It applies a numeric cutoff of \(0.40\) to each model. Model E produces calibrated probability estimates; Model F produces an uncalibrated score that also ranges from zero to one. At the cutoff, Model E has \(TP=8\) and \(FP=4\); Model F has \(TP=9\) and \(FP=8\). Calculate the remaining counts and compare their metrics.

For Model E, \(FN=10-8=2\), and \(TN=20-4=16\). The counts sum to \(8+2+4+16=30\). Sensitivity is \(8/10=0.8000\); specificity is \(16/20=0.8000\); precision is \(8/(8+4)=8/12=0.6667\); and accuracy is \((8+16)/30=24/30=0.8000\).

For Model F, \(FN=10-9=1\), and \(TN=20-8=12\). The counts sum to \(9+1+8+12=30\). Sensitivity is \(9/10=0.9000\); specificity is \(12/20=0.6000\); precision is \(9/(9+8)=9/17=0.5294\); and accuracy is \((9+12)/30=21/30=0.7000\).

At the chosen numeric cutoff, Model F has higher sensitivity but lower specificity, precision and accuracy. However, \(0.40\) means a model-estimated probability of 40% for Model E, while it has no such interpretation for Model F. Their confusion matrices can be compared descriptively on the same cases, but the common number alone does not establish that they were used under equivalent probability or cost conditions. Common mistake avoided: do not assume that scores with the same numerical range share the same calibration or meaning.

A Reliable Comparison Workflow

A careful comparison keeps the threshold rule, case set and metric interpretation together. The following sequence reduces arithmetic and reporting errors.

State the positive class and threshold convention.
Specify which outcome is positive and whether equality with the threshold is classified positive. For example, state \(s_j \ge c\) rather than leaving boundary handling implicit.

Confirm the evaluation basis.
Check whether both classifiers are assessed on the same cases with the same observed outcomes. If not, differences in class mix or case difficulty may affect the comparison.

Build and check each matrix.
Use actual class totals and predicted-positive totals, where available, to obtain all four counts. Verify that \(TP+FN\) and \(FP+TN\) equal the actual class totals, and that all four cells sum to the number of cases.

Calculate metrics using their correct denominators.
Use actual positives for sensitivity, actual negatives for specificity, predicted positives for precision, and all cases for accuracy. State the metric values and the direction of any trade-off.

Interpret the score scale and decision purpose.
Explain what the shared cutoff means for each model. Relate the metric comparison to the intended decision, and avoid claiming a general winner from one threshold alone.

The workflow also provides simple checks. A sensitivity outside \([0,1]\), a negative cell count, or a matrix total that does not match the evaluation-set size indicates an error. If a denominator is zero, the corresponding metric is undefined rather than a number to be filled in by assumption; this is especially relevant if a model predicts no positive cases or the evaluation set contains no actual positives.

Key takeaway: A common threshold makes the classification rule explicit, but it does not force classifiers to have equal sensitivity, specificity, precision, or numbers of positive predictions. Compare the confusion matrices on a clearly defined evaluation set, and verify that the score scales make the threshold interpretation meaningful.

Check Your Understanding

For each problem, use the stated positive-class convention and threshold rule. Round reported proportions to four decimal places.

A classifier is evaluated on 50 cases, 20 of which are actual positives. At its stated threshold, it has \(TP=10\) and \(FP=5\). Find FN, TN, sensitivity, specificity, precision and accuracy. (Answer: \(FN=10\), \(TN=25\), sensitivity \(=0.5000\), specificity \(=0.8333\), precision \(=0.6667\), accuracy \(=0.7000\).)
On 80 cases there are 16 actual positives. At a threshold of \(0.35\), a classifier has \(TP=12\) and \(FP=16\). Find FN, TN, sensitivity, specificity, precision, accuracy and \(F_1\). (Answer: \(FN=4\), \(TN=48\), sensitivity \(=0.7500\), specificity \(=0.7500\), precision \(=0.4286\), accuracy \(=0.7500\), \(F_1=0.5455\).)
Two classifiers are tested on the same 100 cases, with 40 actual positives and 60 actual negatives, at a common threshold. Classifier A has \(TP=30\), \(FP=12\); classifier B has \(TP=28\), \(FP=6\). Find each classifier's sensitivity, specificity, precision and accuracy, and identify which has higher sensitivity. (Answer: A: sensitivity \(=0.7500\), specificity \(=0.8000\), precision \(=0.7143\), accuracy \(=0.7800\); B: sensitivity \(=0.7000\), specificity \(=0.9000\), precision \(=0.8235\), accuracy \(=0.8200\); A has higher sensitivity.)
A test set has 15 actual positives and 25 actual negatives. At threshold \(0.60\), 7 positive cases score above the threshold and 2 equal it; among negatives, 3 score above and 1 equals it. Equality counts as positive. Find the confusion matrix, sensitivity, specificity and precision. (Answer: \(TP=9\), \(FN=6\), \(FP=4\), \(TN=21\); sensitivity \(=0.6000\), specificity \(=0.8400\), precision \(=0.6923\).)
Two classifiers are evaluated on the same 30 cases, including 10 actual positives and 20 actual negatives, at the same numeric cutoff of \(0.40\). Model G has \(TP=7\), \(FP=3\); Model H has \(TP=8\), \(FP=6\). Calculate sensitivity, specificity and precision for each. If G's scores are calibrated probabilities and H's are uncalibrated scores, what limitation should be stated about the shared cutoff? (Answer: G: sensitivity \(=0.7000\), specificity \(=0.8500\), precision \(=0.7000\); H: sensitivity \(=0.8000\), specificity \(=0.7000\), precision \(=0.5714\). The cutoff has a probability interpretation for G but not necessarily for H, so the shared numeric value does not establish equivalent operating conditions.)

What You'll Learn