From Predicted Probabilities to Decisions
A binary-choice model produces a predicted probability, but many decisions require a category: flag or do not flag, intervene or do not intervene, approve or decline. Converting a probability into a category requires a classification threshold. That threshold is a decision rule, not a new estimate of the model’s coefficients or a property of the outcome itself.
The choice matters because a lower threshold labels more cases positive. This may identify more actual positives, but it can also label more actual negatives as positive. A higher threshold generally has the reverse effect. Sensitivity and specificity summarize these two sides of performance.
The positive class must be stated in context. For example, in an application about hospital readmission, the positive outcome might be readmission within 30 days. Reversing which outcome is called positive changes the meanings of sensitivity and specificity.
The Confusion Table and Performance Measures
For a set of \(n\) evaluated cases with observed outcomes, count the four combinations of actual and predicted class. These counts form the confusion table. They depend on the chosen threshold and on the cases used for evaluation.
| Observed positive (\(Y_i=1\)) | Observed negative (\(Y_i=0\)) | |
|---|---|---|
| Classified positive (\(\hat p_i\ge t\)) | True positives (TP) | False positives (FP) |
| Classified negative (\(\hat p_i<t\)) | False negatives (FN) | True negatives (TN) |
Sensitivity uses the actual positives as its denominator: among cases whose observed outcome is positive, what fraction did the rule classify as positive? Specificity uses the actual negatives: among cases whose observed outcome is negative, what fraction did the rule classify as negative? Their complementary error rates are the false-negative rate and false-positive rate.
These rates are proportions, not counts, so they have no physical units. Sensitivity and specificity are undefined in an evaluation sample with no observed positives or no observed negatives, respectively, because the corresponding denominator is zero. In a population, the analogous quantities describe conditional probabilities, such as the probability of a positive classification among cases with \(Y=1\). Empirical rates estimate those quantities only to the extent that the evaluation cases are informative about the population of interest.
Worked Example: Calculating Sensitivity and Specificity
Suppose eight hypothetical patients are evaluated for a binary event. The table gives each patient’s predicted probability of the event and observed outcome. Use threshold \(t=0.50\), classifying a patient as positive when the predicted probability is at least 0.50.
| Patient | Predicted probability | Observed outcome |
|---|---|---|
| A | 0.90 | Positive |
| B | 0.70 | Positive |
| C | 0.40 | Positive |
| D | 0.20 | Positive |
| E | 0.80 | Negative |
| F | 0.60 | Negative |
| G | 0.30 | Negative |
| H | 0.10 | Negative |
Patients A, B, E, and F are classified positive. A and B are true positives; E and F are false positives. C and D are false negatives, while G and H are true negatives. Thus, \(\mathrm{TP}=2\), \(\mathrm{FP}=2\), \(\mathrm{FN}=2\), and \(\mathrm{TN}=2\).
There are four observed positives and four observed negatives, so
At this threshold, the rule identifies half of the observed positives and correctly labels half of the observed negatives. Its false-negative and false-positive rates are both \(0.500\). These are descriptions of this hypothetical set of eight cases, not guarantees about performance on future patients.
How the Threshold Changes the Error Trade-Off
Keep the predicted probabilities and observed outcomes fixed, and change only the threshold. Lowering \(t\) cannot turn a previously positive classification into a negative one: any case with \(\hat p_i\ge t\) also satisfies \(\hat p_i\ge t'\) when \(t'<t\). Therefore, the set classified positive can only grow as the threshold falls. Some newly positive cases may be actual positives, and some may be actual negatives.
Consequently, as the threshold falls, sensitivity cannot decrease and specificity cannot increase. Equivalently, the false-negative rate cannot increase and the false-positive rate cannot decrease. These are weak inequalities because a threshold change may classify no additional cases, particularly when predicted probabilities are tied.
Worked Example: Comparing Two Thresholds
Return to the eight hypothetical patients above. At \(t=0.75\), patients A and E are classified positive. A is a true positive and E is a false positive; B, C, and D are false negatives, while F, G, and H are true negatives. Thus, \(\mathrm{TP}=1\), \(\mathrm{FP}=1\), \(\mathrm{FN}=3\), and \(\mathrm{TN}=3\).
There are still four observed positives and four observed negatives:
Now lower the threshold to \(t=0.25\). Patients A, B, C, E, F, and G are classified positive. This gives \(\mathrm{TP}=3\), \(\mathrm{FP}=3\), \(\mathrm{FN}=1\), and \(\mathrm{TN}=1\), so
Lowering the threshold from 0.75 to 0.25 increased sensitivity from 0.250 to 0.750, while specificity fell from 0.750 to 0.250. It identified more actual positives but also misclassified more actual negatives. Neither threshold is automatically better: the preferred trade-off depends on the consequences of the two error types and the practical objective.
Choosing a Threshold Using Error Costs
A threshold of 0.50 is sometimes used as a default, but it is optimal only under particular conditions. To see how consequences can imply a different threshold, suppose a decision-maker has two actions—classify positive or classify negative—and two possible errors. Let \(C_{\mathrm{FP}}>0\) be the cost of a false positive and \(C_{\mathrm{FN}}>0\) the cost of a false negative. Assume correct classifications have zero cost, the costs are constant across cases, and \(\hat p_i\) is the relevant conditional probability of \(Y_i=1\) for decision-making.
Given a case with predicted probability \(p\), classifying it positive has expected error cost \((1-p)C_{\mathrm{FP}}\), because that action incurs a cost only when the outcome is negative. Classifying it negative has expected error cost \(pC_{\mathrm{FN}}\), because that action incurs a cost only when the outcome is positive. Choose the positive action when its expected cost is no larger than the negative action’s expected cost:
Thus, under these assumptions, the cost-minimizing threshold is \(C_{\mathrm{FP}}/(C_{\mathrm{FP}}+C_{\mathrm{FN}})\). A larger false-negative cost lowers the threshold, making positive classifications more common; a larger false-positive cost raises it. This result does not say that the cost-minimizing threshold always meets a separate target, such as a minimum sensitivity. Such operational constraints can require a different choice.
Worked Example: Applying a Cost-Based Threshold
Suppose a hypothetical screening decision assigns a cost of 2 units to a false positive and 8 units to a false negative. Assume the two possible actions and zero cost for correct classifications as above. The threshold implied by these costs is
For a hypothetical person with predicted probability \(p=0.12\), the expected cost of a positive classification is \((1-0.12)(2)=1.76\) units, while the expected cost of a negative classification is \((0.12)(8)=0.96\) units. The lower expected cost is to classify negative.
For a person with \(p=0.24\), the corresponding expected costs are \((1-0.24)(2)=1.52\) units for a positive classification and \((0.24)(8)=1.92\) units for a negative classification. The rule classifies this person positive. The same comparison applies to each case: classify positive when its predicted probability is at least 0.200. This choice follows from the stated costs and assumptions; different costs or decision constraints can imply a different threshold.
Why Accuracy Can Give the Wrong Impression
Accuracy is the fraction of all cases classified correctly: \((\mathrm{TP}+\mathrm{TN})/n\). It treats false positives and false negatives symmetrically and combines the two actual-outcome groups. If positives are rare, a rule that almost always classifies negative can have high accuracy while failing to identify most positives.
Worked Example: High Accuracy With Missed Positives
Consider 100 hypothetical cases: 5 have the positive outcome and 95 have the negative outcome. At a chosen threshold, suppose the rule yields \(\mathrm{TP}=4\), \(\mathrm{FN}=1\), \(\mathrm{FP}=8\), and \(\mathrm{TN}=87\). These counts sum to 100, with 5 observed positives and 95 observed negatives.
The rule’s sensitivity is \(4/(4+1)=0.800\), its specificity is \(87/(87+8)=87/95=0.9158\), and its accuracy is \((4+87)/100=0.910\). A rule that classifies every case negative would instead have accuracy \(95/100=0.950\), but sensitivity \(0/5=0\): it would miss all five positives.
The higher accuracy of the all-negative rule does not establish that it is preferable. The comparison makes the positive-class detection rate visible and shows why accuracy alone can conceal a consequential error pattern when the classes are imbalanced.
Common Mistakes and Practical Interpretation
- Treating 0.50 as universally correct. The reasoning “probabilities split at one-half, so this must be the right threshold” ignores unequal error costs and other decision goals. A threshold is a chosen rule; justify it from the decision context.
- Reversing the denominators. Sensitivity is TP divided by all observed positives, \(\mathrm{TP}+\mathrm{FN}\). Specificity is TN divided by all observed negatives, \(\mathrm{TN}+\mathrm{FP}\). Always condition on the actual class named in the measure.
- Calling a lower threshold an unqualified improvement. It may increase sensitivity, but it can also increase false positives and reduce specificity. State which performance measure improves and which may worsen.
- Reporting only accuracy for a rare outcome. Report sensitivity and specificity as well, with the counts or denominators that make them interpretable.
- Claiming a sample rate guarantees future performance. The computed rates describe the evaluated cases. Their relevance to other cases depends on how those cases were selected and whether the setting is comparable.
For a transparent report, identify the positive outcome, state the threshold and its tie rule, show the confusion counts, and report sensitivity and specificity with their denominators. Evaluate the rule on cases suited to the intended use; choosing and assessing a threshold on the same observations can make the reported performance look more favorable than performance on new cases.
Check Your Understanding
Use the definitions and decision rule above to answer the following.
- In a confusion table, which denominator is used for sensitivity, and which is used for specificity?
- If the threshold is lowered while the predicted probabilities and outcomes stay fixed, which way can sensitivity and specificity move?
- A sample has 12 true positives, 3 false negatives, 6 false positives, and 79 true negatives. Calculate sensitivity and specificity.
- Suppose false negatives cost more than false positives under the assumptions used to derive the cost-based threshold. Should the implied threshold be higher or lower than when the two costs are equal? Explain.
- Why can a classifier with high accuracy still be poor at identifying a rare positive outcome?