From Logistic Probabilities to Classification Counts
A fitted logistic model gives an estimated probability for each observation, but a confusion matrix compares classified labels with observed outcomes. To construct one, first identify which outcome is positive, then apply the stated threshold to every fitted probability, and finally count the four possible combinations of predicted and observed labels. The resulting matrix is a summary of classifications at that particular threshold; it is not a summary of the probabilities alone.
For example, suppose a hypothetical insurer uses a logistic model to estimate whether a submitted claim will require additional investigation. The model’s fitted probability is not itself a classification. If the stated threshold is \(0.40\), a fitted probability of \(0.72\) is classified as positive, while one of \(0.18\) is classified as negative. The observed outcome is used separately to determine whether that prediction was correct.
The convention \(\hat p_i\ge c\) must be applied consistently. If a problem specifies a different rule for a probability equal to the threshold, follow that rule instead. The positive class is also a matter of definition: it is the outcome being treated as positive in the problem, not necessarily the more frequent outcome or the more desirable outcome.
Reading the Confusion Matrix
Each observation belongs to exactly one cell. A true positive is an observed positive classified as positive. A false negative is an observed positive classified as negative. A false positive is an observed negative classified as positive, and a true negative is an observed negative classified as negative.
| Observed positive | Observed negative | |
|---|---|---|
| Predicted positive | True positive (TP) | False positive (FP) |
| Predicted negative | False negative (FN) | True negative (TN) |
Rows in this layout are predicted classes, and columns are observed classes. Other layouts are possible, so read the labels rather than relying on a remembered matrix orientation. The actual-positive count is \(N_+=TP+FN\), the actual-negative count is \(N_-=FP+TN\), and the total number of observations is \(N=N_++N_-=TP+FP+FN+TN\).
Sensitivity (also called the true-positive rate): \(TP/(TP+FN)=TP/N_+\), when \(N_+>0\).
Specificity (also called the true-negative rate): \(TN/(TN+FP)=TN/N_-\), when \(N_->0\).
False-positive rate: \(FP/(FP+TN)=FP/N_-\), when \(N_->0\).
Accuracy: \((TP+TN)/N\), when \(N>0\).
Sensitivity answers: among observations that were actually positive, what fraction did the classifier identify as positive? Specificity asks the corresponding question among actual negatives. The false-positive rate is the fraction of actual negatives classified as positive, so when \(N_->0\), it equals one minus specificity:
Accuracy instead asks what fraction of all observations were classified correctly. It uses the entire evaluation sample as its denominator, not just one actual class. If the sample contains no observations, accuracy is undefined because \(N=0\). Sensitivity is undefined when there are no actual positives, and specificity and the false-positive rate are undefined when there are no actual negatives. A zero denominator is not a rate of zero.
Worked Examples
Worked Example: Building the Matrix at a Stated Cutoff
A hypothetical review team evaluates a logistic classifier on ten claims. The positive outcome is that a claim actually required additional investigation. The stated threshold is \(c=0.50\), with fitted probabilities at or above the threshold classified as positive. The observed outcomes and fitted probabilities are shown below. Construct the confusion matrix and calculate sensitivity, specificity, false-positive rate, and accuracy.
| Claim | Observed outcome | Fitted probability | Prediction at 0.50 |
|---|---|---|---|
| A | Positive | 0.82 | Positive |
| B | Positive | 0.48 | Negative |
| C | Positive | 0.67 | Positive |
| D | Positive | 0.21 | Negative |
| E | Negative | 0.73 | Positive |
| F | Negative | 0.52 | Positive |
| G | Negative | 0.40 | Negative |
| H | Negative | 0.09 | Negative |
| I | Negative | 0.61 | Positive |
| J | Negative | 0.31 | Negative |
Step 1: Classify from the probabilities. At threshold \(0.50\), probabilities \(0.82, 0.67, 0.73, 0.52,\) and \(0.61\) are at least \(0.50\), so those five claims are predicted positive. The other five are predicted negative.
Step 2: Count outcomes in each cell. Among the four observed positives, A and C are predicted positive, giving \(TP=2\); B and D are predicted negative, giving \(FN=2\). Among the six observed negatives, E, F, and I are predicted positive, giving \(FP=3\); G, H, and J are predicted negative, giving \(TN=3\). The counts sum to \(2+2+3+3=10\), as required.
Step 3: Calculate each measure with its own denominator. Here \(N_+=TP+FN=2+2=4\) and \(N_-=FP+TN=3+3=6\).
The results are consistent: the false-positive rate is \(1-0.5000=0.5000\), and the number correctly classified is \(TP+TN=5\) out of \(10\). Method and trap avoided: classify each row before counting, and use actual-class totals for sensitivity and specificity rather than the total sample size.
Worked Example: Applying the Tie Rule at a Different Threshold
A hypothetical equipment warranty program defines a positive outcome as a repair occurring within 30 days. On a 14-observation evaluation sample, the actual positives have fitted probabilities \(0.91, 0.66, 0.44, 0.30, 0.29,\) and \(0.12\). The actual negatives have fitted probabilities \(0.80, 0.47, 0.30, 0.25, 0.18, 0.10, 0.04,\) and \(0.02\). Use threshold \(c=0.30\), classifying probabilities equal to \(0.30\) as positive. Find the confusion matrix and all four measures.
Step 1: Classify the actual positives. The probabilities \(0.91, 0.66, 0.44,\) and \(0.30\) are at least \(0.30\). Thus, \(TP=4\). The probabilities \(0.29\) and \(0.12\) are below \(0.30\), so \(FN=2\). Therefore \(N_+=4+2=6\).
Step 2: Classify the actual negatives. The probabilities \(0.80, 0.47,\) and \(0.30\) are at least \(0.30\), so \(FP=3\). The remaining five probabilities \(0.25, 0.18, 0.10, 0.04,\) and \(0.02\) are below \(0.30\), so \(TN=5\). Therefore \(N_-=3+5=8\).
Step 3: Calculate the measures. The total is \(6+8=14\), which also equals \(4+2+3+5=14\).
The complement check gives \(1-0.6250=0.3750\), matching the false-positive rate. The tie at \(0.30\) contributes one true positive and one false positive because the two observations with that probability have different observed outcomes. Method and trap avoided: the threshold acts on fitted probabilities, while the observed outcome determines which matrix column each observation enters.
Worked Example: A Missing Actual Class
A hypothetical fraud-screening evaluation contains five transactions, all of which were observed to be negative. Their fitted probabilities of a positive outcome are \(0.72, 0.41, 0.40, 0.39,\) and \(0.08\). With threshold \(c=0.40\) and the rule that equality is classified positive, construct the counts and state which measures are defined.
Step 1: Classify each transaction. The first three probabilities, \(0.72, 0.41,\) and \(0.40\), are at least \(0.40\), so those transactions are predicted positive. The last two, \(0.39\) and \(0.08\), are predicted negative.
Step 2: Count by observed outcome. There are no actual positives, so \(TP=0\) and \(FN=0\). Of the five actual negatives, three are predicted positive, so \(FP=3\), and two are predicted negative, so \(TN=2\). Thus \(N_+=0\), \(N_-=5\), and \(N=5\).
Step 3: Evaluate the denominators. Sensitivity would be \(TP/N_+=0/0\), so it is undefined. Specificity is \(TN/N_-=2/5=0.4000\), and the false-positive rate is \(FP/N_-=3/5=0.6000\). Accuracy is \((TP+TN)/N=(0+2)/5=0.4000\).
The complement check \(1-0.4000=0.6000\) agrees with the false-positive rate. Method and trap avoided: do not report sensitivity as zero merely because \(TP=0\); without any actual positives, the sample provides no denominator for estimating the fraction of positives detected.
Why Accuracy Is a Weighted Average—When It Is Defined
When the evaluation sample contains both actual classes, accuracy can be expressed as a weighted average of sensitivity and specificity. The weights are the proportions of observations that are actually positive and actually negative. This identity follows by splitting the number of correct classifications into the correct positives and correct negatives, then expressing each count as its class total multiplied by its class-specific rate.
If \(N_+>0\) and \(N_->0\), substituting these equalities into the accuracy formula gives
Both conditions matter: \(N_+>0\) makes sensitivity defined, and \(N_->0\) makes specificity defined. If either actual class is absent, accuracy remains defined as long as \(N>0\), but the absent class’s rate and this weighted-average expression are undefined. In the preceding example, \(N_+=0\); accuracy is \(0.4000\), but it cannot be calculated as a weighted average involving sensitivity.
Common Denominator Errors and Interpretation
Each measure answers a different conditional question, so matching the denominator to the question is essential. Sensitivity uses all actual positives, \(TP+FN\), not all predicted positives. Specificity and the false-positive rate use all actual negatives, \(FP+TN\), not all predicted negatives. Accuracy uses all observations.
A confusion matrix is tied to the evaluation observations and the threshold used to classify them. Changing the threshold can change the predicted labels and therefore the matrix counts and rates, even though the observed outcomes and fitted probabilities stay the same. A confusion matrix does not, by itself, describe every feature of the fitted probabilities; it records how the thresholded classifications compare with the observed labels.
It is also important to name the positive class when interpreting a result. “Sensitivity is \(0.75\)” is incomplete if the reader cannot tell which outcome counts as positive. In an applied report, state the outcome and evaluation sample, such as “the classifier identified 75% of the observed claims requiring additional investigation in this sample.”
Check Your Understanding
For each problem, apply the stated threshold rule first, then use the observed outcomes to count the four cells.
- A hypothetical screening model has \(TP=7\), \(FN=3\), \(FP=4\), and \(TN=16\). Calculate sensitivity, specificity, false-positive rate, and accuracy. (Answer: sensitivity \(=0.7000\), specificity \(=0.8000\), false-positive rate \(=0.2000\), accuracy \(=23/30=0.7667\).)
- A hypothetical maintenance model has four actual positives with fitted probabilities \(0.75, 0.60, 0.58,\) and \(0.22\), and five actual negatives with probabilities \(0.91, 0.61, 0.55, 0.20,\) and \(0.03\). At threshold \(0.60\), equality is classified positive. Find the four counts and four measures. (Answer: \(TP=2\), \(FN=2\), \(FP=2\), \(TN=3\); sensitivity \(=0.5000\), specificity \(=0.6000\), false-positive rate \(=0.4000\), accuracy \(=5/9=0.5556\).)
- A hypothetical alert model produces \(TP=24\), \(FN=8\), \(FP=18\), and \(TN=150\). Calculate the four measures and verify the complement relationship. (Answer: sensitivity \(=24/32=0.7500\), specificity \(=150/168=0.8929\), false-positive rate \(=18/168=0.1071\), accuracy \(=174/200=0.8700\); \(1-\text{specificity}=0.1071\).)
- In an evaluation sample, all 12 observed outcomes are negative. At the stated threshold, 5 are predicted positive and 7 are predicted negative. Which of the four measures are defined, and what are their values? (Answer: \(TP=0\), \(FN=0\), \(FP=5\), \(TN=7\); sensitivity is undefined, specificity \(=7/12=0.5833\), false-positive rate \(=5/12=0.4167\), accuracy \(=7/12=0.5833\).)
- A sample has \(N_+=40\) actual positives and \(N_-=60\) actual negatives. A classifier has sensitivity \(0.75\) and specificity \(0.80\). Find the implied counts and accuracy, and state why the weighted-average formula applies. (Answer: \(TP=30\), \(FN=10\), \(TN=48\), \(FP=12\); accuracy \(=(30+48)/100=0.7800=0.40(0.75)+0.60(0.80)\). Both actual classes are present, so both class-specific rates and the weighted-average expression are defined.)