Why Threshold Changes Matter
A classification model can assign a score to each observation, then use a threshold to turn that score into a positive or negative prediction. The scores and actual outcomes do not change when the threshold changes; the predicted labels do. Consequently, observations may move between cells of the confusion matrix even though the underlying data are fixed.
For example, a hypothetical insurer might use a model score to flag claims for additional review. Lowering the flagging threshold generally sends more claims for review, including more claims that did not have the target outcome. Raising it generally sends fewer. Tracking the four confusion-matrix counts makes this change explicit, without yet reducing those counts to performance rates.
Let \(y_i=1\) indicate an actual positive and \(y_i=0\) an actual negative. Let \(s_i\) be the score for observation \(i\), and let \(I(A)\) equal 1 when statement \(A\) is true and 0 otherwise. Under the \(s_i\ge t\) rule, the counts can be written as follows.
These definitions make the accounting checks clear. If the data contain \(P\) actual positives and \(N\) actual negatives, then at every threshold \( \mathrm{TP}+\mathrm{FN}=P \) and \( \mathrm{FP}+\mathrm{TN}=N \). The total number of observations is therefore both \(P+N\) and \(\mathrm{TP}+\mathrm{FP}+\mathrm{TN}+\mathrm{FN}\). These totals do not change with the threshold.
What Moves When the Threshold Changes?
Lowering the threshold can only add observations to the predicted-positive group; it cannot remove any. In particular, if \(t_2<t_1\), every score satisfying \(s_i\ge t_1\) also satisfies \(s_i\ge t_2\). Thus the predicted-positive set at \(t_1\) is contained in the predicted-positive set at \(t_2\). The derivation below connects this set relationship to the confusion-matrix counts.
To see why the result holds, suppose the threshold moves from \(t_1\) to a lower \(t_2\). The observations whose scores are in the interval \(t_2\le s_i<t_1\) change from predicted negative to predicted positive. If one of these observations is actually positive, it moves from FN to TP. If it is actually negative, it moves from TN to FP. No other observation changes cell. Therefore TP and FP cannot decrease, and FN and TN cannot increase.
A tied score group moves together under this rule: if its score equals the new threshold, the whole group is predicted positive. A threshold change that does not cross a score value leaves the classification of every observation unchanged. For discrete data, counts therefore change in steps rather than necessarily changing smoothly.
Identify which class is positive and whether a score equal to the threshold is predicted positive or negative.
Find \(P\) and \(N\) from the observed labels. These totals must be the same at every threshold.
At each threshold, include every observation whose score meets the stated rule, including all tied scores at the boundary when the rule is \(s_i\ge t\).
Separate the predicted positives by actual label to get TP and FP. Use the fixed actual-class totals to obtain FN and TN, then check that all four counts sum to the sample size.
Worked Examples
Worked Example: Recounting at Three Thresholds
A hypothetical screening data set has the following score groups. The first number in each row is the number of actual positives; the second is the number of actual negatives. Use the rule “predict positive when score is at least the threshold.” Find all four counts at thresholds \(0.75\), \(0.40\), and \(0.10\).
| Score | Actual positives | Actual negatives |
|---|---|---|
| 0.90 | 2 | 1 |
| 0.70 | 2 | 3 |
| 0.40 | 1 | 2 |
| 0.10 | 0 | 2 |
Method and actual totals. First sum the columns: \(P=2+2+1+0=5\) and \(N=1+3+2+2=8\). At each threshold, count score groups meeting \(s_i\ge t\) as predicted positive, and put the remaining observations in the predicted-negative group.
At \(t=0.75\). Only the score-\(0.90\) group is predicted positive, giving TP \(=2\) and FP \(=1\). Below the threshold are three actual positives (2 at \(0.70\) and 1 at \(0.40\)) and seven actual negatives (3 at \(0.70\), 2 at \(0.40\), and 2 at \(0.10\)). Therefore FN \(=3\) and TN \(=7\). Check: \(2+3=5\), \(1+7=8\), and the four counts total \(13\).
At \(t=0.40\). The \(0.90\), \(0.70\), and \(0.40\) groups are predicted positive because equality is included. Thus TP \(=2+2+1=5\) and FP \(=1+3+2=6\). The \(0.10\) group gives FN \(=0\) and TN \(=2\). Check: \(5+0=5\), \(6+2=8\), and the counts total \(13\).
At \(t=0.10\). Every group, including the group tied at \(0.10\), is predicted positive. TP \(=5\), FP \(=8\), FN \(=0\), and TN \(=0\). Both actual-class totals and the overall total still reconcile.
The predicted-positive count rises from \(3\) at \(0.75\), to \(11\) at \(0.40\), to \(13\) at \(0.10\). The method avoids a frequent mistake: counting only the observations above the threshold while forgetting that score \(0.40\) is included when the threshold is exactly \(0.40\).
Worked Example: A Small Threshold Increase Separates a Tied Group
A hypothetical claims triage model produces three score groups: score \(0.88\) has 3 actual positives and 1 actual negative; score \(0.62\) has 1 actual positive and 2 actual negatives; score \(0.25\) has 2 actual positives and 1 actual negative. Compare the confusion-matrix counts at thresholds \(0.62\) and \(0.63\), using the \(s_i\ge t\) rule.
Method and totals. The actual-positive total is \(P=3+1+2=6\), and the actual-negative total is \(N=1+2+1=4\). At \(0.62\), the score-\(0.62\) group is included. At \(0.63\), it is excluded.
At \(t=0.62\). The \(0.88\) and \(0.62\) groups are predicted positive. TP \(=3+1=4\), and FP \(=1+2=3\). The two actual positives at \(0.25\) are predicted negative, so FN \(=2\). The one actual negative at \(0.25\) is also predicted negative, so TN \(=1\). Checks: \(4+2=6\), \(3+1=4\), and \(4+3+2+1=10\).
At \(t=0.63\). Only the \(0.88\) group is predicted positive. TP \(=3\) and FP \(=1\). The predicted-negative groups contain \(1+2=3\) actual positives and \(2+1=3\) actual negatives, giving FN \(=3\) and TN \(=3\). Checks: \(3+3=6\), \(1+3=4\), and the counts again total \(10\).
Raising the threshold by \(0.01\) moves the whole group tied at \(0.62\): one actual positive moves from TP to FN, and two actual negatives move from FP to TN. The common error is to omit the two actual positives at \(0.25\) when counting FN. They are still predicted negative at both thresholds, so FN cannot be zero at \(0.62\), nor can it be only one at \(0.63\).
Worked Example: Comparing a Larger Threshold Reduction
A hypothetical review model assigns the following scores. At threshold \(0.80\) and then threshold \(0.50\), calculate the confusion-matrix counts and identify exactly which groups change prediction.
| Score | Actual positives | Actual negatives |
|---|---|---|
| 0.95 | 1 | 1 |
| 0.80 | 2 | 1 |
| 0.65 | 1 | 2 |
| 0.50 | 1 | 1 |
| 0.20 | 1 | 1 |
Method and totals. There are \(P=1+2+1+1+1=6\) actual positives and \(N=1+1+2+1+1=6\) actual negatives, for 12 observations.
At \(t=0.80\). The \(0.95\) and \(0.80\) groups are predicted positive. TP \(=1+2=3\), and FP \(=1+1=2\). The remaining groups have three actual positives and four actual negatives, so FN \(=3\) and TN \(=4\). Check: \(3+3=6\), \(2+4=6\), and the counts total \(12\).
At \(t=0.50\). The \(0.65\) and \(0.50\) groups join the predicted-positive group. TP \(=1+2+1+1=5\), and FP \(=1+1+2+1=5\). Only the \(0.20\) group remains predicted negative, giving FN \(=1\) and TN \(=1\). Check: \(5+1=6\), \(5+1=6\), and the counts still total \(12\).
The predicted-positive count increases from \(3+2=5\) to \(5+5=10\). The lowered threshold converts two actual positives from FN to TP and three actual negatives from TN to FP. This cell-by-cell account avoids the mistake of assuming a threshold change affects only the correctly classified observations.
Interpreting the Count Changes Carefully
The direction of movement is guaranteed, but its size depends on which observations lie between the old and new thresholds. A large threshold change may leave every count unchanged if no scores fall in the crossed interval. Conversely, a small change can alter several counts if many observations share a score near the boundary. When scores are tied, it is not valid to move only some members of the tied group unless the decision rule explicitly includes another tie-breaker.
Thresholds should also be compared using the same positive-class definition and the same set of evaluated observations. Reversing which outcome is called positive changes the meanings of TP, FP, TN, and FN. Changing the evaluation sample at the same time as the threshold makes it impossible to attribute count differences to the threshold alone.
On an evaluation sample, these counts describe that sample under a particular decision rule. They do not, by themselves, establish how the model will behave in a future population. The earlier material on validation design addresses keeping evaluation separate from model development. Here the task is narrower: hold the scores and actual labels fixed, and recalculate the four cells correctly.
Check Your Understanding
For each problem, use the rule “predict positive when score is at least the threshold.” Check the actual-class totals as well as the requested counts.
- A data set has 2 actual positives and 0 actual negatives at score \(0.90\), 1 actual positive and 2 actual negatives at \(0.60\), and 1 actual positive and 1 actual negative at \(0.30\). Find TP, FP, TN, and FN at threshold \(0.60\). (Answer: TP = 3, FP = 2, TN = 1, FN = 1.)
- Score \(0.82\) has 1 actual positive and 2 actual negatives; score \(0.55\) has 2 actual positives and 1 actual negative; score \(0.20\) has 1 actual positive and 2 actual negatives. Compare all four counts at thresholds \(0.56\) and \(0.55\). (Answer: At \(0.56\), TP = 1, FP = 2, TN = 3, FN = 3; at \(0.55\), TP = 3, FP = 3, TN = 2, FN = 1.)
- A score of \(0.91\) has 2 actual positives and 1 actual negative; \(0.70\) has 1 actual positive and 2 actual negatives; \(0.40\) has 2 actual positives and 1 actual negative; and \(0.15\) has 0 actual positives and 2 actual negatives. Compare the counts at thresholds \(0.70\) and \(0.40\). (Answer: At \(0.70\), TP = 3, FP = 3, TN = 3, FN = 2; at \(0.40\), TP = 5, FP = 4, TN = 2, FN = 0.)
- A score of \(0.95\) has 1 actual positive and 1 actual negative; \(0.75\) has 1 actual positive and 2 actual negatives; \(0.50\) has 0 actual positives and 1 actual negative; and \(0.25\) has 2 actual positives and 0 actual negatives. Find the counts at thresholds \(0.75\), \(0.50\), and \(0.25\). (Answer: At \(0.75\), TP = 2, FP = 3, TN = 1, FN = 2; at \(0.50\), TP = 2, FP = 4, TN = 0, FN = 2; at \(0.25\), TP = 4, FP = 4, TN = 0, FN = 0.)
- At threshold \(0.80\), a hypothetical evaluation has TP = 4, FP = 2, TN = 5, and FN = 3. Lowering the threshold to \(0.60\) newly includes a group with 2 actual positives and 1 actual negative. Find the new four counts and verify the class totals. (Answer: TP = 6, FP = 3, TN = 4, FN = 1; actual positives = 7 and actual negatives = 7 at both thresholds.)