Why Change the Threshold?
A classification model can assign each case a score, such as an estimated probability of a positive outcome. A decision rule turns that score into an action: approve or decline, refer or do not refer, investigate or do not investigate. The threshold sets the score at which the action changes. Even when two groups receive scores from the same fitted model, applying one common threshold can produce different false-positive and false-negative rates across groups.
A group-specific threshold is one possible intervention: use a different cutoff for each group, with the aim of changing one or more group-level error rates. This changes the decision rule, not the underlying model or its scores. Its effects should therefore be measured using outcomes and group definitions appropriate to the intended decision. Earlier material on equalized odds defines the fairness criterion of matching true-positive and false-positive rates; here the focus is how thresholds affect those rates.
These definitions condition on the actual outcome within each group. In a validation sample, estimate each rate by dividing the corresponding count by the number of cases with that outcome in that group. The denominator matters: the false-positive rate is not the fraction of predicted-positive cases that are actually negative. That latter quantity is the false discovery proportion, which has a different denominator.
Unless a problem states otherwise, assume the score and observed outcome are available for a representative validation sample, the positive class is defined consistently across groups, and a score equal to the threshold is classified positive. The calculations below use empirical counts, so no parametric score distribution is assumed. A threshold chosen from validation data may not deliver the same rates in future data.
How Thresholds Change Error Rates
For each group, lowering the threshold can only add predicted-positive cases; it cannot remove them. Thus, as the threshold falls, both the true-positive rate and the false-positive rate are nondecreasing. How quickly each rate changes depends on how many positive and negative cases have scores in the newly included range. A threshold change can improve one error rate while worsening another.
The following justification makes the mechanism explicit. If a threshold is lowered from \(t_1\) to \(t_2\), where \(t_2<t_1\), the newly classified-positive cases have scores in \([t_2,t_1)\). The added true positives are positive-outcome cases in that score interval, and the added false positives are negative-outcome cases there. For a group with \(N_{1g}\) positive outcomes and \(N_{0g}\) negative outcomes, the rate changes are the corresponding added counts divided by \(N_{1g}\) and \(N_{0g}\), respectively. Both changes are nonnegative, but their sizes need not be equal.
This also explains why a group-specific threshold does not automatically achieve a desired fairness target. Each group has its own attainable combinations of true-positive and false-positive rates as the threshold moves. With discrete scores, a deterministic threshold may jump over a target rate. And matching one rate does not ensure that the other rate matches too.
Worked Comparisons
Worked Example: Matching Both Error Rates With Separate Thresholds
A hypothetical referral model has scores in four bands. The table gives validation counts for two groups, each with 100 positive outcomes and 100 negative outcomes. Compare a common threshold of \(0.6\) with a group-specific rule that uses \(0.6\) for Group A and \(0.4\) for Group B. A case is predicted positive when its score is at least the threshold.
| Group and actual outcome | Score at least 0.8 | 0.6 to below 0.8 | 0.4 to below 0.6 | Below 0.4 |
|---|---|---|---|---|
| A, positive outcome | 60 | 20 | 10 | 10 |
| A, negative outcome | 10 | 10 | 20 | 60 |
| B, positive outcome | 40 | 20 | 20 | 20 |
| B, negative outcome | 5 | 10 | 5 | 80 |
Method. Sum the bands at or above each cutoff to obtain the confusion-matrix counts, then divide true positives by all actual positives and false positives by all actual negatives. This avoids comparing error counts with different or inappropriate denominators.
Common threshold, \(0.6\). For Group A, the predicted-positive counts are \(60+20=80\) among positive outcomes and \(10+10=20\) among negative outcomes. Thus \(TP=80\), \(FN=100-80=20\), \(FP=20\), and \(TN=100-20=80\). Its rates are \(\operatorname{TPR}_A=80/100=0.80\) and \(\operatorname{FPR}_A=20/100=0.20\).
For Group B at \(0.6\), \(TP=40+20=60\), \(FN=100-60=40\), \(FP=5+10=15\), and \(TN=100-15=85\). Therefore \(\operatorname{TPR}_B=60/100=0.60\) and \(\operatorname{FPR}_B=15/100=0.15\). The common cutoff gives different rates across groups.
Group-specific thresholds. Keep Group A at \(0.6\). For Group B, lower the threshold to \(0.4\), which adds the \(0.4\)-to-below-\(0.6\) band. Then \(TP=40+20+20=80\), \(FN=100-80=20\), \(FP=5+10+5=20\), and \(TN=100-20=80\). Group B now has \(\operatorname{TPR}_B=80/100=0.80\) and \(\operatorname{FPR}_B=20/100=0.20\), matching Group A in this validation sample.
As a check, the common-threshold total error count is \(20+20+40+15=95\) out of 400 cases, or \(95/400=0.2375\). Under the group-specific rule it is \(20+20+20+20=80\) out of 400, or \(0.2000\). The arithmetic is consistent with the confusion counts. This example shows that equalized rates can be attainable for particular score distributions; it does not show that they will be attainable for every model or population.
Worked Example: Enforcing a False-Positive-Rate Limit
A hypothetical screening program requires each group’s empirical false-positive rate to be no greater than \(0.05\). For each group below, choose the lowest available threshold that satisfies the limit, thereby retaining the greatest possible true-positive rate among the listed thresholds. Each group has 100 positive and 100 negative outcomes.
| Group | Threshold | True positives | False positives |
|---|---|---|---|
| C | 0.8 | 70 | 5 |
| C | 0.6 | 85 | 10 |
| D | 0.8 | 50 | 2 |
| D | 0.6 | 75 | 5 |
Method. Check the false-positive-rate constraint at each candidate threshold, then use the feasible threshold with the larger true-positive count. The common mistake is to choose the threshold with the most true positives without first checking whether it violates the constraint.
For Group C at \(0.8\), \(\operatorname{FPR}=5/100=0.05\) and \(\operatorname{TPR}=70/100=0.70\), so the threshold is feasible. At \(0.6\), \(\operatorname{FPR}=10/100=0.10\), which exceeds the limit, even though \(\operatorname{TPR}=85/100=0.85\). Select \(0.8\). Its counts are \(TP=70\), \(FN=100-70=30\), \(FP=5\), and \(TN=100-5=95\).
For Group D at \(0.8\), the rates are \(2/100=0.02\) and \(50/100=0.50\). At \(0.6\), they are \(5/100=0.05\) and \(75/100=0.75\). Both thresholds meet the limit, and \(0.6\) gives the larger true-positive rate, so select \(0.6\). The resulting counts are \(TP=75\), \(FN=25\), \(FP=5\), and \(TN=95\).
The selected group-specific rule has \(30+5+25+5=65\) errors across 400 cases, or \(65/400=0.1625\). A common threshold of \(0.8\) would also meet the limit in both groups, but it would produce \(30+5+50+2=87\) errors, or \(87/400=0.2175\). The comparison is valid for these stated candidate thresholds and counts; it does not establish that group-specific thresholds are always preferable under other objectives.
Worked Example: Equal Error Rates Do Not Equalize Predictive Value
Two groups each have 100 validation cases. For both groups, suppose a selected threshold gives a true-positive rate of \(0.75\) and a false-positive rate of \(0.20\). Group E has 40 positive outcomes and 60 negative outcomes; Group F has 20 positive outcomes and 80 negative outcomes. Calculate the confusion counts and positive predictive value, defined as the proportion of predicted-positive cases that actually have a positive outcome.
Method. Multiply the true-positive rate by the number of actual positives, and the false-positive rate by the number of actual negatives. Then calculate \(TP/(TP+FP)\). This avoids the common mistake of treating equal error rates as proof that all group-level measures are equal.
For Group E, \(TP=0.75(40)=30\), so \(FN=40-30=10\). Also \(FP=0.20(60)=12\), so \(TN=60-12=48\). The predicted-positive total is \(30+12=42\), and the positive predictive value is \(30/42=0.714285\ldots\), or \(0.7143\) to four decimal places.
For Group F, \(TP=0.75(20)=15\), so \(FN=20-15=5\). Also \(FP=0.20(80)=16\), so \(TN=80-16=64\). The predicted-positive total is \(15+16=31\), and the positive predictive value is \(15/31=0.483870\ldots\), or \(0.4839\) to four decimal places. Both groups have the same true-positive and false-positive rates, but their predictive values differ because their outcome prevalence differs. The counts also check: each group totals 100 cases.
What a Threshold Intervention Can—and Cannot—Do
A useful way to specify a threshold intervention is to state the objective before selecting cutoffs. Examples include requiring a maximum false-positive rate, seeking equal true-positive rates, or seeking equal true-positive and false-positive rates. The selected criterion changes which thresholds count as acceptable. A group-specific rule that matches one rate can leave other rates unequal, as the examples demonstrate.
Specify what a positive prediction triggers and which observed outcome counts as positive. Use the same definitions across groups.
For each candidate threshold, calculate confusion counts and rates using the appropriate within-group outcome denominators.
For example, require an empirical false-positive rate below a stated limit or aim to match both error rates. Check whether the available thresholds can meet it.
State the thresholds, resulting counts and rates, and the evidence and policy assumptions needed to interpret them.
Several limitations matter in actuarial and other high-impact decisions. First, group-specific thresholds require group membership to be available and used in the decision process. That can raise privacy, governance, and legal concerns. Laws vary by jurisdiction, protected characteristic, product, and decision; a statistical result alone cannot determine whether a particular policy is lawful. Appropriate legal and compliance review is essential.
Second, rates estimated from a finite validation sample are uncertain, especially when a group has few positive or negative outcomes. A threshold that appears to meet a target in one sample may miss it in another. The calculations here describe observed counts, not guaranteed future performance. Third, the choice of threshold does not correct biased labels, incomplete outcome measurement, or unequal access to the process that generates the data. Nor does matching error rates establish that the decision is fair in every relevant sense.
Finally, group-specific thresholds may complicate consistent administration and explanation. An organization should be able to state why the chosen criterion is relevant, which tradeoffs it accepts, and how group membership is handled. These are substantive choices, not technical consequences dictated by the score. This treatment of thresholds is aligned with the SOA Exam PA (Predictive Analytics) syllabus.
Check Your Understanding
For each problem, use the stated threshold convention that scores at or above the cutoff are predicted positive.
- A group has 60 positive outcomes and 140 negative outcomes. At a proposed cutoff, there are 42 true positives and 14 false positives. Find the false-negative count, true-negative count, true-positive rate, and false-positive rate. (Answer: \(FN=18\), \(TN=126\), \(\operatorname{TPR}=0.7000\), \(\operatorname{FPR}=0.1000\).)
- Group R has 30 positive and 70 negative outcomes; Group S has 50 positive and 50 negative outcomes. A rule gives both groups \(\operatorname{TPR}=0.80\) and \(\operatorname{FPR}=0.10\). Find the positive predictive value for each group. (Answer: Group R \(=24/(24+7)=0.7742\); Group S \(=40/(40+5)=0.8889\).)
- Each group has 100 positive and 100 negative outcomes. At threshold \(0.7\), Group M has 72 true positives and 8 false positives; at threshold \(0.5\), it has 86 true positives and 18 false positives. If the required false-positive rate is at most \(0.10\), which listed threshold should be selected, and what are its two rates? (Answer: Select \(0.7\); \(\operatorname{TPR}=0.7200\), \(\operatorname{FPR}=0.0800\).)
- For Group U, lowering a threshold adds 12 positive-outcome cases and 6 negative-outcome cases to the predicted-positive set. There are 80 positive outcomes and 120 negative outcomes in the group. By how much do the true-positive and false-positive rates increase? (Answer: \(\Delta\operatorname{TPR}=12/80=0.1500\); \(\Delta\operatorname{FPR}=6/120=0.0500\).)
- Two groups have equal false-positive rates, but one has a higher prevalence of positive outcomes. Is equal positive predictive value guaranteed? State the conclusion and the relevant reason. (Answer: No. Predictive value also depends on prevalence, so equal false-positive rates alone do not guarantee equal positive predictive values.)