Choosing Thresholds by F1

Why Choose a Threshold by F1?

A classifier can assign each case a score, such as an estimated probability of belonging to the positive class. A threshold turns those scores into decisions: predict positive when the score is at least the threshold, and predict negative otherwise. Raising or lowering the threshold changes the confusion matrix, so it can change the balance between finding actual positives and avoiding false alarms.

F1 summarizes precision and recall in one number. It is useful when both matter, but it does not encode every consequence of a decision. In particular, a threshold with the largest F1 need not minimize financial cost or meet a required service level. A sound selection process calculates F1 consistently, then checks whether the selected operating point is suitable for the actual decision.

The definitions and method below are aligned with the SOA Exam PA (Predictive Analytics) syllabus.

Definition: For a specified threshold, let $TP$ be the number of true positives, $FP$ the number of false positives, and $FN$ the number of false negatives. When their denominators are positive, precision is $P=TP/(TP+FP)$, and recall (sensitivity) is $R=TP/(TP+FN)$. The F1 score is the harmonic mean of precision and recall, $F_1=2PR/(P+R)$. Equivalently, when $2TP+FP+FN>0$, $F_1=2TP/(2TP+FP+FN)$. For cases where this count-based denominator is zero, use the convention $F_1=0$.

Deriving the Count-Based Formula

The count-based formula makes threshold comparisons convenient because it uses only the three confusion-matrix counts that affect F1. The derivation also clarifies when the harmonic-mean expression can be used directly.

Suppose $TP>0$. Then both $TP+FP$ and $TP+FN$ are positive, so precision and recall are defined and positive. Substitute their formulas into the harmonic mean:

\begin{aligned} F_1 &= \frac{2PR}{P+R} \\ &= \frac{2\left(\frac{TP}{TP+FP}\right) \left(\frac{TP}{TP+FN}\right)} {\frac{TP}{TP+FP}+\frac{TP}{TP+FN}} \\ &= \frac{\frac{2TP^2}{(TP+FP)(TP+FN)}} {\frac{TP(2TP+FP+FN)}{(TP+FP)(TP+FN)}} \\ &= \frac{2TP}{2TP+FP+FN}. \end{aligned}

The cancellation in the final line divides by $TP$, so it is valid under the stated condition $TP>0$. The count formula also gives a useful extension: if $TP=0$ but $2TP+FP+FN>0$, it returns $F_1=0$. In some such cases the precision-recall expression is undefined—for example, if both precision and recall equal zero, its denominator $P+R$ is zero. If $TP=FP=FN=0$, the count formula also has a zero denominator; the convention stated above assigns $F_1=0$. State the convention rather than attempting to divide by zero.

A Consistent Method for Comparing Thresholds

For a fixed validation set and a fixed definition of which class is positive, calculate a confusion matrix at each candidate threshold. Then calculate F1 from the counts and compare the results. The same observations, positive-class definition, and scoring rule must be used for every candidate; otherwise the comparison does not isolate the effect of changing the threshold.

Set the decision rule.
Specify which class is positive and whether a score equal to the threshold is classified as positive. For example, the rule may be “predict positive if score is at least $t$.”

Count the outcomes.
At each candidate threshold, obtain $TP$, $FP$, $FN$, and $TN$ on the same validation observations.

Calculate and compare F1.
Use $2TP/(2TP+FP+FN)$ when the denominator is positive, and apply the stated convention if it is zero. Select the largest value only if maximizing F1 is the decision objective.

Check the decision requirements.
Assess whether the selected threshold satisfies error-cost, workload, sensitivity, or other operational requirements. If it does not, select from the feasible thresholds or use the criterion that directly represents the objective.

Worked Example: Comparing Three Candidate Thresholds

A hypothetical claims review model assigns a score to each of 100 claims. The review team will classify a claim as positive when its score is at least the candidate threshold. The following confusion matrices were obtained on the same validation set. Which threshold has the highest F1?

Threshold	TP	FP	FN	TN
0.70	18	2	12	68
0.40	25	10	5	60
0.20	29	21	1	49

Method. Calculate $F_1=2TP/(2TP+FP+FN)$ for each threshold. First check that each row totals 100: $18+2+12+68=100$, $25+10+5+60=100$, and $29+21+1+49=100$. Each row also has 30 actual positives and 70 actual negatives, as expected from the fixed validation set.

At threshold 0.70, $2TP=2(18)=36$, and the denominator is $36+2+12=50$. Thus $F_1=36/50=0.7200$.

At threshold 0.40, $2TP=2(25)=50$, and the denominator is $50+10+5=65$. Thus $F_1=50/65=0.7692$, rounded to four decimal places.

At threshold 0.20, $2TP=2(29)=58$, and the denominator is $58+21+1=80$. Thus $F_1=58/80=0.7250$.

The highest F1 is 0.7692, at threshold 0.40. The counts also check against the stated class totals: for this threshold, $TP+FN=25+5=30$ actual positives and $FP+TN=10+60=70$ actual negatives.

Common mistake avoided. The arithmetic mean of precision and recall at threshold 0.40 is not F1. Here precision is $25/35=0.7143$ and recall is $25/30=0.8333$, so their arithmetic mean is 0.7738. Their harmonic mean is $2(0.7143)(0.8333)/(0.7143+0.8333)$, approximately 0.7692, agreeing with the count formula.

F1 Does Not Represent Every Decision Objective

F1 balances precision and recall but does not include true negatives or attach different monetary values to different mistakes. Consequently, maximizing F1 is not the same as minimizing an explicitly specified error cost. It also does not impose an operational limit, such as a maximum number of cases that a team can investigate.

Worked Example: Higher F1 but Higher Error Cost

A hypothetical insurer evaluates two thresholds on the same 100 policies. There are 40 actual positives and 60 actual negatives. A false negative costs $10, while a false positive costs $20. Compare the thresholds by F1 and by total error cost.

Threshold	TP	FP	FN	TN
A	20	10	20	50
B	28	25	12	35

Threshold A. The row totals $20+10+20+50=100$, with $20+20=40$ actual positives and $10+50=60$ actual negatives. Its F1 numerator is $2(20)=40$, and its denominator is $40+10+20=70$. Therefore $F_1=40/70=0.5714$, rounded to four decimal places. Its total error cost is $20($10)+10($20)=$200+$200=$400$.

Threshold B. The row totals $28+25+12+35=100$, again with 40 actual positives and 60 actual negatives. Its F1 numerator is $2(28)=56$, and its denominator is $56+25+12=93$. Therefore $F_1=56/93=0.6022$, rounded to four decimal places. Its total error cost is $12($10)+25($20)=$120+$500=$620$.

Threshold B has higher F1, but its total error cost is $220 higher. If minimizing the given cost is the governing objective, threshold A is preferable. F1 alone does not know that false positives cost twice as much as false negatives.

Common mistake avoided. Do not treat “largest F1” as a synonym for “best decision.” First identify the objective. If costs are specified, calculate them from the confusion matrix and compare thresholds on that basis.

F1 also depends on the number of actual positives and negatives in the evaluated population. It is not determined by sensitivity alone: the number of false positives, which contributes to precision, matters as well. Thus F1 measured in a population with one positive-class prevalence need not carry over unchanged to a population with another prevalence, even if sensitivity and false-positive rate are held fixed.

Worked Example: Same Operating Rates, Different Prevalence

Consider a hypothetical classifier with sensitivity 0.80 and false-positive rate 0.10. Compare its F1 in two populations of 1,000 cases: population A has 20% actual positives, while population B has 5% actual positives.

Population A. There are $0.20(1000)=200$ actual positives and $1000-200=800$ actual negatives. Sensitivity 0.80 gives $TP=0.80(200)=160$ and $FN=200-160=40$. The false-positive rate 0.10 gives $FP=0.10(800)=80$, so $TN=800-80=720$. The F1 numerator is $2(160)=320$, and its denominator is $320+80+40=440$. Therefore $F_1=320/440=0.7273$, rounded to four decimal places.

Population B. There are $0.05(1000)=50$ actual positives and $1000-50=950$ actual negatives. Sensitivity 0.80 gives $TP=0.80(50)=40$ and $FN=50-40=10$. The false-positive rate 0.10 gives $FP=0.10(950)=95$, so $TN=950-95=855$. The F1 numerator is $2(40)=80$, and its denominator is $80+95+10=185$. Therefore $F_1=80/185=0.4324$, rounded to four decimal places.

Both populations have the same sensitivity and false-positive rate, but their F1 values differ because the positive and negative class counts differ. The example does not imply that one threshold has changed its operating rates; it shows why an F1 value should be interpreted in the context of the population on which it was calculated.

Common mistake avoided. Do not infer that two F1 scores are directly comparable without checking the evaluated populations and their class composition. In particular, a score calculated in one population is not automatically a guaranteed score in another.

Operational Requirements Can Restrict the Choice

A threshold that maximizes F1 may be unusable if it produces too many cases for review, misses a required sensitivity target, or violates another explicit constraint. In that situation, first identify the feasible thresholds, then compare F1 among those that satisfy the requirements if F1 remains the chosen objective. If no candidate is feasible, the decision-maker must revisit the requirements, candidate thresholds, or model rather than silently choosing an infeasible maximum.

Worked Example: Selecting Among Feasible Thresholds

A hypothetical screening program has 100 actual positive cases and 900 actual negative cases. It requires sensitivity of at least 0.65 and can review no more than 75 false-positive cases. Three candidate thresholds produce the following counts. Choose the best feasible threshold by F1.

Threshold	TP	FP	FN	TN
High	55	18	45	882
Middle	70	60	30	840
Low	82	150	18	750

Check the data totals. Each row has $TP+FN=100$ actual positives and $FP+TN=900$ actual negatives. For example, the Middle row totals $70+60+30+840=1000$.

High threshold. Sensitivity is $55/100=0.55$, below the required 0.65, so this threshold is infeasible. Its F1 is $110/(110+18+45)=110/173=0.6358$, rounded to four decimal places.

Middle threshold. Sensitivity is $70/100=0.70$, meeting the minimum, and $FP=60$, within the review limit. It is feasible. Its F1 is $140/(140+60+30)=140/230=0.6087$, rounded to four decimal places.

Low threshold. Sensitivity is $82/100=0.82$, but $FP=150$, exceeding the limit of 75. It is infeasible. Its F1 is $164/(164+150+18)=164/332=0.4940$, rounded to four decimal places.

Only the Middle threshold meets both requirements, so it is the best feasible choice. The High threshold has the largest F1 of the three, but it fails the sensitivity requirement. Maximizing F1 without checking feasibility would produce an unacceptable recommendation.

Practical Checks Before Reporting an F1-Based Choice

When reporting a threshold selected by F1, make the basis of the comparison explicit. State the positive class, threshold rule, data set used for evaluation, and the F1 convention for a zero denominator. Report the confusion-matrix counts alongside the score: the counts show how the score arose and help a decision-maker assess the underlying trade-off.

Use the same evaluation cases. A threshold comparison is meaningful when each candidate is applied to the same cases.
Keep the class definition fixed. Reversing which outcome is positive changes the interpretation of precision, recall, and F1.
Do not substitute accuracy for F1. F1 excludes true negatives, while accuracy includes them; they answer different questions.
Check the denominator. If $2TP+FP+FN=0$, state the chosen convention rather than dividing by zero.
Check costs and constraints separately. If a decision has unequal error consequences or hard operating limits, F1 alone does not represent the full objective.

Key takeaway: F1 is the harmonic mean of precision and recall, and can be calculated as $2TP/(2TP+FP+FN)$ when its denominator is positive. It offers a compact comparison across thresholds, but the maximum-F1 threshold is not automatically the lowest-cost or operationally acceptable threshold.

Check Your Understanding

For each question, use the stated convention $F_1=0$ if the count-based denominator is zero. Round F1 values to four decimal places where needed.

A validation set has $TP=24$, $FP=6$, and $FN=10$ at one threshold. Calculate F1. (Answer: $48/(48+6+10)=0.7500$)
Two thresholds have counts $(TP,FP,FN)=(30,15,10)$ and $(34,25,6)$, respectively. Which has the higher F1? (Answer: The first threshold; $0.7059$ versus $0.6869$)
A population has 500 cases, 40% actual positives, sensitivity 0.75, and false-positive rate 0.08. Calculate $TP$, $FP$, $FN$, and F1. (Answer: $TP=150$, $FP=24$, $FN=50$, $F_1=300/374=0.8021$)
A program requires sensitivity of at least 0.60 and permits at most 20 false positives. Candidate A has $TP=48, FP=12, FN=32$; candidate B has $TP=60, FP=25, FN=20$. Which candidate is feasible, and what is its F1? (Answer: Candidate A is feasible, with sensitivity $48/80=0.60$, $FP=12$, and $F_1=96/(96+12+32)=0.6857$; candidate B exceeds the false-positive limit.)
At a threshold, $TP=0$, $FP=7$, and $FN=9$. Calculate F1 using the count formula and explain why the harmonic-mean formula cannot be evaluated as $2PR/(P+R)$ here. (Answer: $F_1=0/(0+7+9)=0$; precision and recall are both zero, so $P+R=0$ and the harmonic-mean expression divides by zero.)

What You'll Learn