Choosing a Classification Threshold

Why the Classification Threshold Is a Decision

A logistic model estimates the probability of an outcome, but a probability is not itself a classification. To label a case positive or negative, a decision-maker chooses a cutoff: cases at or above it are classified as positive. The familiar cutoff of 0.50 is only one possible choice. It is not automatically appropriate when missing a true positive and raising a false alarm have different consequences.

A useful threshold depends on the purpose of the classification. A screening process may require high sensitivity so that few true cases are missed. A hypothetical insurer’s review process might instead assign a monetary cost to each false positive and false negative. Prevalence also matters: even when sensitivity and specificity stay fixed, the proportion of positive classifications that are true positives can change substantially as the outcome becomes more or less common.

Definition: Let $Y=1$ denote the positive outcome and $Y=0$ the negative outcome. For a case with estimated probability $p=P(Y=1\mid \text{available predictors})$, a threshold $t$ classifies the case as positive if $p\ge t$, and negative if $p<t$. Sensitivity is $P(\text{classified positive}\mid Y=1)$; specificity is $P(\text{classified negative}\mid Y=0)$. A false positive has $Y=0$ but is classified positive; a false negative has $Y=1$ but is classified negative.

The threshold operates on predicted probabilities, whereas sensitivity and specificity describe how a chosen rule performs across cases whose outcomes are known. The confusion matrix introduced in the earlier material on confusion matrices summarizes these classifications. Changing the threshold changes which cases are called positive, and therefore usually changes both sensitivity and specificity.

Choosing a Threshold From Error Costs

Suppose the cost of a false positive is $C_{\mathrm{FP}}>0$, the cost of a false negative is $C_{\mathrm{FN}}>0$, and correct classifications have zero cost. For a case with estimated probability $p$, the expected cost of classifying it as positive is the false-positive cost multiplied by the chance the case is actually negative. The expected cost of classifying it as negative is the false-negative cost multiplied by the chance the case is actually positive.

L(\text{positive}\mid p)=C_{\mathrm{FP}}(1-p), \qquad L(\text{negative}\mid p)=C_{\mathrm{FN}}p.

Choose the positive classification when its expected cost is no greater than the negative classification’s expected cost. Solving that comparison gives the cost-minimizing threshold.

C_{\mathrm{FP}}(1-p)\le C_{\mathrm{FN}}p \quad\Longleftrightarrow\quad p\ge \frac{C_{\mathrm{FP}}}{C_{\mathrm{FP}}+C_{\mathrm{FN}}}.

Theorem: Under the stated two-action cost model, with correct-classification costs set to zero and with $p$ representing the probability of the positive outcome for the case being considered, the expected-cost-minimizing probability threshold is $t=C_{\mathrm{FP}}/(C_{\mathrm{FP}}+C_{\mathrm{FN}})$. At equality, the two actions have equal expected cost.

The result follows by comparing the costs of the two available actions for the same case. It does not require that the threshold be 0.50. If false negatives are more costly than false positives, the threshold is below 0.50; if false positives are more costly, it is above 0.50. This rule applies to the specified costs and probability. It does not account for other costs, capacity limits, or constraints unless they are included in the decision problem.

Worked Example: Minimizing Cost for a Hypothetical Claim Review

Method and assumptions. A hypothetical claim-screening team assigns a cost of $30 to a false positive and $120 to a false negative; correct decisions have zero cost. For each case, use the stated logistic probability as $p$. There is no interest or mortality assumption, and no additional distributional assumption is needed for these individual expected-cost comparisons.

Step 1: Find the threshold.

t=\frac{30}{30+120}=\frac{30}{150}=0.20.

Step 2: Compare actions for each case. For $p=0.16$, the expected cost of a positive classification is $30(1-0.16)=30(0.84)=\$25.20$. The expected cost of a negative classification is $120(0.16)=\$19.20$, so classify this case as negative.

For $p=0.24$, the positive-classification cost is $30(0.76)=\$22.80$, and the negative-classification cost is $120(0.24)=\$28.80$. Classify this case as positive.

For $p=0.70$, the positive-classification cost is $30(0.30)=\$9.00$, while the negative-classification cost is $120(0.70)=\$84.00$. Classify this case as positive.

Check. The decisions agree with the rule $p\ge0.20$. At $p=0.20$, both expected costs equal $30(0.80)=\$24.00$. Common mistake avoided: using 0.50 by habit would ignore that a false negative costs four times as much as a false positive.

Choosing a Threshold From Sensitivity or Specificity Goals

Sometimes the objective is stated as a performance requirement rather than a cost. For example, a screening rule may need sensitivity of at least 0.75. To choose a threshold, calculate sensitivity and specificity at candidate cutoffs using cases with known outcomes. As the threshold decreases, more cases are classified positive: sensitivity cannot decrease, while specificity cannot increase.

If the requirement is “sensitivity at least $s_0$,” a common selection rule is to choose the highest threshold that still achieves that sensitivity. This favors specificity among the thresholds satisfying the requirement. If the requirement instead sets a minimum specificity, one can choose the lowest threshold that meets it, favoring sensitivity. A threshold selected from a finite data set may not deliver exactly the same performance in future cases.

Worked Example: Meeting a Sensitivity Requirement

Method and assumptions. A hypothetical screening study has ten cases with known outcomes and the following predicted probabilities. Four cases are positive, with probabilities 0.82, 0.61, 0.38, and 0.22. Six are negative, with probabilities 0.70, 0.44, 0.31, 0.18, 0.09, and 0.05. The requirement is sensitivity of at least 0.75; among qualifying cutoffs, choose the highest. There is no interest or mortality assumption, and no probability distribution beyond these stated records is assumed.

Step 1: Determine the number of true positives required. There are four actual positive cases, so sensitivity of at least 0.75 requires at least $0.75(4)=3$ true positives.

Step 2: Find the highest threshold that retains at least three positive cases. The third-highest probability among actual positives is 0.38. A threshold above 0.38 would exclude that case and leave only two true positives, giving sensitivity $2/4=0.50$. At threshold $t=0.38$, the three actual positives with probabilities 0.82, 0.61, and 0.38 are classified positive. Thus sensitivity is $3/4=0.75$.

Step 3: Count false positives and true negatives. Among the actual negatives, probabilities 0.70 and 0.44 meet or exceed 0.38. Therefore, there are 2 false positives and $6-2=4$ true negatives. Specificity is $4/6=0.6667$, to four decimals.

Check. The confusion counts are 3 true positives, 1 false negative, 2 false positives, and 4 true negatives. Sensitivity is $3/(3+1)=0.75$; specificity is $4/(4+2)=0.6667$. A threshold just above 0.38 would fail the stated sensitivity requirement. Common mistake avoided: choosing a cutoff for a desired sensitivity without checking how many actual positive cases the data contain.

How Prevalence Changes the Decision

Prevalence is the proportion of cases in the population that have the positive outcome. With sensitivity $Se$, specificity $Sp$, and prevalence $\pi=P(Y=1)$, the positive predictive value is the probability that a case is truly positive given that it was classified positive:

P(Y=1\mid \text{classified positive}) = \frac{Se\,\pi}{Se\,\pi+(1-Sp)(1-\pi)}.

The numerator is the proportion of all cases that are true positives. The denominator is the proportion classified positive, which includes both true positives and false positives. Thus, even if sensitivity and specificity stay the same, a lower prevalence generally means a smaller positive predictive value: the false positives are drawn from a larger group of actual negatives relative to the number of actual positives.

There are two related but distinct ways to express the role of prevalence in cost-based decisions. If $p$ is already the probability for the case under the population being considered, the cost-based threshold $C_{\mathrm{FP}}/(C_{\mathrm{FP}}+C_{\mathrm{FN}})$ does not contain prevalence as a separate term. Prevalence is reflected in that case’s probability. Alternatively, a decision can be written in terms of how strongly a score supports the positive outcome relative to the negative outcome. In that form, the prevalence appears explicitly.

Derivation: Let $S$ be a score with conditional densities $f_1(s)$ given $Y=1$ and $f_0(s)$ given $Y=0$, and let $\pi=P(Y=1)$. Bayes’ rule gives posterior odds $\frac{P(Y=1\mid S=s)}{P(Y=0\mid S=s)} =\frac{f_1(s)}{f_0(s)}\frac{\pi}{1-\pi}$. The expected-cost rule classifies positive when posterior odds are at least $C_{\mathrm{FP}}/C_{\mathrm{FN}}$, because $p\ge C_{\mathrm{FP}}/(C_{\mathrm{FP}}+C_{\mathrm{FN}})$ is equivalent to $p/(1-p)\ge C_{\mathrm{FP}}/C_{\mathrm{FN}}$. Therefore, the score evidence must satisfy $\frac{f_1(s)}{f_0(s)}\ge\frac{C_{\mathrm{FP}}(1-\pi)}{C_{\mathrm{FN}}\pi}$. This likelihood-ratio form makes the effect of prevalence explicit. It assumes the conditional score distributions and costs in the comparison apply to the population being considered.

Worked Example: Prevalence and the Evidence Needed for Action

Method and assumptions. Consider a hypothetical score whose likelihood ratio $f_1(s)/f_0(s)$ is 3 for a particular case. A false positive costs $10 and a false negative costs $90, with zero cost for correct classifications. Compare two populations with prevalences 0.02 and 0.10. Treat the stated likelihood ratio as given; no parametric distribution for the score is assumed. There is no interest or mortality assumption.

Step 1: Find the posterior-probability cutoff.

t=\frac{10}{10+90}=0.10.

Step 2: Evaluate the case at prevalence 0.02. The likelihood-ratio cutoff is $\frac{10(1-0.02)}{90(0.02)}=\frac{9.8}{1.8}=5.4444$, to four decimals. The case’s likelihood ratio of 3 is below this cutoff, so classify it as negative. Equivalently, its posterior odds are $3(0.02/0.98)=0.0612245$, giving posterior probability $0.0612245/(1+0.0612245)=0.0577$, to four decimals. This is below 0.10.

Step 3: Evaluate the case at prevalence 0.10. The likelihood-ratio cutoff is $\frac{10(1-0.10)}{90(0.10)}=\frac{9}{9}=1$. The likelihood ratio of 3 exceeds this cutoff, so classify the case as positive. Its posterior odds are $3(0.10/0.90)=0.3333333$, giving posterior probability $0.3333333/(1+0.3333333)=0.2500$, to four decimals.

Check. Both posterior probabilities are compared with the same cost-based cutoff, 0.10, and produce the same decisions as the likelihood-ratio comparisons. Common mistake avoided: treating the score’s likelihood ratio as if it were the posterior probability. The score evidence alone does not determine the posterior; the prevalence also matters.

Practical Checks Before Reporting a Threshold

A threshold is meaningful only in relation to the decision objective and the population to which it will be applied. Before reporting one, state whether it was chosen to minimize specified costs, meet a sensitivity or specificity requirement, or serve some other explicit purpose. Then report the resulting confusion counts or performance measures, not just the cutoff.

Use the stated costs. A threshold chosen from costs assumes those costs are the relevant consequences of the two types of error. If correct decisions also have costs, or if actions have other consequences, include them in the expected-cost comparison.
Keep probability and evidence distinct. A probability threshold applies to a probability for the population of interest. A likelihood-ratio threshold must include the prior prevalence as well as the cost ratio.
Do not confuse sensitivity with predictive value. Sensitivity conditions on actual positives; positive predictive value conditions on positive classifications. The latter depends on prevalence.
Check the threshold convention. Here, a case with probability exactly equal to the threshold is classified positive. If a different convention is used, state it, especially when tied scores occur.

Threshold selection is therefore a decision step after probabilities have been produced. The cutoff of 0.50 is appropriate only when the decision setup supports it; unequal error costs, a required screening sensitivity, and a change in prevalence can all lead to a different operational rule.

Check Your Understanding

For each hypothetical problem, use the stated rule and show the key comparison before checking the final answer.

A review system assigns false-positive cost $18 and false-negative cost $72, with zero costs for correct classifications. Find the cost-minimizing probability threshold and state the decision for a case with probability 0.30. (Answer: threshold 0.20; classify positive.)
A sample contains 20 actual positives. A threshold classifies 17 of them as positive. What is sensitivity, and how many false negatives are there? (Answer: sensitivity $17/20=0.8500$; 3 false negatives.)
A classifier has sensitivity 0.90, specificity 0.80, and prevalence 0.10. Find the positive predictive value. (Answer: $0.09/[0.09+0.18]=0.3333$, to four decimals.)
False-positive cost is $5, false-negative cost is $45, and prevalence is 0.05. Find the likelihood-ratio cutoff for a positive decision. (Answer: $5(0.95)/[45(0.05)]=2.1111$, to four decimals.)
A screening goal requires sensitivity of at least 0.80. There are five actual positive cases with probabilities 0.91, 0.73, 0.52, 0.36, and 0.14. What is the highest threshold that meets the goal? (Answer: 0.36; it retains 4 of 5 positives, giving sensitivity 0.80.)

What You'll Learn