Deriving Cost-Sensitive Probability Thresholds

Why a Probability Threshold Depends on Costs

A model can estimate the probability that an outcome belongs to the positive class, but that probability alone does not determine the best action. If a false negative is much more costly than a false positive, acting on a relatively small estimated probability may be sensible. If false positives are more costly, the decision rule may require stronger evidence before assigning the positive class.

The threshold can be derived by comparing the expected costs of the two available actions for a case with estimated probability $p$. This is a conditional, case-by-case comparison. It differs from selecting a threshold to meet a sensitivity, specificity, or capacity requirement: here the objective is to minimize expected cost under the stated cost assumptions. The derivation and examples are aligned with the SOA Exam PA (Predictive Analytics) syllabus.

Definition: Let $Y=1$ denote the positive outcome and $Y=0$ the negative outcome. For a case with calibrated probability $p=P(Y=1\mid\text{case information})$, a false positive occurs when the positive class is predicted but $Y=0$, and a false negative occurs when the negative class is predicted but $Y=1$. A calibrated probability is one that represents the event probability for cases with that information; in particular, among cases assigned probability $p$, the positive-outcome frequency is expected to be $p$.

Deriving the Threshold When Correct Decisions Cost Nothing

First suppose the only costs are $C_{\mathrm{FP}}>0$ for a false positive and $C_{\mathrm{FN}}>0$ for a false negative. A correct prediction has cost zero. For a case with probability $p$, predicting positive incurs a false-positive cost only when $Y=0$, which has probability $1-p$. Predicting negative incurs a false-negative cost only when $Y=1$, which has probability $p$.

\begin{aligned} E[\text{cost}\mid\text{predict positive}]&=(1-p)C_{\mathrm{FP}},\\ E[\text{cost}\mid\text{predict negative}]&=pC_{\mathrm{FN}}. \end{aligned}

Choose the positive prediction when its conditional expected cost is no greater than the negative prediction's cost. The inequality and its rearrangement give the minimum-cost threshold:

(1-p)C_{\mathrm{FP}}\le pC_{\mathrm{FN}} \quad\Longleftrightarrow\quad C_{\mathrm{FP}}\le p(C_{\mathrm{FP}}+C_{\mathrm{FN}}) \quad\Longleftrightarrow\quad p\ge \frac{C_{\mathrm{FP}}}{C_{\mathrm{FP}}+C_{\mathrm{FN}}}.

Theorem: If correct predictions have zero cost and both error costs are positive, the expected-cost-minimizing rule is to predict positive when $p\ge t^*$, where $t^*=C_{\mathrm{FP}}/(C_{\mathrm{FP}}+C_{\mathrm{FN}})$. At equality, the two actions have equal expected cost, so either action minimizes cost. If $C_{\mathrm{FP}}=C_{\mathrm{FN}}$, the threshold is $0.5$; a larger false-negative cost lowers the threshold, while a larger false-positive cost raises it.

The derivation compares costs conditional on the case's probability. It does not require a separate prevalence multiplier in the threshold: prevalence may affect the probabilities produced by a model, but once $p$ is calibrated for the case, the two conditional expected costs use $p$ and $1-p$. For a collection of cases, expected total cost is the sum of the case-level expected costs, so minimizing each case's cost also minimizes that sum when decisions do not interact and the costs are additive.

Worked Example: Unequal False-Positive and False-Negative Costs

A hypothetical review process assigns a cost of $80 to each false positive and $320 to each false negative. Correct decisions cost zero. A calibrated model assigns probabilities $0.18$ and $0.27$ to two separate cases. Derive the threshold, decide how to classify each case, and calculate the chosen expected cost for each.

Method. Use the zero-correct-cost threshold $C_{\mathrm{FP}}/(C_{\mathrm{FP}}+C_{\mathrm{FN}})$, then compare conditional expected costs for each probability. This avoids treating $0.5$ as a universal decision threshold.

Threshold. The cost ratio gives $t^*=80/(80+320)=80/400=0.20$. Thus predict positive at probabilities of at least $0.20$.

Case with $p=0.18$. Predicting positive has expected cost $(1-0.18)(80)=0.82(80)=\$65.60$. Predicting negative has expected cost $0.18(320)=\$57.60$. Since $\$57.60<\$65.60$, predict negative; its expected cost is $\$57.60$.

Case with $p=0.27$. Predicting positive has expected cost $(1-0.27)(80)=0.73(80)=\$58.40$. Predicting negative has expected cost $0.27(320)=\$86.40$. Since $\$58.40<\$86.40$, predict positive; its expected cost is $\$58.40$.

Check. The first probability is below $0.20$ and is assigned negative; the second is above $0.20$ and is assigned positive. The two direct cost comparisons agree with the threshold rule.

When Correct Decisions Also Have Costs

The zero-cost assumption for correct decisions is convenient, not essential. For example, a positive action might involve a treatment or review expense even when the positive outcome is present. A negative action might also have a routine administration cost. The decision should then compare all four outcome-action costs, rather than counting only errors.

Definition: Let $C_{\mathrm{FP}}$, $C_{\mathrm{FN}}$, $C_{\mathrm{TP}}$, and $C_{\mathrm{TN}}$ denote the costs of a false positive, false negative, true positive, and true negative, respectively. For a case with positive-outcome probability $p$, the expected cost of predicting positive is $(1-p)C_{\mathrm{FP}}+pC_{\mathrm{TP}}$. The expected cost of predicting negative is $pC_{\mathrm{FN}}+(1-p)C_{\mathrm{TN}}$.

To derive the decision rule, compare the two expressions. Subtracting the true-negative cost from the false-positive cost and the true-positive cost from the false-negative cost gives the incremental costs $\Delta_{\mathrm{FP}}=C_{\mathrm{FP}}-C_{\mathrm{TN}}$ and $\Delta_{\mathrm{FN}}=C_{\mathrm{FN}}-C_{\mathrm{TP}}$.

\begin{aligned} (1-p)C_{\mathrm{FP}}+pC_{\mathrm{TP}} &\le pC_{\mathrm{FN}}+(1-p)C_{\mathrm{TN}}\\ (1-p)\Delta_{\mathrm{FP}} &\le p\Delta_{\mathrm{FN}}. \end{aligned}

When both incremental costs are positive, rearranging yields the threshold $t^*=\Delta_{\mathrm{FP}}/(\Delta_{\mathrm{FP}}+\Delta_{\mathrm{FN}})$. Thus it is the incremental costs, relative to the costs of the alternative correct outcomes, that determine the threshold. If these differences are not both positive, do not apply this formula automatically: compare the two original expected-cost expressions over $0\le p\le1$. Depending on the costs, one action may be optimal for every probability, or the preferred action may change in the opposite direction.

Worked Example: A Nonzero Cost for a True Positive

In a hypothetical screening program, a false positive costs $90, a true positive costs $20, a false negative costs $210, and a true negative costs $0. Derive the threshold, decide the action at $p=0.30$, and verify the costs at the exact threshold. Also check the costs when the threshold is rounded to four decimal places.

Method. Compare the full expected costs, including the $20 true-positive cost. This avoids incorrectly using the zero-correct-cost formula with $90$ and $210$ alone.

Expected costs. Predicting positive costs $(1-p)90+p20=90-70p$. Predicting negative costs $p210+(1-p)0=210p$. Set them equal to find the point where the preferred action changes: $90-70p=210p$, so $90=280p$ and $t^*=90/280=9/28\approx0.3214$.

Decision at $p=0.30$. The positive-action cost is $90-70(0.30)=90-21=\$69.00$. The negative-action cost is $210(0.30)=\$63.00$. Therefore, predict negative, with expected cost $\$63.00$.

Exact-threshold check. At $p=9/28$, the positive-action cost is $90-70(9/28)=90-22.50=\$67.50$. The negative-action cost is $210(9/28)=\$67.50$. The costs agree, as they must at the threshold.

Rounded-threshold check. Using $p=0.3214$, the positive-action cost is $90(0.6786)+20(0.3214)=61.074+6.428=\$67.502$, or $\$67.50$ to the nearest cent. The negative-action cost is $210(0.3214)=\$67.494$, or $\$67.49$ to the nearest cent. The one-cent difference arises because the displayed probability is rounded; the exact threshold gives equal costs.

Check. The threshold is below $0.5$ because the incremental false-negative cost, $210-20=\$190$, exceeds the incremental false-positive cost, $90-0=\$90$. The threshold is $90/(90+190)=90/280$, matching the direct equality calculation.

General Cost-Matrix Example and a Common Trap

A general cost matrix can change the threshold even when both correct outcomes have nonzero costs. The reliable procedure is to calculate each action's conditional expected cost first, then simplify the comparison. This also makes the rule auditable in a report or an exam response.

Worked Example: Comparing Actions With Costs in All Four Cells

A hypothetical service triage system has costs $C_{\mathrm{FP}}=\$150$, $C_{\mathrm{TP}}=\$40$, $C_{\mathrm{FN}}=\$360$, and $C_{\mathrm{TN}}=\$10$. Derive the threshold, choose an action for a case with $p=0.35$, and confirm that the actions tie at the exact threshold.

Method. Use the full cost matrix and its incremental costs. This avoids ignoring the cost of true-positive and true-negative decisions.

The incremental false-positive cost is $\Delta_{\mathrm{FP}}=150-10=\$140$. The incremental false-negative cost is $\Delta_{\mathrm{FN}}=360-40=\$320$. Both are positive, so the positive-action rule is $p\ge140/(140+320)=140/460=7/23\approx0.3043$.

At $p=0.35$, predicting positive costs $0.65(150)+0.35(40)=97.50+14.00=\$111.50$. Predicting negative costs $0.35(360)+0.65(10)=126.00+6.50=\$132.50$. Therefore, predict positive.

At the exact threshold $p=7/23$, the positive-action cost is $150-110(7/23)=150-770/23=2680/23\approx\$116.52$. The negative-action cost is $10+350(7/23)=10+2450/23=2680/23\approx\$116.52$. The costs are equal before rounding.

Check. Since $0.35>7/23$, the threshold rule selects positive, consistent with the direct comparison. The displayed cost at the tie is rounded to cents only after both exact expressions are shown.

A frequent exam trap is to reverse the error-cost ratio. For example, with $C_{\mathrm{FP}}=\$80$ and $C_{\mathrm{FN}}=\$320$, the incorrect threshold $320/(80+320)=0.80$ assigns a very high threshold even though false negatives are the more expensive error. The correct derivation compares $(1-p)80$ with $p320$, giving $80/400=0.20$. At $p=0.30$, the positive action costs $0.70(80)=\$56$, while the negative action costs $0.30(320)=\$96$; positive is clearly less costly. The ratio with false-positive cost in the numerator follows from the inequality, not from memorizing which cost “sounds more important.”

Key takeaway: For calibrated probabilities and additive costs, select the action with the lower conditional expected cost. With zero costs for correct decisions, the threshold is false-positive cost divided by the sum of the two error costs. With nonzero correct-decision costs, compare incremental costs relative to the alternative correct outcomes, and verify the rule using the original expected-cost expressions.

Check Your Understanding

For each original scenario, derive or apply the relevant cost rule and compare expected costs where requested. Round probabilities to four decimals and money to cents where needed.

A hypothetical inspection has false-positive cost $50 and false-negative cost $250, with correct decisions costing zero. Find the threshold and choose an action for a calibrated probability of $0.14$. (Answer: $t^*=1/6\approx0.1667$; predict negative, with expected cost $\$35.00$.)
A hypothetical alert system has false-positive cost $120 and false-negative cost $280, with zero correct-decision costs. Find the threshold and both action costs at $p=0.30$. (Answer: $t^*=0.3000$; both actions cost $\$84.00$, so either minimizes expected cost.)
A hypothetical review has $C_{\mathrm{FP}}=\$100$, $C_{\mathrm{TP}}=\$25$, $C_{\mathrm{FN}}=\$175$, and $C_{\mathrm{TN}}=\$5$. Find the threshold and choose an action for $p=0.40$, verifying by costs. (Answer: $t^*=95/(95+150)=19/49\approx0.3878$; predict positive, since its expected cost is $\$70.00$, versus $\$73.00$ for negative.)
A hypothetical classification task has false-positive cost $40 and false-negative cost $160, with zero correct-decision costs. For three independent cases with calibrated probabilities $0.10$, $0.25$, and $0.60$, use the cost-minimizing threshold and find the total expected cost of the chosen decisions. (Answer: $t^*=0.20$; classify negative, positive, positive; total expected cost $=\$16+\$30+\$16=\$62.00$.)
A hypothetical decision system has $C_{\mathrm{FP}}=\$30$, $C_{\mathrm{TP}}=\$50$, $C_{\mathrm{FN}}=\$20$, and $C_{\mathrm{TN}}=\$0$. Determine whether there is a probability at which the preferred action changes, and state the optimal action for all $p\in[0,1]$. (Answer: The positive-action cost is $30+20p$ and the negative-action cost is $20p$; predicting negative is cheaper by $30 for every $p$, so there is no threshold at which the preferred action changes.)

What You'll Learn