Classification Thresholds and Unequal Error Costs

From a Leaf Probability to a Decision

A classification tree can provide more information than a predicted label. A terminal node may estimate the probability that an observation belongs to the positive class. To turn that probability into an action, a decision rule needs a threshold. A threshold of one-half is a common convention, but it is optimal only under particular cost assumptions. When a false positive and a false negative have different consequences, a different threshold may minimize expected cost.

For instance, a hypothetical insurer might use a tree to estimate whether a submitted claim requires additional review. The estimate is a probability, not a certainty that the claim belongs to either class. A decision rule determines whether to send it for review. The cost of reviewing a claim that did not need review may differ from the cost of failing to review one that did.

Definition: Let the positive class be the event of interest, and let $p$ be its estimated probability for an observation, such as the probability assigned to that observation’s terminal node. A classification threshold $t$ is a chosen probability cutoff: predict the positive class when $p$ is at least $t$, and predict the negative class otherwise. The probability estimate and the predicted label are distinct; changing $t$ can change the label without changing $p$.

The decision depends on both $p$ and the consequences of the two possible errors. A false positive means predicting positive when the true class is negative. A false negative means predicting negative when the true class is positive. In the standard two-class calculation below, correct classifications have zero cost, and the error costs are fixed and nonnegative. Costs are measured on the same scale, such as dollars or loss points.

Deriving the Cost-Minimizing Threshold

Suppose the cost of a false positive is $C_{\mathrm{FP}}$, and the cost of a false negative is $C_{\mathrm{FN}}$, with both costs positive. Conditional on the observation’s estimated positive-class probability $p$, predicting positive can incur a false-positive cost only if the observation is actually negative. Its expected cost is therefore $C_{\mathrm{FP}}(1-p)$. Predicting negative can incur a false-negative cost only if the observation is actually positive, so its expected cost is $C_{\mathrm{FN}}p$.

Choose the positive class when its conditional expected cost is no greater than the cost of choosing negative:

C_{\mathrm{FP}}(1-p) \leq C_{\mathrm{FN}}p.

Rearranging gives $C_{\mathrm{FP}} \leq p(C_{\mathrm{FP}}+C_{\mathrm{FN}})$. Because the sum of the two positive costs is positive, division preserves the inequality. This yields the cost-minimizing threshold.

Theorem: With two classes, zero cost for a correct classification, and fixed positive error costs $C_{\mathrm{FP}}$ and $C_{\mathrm{FN}}$, predict positive when

p \geq t^* = \frac{C_{\mathrm{FP}}}{C_{\mathrm{FP}}+C_{\mathrm{FN}}}.

At equality, both decisions have the same expected cost. A stated rule that predicts positive when $p \geq t^*$ resolves the tie in favor of positive; a rule using $p > t^*$ resolves it in favor of negative. Either choice has the same expected cost at the threshold.

The formula explains why the threshold need not be one-half. If the two error costs are equal, they cancel symmetrically and $t^*=0.5$. If false negatives are more costly, $C_{\mathrm{FN}}>C_{\mathrm{FP}}$, the threshold is below one-half, so the rule predicts positive more readily. If false positives are more costly, the threshold is above one-half, so stronger evidence is required before predicting positive.

Worked Example: Setting a Review Threshold

A hypothetical insurer uses a classification tree to estimate whether a claim needs additional review. For each claim, a false positive (reviewing a claim that does not need review) costs $120 in staff time. A false negative (not reviewing a claim that needs review) has an estimated cost of $480. Correct decisions have zero cost for this calculation. The tree assigns one claim a positive-class probability of 0.25. Find the cost-minimizing threshold and the decision for this claim.

Method and assumptions. Apply the two-class expected-cost rule with $C_{\mathrm{FP}}=\$120$ and $C_{\mathrm{FN}}=\$480$. The costs are treated as fixed per claim, and the stated probability is the estimated probability that review is needed.

Step 1: Calculate the threshold.

t^*=\frac{120}{120+480}=\frac{120}{600}=0.20.

Step 2: Compare the claim’s probability with the threshold. The claim has $p=0.25$, which is greater than 0.20, so the rule predicts that review is needed.

Step 3: Check by comparing expected costs. If the insurer reviews the claim, the false-positive probability is $1-0.25=0.75$, giving expected cost $120(0.75)=\$90$. If it does not review, the false-negative probability is 0.25, giving expected cost $480(0.25)=\$120$. Reviewing has the lower expected cost.

Check. The threshold rule and the direct cost comparison both select review, with an expected cost of $90 rather than $120. The method avoids the mistake of treating one-half as automatically optimal just because the output is a probability.

Using One Threshold Across Several Leaves

If the same error costs apply to every observation, the cost-minimizing threshold applies to probabilities from every terminal node. The probabilities themselves may differ by leaf, but the decision rule is unchanged. To evaluate the rule across a group, calculate each leaf’s expected cost per observation and multiply by the number of observations represented by that leaf. These are expected costs based on the estimated probabilities; they are not guaranteed realized costs.

Worked Example: Comparing a Threshold With One-Half

A hypothetical risk-screening tree has three terminal nodes. The estimated positive-class probabilities and numbers of future cases assigned to them are 0.08 for 100 cases, 0.16 for 50 cases, and 0.40 for 20 cases. A false positive costs $250 per case, while a false negative costs $1,750. Correct classifications have zero cost. Find the decisions and total expected cost under the cost-minimizing threshold, then compare with a threshold of 0.50.

Method and assumptions. Use the same fixed costs and probability interpretation in all three leaves. First calculate $t^*$, then determine each leaf’s decision and expected cost per case.

Step 1: Find the threshold.

t^*=\frac{250}{250+1{,}750}=\frac{250}{2{,}000}=0.125.

Step 2: Decide for the leaf with $p=0.08$. Since $0.08<0.125$, predict negative. The expected cost per case is $1{,}750(0.08)=\$140$. For comparison, predicting positive would cost $250(1-0.08)=250(0.92)=\$230$. Across 100 cases, the selected decision has expected cost $100(140)=\$14{,}000$.

Step 3: Decide for the leaf with $p=0.16$. Since $0.16\geq0.125$, predict positive. The expected cost per case is $250(1-0.16)=250(0.84)=\$210$. Predicting negative would cost $1{,}750(0.16)=\$280$. Across 50 cases, the selected decision has expected cost $50(210)=\$10{,}500$.

Step 4: Decide for the leaf with $p=0.40$. Since $0.40\geq0.125$, predict positive. The expected cost per case is $250(1-0.40)=250(0.60)=\$150$. Predicting negative would cost $1{,}750(0.40)=\$700$. Across 20 cases, the selected decision has expected cost $20(150)=\$3{,}000$.

Step 5: Add the expected costs. The total is $\$14{,}000+\$10{,}500+\$3{,}000=\$27{,}500$.

Step 6: Compare with the one-half rule. At a threshold of 0.50, the first two leaves are predicted negative and the third positive. The first leaf still costs $140 per case, or $14,000. The second now costs $1{,}750(0.16)=\$280$ per case, or $14,000. The third costs $150 per case, or $3,000. The total is $\$14{,}000+\$14{,}000+\$3{,}000=\$31{,}000$.

Check. The cost-minimizing decisions reduce the calculated expected cost by $\$31{,}000-\$27{,}500=\$3{,}500$. The method avoids applying a single default label to all leaves or assuming that a probability below one-half must imply a negative decision.

When Decision Costs Are More General

The simple threshold formula assumes zero cost for both correct classifications. In some decision settings, a positive action has a cost even when it is appropriate, or a correct negative action has a benefit represented as a negative cost. In that case, use a cost table that states the cost of each action under each true class. A threshold can still be derived by comparing the expected costs of the two actions, provided the resulting inequality has the usual direction.

Let $C_{+,+}$ denote the cost of predicting positive when the true class is positive, and $C_{+,-}$ the cost of predicting positive when the true class is negative. Similarly, let $C_{-,+}$ and $C_{-,-}$ be the costs of predicting negative under the positive and negative true classes. The expected costs are $pC_{+,+}+(1-p)C_{+,-}$ for predicting positive and $pC_{-,+}+(1-p)C_{-,-}$ for predicting negative. Choose positive when the first is no greater than the second.

Result: If the incremental cost of predicting positive on a negative case, $C_{+,-}-C_{-,-}$, is positive, and the incremental cost of predicting negative on a positive case, $C_{-,+}-C_{+,+}$, is positive, the threshold is

t^* = \frac{C_{+,-}-C_{-,-}} {(C_{+,-}-C_{-,-})+(C_{-,+}-C_{+,+})}.

To see why, subtract the expected cost of predicting negative from that of predicting positive. The difference is $(1-p)(C_{+,-}-C_{-,-})-p(C_{-,+}-C_{+,+})$. Predict positive when this difference is at most zero. Rearranging and dividing by the sum of the two positive incremental costs gives the stated threshold. When correct classifications cost zero, this reduces to the earlier formula.

Worked Example: Including the Cost of an Appropriate Intervention

A hypothetical health program uses a tree to decide whether to offer an additional screening intervention. The program assigns the following costs per person: offering the intervention to someone who does not need it costs $340; offering it to someone who does need it costs $40; withholding it from someone who needs it costs $1,540; and withholding it from someone who does not need it costs $0. A person’s estimated probability of needing the intervention is 0.18. Find the cost-minimizing threshold and decision.

Method and assumptions. Compare the two expected costs using the full cost table. The costs already include the relevant consequences, so no additional penalty is added.

Step 1: Identify the four costs. Predict positive (offer) when positive is true costs $C_{+,+}=\$40$. Predict positive when negative is true costs $C_{+,-}=\$340$. Predict negative (withhold) when positive is true costs $C_{-,+}=\$1{,}540$. Predict negative when negative is true costs $C_{-,-}=\$0$.

Step 2: Calculate the incremental costs. The added cost of offering the intervention to a person who does not need it, relative to withholding it from that person, is $\$340-\$0=\$340$. The added cost of withholding it from a person who does need it, relative to offering it, is $\$1{,}540-\$40=\$1{,}500$.

Step 3: Calculate the threshold.

t^*=\frac{340}{340+1{,}500} =\frac{340}{1{,}840} \approx 0.1848.

Step 4: Apply the threshold. The person’s probability is 0.18, which is below 0.1848, so withhold the intervention.

Step 5: Verify by direct expected-cost calculation. Offering costs $0.18(40)+0.82(340)=7.20+278.80=\$286.00$. Withholding costs $0.18(1{,}540)+0.82(0)=\$277.20$. Withholding is less costly by $\$286.00-\$277.20=\$8.80$.

Check. Both the threshold rule and direct calculation select withholding. The method avoids treating all costs incurred by a positive action as a false-positive penalty; the threshold depends on the difference in costs between actions within each true class.

Common Mistakes and Practical Checks

Using one-half by default. A threshold of 0.50 follows from equal false-positive and false-negative costs under the zero-correct-cost model. It is not a universal property of classification probabilities.
Reversing the costs in the formula. The numerator for the positive-class threshold is $C_{\mathrm{FP}}$, not $C_{\mathrm{FN}}$. A more expensive false negative lowers the threshold because predicting positive becomes worthwhile at a smaller $p$.
Comparing costs without conditioning on the class probabilities. For a positive decision, the false-positive probability is $1-p$, not $p$. For a negative decision, the false-negative probability is $p$.
Confusing probability estimation with the decision rule. A threshold changes which label is assigned to a probability; it does not, by itself, change the probability estimated by a leaf.
Combining costs measured on different scales. Both error costs must use a comparable scale. If consequences are converted into dollars, apply the same valuation basis to both types of error.

A cost threshold is a decision rule, not a claim that one class is more probable in the population or that a tree’s probability estimates have changed. Its usefulness depends on clearly defining the positive class, the actions, the error consequences, and the probability to which the rule is applied. Those distinctions are central to decision-focused classification and are aligned with the SOA Exam SRM (Statistics for Risk Modeling) syllabus.

Check Your Understanding

For each problem, use conditional expected costs and show which action has the lower expected cost.

A hypothetical claims triage rule has a false-positive cost of $90 and a false-negative cost of $510; correct decisions cost zero. Find the threshold and the decision for a case with estimated positive probability 0.14. (Answer: $t^*=0.15$; predict negative.)
A positive action has false-positive cost $400 and false-negative cost $1,600 applies if the action is withheld from a positive case. Correct decisions cost zero. A case has estimated positive probability 0.23. Find the threshold and the minimum expected cost for that case. (Answer: $t^*=0.20$; predict positive; minimum expected cost is $308.)
A tree assigns 40 cases to a leaf with $p=0.10$, 30 cases to a leaf with $p=0.30$, and 10 cases to a leaf with $p=0.70$. A false positive costs $200 and a false negative costs $800; correct decisions cost zero. Find the cost-minimizing decisions and total expected cost. (Answer: threshold $0.20$; predict negative, positive, and positive, respectively; total expected cost is $8,000.)
A decision table has costs $C_{+,-}=\$500$, $C_{-,-}=\$50$, $C_{-,+}=\$2{,}050$, and $C_{+,+}=\$50$. Find the threshold and decide for an observation with $p=0.18$. (Answer: $t^*=450/(450+2{,}000)\approx0.1837$; predict negative.)
Under the zero-cost-for-correct-decisions model, false positives cost $300 and false negatives cost $900. What happens when $p=0.25$, including the expected cost of either decision? (Answer: $t^*=0.25$; the decisions tie, each with expected cost $225.)

What You'll Learn