Cost-Sensitive Threshold Selection

Why the Threshold Depends on the Cost

A classifier that reports a probability does not, by itself, determine which class to assign. A cutoff of one-half is appropriate only under particular error costs. If missing a harmful condition is much more costly than raising a false alarm, a lower cutoff may be preferable; if false alarms are especially costly, the cutoff may be higher. The decision should follow from the costs of the possible outcomes, rather than from habit.

Let \(Y\in\{0,1\}\) be the true class, and let \(X\) denote the observed features. A fitted classifier supplies a probability \(p(x)=P(Y=1\mid X=x)\), interpreted as a posterior probability for the population in which the decision will be used. We consider the action of predicting class 1 or class 0. First suppose correct classifications have zero cost, a false positive costs \(C_{\mathrm{FP}}>0\), and a false negative costs \(C_{\mathrm{FN}}>0\).

Deriving the Bayes Cost Threshold

At a fixed feature value \(x\), predicting class 1 incurs a false-positive cost when \(Y=0\); predicting class 0 incurs a false-negative cost when \(Y=1\). Their conditional expected costs are therefore

Choose class 1 when its conditional cost is no larger than the cost of class 0. Since both costs are positive, rearranging the inequality gives the threshold below.

Proof. Predicting class 1 is optimal at \(x\) precisely when

Adding \(C_{\mathrm{FP}}p(x)\) to each side and dividing by the positive number \(C_{\mathrm{FP}}+C_{\mathrm{FN}}\) yields

The choice is made separately at each \(x\). Any rule that chooses the higher-cost action at some feature values cannot have smaller expected cost: at those values its conditional expected cost is at least that of the minimizing action, and averaging conditional costs over \(X\) preserves the inequality. Equality permits either action without changing conditional cost. \(\square\)

Equivalent Likelihood-Ratio Form

The same decision can be written using class-conditional densities. Let \(\pi_1=P(Y=1)\), \(\pi_0=P(Y=0)\), and let \(f_1(x)\) and \(f_0(x)\) be the densities of \(X\) conditional on classes 1 and 0, respectively. Where the denominator is positive, Bayes' rule gives \(p(x)=\pi_1 f_1(x)/\{\pi_1 f_1(x)+\pi_0 f_0(x)\}\). Substituting this expression into the probability threshold gives the likelihood-ratio rule.

This form makes clear that the decision reflects both the class-conditional evidence in the data and the probabilities of the classes. It is an equivalent expression for the same Bayes rule, not a separate choice of costs.

More General Costs and the Role of Probabilities

The zero-cost convention for correct classifications is often convenient, but the derivation extends to a general two-action cost table. Let \(c_{ay}\) be the cost of predicting class \(a\) when the true class is \(y\). Define the incremental false-positive cost as \(d_{\mathrm{FP}}=c_{10}-c_{00}\), and the incremental false-negative cost as \(d_{\mathrm{FN}}=c_{01}-c_{11}\). If both increments are positive, predicting class 1 is preferable when

Only cost differences between actions matter; adding the same amount to both actions for a given true class does not change the decision. The simpler threshold uses \(d_{\mathrm{FP}}=C_{\mathrm{FP}}\) and \(d_{\mathrm{FN}}=C_{\mathrm{FN}}\).

The posterior probability must correspond to the population for which the decision is made. If the supplied score is not a calibrated estimate of that posterior, inserting it into the theoretical probability cutoff does not, in general, yield the Bayes cost rule. For an arbitrary score, threshold performance can instead be assessed empirically on representative validation data. Cross-fitted calibration of discriminant scores can provide probabilities for thresholding; the threshold calculation itself does not make an uncalibrated score into a probability.

Selecting a Threshold on Validation Data

When no trustworthy posterior probabilities are available, or when the fitted model's behavior needs to be assessed directly, compare candidate thresholds using validation observations. Suppose the classifier and any score calibration have been fitted without using the validation observations. For validation pairs \((s_i,y_i)\), where \(s_i\) is the score and \(y_i\in\{0,1\}\), define the empirical cost at threshold \(t\) by

Here \(n_{\mathrm{val}}\) is the number of validation observations and \(\mathbf{1}\{\cdot\}\) equals 1 when its condition holds and 0 otherwise. The selected threshold minimizes this quantity over a prespecified candidate set. A threshold grid is one option. Since predictions change only when the threshold crosses a validation score, a search may also consider the distinct score values and the intervals between them. The convention \(s_i\ge t\) assigns observations exactly at the threshold to class 1 and should be used consistently.

What Validation Minimization Guarantees

A validation cost is a random estimate of a threshold's population cost. Under a clear sampling condition, its expectation has a direct interpretation: conditional on everything used to fit the classifier, the validation estimate is unbiased for that fixed classifier's risk.

Proof. Conditional on the fitted classifier, each validation score and label has the target-population distribution, and the threshold rule is fixed. Let \(Z_i(t)\) be the cost for validation observation \(i\). Then \(E[Z_i(t)\mid\text{fitted classifier}]=R(t)\) for every \(i\). By linearity of conditional expectation,

This statement concerns a fixed threshold. It does not say that the minimum over many validation thresholds is an unbiased estimate of the cost of the selected threshold. Taking a minimum tends to favor thresholds that benefited from validation-sample noise. Therefore, report the validation cost as a selection criterion, not as an independent final performance estimate. Use separate test data or an appropriate cross-validation design for final assessment. \(\square\)

For a prespecified finite grid, the effect of validation noise can also be bounded. Suppose there are \(K\) candidate thresholds, validation observations are independent and identically distributed, and each per-observation cost lies in \([0,M]\), where \(M=\max(C_{\mathrm{FP}},C_{\mathrm{FN}})\). Let \(\hat t\) minimize validation cost over the grid, and let \(t^*\) minimize population cost over that same grid. Hoeffding's inequality and a union bound give, for \(\epsilon>0\),

On the event that every grid estimate is within \(\epsilon\) of its population cost, the definition of \(\hat t\) implies

Thus, with probability at least \(1-2K\exp\{-2n_{\mathrm{val}}\epsilon^2/M^2\}\), the selected grid threshold has population cost at most \(2\epsilon\) above the best threshold on that grid. The guarantee is relative to the candidate grid, not to every possible threshold, and its assumptions require validation data independent of fitting and representative of the target population. \(\square\)

Common Mistakes and Practical Implications

• Reversing the costs. A common mistake is to put \(C_{\mathrm{FN}}\) in the numerator of the cutoff. The comparison of conditional costs shows that the numerator is \(C_{\mathrm{FP}}\): increasing false-positive cost raises the threshold; increasing false-negative cost lowers it.
• Always using one-half. A cutoff of \(0.5\) follows only when the two error costs are equal under the stated zero-correct-cost convention. It is not a universal decision rule.
• Thresholding any score as if it were a probability. The cost-derived probability cutoff applies to posterior probabilities. A score with a different scale requires validation-based threshold selection or a justified probability interpretation.
• Reporting the smallest validation cost as final performance. Threshold selection uses the validation labels, so the minimum observed cost can be optimistic. Assess the selected rule on data not used to choose the threshold.
• Ignoring the population in the probability. The threshold formula is conditional on a posterior for the intended decision population. A probability estimate for a different population may not yield the intended cost-minimizing decisions.

Cost-sensitive thresholding makes the decision objective explicit: probabilities describe uncertainty about the class, while the cost matrix determines how that uncertainty is converted into an action. The theoretical threshold is useful when probabilities and costs are specified; validation selection is useful when performance must be measured for a fitted scoring rule. In either case, threshold selection and final risk assessment are distinct tasks.