Cost-Sensitive Thresholds and Their Margin Distributions

Why Error Costs Change the Threshold

A classification rule does not always face equal consequences for its two kinds of error. In a screening setting, for example, missing a true case may be more costly than investigating a person who is ultimately disease-free. When false positives and false negatives have different costs, the optimal decision threshold generally moves away from one half. The relevant question is not simply whether the conditional probability of class 1 exceeds one half, but whether the expected cost of predicting class 1 is smaller than the expected cost of predicting class 0.

Let $Y\in\{0,1\}$ be the label and $X$ the feature vector. Write $\eta(x)=P(Y=1\mid X=x)$. A classifier $g$ takes values in $\{0,1\}$. Let $c_{\mathrm{FP}}>0$ be the cost of predicting 1 when $Y=0$, and let $c_{\mathrm{FN}}>0$ be the cost of predicting 0 when $Y=1$. Correct classifications have cost zero. Assume $\eta$ is measurable and classifiers are measurable, so the risks and expectations below are defined.

Definition: The cost-sensitive risk of a classifier $g$ is $$ R_c(g)=E\!\left[c_{\mathrm{FP}}\mathbf{1}\{g(X)=1,Y=0\} +c_{\mathrm{FN}}\mathbf{1}\{g(X)=0,Y=1\}\right]. $$ Here the subscript $c$ indicates that risk is evaluated using the specified error costs.

Deriving the Cost-Sensitive Bayes Threshold

At a fixed feature value $x$, predicting 1 incurs a false-positive cost only when $Y=0$, while predicting 0 incurs a false-negative cost only when $Y=1$. The two conditional risks are therefore

r_c(1\mid x)=c_{\mathrm{FP}}\{1-\eta(x)\}, \qquad r_c(0\mid x)=c_{\mathrm{FN}}\eta(x).

The optimal action is the one with smaller conditional risk. This comparison gives the cost-sensitive threshold.

Theorem: For positive costs $c_{\mathrm{FP}}$ and $c_{\mathrm{FN}}$, define $$ t_c=\frac{c_{\mathrm{FP}}}{c_{\mathrm{FP}}+c_{\mathrm{FN}}}. $$ A Bayes classifier for $R_c$ is $$ g_c^*(x)=\mathbf{1}\{\eta(x)\ge t_c\}. $$ At points where $\eta(x)=t_c$, either action has the same conditional risk, so the displayed rule is one valid tie convention.

Proof. Predicting 1 has no greater conditional risk than predicting 0 exactly when $$ c_{\mathrm{FP}}\{1-\eta(x)\}\le c_{\mathrm{FN}}\eta(x). $$ Rearranging, using the positivity of both costs, $$ c_{\mathrm{FP}}\le \{c_{\mathrm{FP}}+c_{\mathrm{FN}}\}\eta(x) \quad\Longleftrightarrow\quad \eta(x)\ge \frac{c_{\mathrm{FP}}}{c_{\mathrm{FP}}+c_{\mathrm{FN}}}=t_c. $$ Thus the stated rule minimizes conditional risk at every $x$. For any measurable classifier $g$, its risk is the expectation of its conditional risk given $X$. Pointwise minimization therefore gives $R_c(g_c^*)\le R_c(g)$, proving Bayes optimality. At equality, the two conditional risks coincide, which justifies either tie choice. $\square$

The threshold lies strictly between zero and one. If the costs are equal, it is one half and the usual equal-cost Bayes rule is recovered. If false negatives are more costly, $c_{\mathrm{FN}}>c_{\mathrm{FP}}$, then $t_c<1/2$: the rule predicts class 1 at a lower posterior probability. If false positives are more costly, the threshold exceeds one half. Multiplying both costs by the same positive constant leaves $t_c$ unchanged, though it scales the numerical risk.

Worked Example: A Higher Cost for False Positives

Suppose a false positive costs 4 units and a false negative costs 1 unit. Then $$ t_c=\frac{4}{4+1}=0.8. $$ At a feature value with $\eta(x)=0.7$, predicting 1 has conditional risk $4(1-0.7)=1.2$, while predicting 0 has conditional risk $1(0.7)=0.7$. The Bayes action is 0. At a feature value with $\eta(x)=0.9$, those risks are $4(0.1)=0.4$ and $1(0.9)=0.9$, respectively, so the Bayes action is 1. The decision changes at posterior probability 0.8, not at 0.5.

The Margin Distribution Around the Cost-Sensitive Threshold

Once the threshold has changed, the relevant distance to the decision boundary changes with it. Define the cost-sensitive margin distance by $D_c(X)=|\eta(X)-t_c|$. It measures how far the conditional class probability is from the threshold, in probability units. Small values identify feature values near the decision boundary; large values identify points farther from it.

Definition: The cost-sensitive margin distribution is the distribution of $D_c(X)=|\eta(X)-t_c|$. Its cumulative form is $$ F_c(u)=P\{|\eta(X)-t_c|\le u\},\qquad u\ge 0. $$ The punctured margin distribution is $$ M_c(u)=P\{0<|\eta(X)-t_c|\le u\},\qquad u>0, $$ and the boundary mass is $b_c=P\{\eta(X)=t_c\}$. Thus $F_c(u)=b_c+M_c(u)$ for $u>0$.

The distinction between $F_c$ and $M_c$ matters when the feature distribution places positive probability exactly on the threshold. Those points have equal conditional risk under either decision, so they contribute no excess risk from disagreement with a chosen Bayes rule. The punctured distribution describes points close to, but not exactly on, the threshold. For example, a margin condition can be stated as $M_c(u)\le A u^\alpha$ for $0<u\le u_0$, with constants $A>0$, $u_0>0$, and exponent $\alpha\ge 0$. This is a condition on probability mass near the cost-sensitive boundary; it is not an assumption that the boundary mass is zero.

Worked Example: A Discrete Margin Distribution

Let $c_{\mathrm{FP}}=1$ and $c_{\mathrm{FN}}=3$, so $t_c=1/4$. Suppose $X$ takes three values with probabilities and regression-function values $$ \begin{array}{c|ccc} & x_1 & x_2 & x_3\\ \hline P(X=x_j) & 0.40 & 0.35 & 0.25\\ \eta(x_j) & 0.20 & 0.80 & 0.95 \end{array} $$ The Bayes classifier predicts 0 at $x_1$ and 1 at $x_2,x_3$, since only the first regression value is below $1/4$. The three margin distances are $$ |0.20-0.25|=0.05,\qquad |0.80-0.25|=0.55,\qquad |0.95-0.25|=0.70. $$ There is no boundary mass. Consequently, $$ M_c(u)=0.40\mathbf{1}\{u\ge0.05\} +0.35\mathbf{1}\{u\ge0.55\} +0.25\mathbf{1}\{u\ge0.70\}. $$ For instance, $M_c(0.10)=0.40$, while $M_c(0.60)=0.75$. To check the cost scale, the Bayes conditional risks at $x_1,x_2,x_3$ are $3(0.20)=0.60$, $1(0.20)=0.20$, and $1(0.05)=0.05$. Weighting by the probabilities gives $R_c(g_c^*)=0.40(0.60)+0.35(0.20)+0.25(0.05)=0.3225$.

Disagreement and Cost-Sensitive Excess Risk

The margin distance also quantifies the cost of making the wrong decision. Let $g_c^*$ be the threshold rule above and let $g$ be any measurable classifier. At points where $g$ and $g_c^*$ agree, their conditional risks are equal. Where they disagree away from the threshold, the difference in conditional risk is the absolute difference between the two possible conditional risks.

Result: For every measurable classifier $g$, $$ R_c(g)-R_c(g_c^*)= (c_{\mathrm{FP}}+c_{\mathrm{FN}}) E\!\left[|\eta(X)-t_c|\, \mathbf{1}\{g(X)\ne g_c^*(X)\}\right]. $$

Derivation. Write $S=c_{\mathrm{FP}}+c_{\mathrm{FN}}$. Conditional on $X=x$, the risk difference is zero if the classifiers agree. If $\eta(x)\ge t_c$ and they disagree, $g_c^*(x)=1$ and $g(x)=0$, so the difference is $$ r_c(0\mid x)-r_c(1\mid x) =c_{\mathrm{FN}}\eta(x)-c_{\mathrm{FP}}\{1-\eta(x)\} =S\{\eta(x)-t_c\}. $$ If $\eta(x)<t_c$ and they disagree, $g_c^*(x)=0$ and $g(x)=1$, so the difference is $$ r_c(1\mid x)-r_c(0\mid x) =c_{\mathrm{FP}}\{1-\eta(x)\}-c_{\mathrm{FN}}\eta(x) =S\{t_c-\eta(x)\}. $$ In both cases the difference is $S|\eta(x)-t_c|$. At equality, this expression is zero, regardless of the tie convention. Therefore, for every $x$, the conditional risk difference equals $S|\eta(x)-t_c|\mathbf{1}\{g(x)\ne g_c^*(x)\}$. Taking expectations proves the result. $\square$

This identity shows precisely how the margin distribution enters: disagreement near the threshold can have small cost per observation, while disagreement farther away has a larger cost. For any $u>0$, splitting the expectation according to whether $D_c(X)\le u$ gives $$ R_c(g)-R_c(g_c^*)\le S\left[u\,P\{g(X)\ne g_c^*(X),D_c(X)\le u\} +E\{D_c(X)\mathbf{1}\{g(X)\ne g_c^*(X),D_c(X)>u\}\}\right]. $$ The first term is governed by the probability of being near the threshold; the second concerns disagreements farther from it. The margin distribution describes the first quantity’s available probability mass, while the identity retains the distance that determines each point’s cost.

Worked Example: A Continuous Margin Distribution

Suppose $Z=\eta(X)$ is uniformly distributed on $[0,1]$, and the costs are $c_{\mathrm{FP}}=2$ and $c_{\mathrm{FN}}=1$. The threshold is $t_c=2/3$. For $0<u\le1/3$, the interval of $Z$-values within distance $u$ of $2/3$ is $[2/3-u,2/3+u]$, fully inside $[0,1]$, so $M_c(u)=2u$. For $1/3<u\le2/3$, the upper end is truncated at 1, giving $$ M_c(u)=1-(2/3-u)=1/3+u. $$ For $u\ge2/3$, all values lie within distance $u$, so $M_c(u)=1$. There is no boundary mass because a uniform random variable assigns probability zero to the single value $2/3$.

As a check on the excess-risk identity, consider a classifier that disagrees with the Bayes rule only when $Z\in[2/3-0.1,2/3+0.1]$. The total cost multiplier is $S=3$, and the uniform density is 1. Its excess risk is $$ 3\int_{-0.1}^{0.1}|z|\,dz =3\left(2\int_0^{0.1}z\,dz\right) =3(0.01)=0.03. $$ The calculation weights each disagreement by its distance from the threshold, rather than treating all points in the interval as equally costly.

Common Pitfall and Practical Interpretation

A common mistake is to retain the one-half threshold after changing the error costs. That rule is optimal only when false positives and false negatives have equal costs. For example, with costs 1 and 3, a posterior probability of 0.30 exceeds one half’s lower? It does not; a one-half rule predicts 0, whereas the correct cost-sensitive threshold $1/4$ predicts 1. The correction is to compare the two conditional costs, not to apply a fixed threshold independently of the loss.

A second potential confusion is to use the old boundary when describing the margin. The margin must be centered at the threshold induced by the costs, $t_c$, because that is where the two conditional costs are equal. The factor $c_{\mathrm{FP}}+c_{\mathrm{FN}}$ then converts distance in posterior-probability units into conditional excess cost. Together, the threshold and its margin distribution summarize which observations are near the decision boundary and how costly disagreement can be.

Check Your Understanding

Use the definitions and derivations above to answer these questions.

If $c_{\mathrm{FP}}=2$ and $c_{\mathrm{FN}}=6$, what is the cost-sensitive threshold, and which class is chosen when $\eta(x)=0.30$?
Why can a point exactly on the threshold have positive probability without contributing to cost-sensitive excess risk?
For $c_{\mathrm{FP}}=3$ and $c_{\mathrm{FN}}=1$, write the conditional risk of each action and verify the threshold by comparing them.
In the discrete example, what is $M_c(0.50)$, and what probability mass does it include?
State the cost-sensitive excess-risk identity and explain the role of the total cost multiplier.