Asymmetric Costs and Shifted Calibration Thresholds

Why Unequal Costs Shift the Decision Boundary

In binary classification, predicting the less common class can still be the rational choice when missing that class is especially costly. A screening rule, for example, may place a high cost on a false negative and a smaller cost on a false positive. The optimal decision threshold then need not be one half. A surrogate loss must be assessed relative to this shifted threshold, not automatically relative to the usual zero-score boundary.

Let $Y\in\{-1,+1\}$, and at a fixed feature value $x$, write $\eta=P(Y=+1\mid X=x)$. A false negative means predicting $-1$ when $Y=+1$, with cost $C_{\mathrm{FN}}>0$. A false positive means predicting $+1$ when $Y=-1$, with cost $C_{\mathrm{FP}}>0$. Correct predictions have cost zero. We use a real-valued score $t$, predicting $+1$ when $t>0$ and $-1$ when $t\leq 0$.

Definition: The cost-sensitive conditional risk of an action is its expected misclassification cost given $X=x$. The Bayes action is an action that minimizes this conditional risk. A margin surrogate $\phi$ has cost-sensitive conditional surrogate risk $$ R_{\phi}(t;\eta) = C_{\mathrm{FN}}\eta\,\phi(t) + C_{\mathrm{FP}}(1-\eta)\,\phi(-t). $$ Here $\phi(t)$ is the loss when the true label is $+1$, and $\phi(-t)$ is the loss when the true label is $-1$.

Deriving the Cost-Sensitive Bayes Threshold

There are only two possible actions, so their conditional risks can be compared directly. If we predict $+1$, the error occurs when $Y=-1$; if we predict $-1$, the error occurs when $Y=+1$.

R(+1\mid x)=C_{\mathrm{FP}}(1-\eta), \qquad R(-1\mid x)=C_{\mathrm{FN}}\eta.

Thus predicting $+1$ is optimal when its risk is no greater than the risk of predicting $-1$. Solving that inequality gives the threshold.

Theorem: For positive costs $C_{\mathrm{FN}}$ and $C_{\mathrm{FP}}$, a Bayes-optimal rule predicts $+1$ when $$ \eta\geq \eta_*, \qquad \eta_*=\frac{C_{\mathrm{FP}}}{C_{\mathrm{FN}}+C_{\mathrm{FP}}}, $$ and predicts $-1$ when $\eta\leq\eta_*$. At equality, both actions have the same risk.

Proof. The condition for predicting $+1$ is $C_{\mathrm{FP}}(1-\eta)\leq C_{\mathrm{FN}}\eta$. Rearranging, $C_{\mathrm{FP}}\leq (C_{\mathrm{FN}}+C_{\mathrm{FP}})\eta$. The sum of the costs is positive, so division gives $\eta\geq\eta_*$. Reversing the inequality gives the condition for predicting $-1$. At equality, the two conditional risks coincide, so either action minimizes them. $\square$

Worked Example: Finding the Threshold From the Costs

Suppose a missed positive case costs $C_{\mathrm{FN}}=4$, while a false alarm costs $C_{\mathrm{FP}}=1$. The threshold is $$ \eta_*=\frac{1}{4+1}=0.2. $$ At $\eta=0.15$, predicting $+1$ has risk $1(1-0.15)=0.85$, while predicting $-1$ has risk $4(0.15)=0.60$; predict $-1$. At $\eta=0.30$, the risks are $1(0.70)=0.70$ and $4(0.30)=1.20$; predict $+1$. The threshold is below one half because false negatives are more costly.

A Cost-Adjusted Probability

The shifted threshold can be expressed as an ordinary one-half threshold after transforming the conditional probability. This makes the connection to margin-loss calibration precise.

Definition: For $0\leq\eta\leq1$, define the cost-adjusted probability $$ q(\eta)= \frac{C_{\mathrm{FN}}\eta} {C_{\mathrm{FN}}\eta+C_{\mathrm{FP}}(1-\eta)}. $$ The denominator is positive for every $\eta\in[0,1]$, since the costs are positive.

The transformation is strictly increasing: differentiating gives $q'(\eta)=C_{\mathrm{FN}}C_{\mathrm{FP}}/[C_{\mathrm{FN}}\eta+C_{\mathrm{FP}}(1-\eta)]^2>0$. Also, $q(\eta_*)=1/2$. More directly, $q(\eta)>1/2$ exactly when $C_{\mathrm{FN}}\eta>C_{\mathrm{FP}}(1-\eta)$, which is exactly the condition $\eta>\eta_*$. The transformed probability therefore preserves which side of the Bayes boundary $\eta$ lies on.

Worked Example: Transforming the Posterior

For the costs $C_{\mathrm{FN}}=4$ and $C_{\mathrm{FP}}=1$, at $\eta=0.30$, $$ q(0.30)=\frac{4(0.30)}{4(0.30)+1(0.70)} =\frac{1.20}{1.90} =\frac{12}{19}> \frac12. $$ At $\eta=0.15$, $q(0.15)=0.60/(0.60+0.85)=12/29<1/2$. These transformed values give the same decisions as the direct risk comparisons, with the dividing point now at one half.

The Shifted Calibration Condition

For calibration, compare the best surrogate risk without a decision constraint to the best risk among scores that make the wrong decision. To ensure these differences are well-defined and finite, assume throughout this section that $\phi:\mathbb{R}\to[0,\infty)$ is finite-valued. Then $R_\phi(t;\eta)$ is finite at every finite $t$ and nonnegative, so its unrestricted and constrained infima are finite real numbers.

Definition: Write $R_\phi^*(\eta)=\inf_{t\in\mathbb{R}}R_\phi(t;\eta)$. With the convention that a zero score predicts $-1$, define the cost-sensitive calibration gap by $$ \Delta_{\phi,C}(\eta)= \begin{cases} \displaystyle\inf_{t\leq0}R_\phi(t;\eta)-R_\phi^*(\eta), &\eta>\eta_*,\\[6pt] \displaystyle\inf_{t>0}R_\phi(t;\eta)-R_\phi^*(\eta), &\eta<\eta_*. \end{cases} $$ The surrogate is cost-sensitive calibrated if $\Delta_{\phi,C}(\eta)>0$ for every $\eta\ne\eta_*$. No positive gap is required at $\eta=\eta_*$, where both decisions are Bayes-optimal.

The constraints match the decision rule: above the threshold the Bayes action is $+1$, so scores $t\leq0$ are wrong; below the threshold the Bayes action is $-1$, so scores $t>0$ are wrong. The strict positivity condition rules out sequences of wrong-decision scores whose surrogate risks approach the unrestricted infimum. It is an infimum condition, so it does not require that a constrained minimizer be attained.

Theorem: Suppose $C_{\mathrm{FN}},C_{\mathrm{FP}}>0$ and $\phi:\mathbb{R}\to[0,\infty)$ is finite-valued. For every $\eta$, let $S(\eta)=C_{\mathrm{FN}}\eta+C_{\mathrm{FP}}(1-\eta)$. Then $$ R_\phi(t;\eta) = S(\eta)\bigl[q(\eta)\phi(t)+(1-q(\eta))\phi(-t)\bigr]. $$ Consequently, cost-sensitive calibration at $\eta$ is equivalent to positive wrong-side calibration gap for the ordinary symmetric margin risk $q\phi(t)+(1-q)\phi(-t)$ at $q=q(\eta)$, using the same score convention.

Proof. By definition, $S(\eta)q(\eta)=C_{\mathrm{FN}}\eta$ and $S(\eta)(1-q(\eta))=C_{\mathrm{FP}}(1-\eta)$. Substitution gives the displayed factorization. Since $S(\eta)>0$, multiplication by $S(\eta)$ multiplies each constrained infimum and the unrestricted infimum by the same positive number. It therefore multiplies their difference by $S(\eta)$ and preserves whether the gap is strictly positive. Finally, $q(\eta)>1/2$ exactly when $\eta>\eta_*$, and $q(\eta)<1/2$ exactly when $\eta<\eta_*$. Thus the wrong-side constraints in the two formulations agree, including the convention at zero. $\square$

Worked Surrogate Examples

Worked Example: Weighted Logistic Loss

Take $\phi(z)=\log(1+e^{-z})$, which is nonnegative and finite-valued. Write $A=C_{\mathrm{FN}}\eta$ and $B=C_{\mathrm{FP}}(1-\eta)$. For $0<\eta<1$, both are positive, and $$ R_\phi(t;\eta)=A\log(1+e^{-t})+B\log(1+e^t). $$ Its derivative is $(-A+Be^t)/(1+e^t)$. It vanishes at $t=\log(A/B)$, is negative below that value, and is positive above it. Hence this is the unique minimizer. Its sign is positive exactly when $A>B$, or equivalently $\eta>\eta_*$.

With costs $4$ and $1$ and $\eta=0.30$, $A=1.20$, $B=0.70$, and the minimizing score is $\log(12/7)>0$. At $\eta=0.15$, $A=0.60$, $B=0.85$, and the minimizing score is $\log(12/17)<0$. Thus the weighted logistic conditional minimizer changes sign at the cost-sensitive threshold, not at $\eta=1/2$. At $\eta=0$ or $1$, the infimum may be approached only as the score tends to infinity in magnitude; the calibration-gap definition uses infima and remains applicable.

Worked Example: Weighted Hinge Loss

For $\phi(z)=\max(0,1-z)$, the conditional risk is $A\max(0,1-t)+B\max(0,1+t)$, with $A=C_{\mathrm{FN}}\eta$ and $B=C_{\mathrm{FP}}(1-\eta)$. On $[-1,1]$, this equals $A+B+(B-A)t$. For $t\geq1$, it equals $B(1+t)$, which is nondecreasing; for $t\leq-1$, it equals $A(1-t)$, which is nonincreasing as $t$ increases. Therefore, if $A>B$, a global minimum occurs at $t=1$; if $B>A$, a global minimum occurs at $t=-1$; and if $A=B$, every $t\in[-1,1]$ is a minimizer.

For costs $4$ and $1$ at $\eta=0.30$, $A=1.20>B=0.70$. The unrestricted minimum is $2B=1.40$, attained at $t=1$. Over the wrong-decision set $t\leq0$, the minimum is $A+B=1.90$, attained at $t=0$, so the gap is $1.90-1.40=0.50$. At $\eta=0.15$, $B=0.85>A=0.60$. The unrestricted minimum is $2A=1.20$, attained at $-1$; over the wrong-decision set $t>0$, the infimum is $A+B=1.45$, approached as $t$ decreases to zero. The gap is $1.45-1.20=0.25$.

Common Mistake and Practical Meaning

A common mistake is to apply the one-half posterior threshold after changing the error costs. For example, with false negatives costing four times as much as false positives, a posterior of $0.30$ is above the correct threshold $0.20$, even though it is below one half. Using one half would make the wrong decision under the stated costs.

A related distinction is between estimating $\eta$ and making the cost-sensitive decision. An accurate estimate of $\eta$ can be compared directly with $\eta_*$. Alternatively, a cost-weighted margin surrogate can target the transformed probability $q(\eta)$, whose one-half boundary corresponds to $\eta_*$. These are different targets: a score that estimates $q$ is not, without a transformation, an estimate of the original probability $\eta$.

The calibration condition makes the decision target explicit. It asks whether the surrogate risk has a strictly positive penalty for scores on the wrong side of the cost-sensitive boundary. The factorization theorem shows that unequal costs do not require a new symmetric-margin argument: they rescale the conditional risk and transform the posterior, shifting the relevant boundary to the correct location.

Check Your Understanding

Use the definitions and calculations above to answer these questions.

If $C_{\mathrm{FN}}=2$ and $C_{\mathrm{FP}}=3$, what is $\eta_*$, and which action is Bayes-optimal at $\eta=0.7$? Show the two conditional risks.
Why does the calibration gap use $t\leq0$ above the threshold but $t>0$ below it?
For positive costs, explain why $q(\eta)>1/2$ exactly when $\eta>\eta_*$.
In the weighted hinge example, compute the wrong-side gap when $A>B>0$.
Why is nonnegativity of $\phi$ a useful assumption when defining these infimum differences?