Cost-Sensitive Kernel Classification by Thresholding Scores

Why the Classification Threshold Depends on Costs

A classifier that treats false positives and false negatives equally uses a different decision rule from one in which the errors have different consequences. For example, a screening procedure may accept more false alarms to avoid missing a condition. The key is to compare the costs of the two possible actions at each feature value, rather than to use a probability threshold of one half automatically.

Let $Y\in\{-1,+1\}$ be the class label and $X$ the feature vector. Write $\eta(x)=P(Y=+1\mid X=x)$, using a measurable version of the conditional probability. A decision $+1$ when $Y=-1$ incurs false-positive cost $c_{\mathrm{FP}}>0$; a decision $-1$ when $Y=+1$ incurs false-negative cost $c_{\mathrm{FN}}>0$. Correct decisions have zero cost. The costs are fixed and do not depend on $x$.

Definition: The cost-sensitive classification risk of a measurable classifier $h:\mathcal X\to\{-1,+1\}$ is $$ R_c(h)=E\!\left[c_{\mathrm{FN}}\,\mathbf 1\{Y=+1,\ h(X)=-1\} +c_{\mathrm{FP}}\,\mathbf 1\{Y=-1,\ h(X)=+1\}\right]. $$ Its conditional risk at $X=x$ is the expected cost given $X=x$.

At a fixed $x$, predicting $+1$ has conditional risk $c_{\mathrm{FP}}(1-\eta(x))$, while predicting $-1$ has conditional risk $c_{\mathrm{FN}}\eta(x)$. Comparing these quantities gives the Bayes rule.

Theorem: For positive costs $c_{\mathrm{FP}}$ and $c_{\mathrm{FN}}$, a Bayes classifier is $$ h_c^*(x)= \begin{cases} +1, & \eta(x)\ge \tau_c,\\ -1, & \eta(x)<\tau_c, \end{cases} \qquad \tau_c=\frac{c_{\mathrm{FP}}}{c_{\mathrm{FP}}+c_{\mathrm{FN}}}. $$ At $\eta(x)=\tau_c$, either decision has the same conditional risk.

To verify the rule, let $r_+(x)=c_{\mathrm{FP}}(1-\eta(x))$ and $r_-(x)=c_{\mathrm{FN}}\eta(x)$. Predicting $+1$ is no more costly exactly when $$ c_{\mathrm{FP}}(1-\eta(x))\le c_{\mathrm{FN}}\eta(x) \quad\Longleftrightarrow\quad c_{\mathrm{FP}}\le (c_{\mathrm{FP}}+c_{\mathrm{FN}})\eta(x) \quad\Longleftrightarrow\quad \eta(x)\ge\tau_c. $$ Because this comparison holds separately at every $x$, choosing the lower conditional risk minimizes the overall risk: the risk of any classifier is the expectation of its conditional risk. Moreover, when a classifier chooses the higher-cost action at $x$, its conditional excess risk is $|r_+(x)-r_-(x)|$. Integrating these nonnegative differences proves the Bayes optimality claim. $\square$

Worked Example: A High Cost for False Positives

Suppose a false positive costs $4$ units and a false negative costs $1$ unit. The threshold is $$ \tau_c=\frac{4}{4+1}=0.8. $$

At a feature value where $\eta(x)=0.7$, predicting positive costs $4(1-0.7)=1.2$, while predicting negative costs $1(0.7)=0.7$. The Bayes decision is negative. At $\eta(x)=0.85$, the corresponding costs are $4(0.15)=0.6$ and $1(0.85)=0.85$, so the Bayes decision is positive. This comparison also shows why “predict positive when the posterior exceeds one half” is not the right rule for these costs.

Thresholds for Scores

A score-based classifier predicts positive when a real-valued score exceeds a chosen threshold. If $g(x)=\log\{\eta(x)/(1-\eta(x))\}$ is the posterior log-odds for $0<\eta(x)<1$, then the probability threshold translates into a score threshold: $$ \eta(x)\ge\tau_c \quad\Longleftrightarrow\quad g(x)\ge \log\!\left(\frac{c_{\mathrm{FP}}}{c_{\mathrm{FN}}}\right). $$ Thus an ordinary log-odds score can be used with a cost-dependent threshold. Equivalently, subtract $\log(c_{\mathrm{FP}}/c_{\mathrm{FN}})$ from that score and compare the result with zero.

This distinction matters for kernel classifiers. A kernel score $g$ need not include a constant shift as an element of its reproducing kernel Hilbert space (RKHS). Applying a threshold to the score does not require the shifted function to belong to that space: the fitted function can remain $g$, with the decision made by comparing $g(x)$ to the threshold. If a shifted score is explicitly fitted or penalized, its membership and penalty in the chosen RKHS are separate issues.

Worked Example: Shifting a Kernel Score Threshold

Let $c_{\mathrm{FP}}=3$ and $c_{\mathrm{FN}}=2$. Then $\tau_c=3/5=0.6$, and the log-odds threshold is $\log(3/2)$. Suppose a fitted score is the true posterior log-odds at a point with $\eta(x)=0.75$. Its value is $$ g(x)=\log\!\left(\frac{0.75}{0.25}\right)=\log 3. $$

Since $\log 3>\log(3/2)$, classify the point as positive. Alternatively, use the shifted score $$ g(x)-\log(3/2)=\log 3-\log(3/2)=\log 2>0, $$ which gives the same decision by thresholding at zero. The threshold comparison is valid whether $g$ is a kernel score or another real-valued score; it does not by itself establish that $g$ estimates the true log-odds.

How Weighted Surrogate Losses Encode the Costs

A surrogate loss replaces the discontinuous classification loss with a more convenient loss of the signed margin $Yf(X)$, where $f$ is a real-valued score. To incorporate unequal costs, weight positive-label losses by $c_{\mathrm{FN}}$ and negative-label losses by $c_{\mathrm{FP}}$. For a margin loss $\phi$, the corresponding population surrogate risk is $$ \mathcal R_{c,\phi}(f) =E\!\left[c_{\mathrm{FN}}\mathbf 1\{Y=+1\}\phi(f(X)) +c_{\mathrm{FP}}\mathbf 1\{Y=-1\}\phi(-f(X))\right]. $$

Conditioning on $X=x$, define $A=c_{\mathrm{FN}}\eta(x)$ and $B=c_{\mathrm{FP}}(1-\eta(x))$. The conditional surrogate risk is $A\phi(f(x))+B\phi(-f(x))$. The comparison $A\ge B$ is exactly the Bayes positive-decision condition. The following two losses make that connection explicit.

Result: For logistic loss $\phi(u)=\log(1+e^{-u})$, when $0<\eta(x)<1$, the unique minimizer of the conditional weighted logistic risk is $$ f_c^*(x)=\log\!\left(\frac{c_{\mathrm{FN}}\eta(x)} {c_{\mathrm{FP}}(1-\eta(x))}\right). $$ Its sign is positive exactly when $\eta(x)>\tau_c$, zero at equality, and negative below the threshold.

For a complete derivation, the conditional risk as a function of a scalar $f$ is $$ r(f)=A\log(1+e^{-f})+B\log(1+e^f). $$ For $0<\eta(x)<1$, both $A$ and $B$ are positive. Differentiation gives $$ r'(f)=\frac{-A+Be^f}{1+e^f}, \qquad r''(f)=\frac{(A+B)e^f}{(1+e^f)^2}>0. $$ The strictly positive second derivative makes $r$ strictly convex. Setting $r'(f)=0$ gives $e^f=A/B$, hence the stated unique minimizer. Its sign is the sign of $A-B$, and $$ A\ge B \quad\Longleftrightarrow\quad c_{\mathrm{FN}}\eta(x)\ge c_{\mathrm{FP}}(1-\eta(x)) \quad\Longleftrightarrow\quad \eta(x)\ge\tau_c. $$ At $\eta(x)=0$ or $1$, the infimum is approached as $f$ tends to negative or positive infinity, respectively; there is no finite minimizer. The limiting sign still agrees with the Bayes decision. $\square$

For comparison, unweighted logistic loss has conditional minimizer $\log\{\eta(x)/(1-\eta(x))\}$ at interior probabilities. Weighting the two classes changes that minimizer by subtracting $\log(c_{\mathrm{FP}}/c_{\mathrm{FN}})$. It therefore builds the cost-dependent threshold into the score itself.

Worked Example: Weighted Logistic Loss at the Threshold

Take $c_{\mathrm{FP}}=3$, $c_{\mathrm{FN}}=2$, and $\eta(x)=0.6$. Then $A=2(0.6)=1.2$ and $B=3(0.4)=1.2$. The weighted logistic minimizer is $$ f_c^*(x)=\log(A/B)=\log(1)=0. $$ The posterior is exactly the Bayes threshold, so either class decision has equal cost.

If instead $\eta(x)=0.75$, then $A=1.5$ and $B=0.75$, giving $f_c^*(x)=\log 2>0$. The classifier predicts positive, as the Bayes rule requires. The unweighted log-odds score at this posterior is $\log 3$; subtracting $\log(3/2)$ again gives $\log 2$.

Weighted Hinge Loss, Including Endpoint Cases

For hinge loss $\phi(u)=(1-u)_+$, where $z_+=\max(z,0)$, the conditional weighted risk is $$ r(f)=A(1-f)_+ + B(1+f)_+. $$ Its minimizers make the cost-sensitive decision visible directly, while also illustrating why endpoint posteriors need care.

Result: If $A>B$, every minimizer of the weighted hinge conditional risk is at least $1$. If $A<B$, every minimizer is at most $-1$. If $A=B$, the minimizer set is $[-1,1]$. Consequently, whenever $\eta(x)\ne\tau_c$, every conditional minimizer has the Bayes sign.

To derive this result, divide the real line into three regions. For $f<-1$, the positive-part terms give $r(f)=A(1-f)$, which is nonincreasing as $f$ increases toward $-1$, and strictly decreasing if $A>0$. For $-1\le f\le1$, they give $r(f)=A+B+(B-A)f$. For $f>1$, they give $r(f)=B(1+f)$, which is nondecreasing as $f$ increases beyond $1$, and strictly increasing if $B>0$.

If $A>B$, the middle expression decreases with $f$, so it is minimized at $f=1$; the right-hand expression cannot improve on that value. If $A=0$, the left-hand expression is identically zero, so all $f\le-1$ also minimize, but this case has $A<B$, not $A>B$. Thus for $A>B$, all minimizers are at least $1$. The symmetric argument gives minimizers at most $-1$ when $A<B$, allowing a flat right-hand region if $B=0$. When $A=B$, the middle expression is constant and each outer region has no smaller value, so the full minimizer set is $[-1,1]$. Since $A\ge B$ is equivalent to $\eta(x)\ge\tau_c$, these cases give the claimed Bayes signs away from equality. At equality, the Bayes decision itself is tied. $\square$

Worked Example: Hinge-Loss Ties and Posterior Endpoints

Let $c_{\mathrm{FP}}=2$ and $c_{\mathrm{FN}}=3$, so $\tau_c=2/5=0.4$. At $\eta(x)=0.4$, $A=3(0.4)=1.2$ and $B=2(0.6)=1.2$. The conditional hinge risk is constant on $[-1,1]$, so every score in that interval minimizes it; zero score is not evidence of a uniquely determined class.

At the endpoint $\eta(x)=0$, $A=0$ and $B=2$. The risk is $2(1+f)_+$, which equals zero for every $f\le-1$. It is flat there, not strictly decreasing. At $\eta(x)=1$, $A=3$ and $B=0$, so the risk is $3(1-f)_+$, equal to zero for every $f\ge1$. It is flat there, not strictly increasing. In both endpoint cases, the minimizing scores have the correct Bayes sign.

Using the Rule in Kernel Training

For a kernel score $f$ in an RKHS $\mathcal H$, a cost-weighted empirical surrogate objective can take the form $$ \frac{1}{n}\sum_{i=1}^n \left[ c_{\mathrm{FN}}\mathbf 1\{Y_i=+1\}\phi(f(X_i)) +c_{\mathrm{FP}}\mathbf 1\{Y_i=-1\}\phi(-f(X_i)) \right] +\lambda\|f\|_{\mathcal H}^2, \qquad \lambda>0. $$ The weights encode the desired cost comparison in the population conditional risk. For weighted logistic loss, zero-thresholding its ideal conditional score targets the cost-sensitive Bayes rule; for weighted hinge loss, the conditional minimizers have the required sign except where the Bayes decision is tied.

This population calculation is not, on its own, a finite-sample guarantee. A regularized fit is constrained by the chosen function class and the observed sample; its score need not equal the pointwise conditional minimizer. In particular, weighting does not remove approximation error or make an estimated score exact. The safe interpretation is that costs determine the target rule and the corresponding population loss, while estimation and regularization determine how closely a fitted kernel score reaches that target.

Common Mistakes

Using the wrong cost in the threshold numerator. The threshold is $c_{\mathrm{FP}}/(c_{\mathrm{FP}}+c_{\mathrm{FN}})$: it follows by comparing the cost of predicting positive with the cost of predicting negative.
Applying a one-half threshold by habit. That threshold is appropriate for equal costs, not automatically for unequal costs.
Confusing weighted and unweighted scores. An ordinary log-odds score uses threshold $\log(c_{\mathrm{FP}}/c_{\mathrm{FN}})$; a cost-weighted logistic score has that shift built into its population target and is thresholded at zero.
Claiming strict hinge monotonicity at all posteriors. At $\eta(x)=0$, the hinge risk is flat for $f\le-1$; at $\eta(x)=1$, it is flat for $f\ge1$. The outer-region monotonicity is non-strict in general and strict only when its corresponding coefficient is positive.
Assuming a threshold shift is an RKHS function. A decision can compare an unshifted kernel score with a scalar threshold; this does not require a constant function to belong to the RKHS.

Key takeaway: Compare $c_{\mathrm{FP}}(1-\eta(x))$ with $c_{\mathrm{FN}}\eta(x)$. This yields the Bayes probability threshold, the corresponding log-odds shift, and the class weights used in cost-sensitive surrogate training.

Check Your Understanding

Use the cost comparison and conditional-risk calculations, not a default one-half threshold.

If $c_{\mathrm{FP}}=1$ and $c_{\mathrm{FN}}=4$, what is the Bayes probability threshold?
For general positive costs, derive the log-odds threshold from the probability threshold.
At $\eta(x)=\tau_c$, what are the minimizers of the weighted hinge conditional risk?
Why can an unshifted kernel score be thresholded without requiring a constant function in its RKHS?
At posterior endpoints, why may the weighted hinge risk be flat on an unbounded interval?