Discontinuous Zero-One Loss

Introduction

A classification error is easy to describe: the predicted label either agrees with the true label or it does not. Zero-one loss records exactly that distinction, assigning zero to a correct prediction and one to an incorrect prediction. This simplicity is useful, but it comes with an important mathematical cost: small changes in a numerical prediction can abruptly change the loss.

The distinction between a predicted class and a numerical score matters. If predictions are labels in a finite set, zero-one loss is a function on that set. In many learning problems, however, a model first produces a real-valued score and the score's sign determines the class. We will examine zero-one loss in that score representation, where its discontinuity and nonconvexity can be stated precisely.

Definition and Conventions

Definition: For a binary response $y\in\{-1,+1\}$ and a real-valued score $t\in\mathbb{R}$, define the zero-one loss by $$ \ell_{01}(y,t)=\mathbf{1}\{yt\leq 0\}, $$ where $\mathbf{1}\{A\}$ is one when condition $A$ holds and zero otherwise. A positive score predicts $+1$, and a negative score predicts $-1$. A score of zero is counted as an error for either label.

When $t\neq 0$, this loss is precisely the indicator that the predicted label $\operatorname{sign}(t)$ differs from $y$. The convention at zero removes ambiguity: a tie is counted as an error. It is a convention for this score-based loss, not a claim that every classification system must handle ties this way.

For a predictor $g$ that maps an input $x$ to a real score $g(x)$, its zero-one risk is the expected loss $R(g)=E[\ell_{01}(Y,g(X))]$. When the joint distribution of $(X,Y)$ has finite support, this expectation is a finite weighted sum. The risk is the average penalty under the specified distribution; calculating it does not require choosing or optimizing a predictor.

Worked Risk Calculations

Worked Example: Risk for Two Score Predictors

Suppose $(X,Y)$ has the following finite distribution, and two predictors produce the displayed scores. Calculate each predictor's risk.

Outcome	Probability	Label $Y$	Score from $g_A$	Score from $g_B$
1	0.20	$+1$	$+0.7$	$-0.2$
2	0.30	$+1$	$-0.1$	$+0.4$
3	0.10	$-1$	$-0.8$	$+0.3$
4	0.40	$-1$	$+0.2$	$-0.6$

For $g_A$, outcome 1 is correct because $Yg_A(X)=1(0.7)=0.7>0$, so its loss is zero. Outcome 2 is incorrect because $1(-0.1)=-0.1\leq 0$, so its loss is one. Outcome 3 is correct because $(-1)(-0.8)=0.8>0$, and outcome 4 is incorrect because $(-1)(0.2)=-0.2\leq 0$. Thus $$ R(g_A)=0.20(0)+0.30(1)+0.10(0)+0.40(1)=0.70. $$

For $g_B$, the four products $Yg_B(X)$ are $-0.2,\ 0.4,\ -0.3,\ 0.6$, respectively. The losses are therefore $1,0,1,0$, giving $$ R(g_B)=0.20(1)+0.30(0)+0.10(1)+0.40(0)=0.30. $$

Each risk is the probability mass of the outcomes on which that predictor makes an error. The probabilities sum to $0.20+0.30+0.10+0.40=1$, and both risks lie between zero and one.

Worked Example: A Zero Score and the Tie Convention

Let $P(Y=+1)=0.6$ and $P(Y=-1)=0.4$. Consider a constant predictor with score $t=0$. For either possible label, $Yt=0$, so the loss is one. Its risk is $$ R(0)=0.6(1)+0.4(1)=1. $$

Now take the constant score $t=0.5$. A positive label incurs loss zero because $1(0.5)>0$; a negative label incurs loss one because $(-1)(0.5)<0$. Therefore $$ R(0.5)=0.6(0)+0.4(1)=0.4. $$

For $t=-0.5$, the positive label incurs loss one and the negative label loss zero, so $R(-0.5)=0.6$. The calculation at zero differs from either nonzero score because the stated convention counts every tie as an error.

Boundedness, Discontinuity, and Nonconvexity

Theorem: For every $y\in\{-1,+1\}$ and $t\in\mathbb{R}$, $0\leq \ell_{01}(y,t)\leq 1$. As a function of $t$, $\ell_{01}(y,t)$ is discontinuous at $t=0$, is not globally Lipschitz, and is not convex.

Proof of boundedness. The loss is an indicator, so it can take only the values zero and one. Both satisfy $0\leq \ell_{01}(y,t)\leq 1$, for every label and score. In particular, for any random pair $(X,Y)$ and measurable score predictor $g$, the loss is bounded and hence has a finite expectation. Monotonicity of expectation gives $$ 0\leq R(g)=E[\ell_{01}(Y,g(X))]\leq 1. $$ This argument requires no finite-support assumption: boundedness alone ensures integrability.

Proof of discontinuity. Fix $y=+1$. For every $t<0$, $\ell_{01}(+1,t)=1$, while for every $t>0$, $\ell_{01}(+1,t)=0$. At $t=0$, the loss is one. In particular, along the positive sequence $t_n=1/n$, which converges to zero, $\ell_{01}(+1,t_n)=0$ for all $n$, but $\ell_{01}(+1,0)=1$. The function is not continuous at zero. For $y=-1$, the loss equals zero when $t<0$ and one when $t\geq0$, so it is also discontinuous there.

Proof that the loss is not globally Lipschitz. A function is globally Lipschitz if there is a finite constant $L$ such that the absolute difference in its values at any two arguments is at most $L$ times the distance between those arguments. Fix $y=+1$, and compare $t=-\varepsilon$ and $s=\varepsilon$, where $\varepsilon>0$. The loss difference is one, while the distance between the scores is $2\varepsilon$. A Lipschitz bound would require $1\leq 2L\varepsilon$ for every $\varepsilon>0$, which fails for any fixed finite $L$ by taking $\varepsilon<1/(2L)$. Thus no such global constant exists.

Proof of nonconvexity. A convex function $h$ must satisfy $h((u+v)/2)\leq [h(u)+h(v)]/2$ for every pair $u,v$. For $h(t)=\ell_{01}(+1,t)$, take $u=-1$ and $v=1$. Then $h(-1)=1$, $h(1)=0$, and $h(0)=1$, so $$ h(0)=1>\frac{h(-1)+h(1)}{2}=\frac{1}{2}. $$ The defining inequality fails, and the loss is not convex. Since a differentiable function must be continuous, the discontinuity also rules out differentiability at zero. $\square$

What the Risk Calculation Shows

The boundedness result transfers directly from the individual loss to risk: a model cannot have zero-one risk below zero or above one. In a finite distribution with outcomes indexed by $i$, probabilities $p_i$, labels $y_i$, and scores $t_i$, the calculation is $$ R=\sum_i p_i\mathbf{1}\{y_it_i\leq0\}. $$ Each summand contributes either its full probability $p_i$, if it is an error, or zero, if it is correct. The formula makes clear why finite risk calculations are transparent even though the loss itself is discontinuous.

A discontinuous loss need not make every risk function discontinuous in every setting. For example, a distribution or a restricted set of possible scores can affect how the weighted errors change. But when the score crosses zero for an outcome with positive probability, that outcome's contribution can jump by its probability mass. The finite calculation below isolates this effect.

Worked Example: A Risk Jump Across the Decision Boundary

Suppose a constant score $t$ is used, with $P(Y=+1)=0.7$ and $P(Y=-1)=0.3$. For $t<0$, positive labels are errors and negative labels are correct, so $$ R(t)=0.7 \qquad (t<0). $$ For $t>0$, negative labels are errors and positive labels are correct, so $$ R(t)=0.3 \qquad (t>0). $$ At $t=0$, both labels are counted as errors, giving $R(0)=1$. Thus the risk has a jump at zero under this distribution. The change from a small positive score to a zero score changes the risk from $0.3$ to $1$, a difference of $0.7$, however small the positive score is.

This is one concrete way the loss's discontinuity can appear in risk calculations. It does not imply that every parameterized risk has a jump at every boundary; the distribution and the predictor determine which outcomes contribute.

Class Labels Versus Scores

There is a useful qualification to the statement that zero-one loss is discontinuous. If the prediction is directly a label $a$ in a finite label set $\mathcal{Y}$, the loss is $\mathbf{1}\{y\neq a\}$. With the discrete topology on $\mathcal{Y}$, every function on that finite set is continuous: each label is an isolated point. The discontinuity discussed above concerns a real-valued score $t$, where values can approach the boundary $0$ from either side.

The score formulation is important because models commonly output real numbers before converting them into labels. A tiny change in score near zero can reverse the predicted class, and zero-one loss records that reversal as a full change between zero and one. This is why boundedness alone should not be confused with smoothness or stability.

Common Mistake

Common mistake: “The loss is bounded between zero and one, so changing the score slightly cannot change the loss very much.” Boundedness limits the size of the loss values; it does not limit how rapidly the loss can change with the score. For $y=+1$, scores $-\varepsilon$ and $+\varepsilon$ are only $2\varepsilon$ apart, yet their losses differ by one. The correction is to check boundedness and continuity or Lipschitz regularity as separate properties.

Another related error is to apply the convex-risk implication in the wrong direction. A result that convex losses give convex risks does not make a particular loss convex. Here convexity fails directly by the midpoint calculation, regardless of whether a specific risk happens to have other favorable properties.

Key takeaway: Binary zero-one loss is an indicator penalty: it is bounded, but in a real-valued score representation it jumps at the decision boundary, is not globally Lipschitz, and is nonconvex. Finite risks are obtained by summing the probabilities of the misclassified outcomes.

Check Your Understanding

Answer each question using the definition and calculations above.

For $y=-1$, what is $\ell_{01}(y,t)$ at $t=-0.4$, $t=0$, and $t=0.4$?
A finite distribution has three outcomes with probabilities $0.2, 0.5, 0.3$, and a predictor makes errors on the first and third outcomes only. What is its risk?
Which two scores demonstrate the midpoint failure of convexity for the loss with $y=+1$, and what are the three loss values?
Why does boundedness of zero-one loss imply a finite risk even when the distribution is not finite?
Explain why saying “zero-one loss is discontinuous” needs qualification when predictions are class labels rather than real-valued scores.

Outcome	Probability	Label \(Y\)	Score from \(g_A\)	Score from \(g_B\)
1	0.20	\(+1\)	\(+0.7\)	\(-0.2\)
2	0.30	\(+1\)	\(-0.1\)	\(+0.4\)
3	0.10	\(-1\)	\(-0.8\)	\(+0.3\)
4	0.40	\(-1\)	\(+0.2\)	\(-0.6\)

What You'll Learn