Nondifferentiable Losses and Subgradients

Why Subgradients Are Needed

A convex loss need not have an ordinary derivative at every prediction. Absolute loss has a sharp corner when the prediction equals the observed value; hinge loss has a corner at the margin. At such a point, asking for “the slope” has no single answer. A subgradient gives a precise replacement: it is the slope of a line that supports the loss from below at the point in question.

The distinction is useful in optimization because many important convex losses are not differentiable everywhere. The definition does not require choosing one arbitrarily preferred slope at a corner: it identifies the entire set of slopes that satisfy the supporting-line condition.

The inequality is required for every possible comparison point \(u\), not merely for nearby values. Geometrically, the line through \((t,\phi(t))\) with slope \(g\) must never lie above the graph. Convexity is essential to the familiar supporting-line interpretation and, for finite convex functions on the real line, ensures that subgradients exist.

Ordinary Derivatives and Subgradients

At a differentiable point of a convex function, the subdifferential contains exactly the ordinary derivative. First, convexity gives the supporting-line inequality with slope \(\phi'(t)\). To see why that slope is unique, suppose \(g\) is any subgradient and set \(u=t+h\). For \(h>0\), the defining inequality implies \(\bigl(\phi(t+h)-\phi(t)\bigr)/h\geq g\). Taking \(h\) down to zero gives \(\phi'(t)\geq g\). For \(h<0\), division reverses the inequality, so \(\bigl(\phi(t+h)-\phi(t)\bigr)/h\leq g\). Taking \(h\) up to zero gives \(\phi'(t)\leq g\). Thus \(g=\phi'(t)\).

For completeness, the supporting-line inequality at a differentiable point follows directly from convexity. If \(u>t\), convexity applied to \(t+h\), for \(0<h<u-t\), gives \(\phi(t+h)\leq \phi(t)+\frac{h}{u-t}\bigl(\phi(u)-\phi(t)\bigr)\). Divide by \(h\) and let \(h\downarrow0\) to obtain \(\phi'(t)\leq(\phi(u)-\phi(t))/(u-t)\). If \(u<t\), apply the same convexity argument to \(t-h\), with \(0<h<t-u\), and let \(h\downarrow0\); it gives \((\phi(t)-\phi(u))/(t-u)\leq\phi'(t)\). Rearranging either secant-slope inequality yields \(\phi(u)\geq\phi(t)+\phi'(t)(u-t)\). This proves the stated result. \(\square\)

Absolute Loss

For an observed value \(y\), define absolute loss as a function of prediction \(t\) by \(\phi_y(t)=|t-y|\). To the left of \(y\), it equals \(y-t\), with slope \(-1\); to the right, it equals \(t-y\), with slope \(1\). At \(t=y\), the left and right slopes differ, so there is no ordinary derivative.

Away from \(y\), the loss is affine in a neighborhood of \(t\). Its slope is its derivative, and the differentiable-point result shows that this is its only subgradient. At \(t=y\), write \(u=y+h\). The defining inequality becomes \(|h|\geq gh\) for every real \(h\). For \(h>0\), it requires \(g\leq1\); for \(h<0\), it requires \(g\geq-1\). Together these conditions give \(-1\leq g\leq1\). Conversely, any \(g\) in that interval satisfies the inequality: for \(h\geq0\), \(gh\leq h=|h|\), and for \(h<0\), \(gh\leq -h=|h|\). This proves the full subdifferential at the kink.

Hinge Loss

For a binary label \(y\in\{-1,1\}\), the hinge loss at prediction \(t\) is \(\phi_y(t)=\max(0,1-yt)\). The quantity \(yt\) is the signed margin: the loss is linear when \(yt<1\), zero when \(yt>1\), and has a kink at \(yt=1\). On the linear side its slope with respect to \(t\) is \(-y\), not \(-1\) for both labels.

When \(yt<1\), the prediction lies in the linear part of the loss, whose slope is \(-y\); when \(yt>1\), the loss is locally constant with slope \(0\). In each case the loss is differentiable, so these are the only subgradients.

To verify the kink formula, suppose \(yt=1\). For any \(u\), put \(h=u-t\). Then \(1-yu=-yh\), and the loss at \(u\) is \(\max(0,-yh)\), while the loss at \(t\) is zero. A proposed slope \(g\) is valid precisely when \(\max(0,-yh)\geq gh\) for every \(h\). Taking \(h=ys\) with \(s>0\) makes the left side zero, so \(gy\leq0\). Taking \(h=-ys\) with \(s>0\) makes the left side \(s\) and the right side \(-gys\), so \(gy\geq-1\). Therefore \(-1\leq gy\leq0\), which is equivalent to \(g=-\alpha y\) for some \(\alpha\in[0,1]\). Conversely, if these bounds hold, then for \(h=ys\), \(s>0\), \(gh=gys\leq0\); for \(h=-ys\), \(s>0\), \(gh=-gys\leq s\). Those are exactly the required inequalities on the two sides. The formula is proved. \(\square\)

Common Mistakes and Practical Checks

• Treating a subgradient as an ordinary derivative. At a kink there is no single derivative. The subdifferential is a set of slopes that satisfy a global inequality.
• Using only one side of the kink. A candidate slope must support the loss on both sides and for every \(u\). A one-sided slope may be valid, but it is not the only valid subgradient.
• Ignoring the label in hinge loss. On the linear part the slope is \(-y\). It equals \(-1\) for \(y=1\) and \(1\) for \(y=-1\).
• Reversing the hinge slopes for \(y=-1\). The kink is at \(t=-1\); the loss is flat to its left and increasing to its right. The left and right slopes are \(0\) and \(1\), respectively.
• Checking an inequality at just one comparison value. A single successful check does not establish that \(g\) is a subgradient. The definition quantifies over every \(u\); use the derived formula or verify both sides for arbitrary \(u\).