Convexity of Loss in the Prediction

Why Convexity in the Prediction Matters

A loss assigns a penalty to a prediction after the outcome is known. For a fixed outcome, convexity asks how that penalty changes as the prediction moves between two values. If a loss is convex, the penalty at an intermediate prediction is no greater than the corresponding weighted average of the penalties at the endpoints. This shape property is useful when comparing candidate predictions and is one reason convex losses are common in learning problems.

Here we focus on the loss as a function of the prediction, with the outcome held fixed. The claim is not that every function of the data is convex, nor that a loss must be differentiable. Absolute and hinge losses, for example, have corners but are convex.

Definition: Let $\mathcal{T}$ be a convex set of possible predictions, and let $\ell(y,t)$ be a loss for outcome $y$ and prediction $t\in\mathcal{T}$. The loss is convex in the prediction if, for every fixed $y$, every $t_0,t_1\in\mathcal{T}$, and every $\alpha\in[0,1]$, $$ \ell\bigl(y,\alpha t_0+(1-\alpha)t_1\bigr) \leq \alpha\ell(y,t_0)+(1-\alpha)\ell(y,t_1). $$

The set $\mathcal{T}$ must itself be convex so that the intermediate prediction $\alpha t_0+(1-\alpha)t_1$ is allowed. In the examples below, the prediction is real-valued and $\mathcal{T}=\mathbb{R}$. Convexity is checked separately for each fixed outcome; it does not require a probability distribution for the outcomes.

Squared and Absolute Loss

For a real-valued outcome $y$ and prediction $t$, squared loss is $\ell(y,t)=(y-t)^2$. Its graph as a function of $t$ is a parabola, and the algebra below verifies the defining inequality directly.

Result: Squared loss is convex in its prediction. For $t_0,t_1\in\mathbb{R}$ and $\alpha\in[0,1]$, $$ \alpha(y-t_0)^2+(1-\alpha)(y-t_1)^2 -\bigl(y-\alpha t_0-(1-\alpha)t_1\bigr)^2 = \alpha(1-\alpha)(t_0-t_1)^2 \geq 0. $$ Thus the value at the intermediate prediction is at most the weighted average of the endpoint values.

Worked Example: Squared Loss Between Two Predictions

Take outcome $y=3$, endpoint predictions $t_0=1$ and $t_1=5$, and weight $\alpha=1/4$. The intermediate prediction is $t=(1/4)(1)+(3/4)(5)=4$. Its squared loss is $(3-4)^2=1$.

The weighted average of the endpoint losses is $(1/4)(3-1)^2+(3/4)(3-5)^2=(1/4)(4)+(3/4)(4)=4$. The convexity inequality is therefore $1\leq4$, as required. The algebraic difference in the result is $(1/4)(3/4)(1-5)^2=3$, which also equals $4-1$.

Absolute loss is $\ell(y,t)=|y-t|$. Its convexity follows from the triangle inequality, so no derivative at the point $t=y$ is needed.

Proof: For $\alpha\in[0,1]$, $$ \left|y-\{\alpha t_0+(1-\alpha)t_1\}\right| = \left|\alpha(y-t_0)+(1-\alpha)(y-t_1)\right| \leq \alpha|y-t_0|+(1-\alpha)|y-t_1|. $$ The inequality is the triangle inequality together with the nonnegative weights $\alpha$ and $1-\alpha$. This is exactly the definition of convexity in the prediction. $\square$

Worked Example: Absolute Loss at an Intermediate Prediction

Let $y=2$, $t_0=0$, $t_1=4$, and $\alpha=1/2$. The intermediate prediction is $2$, with loss $|2-2|=0$. The average endpoint loss is $(1/2)|2-0|+(1/2)|2-4|=2$. Thus $0\leq2$.

If instead the outcome is $y=1$ with the same endpoint predictions and weight, the intermediate loss is $|1-2|=1$, while the average endpoint loss remains $(1/2)(1)+(1/2)(3)=2$. Again the inequality holds. Convexity does not require the intermediate prediction to have zero loss.

Logistic Loss for a Margin Prediction

For binary outcomes coded as $y\in\{-1,+1\}$, a real-valued prediction $t$ is often interpreted as a score. The product $yt$ is the signed margin: it is positive when the score points toward the observed label and negative when it points away. The logistic loss in this convention is $\ell(y,t)=\log(1+\exp(-yt))$.

Result: For fixed $y\in\{-1,+1\}$, logistic loss is convex in $t$. Its first and second derivatives are $$ \frac{\partial \ell(y,t)}{\partial t} = \frac{-y}{1+\exp(yt)}, \qquad \frac{\partial^2 \ell(y,t)}{\partial t^2} = \frac{\exp(yt)}{(1+\exp(yt))^2} \geq 0. $$

For completeness, the second-derivative test used here follows from elementary calculus. If a twice-differentiable function $g$ on an interval has $g''\geq0$, then $g'$ is nondecreasing by the mean value theorem. For any $a<z<b$, apply that theorem on $[a,z]$ and $[z,b]$: the two secant slopes are $g'(u)$ and $g'(v)$ for some $u\in(a,z)$ and $v\in(z,b)$, so $g'(u)\leq g'(v)$. Hence $(g(z)-g(a))/(z-a)\leq(g(b)-g(z))/(b-z)$. Rearranging gives $g(z)\leq ((b-z)/(b-a))g(a)+((z-a)/(b-a))g(b)$, the convexity inequality. The endpoints follow by equality. The displayed second derivative is nonnegative everywhere, so this test applies to logistic loss on $\mathbb{R}$.

Worked Example: Evaluating Logistic Loss

Take $y=+1$. At $t=0$, the loss is $\log(2)$. At $t=2$, it is $\log(1+\exp(-2))$, approximately $0.127$. At the midpoint $t=1$, it is $\log(1+\exp(-1))$, approximately $0.313$.

The average of the endpoint losses is $\{\log(2)+\log(1+\exp(-2))\}/2$, approximately $(0.693+0.127)/2=0.410$. Thus the midpoint loss $0.313$ is below the average $0.410$, consistent with convexity. This numerical check illustrates the inequality; the second-derivative argument establishes it for every pair of predictions.

Hinge Loss and Zero-One Loss

Using the same binary labels and score prediction, hinge loss is $\ell(y,t)=\max(0,1-yt)$. It penalizes a score that does not reach a unit positive margin. Its convexity can be verified without differentiating at the corner.

Proof: For fixed $y$, the two functions of $t$ given by $0$ and $1-yt$ are affine. For $t_\alpha=\alpha t_0+(1-\alpha)t_1$, each affine function at $t_\alpha$ equals the same weighted average of its values at $t_0,t_1$. Therefore, for either affine function $q$, $$ q(t_\alpha)=\alpha q(t_0)+(1-\alpha)q(t_1) \leq \alpha\max\{0,1-yt_0\}+(1-\alpha)\max\{0,1-yt_1\}. $$ Taking the maximum over the two choices $q(t_\alpha)$ on the left gives $\max\{0,1-yt_\alpha\}$ and preserves this upper bound. Hence hinge loss satisfies the convexity inequality. $\square$

Worked Example: Hinge Loss Across Its Corner

Let $y=+1$, $t_0=0$, and $t_1=2$, with $\alpha=1/2$. The endpoint hinge losses are $\max(0,1)=1$ and $\max(0,-1)=0$. The midpoint prediction is $t=1$, whose loss is $\max(0,0)=0$. The required inequality is $0\leq(1/2)(1)+(1/2)(0)=1/2$.

This pair crosses the corner at $t=1$. The lack of a derivative at that point does not obstruct the convexity check: the maximum-of-affine-functions proof applies there as well.

Zero-one loss makes a different choice. For a real score $t$ and label $y\in\{-1,+1\}$, define $\ell_{01}(y,t)=\mathbf{1}\{yt\leq0\}$, where $\mathbf{1}$ is one when its condition holds and zero otherwise. This convention counts a zero score as an error. Unlike the four losses above, zero-one loss is not convex in the score.

Worked Example: A Convexity Violation for Zero-One Loss

Fix $y=+1$, and choose $t_0=-1$, $t_1=1$, and $\alpha=1/2$. At $t_0$, $yt_0=-1\leq0$, so the loss is $1$. At $t_1$, $yt_1=1>0$, so the loss is $0$. The midpoint prediction is $t=0$; by the stated tie convention, its loss is $1$.

Convexity would require $1\leq(1/2)(1)+(1/2)(0)=1/2$, which is false. Thus zero-one loss is nonconvex in the score prediction. The conclusion is not changed by choosing the opposite tie convention: using endpoints $t_0=-2$ and $t_1=1$ gives a midpoint of $-1/2$, where the loss is $1$, but the endpoint average is still $1/2$.

What Convexity Does—and Does Not—Say

The property examined here belongs to the function $t\mapsto\ell(y,t)$ with $y$ fixed. It is a pointwise statement about the shape of a loss as the prediction changes. It does not, by itself, describe an optimization algorithm or establish a property of a population or empirical risk; those quantities involve additional operations on losses.

A frequent mistake is to infer convexity from a loss being nonnegative, continuous, or intuitive. None of those properties is enough: zero-one loss is nonnegative and is not convex. The correct check is the defining inequality for all pairs of allowed predictions and all weights in $[0,1]$. Another common mistake is to assume that a corner rules out convexity. Absolute and hinge losses show why that is wrong: differentiability is not part of the definition.

Loss	Prediction setup	Convex in prediction?	Reason
Squared: $(y-t)^2$	Real outcome and prediction	Yes	Direct convexity identity
Absolute: $\|y-t\|$	Real outcome and prediction	Yes	Triangle inequality
Logistic: $\log(1+\exp(-yt))$	$y\in\{-1,+1\}$, real score	Yes	Nonnegative second derivative
Hinge: $\max(0,1-yt)$	$y\in\{-1,+1\}$, real score	Yes	Maximum of two affine functions
Zero-one: $\mathbf{1}\{yt\leq0\}$	$y\in\{-1,+1\}$, real score	No	Midpoint counterexample

Key takeaway: Convexity in the prediction is the Jensen inequality with the outcome held fixed. Squared, absolute, logistic, and hinge losses satisfy it; zero-one loss fails it. A loss may be convex even when it has a corner, and a nonnegative loss need not be convex.

Check Your Understanding

Use the definition and the examples above to answer the following.

In the definition of convexity in the prediction, which quantities are held fixed and which are allowed to vary?
For squared loss, what is the nonnegative difference between the weighted endpoint losses and the loss at the interpolated prediction?
Why does the second derivative of logistic loss establish convexity for every fixed binary label?
Explain why the corner of hinge loss does not prevent it from being convex.
Give a pair of predictions and a weight that demonstrate the failure of convexity for zero-one loss under the stated tie convention.

Loss	Prediction setup	Convex in prediction?	Reason
Squared: \((y-t)^2\)	Real outcome and prediction	Yes	Direct convexity identity
Absolute: \(\|y-t\|\)	Real outcome and prediction	Yes	Triangle inequality
Logistic: \(\log(1+\exp(-yt))\)	\(y\in\{-1,+1\}\), real score	Yes	Nonnegative second derivative
Hinge: \(\max(0,1-yt)\)	\(y\in\{-1,+1\}\), real score	Yes	Maximum of two affine functions
Zero-one: \(\mathbf{1}\{yt\leq0\}\)	\(y\in\{-1,+1\}\), real score	No	Midpoint counterexample