Why Convex Losses Give Convex Risks

Why Convexity Passes to Risk

A loss evaluates the penalty for a prediction at a particular observation; risk averages that penalty over the distribution of observations. If, for every fixed observation, blending two predictions cannot increase loss beyond the corresponding blend of their losses, then taking an expectation preserves that inequality. The result is a convexity property of risk, not just of the loss at a single data point.

The conclusion has practical limits. A blend of two possible predictions must itself be an allowed prediction, the resulting rule must be measurable, and expectations must be handled carefully if risk can be infinite. In particular, the usual convex-combination expression must not be used at an endpoint when it contains $0\cdot\infty$.

Setup and Definitions

Let $Z$ be a random observation taking values in a measurable space $\mathcal Z$. Predictions take values in a convex set $\mathcal A\subseteq\mathbb R^d$. A prediction rule $h:\mathcal Z\to\mathcal A$ assigns a prediction to each observation. Let $\ell(a;z)$ denote the loss from predicting $a$ when the observation is $z$. Assume that $\ell$ is jointly measurable and nonnegative, so that its expectation is well-defined as an extended nonnegative real number. The risk of $h$ is

R(h)=E[\ell(h(Z);Z)]\in[0,+\infty].

For this tutorial, convexity of the loss means convexity in the prediction while the observation is held fixed. Thus, for every $z\in\mathcal Z$, every $a,b\in\mathcal A$, and every $t\in[0,1]$,

\ell((1-t)a+tb;z)\leq (1-t)\ell(a;z)+t\ell(b;z).

This is the two-point form of Jensen’s inequality for a convex function. It compares the loss at an interpolated prediction with the interpolation of the two losses. It makes no claim that the observation itself, or the distribution of observations, is being mixed.

Definition: For two prediction rules $h,g:\mathcal Z\to\mathcal A$, their pointwise mixture at weight $t\in[0,1]$ is the rule $h_t$ defined by $h_t(z)=(1-t)h(z)+tg(z)$. This is an interpolation of their predictions at each observation, not a random choice of one rule or the other.

Convexity of $\mathcal A$ ensures that $h_t(z)\in\mathcal A$. If $h$ and $g$ are measurable, then their coordinatewise affine combination $h_t$ is measurable. Joint measurability of the loss then makes $\ell(h_t(Z);Z)$ a measurable random variable. These conditions let us define all three risks in the claim below.

The Convexity Theorem for Risk

Theorem: Suppose $\mathcal A\subseteq\mathbb R^d$ is convex, $h$ and $g$ are measurable prediction rules into $\mathcal A$, and $\ell$ is jointly measurable, nonnegative, and convex in its prediction for every fixed observation. For $0<t<1$, $$ R(h_t)\leq (1-t)R(h)+tR(g). $$ At $t=0$, $h_t=h$ and $R(h_t)=R(h)$; at $t=1$, $h_t=g$ and $R(h_t)=R(g)$. These endpoint statements apply whether the risks are finite or infinite.

Proof. Fix $z\in\mathcal Z$. Apply convexity of $\ell(\,\cdot\,;z)$ to the two predictions $h(z)$ and $g(z)$:

\ell(h_t(z);z) \leq (1-t)\ell(h(z);z)+t\ell(g(z);z).

Now substitute the random observation $Z$. The inequality holds pointwise, so monotonicity of expectation for nonnegative random variables gives

E[\ell(h_t(Z);Z)] \leq E[(1-t)\ell(h(Z);Z)+t\ell(g(Z);Z)].

For $0<t<1$, both coefficients are strictly positive. Homogeneity and additivity of the integral for nonnegative functions, including when an integral is $+\infty$, give

E[(1-t)\ell(h(Z);Z)+t\ell(g(Z);Z)] =(1-t)R(h)+tR(g).

Combining the inequalities proves the claim for interior weights. At $t=0$, the definition gives $h_t=h$, and at $t=1$ it gives $h_t=g$; hence the risks equal the corresponding endpoint risks. This direct treatment avoids expressions such as $0\cdot(+\infty)$, which are undefined under standard extended-real arithmetic. $\square$

The theorem says that the risk functional is convex along every segment of prediction rules that is available in this way. It does not say that risk must be finite: the inequality for an interior weight remains valid as an extended-real inequality, including when its right-hand side is $+\infty$.

Worked Examples

Worked Example: Scalar Squared Loss

Let $Y$ equal 0 with probability $1/4$ and 2 with probability $3/4$. Use scalar predictions $a\in\mathbb R$ and squared loss $\ell(a;Y)=(Y-a)^2$. Consider constant rules predicting $a=0$ and $b=2$. Their risks are

R(0)=\frac14(0-0)^2+\frac34(2-0)^2=0+\frac34(4)=3,

R(2)=\frac14(0-2)^2+\frac34(2-2)^2=\frac14(4)+0=1.

With $t=1/2$, the interpolated prediction is $a_t=(1/2)0+(1/2)2=1$. Its risk is

R(1)=\frac14(0-1)^2+\frac34(2-1)^2 =\frac14+\frac34=1.

The convexity bound is $R(1)\leq \frac12R(0)+\frac12R(2)=\frac12(3)+\frac12(1)=2$, and indeed $1\leq2$. The bound need not be an equality: convexity gives an upper bound on the risk of the interpolated prediction, not an exact formula for it.

Worked Example: Vector Squared Loss

Let the response $Y$ take the values $(1,0)$ and $(-1,2)$, each with probability $1/2$. Predictions lie in $\mathbb R^2$, and the loss is squared Euclidean distance, $\ell(a;Y)=\|Y-a\|_2^2$. Compare the constant predictions $u=(0,0)$ and $v=(2,0)$, using $t=1/4$.

At $u$, the two squared distances are $1^2+0^2=1$ and $(-1)^2+2^2=5$, so

R(u)=\frac12(1)+\frac12(5)=3.

At $v$, the squared distances are $(1-2)^2+0^2=1$ and $(-1-2)^2+2^2=9+4=13$, so

R(v)=\frac12(1)+\frac12(13)=7.

The blended prediction is $u_t=(3/4)u+(1/4)v=(1/2,0)$. Its squared distances are $(1-1/2)^2+0^2=1/4$ and $(-1-1/2)^2+2^2=9/4+4=25/4$. Therefore

R(u_t)=\frac12\left(\frac14\right)+\frac12\left(\frac{25}{4}\right) =\frac{26}{8}=\frac{13}{4}.

The risk bound is $(3/4)R(u)+(1/4)R(v)=(3/4)(3)+(1/4)(7)=9/4+7/4=4$. Thus $13/4\leq4$, as required. The calculation shows that the same argument works for vector predictions: the convex combination is taken coordinatewise, while the loss is evaluated using the vector prediction.

Worked Example: Hinge Loss With a Scalar Prediction

Let the label $Y\in\{-1,+1\}$ have probability $1/2$ at each value. For scalar score $a\in\mathbb R$, take hinge loss $\ell(a;Y)=\max(0,1-Ya)$. The maximum of two affine functions of $a$ is convex, so this loss is convex in the prediction for each fixed label.

At prediction $a=-1$, the loss is 2 when $Y=+1$ and 0 when $Y=-1$, giving $R(-1)=(1/2)(2)+(1/2)(0)=1$. At prediction $a=1$, the losses are 0 and 2, so $R(1)=1$. Their midpoint is $a_0=0$, where the loss is 1 for either label; hence $R(0)=1$. The mixture inequality reads $1\leq (1/2)(1)+(1/2)(1)=1$, and holds with equality. This example also shows that convexity does not require the risk to decrease strictly when predictions are blended.

A Fully Derived Squared-Loss Identity

For squared Euclidean loss, there is a more precise relation than the general inequality. Let $y,a,b\in\mathbb R^d$ and $0\leq t\leq1$. Write $x=a-y$ and $w=b-y$. Expanding the squared norm using the inner product gives

\begin{aligned} \|(1-t)a+tb-y\|_2^2 &=\|(1-t)x+tw\|_2^2\\ &=(1-t)^2\|x\|_2^2+2t(1-t)\langle x,w\rangle+t^2\|w\|_2^2. \end{aligned}

On the other hand, subtracting this expression from the weighted endpoint losses yields

\begin{aligned} &(1-t)\|x\|_2^2+t\|w\|_2^2-\|(1-t)x+tw\|_2^2\\ &=t(1-t)\bigl(\|x\|_2^2+\|w\|_2^2-2\langle x,w\rangle\bigr)\\ &=t(1-t)\|x-w\|_2^2 =t(1-t)\|a-b\|_2^2. \end{aligned}

Rearranging proves the pointwise identity

\|(1-t)a+tb-y\|_2^2 =(1-t)\|a-y\|_2^2+t\|b-y\|_2^2-t(1-t)\|a-b\|_2^2.

If prediction rules $h,g$ have finite squared-loss risks, apply this identity at $a=h(Z)$, $b=g(Z)$, and $y=Y$, then take expectations. All terms have finite expectations: in particular, $\|h(Z)-g(Z)\|_2^2\leq2\|h(Z)-Y\|_2^2+2\|g(Z)-Y\|_2^2$. Thus

R(h_t)=(1-t)R(h)+tR(g)-t(1-t)E[\|h(Z)-g(Z)\|_2^2].

The subtracted term is nonnegative, which recovers the convexity inequality. The identity also describes why squared loss can give a strict inequality when the two rules disagree on a set of positive probability and $0<t<1$.

What the Result Does—and Does Not—Say

The theorem concerns the risk of predictions formed by interpolation. If a learning procedure can use only a restricted class of rules $\mathcal H$, then convexity of the loss alone does not guarantee that $h_t\in\mathcal H$. For example, a class consisting only of two fixed rules need not contain any intermediate rule. The loss can remain convex pointwise even though the available class is not closed under mixtures. To conclude convexity of risk over a particular class, that class must contain the mixtures under consideration.

Interpolation is also different from randomization. A randomized procedure that selects rule $h$ with probability $1-t$ and $g$ with probability $t$ has expected risk $(1-t)R(h)+tR(g)$ for $0<t<1$, with the sum interpreted in $[0,+\infty]$. At $t=0$ its expected risk is $R(h)$, and at $t=1$ it is $R(g)$, including when either risk is infinite. The mixture rule instead predicts $(1-t)h(z)+tg(z)$ at each $z$, and its risk is bounded above by that quantity under the theorem’s assumptions. These are distinct procedures and need not have the same predictions or risk.

Nonnegativity of the loss is a convenient assumption that permits extended risks in $[0,+\infty]$ and ensures the expectation and order argument are valid. If a loss can take both positive and negative values, one must impose conditions such as integrability of the relevant losses before using linearity of expectation. Otherwise, expressions involving $+\infty$ and $-\infty$ may be undefined.

Common Mistake

Using the interior convexity formula blindly at the endpoints. If $R(g)=+\infty$, writing $R(h_0)\leq 1\cdot R(h)+0\cdot R(g)$ introduces the undefined product $0\cdot(+\infty)$. The correction is to state endpoint cases directly: $h_0=h$ and $h_1=g$. Use the weighted-risk inequality only for $0<t<1$, where both coefficients are positive.
Assuming pointwise convexity automatically makes every model class convex. Convexity of the loss controls the risk of a valid interpolated prediction; it does not put that prediction into a restricted class. Check that the action set and rule class permit the mixture.
Confusing a mixture prediction with a randomized selection. The mixture predicts an intermediate value; randomization selects one of the original predictions. The theorem applies directly to the former.

Key takeaway: When the action set is convex and the loss is measurable, nonnegative, and convex in each prediction, the risk of a pointwise mixture is bounded by the corresponding mixture of risks for interior weights. Handle the endpoints as equalities, especially when risks may be infinite.

Check Your Understanding

Use the conditions and distinctions developed above to answer these questions.

Which assumptions ensure that the pointwise mixture of two prediction rules is a valid measurable rule?
Why can the risk inequality be integrated when the pointwise loss inequality holds and losses are nonnegative?
If $R(h)=2$ and $R(g)=+\infty$, what is the correct risk statement at $t=0$, and why should the weighted formula not be used there?
Explain the difference between interpolating two scalar predictions and randomly selecting one of them.
Can a convex loss guarantee a risk-minimizing mixture belongs to a restricted class that contains only two rules? Explain.