Strong Convexity And Risk Reduction By Averaging

Why Averaging Can Reduce Risk

A weighted average of predictions can have lower risk than the weighted average of the risks of its components. Convexity alone explains why the aggregate risk is no larger: Jensen's inequality applies separately at each observation. Strong convexity gives more information: it quantifies an additional reduction, proportional to how far the component predictions lie from their weighted average.

This is a pointwise property of the loss as a function of the prediction. It does not require the candidate predictors to be independent, and it does not require the weights to have been chosen by a particular algorithm. The weights are fixed for the result below. The key objects are the loss's curvature and the dispersion of predictions at the same input.

Strong Convexity and Prediction Dispersion

Let $C$ be a convex subset of $\mathbb{R}^d$, with Euclidean norm $\|\cdot\|$. A loss $\ell(y,a)$ assigns a loss to outcome $y$ and prediction $a\in C$. The outcome $y$ is held fixed when we examine curvature in the prediction argument.

Definition: For $\lambda>0$, the loss is $\lambda$-strongly convex in its prediction on $C$ if, for every outcome $y$, the function $a\mapsto \ell(y,a)-\frac{\lambda}{2}\|a\|^2$ is convex on $C$. Equivalently, for $a,b\in C$ and $t\in[0,1]$, $$ \ell\bigl(y,ta+(1-t)b\bigr) \leq t\ell(y,a)+(1-t)\ell(y,b) -\frac{\lambda}{2}t(1-t)\|a-b\|^2. $$

The constant $\lambda$ depends on the scale of the loss and the norm. For example, the squared loss $\ell(y,a)=(y-a)^2$ is $2$-strongly convex in the scalar prediction $a$. A valid constant must hold throughout the prediction domain under consideration, not just near a typical prediction.

For $M$ candidate predictions $a_1,\ldots,a_M\in C$, let $w_1,\ldots,w_M$ be nonnegative weights summing to one, and write $\bar a=\sum_{j=1}^M w_j a_j$. The weighted dispersion is

D_w(a_1,\ldots,a_M) =\sum_{j=1}^M w_j\|a_j-\bar a\|^2.

It is zero exactly when every prediction with positive weight equals $\bar a$. Since $C$ is convex, $\bar a$ also belongs to $C$, so the aggregate is an admissible prediction.

Worked Example: Squared Loss at One Observation

Suppose the observed outcome is $y=1$, and two candidates predict $a_1=0$ and $a_2=2$, with weights $w_1=w_2=1/2$. Under squared loss, the individual losses are

\ell(1,0)=(1-0)^2=1,\qquad \ell(1,2)=(1-2)^2=1.

The weighted average prediction is $\bar a=(0+2)/2=1$, with loss $\ell(1,1)=0$. Its improvement over the weighted average component loss is therefore $1-0=1$. The weighted dispersion is

D_w=\frac12(0-1)^2+\frac12(2-1)^2 =\frac12+\frac12=1.

Squared loss is $2$-strongly convex, so $(\lambda/2)D_w=(2/2)(1)=1$. The guaranteed reduction is exactly the reduction observed here.

A Pointwise Inequality for Weighted Predictions

The following result is the basic calculation behind risk reduction. Its correction term is the weighted dispersion, not the unweighted spread of the candidate predictions.

Theorem (Strongly convex Jensen inequality): Suppose $a_1,\ldots,a_M\in C$, $w_j\geq 0$, and $\sum_{j=1}^M w_j=1$. If $a\mapsto\ell(y,a)$ is $\lambda$-strongly convex on $C$, then $$ \ell\left(y,\sum_{j=1}^M w_j a_j\right) \leq \sum_{j=1}^M w_j\ell(y,a_j) -\frac{\lambda}{2}\sum_{j=1}^M w_j\left\|a_j-\sum_{k=1}^M w_k a_k\right\|^2. $$

Proof. Define $g_y(a)=\ell(y,a)-\frac{\lambda}{2}\|a\|^2$. By strong convexity, $g_y$ is convex. Jensen's inequality for the finite weighted average gives

g_y(\bar a)\leq\sum_{j=1}^M w_j g_y(a_j).

Substitute the definition of $g_y$ and rearrange:

\ell(y,\bar a) \leq \sum_{j=1}^M w_j\ell(y,a_j) -\frac{\lambda}{2} \left(\sum_{j=1}^M w_j\|a_j\|^2-\|\bar a\|^2\right).

To identify the term in parentheses, expand each squared distance and use $\sum_jw_ja_j=\bar a$:

\begin{aligned} \sum_{j=1}^M w_j\|a_j-\bar a\|^2 &=\sum_{j=1}^M w_j \left(\|a_j\|^2-2a_j^\mathsf{T}\bar a+\|\bar a\|^2\right)\\ &=\sum_{j=1}^M w_j\|a_j\|^2 -2\bar a^\mathsf{T}\bar a+\|\bar a\|^2\\ &=\sum_{j=1}^M w_j\|a_j\|^2-\|\bar a\|^2. \end{aligned}

Substituting this identity proves the inequality. $\square$

This proof also shows the role of strong convexity: subtracting the quadratic term produces an ordinary convex function, to which Jensen's inequality applies. The dispersion correction is exactly the difference between the weighted second moment and the squared norm of the weighted mean.

From Pointwise Loss to Risk

Now let $(X,Y)$ have a fixed joint distribution, and let $f_1,\ldots,f_M$ be measurable predictors taking values in $C$. For fixed weights $w_j$, define the aggregate predictor $\bar f(x)=\sum_jw_jf_j(x)$. The risk of a predictor $f$ is $R(f)=E[\ell(Y,f(X))]$.

Theorem (Risk reduction by averaging): Suppose $\ell(y,\cdot)$ is $\lambda$-strongly convex on $C$ for every $y$, all predictors take values in $C$, and the component losses and dispersion below are integrable. Then $$ R(\bar f) \leq \sum_{j=1}^M w_jR(f_j) -\frac{\lambda}{2}E\left[\sum_{j=1}^M w_j \|f_j(X)-\bar f(X)\|^2\right]. $$

Proof. For each realized pair $(X,Y)$, apply the pointwise inequality with $a_j=f_j(X)$. This gives

\ell(Y,\bar f(X)) \leq \sum_{j=1}^M w_j\ell(Y,f_j(X)) -\frac{\lambda}{2}\sum_{j=1}^M w_j\|f_j(X)-\bar f(X)\|^2.

All terms are measurable by the measurability assumptions and the measurability of the loss. The stated integrability conditions allow expectations of both sides to be taken and the finite weighted sum to be moved outside the expectation. Thus the left side becomes $R(\bar f)$, and the first term on the right becomes $\sum_jw_jR(f_j)$. The remaining term is the expected weighted dispersion shown in the theorem. This proves the result. $\square$

The conclusion compares the aggregate risk with the weighted average risk, not with the smallest component risk. In particular, it guarantees a strict reduction from that average whenever the expected dispersion is positive. It does not claim that averaging beats every candidate.

Worked Example: Bernoulli Log Loss on Probability Predictions

For a binary outcome $y\in\{0,1\}$ and a probability prediction $p\in(0,1)$, the Bernoulli log loss is $\ell(y,p)=-y\log p-(1-y)\log(1-p)$. Its second derivative in $p$ is

\frac{\partial^2\ell(y,p)}{\partial p^2} =\frac{y}{p^2}+\frac{1-y}{(1-p)^2}\geq 1.

For either possible binary outcome, the inequality holds throughout $(0,1)$. Hence this loss is $1$-strongly convex in $p$ on that domain. Consider one observation with $y=1$, and predictions $p_1=0.2$ and $p_2=0.8$ at equal weights. The aggregate prediction is $0.5$. The weighted component loss is

\frac12[-\log(0.2)]+\frac12[-\log(0.8)] =\frac12\log(6.25)\approx 0.91629.

The aggregate loss is $-\log(0.5)=\log 2\approx0.69315$, so the reduction is approximately $0.22314$. The dispersion is

\frac12(0.2-0.5)^2+\frac12(0.8-0.5)^2 =\frac12(0.09)+\frac12(0.09)=0.09.

The theorem guarantees a reduction of at least $(1/2)(0.09)=0.045$. The actual reduction, approximately $0.22314$, exceeds that lower bound. The bound is guaranteed, not generally an equality.

Worked Example: Expected Dispersion Across Inputs

Let $X$ take values $x_0,x_1$, each with probability $1/2$, and let $Y=0$ at $x_0$ and $Y=1$ at $x_1$. Use squared loss and two predictors with equal weights. At $x_0$, let their predictions be $0$ and $2$; at $x_1$, let them be $1$ and $3$.

At $x_0$, the aggregate predicts $1$. The two component losses are $0$ and $4$, so their weighted average loss is $2$; the aggregate loss is $(0-1)^2=1$. At $x_1$, the aggregate predicts $2$. The component losses are again $0$ and $4$, while the aggregate loss is $(1-2)^2=1$. Consequently,

R(\bar f)=\tfrac12(1)+\tfrac12(1)=1,\qquad \tfrac12R(f_1)+\tfrac12R(f_2)=2.

At each input the predictions are one unit on either side of their mean, so the weighted dispersion is $1$. Its expectation is also $1$. Since squared loss has $\lambda=2$, the guaranteed reduction is $(2/2)(1)=1$, matching the actual reduction $2-1=1$.

What the Bound Does—and Does Not—Say

The bound is useful because it separates two sources of an averaging gain: the curvature constant $\lambda$ and the amount of disagreement among the predictions. If all positively weighted candidates make the same prediction at almost every input, dispersion is zero and this result guarantees no additional reduction. If predictions disagree, strong curvature makes their weighted average quantitatively preferable to the weighted average of their losses.

The prediction domain matters. The aggregate must remain in a region where the loss is strongly convex. For example, Bernoulli log loss has the stated $1$-strong convexity on the open probability interval $(0,1)$; predictions at the endpoints are not finite log-loss predictions for both outcomes. Strong convexity can also depend on restricting predictions to a smaller domain or changing the norm. The constant cannot be transferred without checking its assumptions.

Common mistake: concluding that an average must beat the best candidate. The theorem says $R(\bar f)\leq\sum_jw_jR(f_j)$, with a dispersion correction. A weighted average of risks can be larger than the minimum risk. For example, if one candidate has risk $0$ and another has risk $4$, equal weights give average risk $2$; a risk of $1$ for the aggregate is a reduction from $2$, but still exceeds the best risk $0$. The correction is to identify the comparator correctly: this result improves on the weighted average of component risks, and an additional argument is needed to compare with the best single component.

Another error is to use ordinary convexity as if it supplied a strictly positive correction. Convexity gives Jensen's inequality without the dispersion term. The quantitative gain requires a valid $\lambda>0$ over the relevant prediction domain. Absolute error, for instance, is convex but is not strongly convex in its prediction on an interval, so this particular dispersion guarantee does not follow for it.

Key takeaway: Under $\lambda$-strong convexity, averaging fixed-weight predictions reduces risk relative to their weighted average risk by at least $\lambda/2$ times their expected weighted prediction dispersion. The guarantee is only as strong as the curvature condition and applies to this comparator, not automatically to the best candidate.

Check Your Understanding

Use the definitions and results above to answer these questions.

For three predictions and fixed weights, write the weighted dispersion and explain when it is zero.
Why does strong convexity, rather than ordinary convexity, yield a quantitative dispersion correction?
If squared loss is used with scalar predictions, what value of $\lambda$ appears in the risk-reduction bound?
Does the risk-reduction theorem guarantee that the aggregate beats the lowest-risk candidate? Explain which risks it compares.
What must be checked before applying the Bernoulli log-loss result to a set of probability predictions?