Ensemble Risk Targets and Oracle Benchmarks

Population Risk as an Ensemble Target

An ensemble combines a collection of prediction functions, but the phrase “best ensemble” is incomplete until the target being minimized is specified. The target here is population risk: expected loss on a new observation from a specified population. We compare this risk with three oracle benchmarks: the best single member, the best unrestricted fixed linear combination, and the best fixed convex combination.

Treat the fitted member functions as fixed. Thus, the expectation below is over a fresh test observation, not over the data used to fit the members. This distinction makes the benchmark a population target rather than a training-error calculation. The oracle weights are also fixed across observations: they are not chosen separately for each value of the predictors.

Definition: Let $(X,Y)$ have a specified population distribution, and let $f_1,\ldots,f_M$ be measurable, real-valued prediction functions. For a nonnegative measurable loss $\mathcal L(y,a)$, the population risk of a prediction function $f$ is $$ R(f)=E\!\left[\mathcal L\bigl(Y,f(X)\bigr)\right]. $$ For a weight vector $w=(w_1,\ldots,w_M)\in\mathbb R^M$, define $f_w(x)=\sum_{j=1}^M w_j f_j(x)$ and $R(w)=R(f_w)$. The nonnegative-loss assumption ensures this expectation is defined, possibly with value $+\infty$.

The members and their risks are denoted by $f_j$ and $R_j=R(f_j)$. The three benchmarks differ by which weight vectors they allow. In the unrestricted linear benchmark, weights may be negative and need not sum to one. In the convex benchmark, weights are nonnegative and sum to one. A one-hot vector, with one entry equal to one and the rest zero, selects a single member.

Definition: The best-member, best-linear-combination, and best-convex-combination oracle risks are, respectively, $$ R_{\mathrm{single}}^*=\min_{1\le j\le M}R_j,\qquad R_{\mathrm{lin}}^*=\inf_{w\in\mathbb R^M}R(w),\qquad R_{\mathrm{conv}}^*=\inf_{w\in\Delta_M}R(w), $$ where $\Delta_M=\{w\in\mathbb R^M:w_j\ge 0\text{ for all }j,\ \sum_{j=1}^M w_j=1\}$. The infima define the benchmarks even when an optimum does not exist.

The word oracle matters. These quantities use the population distribution to judge candidate weights; they do not describe how weights are estimated from a finite data set. The linear benchmark also depends on the stated convention: here, “linear combination” allows any coefficient vector in $\mathbb R^M$, without a sum-to-one constraint.

The Basic Ordering of the Benchmarks

The three feasible sets are nested in the relevant way. Every convex weight vector is a possible unrestricted linear weight vector, and every single member can be represented by a convex weight vector. Consequently, enlarging the set of candidates cannot increase the infimum risk.

Theorem: If the member risks are finite, then $$ R_{\mathrm{lin}}^*\le R_{\mathrm{conv}}^*\le R_{\mathrm{single}}^*. $$ The inequalities need not be strict, and the linear or convex infimum need not be attained without further conditions.

Proof. Since $\Delta_M\subseteq\mathbb R^M$, the infimum of $R(w)$ over $\mathbb R^M$ is no larger than its infimum over $\Delta_M$. For each $j$, the $j$th one-hot vector $e_j$ belongs to $\Delta_M$, and $R(e_j)=R_j$. Therefore the infimum over $\Delta_M$ is no larger than $\min_j R_j$. Combining these two inequalities proves the result. $\square$

This ordering compares targets, not the performance of three algorithms. For example, an algorithm that estimates unrestricted weights from a small sample may have greater test risk than the best member, even though the unrestricted population oracle cannot have greater risk. Estimation error and the population optimum are different quantities.

Worked Example: A Convex Combination Improves on Either Member

Let $X$ be constant, let $Y=1$ with probability one, and use squared-error loss $\mathcal L(y,a)=(y-a)^2$. Take two constant members $f_1=0$ and $f_2=2$. Calculate their risks and the best convex-combination risk.

For the first member, $R_1=(1-0)^2=1$. For the second, $R_2=(1-2)^2=1$. Thus $R_{\mathrm{single}}^*=1$.

Write a convex combination as $f_\alpha=(1-\alpha)f_1+\alpha f_2=2\alpha$, where $0\le\alpha\le1$. Its risk is $$ R(f_\alpha)=(1-2\alpha)^2. $$ This is nonnegative and equals zero at $\alpha=\tfrac12$. Hence $R_{\mathrm{conv}}^*=0$, strictly below either member’s risk. The unrestricted linear benchmark is also zero, since it includes this convex choice.

What Convexity of the Loss Adds

The benchmark ordering follows from feasible-set inclusion and does not require a convex loss. If the loss is convex in the prediction, one obtains a further useful bound for each chosen convex weight vector: its risk is no greater than the corresponding weighted average of member risks.

Theorem: Suppose $\mathcal L(y,a)$ is convex in $a$ for every $y$, and each member risk $R_j$ is finite. For every $w\in\Delta_M$, $$ R(w)\le \sum_{j=1}^M w_j R_j. $$

Proof. Fix an observation $(x,y)$. Convexity of the loss and the identity $f_w(x)=\sum_j w_j f_j(x)$ give $$ \mathcal L\!\left(y,\sum_{j=1}^M w_j f_j(x)\right) \le \sum_{j=1}^M w_j\mathcal L\bigl(y,f_j(x)\bigr). $$ Both sides are measurable and nonnegative. Taking expectations preserves the inequality, and the finite sum on the right has expectation $\sum_jw_jR_j$, which is finite by assumption. The left expectation is $R(w)$, proving the claim. $\square$

This result does not say that every convex combination improves on the best member. A poor choice of weights can still perform poorly; the inequality compares its risk with its own weighted average of member risks. The optimum convex benchmark is no worse than the best member because it can select that member, whether or not the loss is convex.

Worked Example: The Convex Benchmark Need Not Improve

Again let $X$ be constant and use squared-error loss, but now let $Y$ equal $-1$ or $1$, each with probability $1/2$. Suppose both members predict zero: $f_1=f_2=0$.

For either member, the squared error is $1$ for both possible outcomes, so $R_1=R_2=1$ and $R_{\mathrm{single}}^*=1$. Every convex combination still predicts zero, since $(1-\alpha)f_1+\alpha f_2=0$ for all $0\le\alpha\le1$. Its risk is therefore always $1$, and $R_{\mathrm{conv}}^*=1$ as well.

The inclusion of the members among convex combinations guarantees that the oracle can match the best member; it does not guarantee a strict gain. Here the members supply no different predictions for the convex weights to combine.

When Does a Best Convex Combination Exist?

An infimum is a valid benchmark even if no weight vector achieves it. For convex combinations, attainment follows under mild regularity conditions. The key points are that the simplex is compact and that nonnegative lower-semicontinuous loss makes the risk lower semicontinuous as a function of the weights.

Theorem: Suppose $M$ is finite, each $f_j$ is measurable and finite-valued, $\mathcal L(y,a)$ is nonnegative and lower semicontinuous in $a$ for every $y$, and at least one member has finite risk. Then there is a $w^*\in\Delta_M$ such that $R(w^*)=R_{\mathrm{conv}}^*$.

Proof. For each fixed $(x,y)$, the map $w\mapsto\sum_jw_jf_j(x)$ is continuous. Lower semicontinuity of the loss therefore makes $w\mapsto\mathcal L(y,\sum_jw_jf_j(x))$ lower semicontinuous on $\Delta_M$. If $w_k\to w$, Fatou’s lemma applies to these nonnegative losses and gives $$ R(w)\le \liminf_{k\to\infty}R(w_k). $$ Thus $R$ is lower semicontinuous. The simplex $\Delta_M$ is compact, so a sequence of weights whose risks approach the infimum has a convergent subsequence with limit $w^*\in\Delta_M$. Lower semicontinuity implies that $R(w^*)$ is no greater than the limiting infimum, and hence equals $R_{\mathrm{conv}}^*$. At least one finite-risk member ensures this minimum is finite. $\square$

Worked Example: A Linear Oracle Can Beat Every Convex Combination

Let $X$ be constant, $Y=2$ with probability one, and use squared-error loss. Take $f_1=0$ and $f_2=1$. The member risks are $R_1=4$ and $R_2=1$, so $R_{\mathrm{single}}^*=1$.

A convex combination has prediction $(1-\alpha)0+\alpha1=\alpha$, with $0\le\alpha\le1$. Its risk is $(2-\alpha)^2$, which decreases over this interval and is minimized at $\alpha=1$. Therefore $R_{\mathrm{conv}}^*=1$.

For an unrestricted linear combination, choose $w_1=0$ and $w_2=2$. The prediction is $0f_1+2f_2=2$, so its risk is zero. Since squared-error risk is nonnegative, $R_{\mathrm{lin}}^*=0$. This gain uses a coefficient greater than one; convex weights cannot produce a prediction above the largest member prediction in this constant-prediction example.

Fixed Weights Are Not Observation-Specific Weights

A fixed oracle chooses one weight vector for the whole population. It does not choose a different member or weight vector after seeing $X=x$. An observation-specific rule has a larger set of possibilities and is a different benchmark. The distinction can be important when member performance varies across regions of the predictor space.

Worked Example: A Fixed Convex Oracle Versus a Pointwise Choice

Let $X$ take values $a$ and $b$, each with probability $1/2$, and let $Y=0$ at $a$ and $Y=2$ at $b$. Under squared-error loss, consider two members: $f_1(a)=f_1(b)=0$ and $f_2(a)=f_2(b)=2$.

Member $f_1$ has errors $0$ and $-2$, so its risk is $\tfrac12(0^2)+\tfrac12((-2)^2)=2$. Member $f_2$ has errors $2$ and $0$, so its risk is also $2$. A fixed convex combination with weight $\alpha$ on $f_2$ predicts $2\alpha$ at both values of $X$. Its risk is $$ \frac12(2\alpha-0)^2+\frac12(2\alpha-2)^2 =2\alpha^2+2(1-\alpha)^2. $$ This is minimized at $\alpha=\tfrac12$, giving risk $1$.

By contrast, an observation-specific rule could use $f_1$ at $a$ and $f_2$ at $b$, predicting $0$ and $2$ and achieving risk zero. That rule is not a fixed convex combination of the two functions. Its lower risk does not contradict the oracle ordering, which compares only fixed weight vectors.

Common Mistakes and Practical Meaning

Confusing oracle risk with training performance. The oracle infimum uses expected loss under the population distribution. Empirical loss on the training observations is a different quantity.
Assuming an unrestricted linear combination must be convex. Linear weights can be negative or have a sum other than one. Convex weights must be nonnegative and sum to one.
Claiming the best convex combination is strictly better than the best member. The feasible-set argument proves “no worse,” not “strictly better.” Identical members, as in the example, give equality.
Allowing weights to vary with $X$ without changing the target. A rule that selects weights separately for each observation is not the fixed-weight oracle defined here.
Assuming every infimum is attained. The benchmark is well-defined as an infimum, but attainment needs conditions. The compactness argument above establishes it for convex weights under the stated assumptions.

Key takeaway: For fixed members and a common population risk, the unrestricted linear oracle is no worse than the convex oracle, which is no worse than the best member. These are population benchmarks; they do not account for the cost of estimating weights.

Check Your Understanding

Answer these questions using the definitions and results above.

Why does the best convex-combination risk never exceed the best single-member risk, even if the loss is not convex?
For a convex loss, what inequality relates the risk of a chosen convex combination to the weighted risks of its members?
In the example with $f_1=0$, $f_2=1$, and constant response $Y=2$, why can the unrestricted linear oracle achieve zero risk while the convex oracle cannot?
What assumptions in the attainment result ensure that a minimum convex risk is achieved?
Why is a rule that chooses different members at $X=a$ and $X=b$ not the fixed convex-combination oracle?