Gated Ensembles With Input-Dependent Weights

Why Let Ensemble Weights Depend on the Input?

A fixed convex ensemble uses the same weights for every case. That is appropriate when its members have similar relative performance throughout the predictor space. But one member may be more accurate in one region and another member elsewhere. A gated ensemble allows the mixture weights to change with the input, while still requiring them to be nonnegative and sum to one.

The comparison in this tutorial is between the best population risk achievable with fixed convex weights and the best population risk achievable with input-dependent convex weights. This is a comparison of function classes: it does not imply that every particular gate, or every method used to estimate a gate, will perform well.

Definition and Population Risk

Suppose \(M\) prediction functions \(f_1,\ldots,f_M\) are available. Treat these functions as fixed, and evaluate them on a fresh pair \((X,Y)\), where \(X\) is the input and \(Y\) is the response. The analysis below uses squared-error loss and assumes \(E[Y^2]<\infty\) and \(E[f_j(X)^2]<\infty\) for each \(j\).

Definition: The convex simplex of \(M\) weights is \(\Delta_M=\{a\in\mathbb{R}^M:a_j\geq 0\text{ for every }j,\ \sum_{j=1}^M a_j=1\}\). A gated ensemble is a prediction rule \(g_\alpha(x)=\sum_{j=1}^M\alpha_j(x)f_j(x)\), where \(\alpha(x)=(\alpha_1(x),\ldots,\alpha_M(x))\) is a measurable function taking values in \(\Delta_M\). Its population risk is \(R(\alpha)=E[(Y-g_\alpha(X))^2]\). A global convex ensemble uses a single \(w\in\Delta_M\) for all inputs, with prediction \(g_w(x)=\sum_{j=1}^M w_j f_j(x)\) and risk \(R(w)=E[(Y-g_w(X))^2]\).

The weights may depend on \(X\), but not on the unobserved response \(Y\) at prediction time. Measurability ensures that the resulting prediction is a legitimate random variable. Since a constant function \(\alpha(x)=w\) is an admissible gate, every global convex ensemble is also a gated ensemble.

The population risks here average over the distribution of a future \((X,Y)\). They are not training errors, and the predictors are regarded as fixed for the comparison. The gate is allowed to use any measurable rule in the stated class; an applied procedure may search over a much smaller class.

Worked Example: A Gate That Uses Different Members in Two Regions

Let \(X\) take the values \(-1\) and \(1\), each with probability \(1/2\). Let \(Y=1+X+\varepsilon\), where \(E[\varepsilon\mid X]=0\) and \(E[\varepsilon^2\mid X]=1\). Consider two constant predictors, \(f_1(x)=0\) and \(f_2(x)=2\).

Define \(\alpha_2(x)=(x+1)/2\) and \(\alpha_1(x)=1-\alpha_2(x)\). At \(X=-1\), the weights are \((1,0)\) and the prediction is \(0\); at \(X=1\), the weights are \((0,1)\) and the prediction is \(2\). Both weight vectors belong to \(\Delta_2\), and the weights vary with the input.

The conditional mean is \(E[Y\mid X]=1+X\), which equals the gate's prediction at both possible inputs. Therefore its squared-error risk is just the conditional noise variance: \(R(\alpha)=E[\varepsilon^2]=1\). For comparison, the global weights \(w=(1/2,1/2)\) predict \(1\) at both inputs. Its error is \(Y-1=X+\varepsilon\), so \(R(w)=E[(X+\varepsilon)^2]=E[X^2]+2E[X\varepsilon]+E[\varepsilon^2]=1+0+1=2\). Here the gate uses different members where their conditional means are better suited.

A Conditional Decomposition of Risk

The following identity separates response variability that no predictor based only on \(X\) can remove from the error in approximating the conditional mean. It applies to any admissible gate and will clarify what input-dependent weights can change.

Theorem: Let \(m(X)=E[Y\mid X]\) and \(v(X)=\operatorname{Var}(Y\mid X)\). For every measurable \(\alpha(X)\in\Delta_M\), \[ R(\alpha)=E[v(X)]+E\left[\left\{m(X)-\sum_{j=1}^M\alpha_j(X)f_j(X)\right\}^2\right]. \]

Proof. Write \(g=g_\alpha(X)\). The gate and all member predictions are functions of \(X\). Conditional on \(X\), expand the squared error around \(m(X)\):

(Y-g)^2=\{Y-m(X)\}^2+2\{Y-m(X)\}\{m(X)-g\}+\{m(X)-g\}^2.

Taking conditional expectation given \(X\), the first term becomes \(v(X)\). The conditional expectation of the cross term is zero, because \(m(X)-g\) is fixed given \(X\) and \(E[Y-m(X)\mid X]=0\). The last term is already determined by \(X\). Thus \(E[(Y-g)^2\mid X]=v(X)+\{m(X)-g\}^2\). Taking expectations proves the identity. All terms are integrable under the stated second-moment assumptions: convex combinations of the \(f_j(X)\) are square-integrable, and \(m(X)\) is square-integrable by conditional Jensen's inequality. \(\square\)

The gate can affect the second term by changing its approximation to \(m(X)\); it cannot remove the first term. This identity does not say that the best gate must reproduce the conditional mean exactly. It can do so only if that mean can be represented, or closely approximated, by convex combinations of the available member predictions at the relevant inputs.

Worked Example: A Gated Risk Cannot Remove Irreducible Noise

Suppose \(X\) is equally likely to be \(-1\) or \(1\), and \(Y=X+\varepsilon\), where \(E[\varepsilon\mid X]=0\) and \(\operatorname{Var}(\varepsilon\mid X)=4\). Use the members \(f_1(x)=x\) and \(f_2(x)=-x\). The gate that always assigns weight one to \(f_1\) predicts \(X\).

Its error is \(Y-f_1(X)=\varepsilon\), so its risk is \(4\). The decomposition gives the same result: \(m(X)=X\), \(v(X)=4\), and the squared conditional-mean approximation error is zero. No choice of weights can lower risk below \(4\), because the decomposition gives \(R(\alpha)=4+E[(m(X)-g_\alpha(X))^2]\geq 4\).

A different gate could even increase risk. For example, always choosing \(f_2\) gives prediction \(-X\), conditional-mean error \(2X\), and hence risk \(4+E[(2X)^2]=8\). Input-dependent weights create options; they do not by themselves guarantee that those options are used well.

Best Gated Risk Versus Best Global Risk

Let the two oracle risks be \(R_{\mathrm{gate}}^*=\inf_{\alpha}R(\alpha)\), where the infimum ranges over all measurable simplex-valued gates, and \(R_{\mathrm{global}}^*=\min_{w\in\Delta_M}R(w)\). The global minimum exists: the simplex is compact, and the risk is continuous in \(w\). In fact, writing \(f(X)=(f_1(X),\ldots,f_M(X))^\mathsf{T}\), \(R(w)=E[Y^2]-2w^\mathsf{T}E[Yf(X)]+w^\mathsf{T}E[f(X)f(X)^\mathsf{T}]w\), a finite quadratic function under the moment assumptions.

Theorem: Under the stated moment assumptions, \(R_{\mathrm{gate}}^*\leq R_{\mathrm{global}}^*\).

Proof. Let \(w^*\in\Delta_M\) be a global minimizer. Define the constant gate \(\alpha(x)=w^*\) for every \(x\). It is measurable, and its values are in \(\Delta_M\), so it is among the gates over which \(R_{\mathrm{gate}}^*\) is defined. Its prediction and risk equal those of the global ensemble: \(g_\alpha(X)=g_{w^*}(X)\) and \(R(\alpha)=R_{\mathrm{global}}^*\). An infimum over a class containing this gate cannot be greater than its risk. Therefore \(R_{\mathrm{gate}}^*\leq R_{\mathrm{global}}^*\). \(\square\)

This is a class-inclusion result, not a claim that the inequality is always strict. It also concerns the best gate in the full class of measurable functions. Restrictions such as a particular tree structure or a limited number of parameters can make the class searched in practice smaller than the class in the theorem.

Worked Example: Strict Improvement Over Every Global Weight

Let \(X\) be equally likely to be \(-1\) or \(1\), let \(Y=X\), and use two constant members \(f_1(x)=1\) and \(f_2(x)=-1\). A gate can choose \(f_1\) when \(X=1\) and \(f_2\) when \(X=-1\). Its prediction is \(X\), so \(R_{\mathrm{gate}}^*=0\); squared error is nonnegative, so zero is indeed the minimum.

For any global weights \(w_1,w_2\geq0\) with \(w_1+w_2=1\), the prediction is the constant \(c=w_1-w_2=2w_1-1\). The two possible squared errors are \((1-c)^2\) and \((-1-c)^2\), each with probability \(1/2\). Thus \(R(w)=\{(1-c)^2+(-1-c)^2\}/2=1+c^2\geq1\). Taking \(w_1=w_2=1/2\) gives \(c=0\) and risk \(1\), so \(R_{\mathrm{global}}^*=1\). The best gated risk is strictly smaller.

Worked Example: When Gating Offers No Improvement

Let \(X\) have any distribution with \(E[X^2]<\infty\), let \(Y=1.5X\), and take \(f_1(x)=x\) and \(f_2(x)=2x\). The global weights \(w_1=w_2=1/2\) give prediction \((1/2)x+(1/2)2x=1.5x\), so their risk is zero.

Because squared error is nonnegative, the best global risk is zero. The best gated risk is at most zero by the comparison theorem and at least zero by nonnegativity. Hence both are zero. In this example a constant convex combination already represents the response exactly; allowing the weights to vary with \(X\) cannot improve on it.

Common Pitfall and Practical Interpretation

Wrong reasoning: “Gated ensembles have a larger function class, so any fitted gate must beat the best global weights.” Correction: The theorem compares the infimum over all admissible gates with the global optimum. A particular gate may be poorly chosen and have higher risk; the example with \(f_2(x)=-x\) demonstrates this directly.
Wrong reasoning: “The weights may use whichever member has the smallest realized error for each case.” Correction: A valid prediction-time gate is a function of the available input \(X\), not of the response \(Y\) that has not yet been observed.
Wrong reasoning: “Input-dependent weights must strictly improve risk.” Correction: They can tie the global optimum, as when a fixed convex combination already gives the best representable conditional mean.

The useful question is therefore not whether gates are automatically better, but whether the conditional-mean structure varies across inputs in a way that the members and the allowed gate can represent. Gating is most compelling when different members provide useful predictions in different regions. The risk decomposition also reminds us that even a perfect gate cannot eliminate conditional response variation.

Check Your Understanding

Use the definitions and results above to answer the following questions.

What two conditions must the weight vector \(\alpha(x)\) satisfy at every input for it to define a convex gate?
In the conditional risk decomposition, which term can the choice of gate change, and which term does it leave unchanged?
Why does inclusion of constant gates prove that the best gated risk is no greater than the best global convex risk?
For two constant members \(f_1(x)=1\), \(f_2(x)=-1\), and a response \(Y=X\) with \(X\in\{-1,1\}\) equally likely, explain why a gate can have zero risk while no global convex weights do.
Does the comparison theorem guarantee that a particular fitted gate has lower population risk than the best global weights? Explain.