Worked Comparison of Averaging, Stacking, and Selection

Why Compare Aggregation Rules by Risk?

Averaging several predictions can reduce error, but the reduction depends on how their errors move together. If the errors are independent, averaging can be effective even when the individual predictors have different accuracies. If errors are strongly correlated, the gain may be small. Bias changes the comparison again: weights that minimize error variance need not minimize mean squared error.

We compare four rules for combining a fixed set of predictors: equal averaging, weights chosen from an error covariance structure, weights fitted using validation data, and selection of one predictor. Throughout, risk means expected squared prediction error on a new observation. This makes each calculation comparable, while keeping clear the distinction between choosing weights with known population quantities and estimating them from data.

Risk Notation for a Weighted Ensemble

Suppose there are \(K\) candidate predictors, and let \(e_j\) be the prediction error of candidate \(j\) on a new observation. Write \(e=(e_1,\ldots,e_K)^\top\), \(b=E[e]\), and let \(\Sigma=\operatorname{Cov}(e)\). A weight vector \(w=(w_1,\ldots,w_K)^\top\) with \(\mathbf{1}^\top w=1\) forms a weighted prediction whose error is \(w^\top e\). The sum-to-one condition ensures that a common shift in all candidate predictions is preserved by the combination.

Definition: The squared-error risk of weights \(w\) is \(R(w)=E[(w^\top e)^2]\). The error second-moment matrix is \(M=E[ee^\top]=\Sigma+bb^\top\), so \(R(w)=w^\top M w\). Equal averaging uses \(w_j=1/K\). Single-member selection uses a coordinate vector, with one weight equal to 1 and the others 0.

The risk identity follows by expanding the square: \(E[(w^\top e)^2]=w^\top E[ee^\top]w\). Also \(E[(e-b)(e-b)^\top]=\Sigma\), so expanding \(ee^\top=(e-b+b)(e-b+b)^\top\) and taking expectations gives \(M=\Sigma+bb^\top\). Thus covariance alone describes risk only when the errors are unbiased, meaning \(b=0\).

Worked Comparisons

Worked Example: Independent Errors with Unequal Variances

Suppose three unbiased prediction errors are independent, with distributions \(e_1\sim N(0,1)\), \(e_2\sim N(0,4)\), and \(e_3\sim N(0,9)\). Calculate the risk of equal averaging, the best single predictor, and the covariance-optimal affine average.

For equal weights \(1/3\), independence makes the variance of the average the sum of the weighted variances: \(R_{\mathrm{avg}}=(1+4+9)/9=14/9\), approximately \(1.556\). The three single-member risks are \(1,4,9\), so selecting the best member has risk \(1\). Equal averaging is worse than that best member in this example.

For independent unbiased errors, the variance-minimizing weights are proportional to inverse variances. Normalizing the inverse variances gives \(w_1=(1)/(1+1/4+1/9)=36/49\), \(w_2=(1/4)/(1+1/4+1/9)=9/49\), and \(w_3=(1/9)/(1+1/4+1/9)=4/49\). They sum to \((36+9+4)/49=1\). The risk is \((36/49)^2+4(9/49)^2+9(4/49)^2 =1296/2401+324/2401+144/2401 =1764/2401=36/49\), approximately \(0.735\).

Here the covariance-optimal average beats both equal averaging and the best single predictor. The calculation also shows why a small weight on a high-variance predictor can matter: its contribution to risk depends on the square of its weight as well as its error variance.

Worked Example: Correlated Errors Change the Gain

Let \(e_1,e_2\) be unbiased errors with covariance matrix \(\Sigma=\begin{pmatrix}1&0.5\\0.5&4\end{pmatrix}\). This is a valid positive-definite covariance matrix because its leading diagonal entry is positive and its determinant is \(4-0.25=3.75>0\). Compare equal averaging, the best single member, and covariance-optimal weights.

Equal averaging has risk \((1+4+2(0.5))/4=6/4=1.5\). The single-member risks are 1 and 4, so the better single predictor has risk 1. The covariance is positive: when one error is high, the other tends to be high as well, limiting the benefit of averaging.

For weights \((w,1-w)\), the risk is \(w^2+4(1-w)^2+2w(1-w)(0.5)\). Expanding gives \(4w^2-3w+4\) only if the cross-term is mistakenly treated as \(w(1-w)\) without its coefficient; here the cross-term is \(w(1-w)\), so the correct expression is \(w^2+4(1-2w+w^2)+w(1-w)=4w^2-7w+4\). Its derivative is \(8w-7\), giving \(w=7/8\) and \(1-w=1/8\).

Substitution yields risk \(49/64+4/64+7/64=60/64=15/16=0.9375\). This is slightly below the best single-member risk of 1, but the improvement is much smaller than in the independent-errors example. The positive covariance has not made averaging useless; it has changed the size of its gain.

Worked Example: Bias Makes Covariance-Optimal Weights Suboptimal

Suppose independent errors have \(e_1\sim N(0.4,0.04)\) and \(e_2\sim N(-0.2,0.25)\), where the second parameter is the variance. Compare covariance-optimal weights with weights that minimize squared-error risk subject to nonnegative weights summing to one.

The covariance-optimal weights for unbiased errors would be proportional to inverse variances. Applying that variance-only rule gives \(w_1=(1/0.04)/(1/0.04+1/0.25)=25/29\) and \(w_2=4/29\). The combined variance is \(1/(25+4)=1/29\). Its bias is \(0.4(25/29)-0.2(4/29)=9.2/29=46/145\). Consequently its risk is \(1/29+(46/145)^2=725/21025+2116/21025=2841/21025\), approximately \(0.1351\).

For a general first weight \(w\in[0,1]\), the second weight is \(1-w\). The combination has variance \(0.04w^2+0.25(1-w)^2\) and bias \(0.4w-0.2(1-w)=0.6w-0.2\). Therefore, \(R(w)=0.04w^2+0.25(1-w)^2+(0.6w-0.2)^2\). Differentiating gives \(R'(w)=1.3w-0.74\), so the minimum is at \(w=37/65\), which lies in \([0,1]\).

At these weights the variance is \((1369/25+196)/4225=6269/105625\), and the squared bias is \((46/325)^2=2116/105625\). Thus the risk is \(8385/105625\), approximately \(0.0794\). For comparison, selecting predictor 1 gives risk \(0.04+0.4^2=0.20\); predictor 2 gives \(0.25+(-0.2)^2=0.29\). In this example, accounting for bias yields a combination that beats both the variance-only combination and either single member.

Worked Example: Fitting a Stack from Validation Residuals

Consider two predictors and two validation observations. Their residual vectors, defined as prediction minus outcome, are \(e_1=(1,0)\) and \(e_2=(0,2)\). Fit a nonnegative two-predictor stack by minimizing mean squared validation error.

If \(w\) is the weight on predictor 1, the weight on predictor 2 is \(1-w\), and the stacked residual vector is \((w,2(1-w))\). Its validation mean squared error is \([w^2+4(1-w)^2]/2\). Differentiating this expression gives \(5w-4\), so the minimizing weight is \(w=4/5\). The fitted validation error is \([ (4/5)^2+4(1/5)^2]/2=(16/25+4/25)/2=0.4\).

The individual validation errors are \(1/2=0.5\) for predictor 1 and \(4/2=2\) for predictor 2; equal averaging has validation error \((1/4+1)/2=0.625\). Thus fitted stacking improves validation error over both equal averaging and selecting the better single member. If, on a separate test observation, the two residuals are 0 and 1, the fitted stack has squared error \((0.2)^2=0.04\), equal averaging has squared error \(0.25\), and selecting predictor 1 has squared error 0. These test values are a separate illustration: validation performance identifies a fitted rule, but does not guarantee that it will beat every alternative on each new observation.

Two Risk Results and Their Proofs

Theorem: If \(e_1,\ldots,e_K\) are unbiased with finite second moments, then equal averaging has risk \(R_{\mathrm{avg}}=K^{-2}\left(\sum_{j=1}^K\operatorname{Var}(e_j)+2\sum_{1\leq j<k\leq K}\operatorname{Cov}(e_j,e_k)\right)\).

Proof. The error of the equal average is \(K^{-1}\sum_j e_j\). Since the errors have mean zero, its risk equals its variance. Expanding the variance of a sum gives \(\operatorname{Var}(K^{-1}\sum_j e_j)=K^{-2}\{\sum_j\operatorname{Var}(e_j)+2\sum_{j<k}\operatorname{Cov}(e_j,e_k)\}\). This is the stated expression. \(\square\)

The theorem makes dependence explicit. If all errors are independent with a common variance \(\sigma^2\), the risk is \(\sigma^2/K\). If all have variance \(\sigma^2\) and every distinct pair has covariance \(c\), the risk is \(\sigma^2/K+(K-1)c/K\). As \(K\) grows, the independent component shrinks, while the shared covariance component remains.

Theorem: Let \(A\) be a positive-definite matrix and minimize \(w^\top A w\) over real vectors \(w\) satisfying \(\mathbf{1}^\top w=1\). The unique minimizer is \(w^*=A^{-1}\mathbf{1}/(\mathbf{1}^\top A^{-1}\mathbf{1})\). For unbiased errors with positive-definite covariance \(\Sigma\), take \(A=\Sigma\). For possibly biased errors with positive-definite second-moment matrix \(M\), take \(A=M\).

Proof. Positive definiteness makes the objective strictly convex, so any feasible stationary point is the unique global minimum. Introduce a multiplier \(\lambda\) for the constraint and form \(Q(w,\lambda)=w^\top A w-\lambda(\mathbf{1}^\top w-1)\). Differentiating with respect to \(w\) gives \(2Aw-\lambda\mathbf{1}=0\), hence \(w=(\lambda/2)A^{-1}\mathbf{1}\). Enforcing \(\mathbf{1}^\top w=1\) gives \(\lambda/2=1/(\mathbf{1}^\top A^{-1}\mathbf{1})\), which yields the formula. The denominator is positive because \(A^{-1}\) is positive definite and \(\mathbf{1}\ne0\). Strict convexity gives uniqueness. \(\square\)

This result allows negative weights. If weights must also be nonnegative, the unconstrained answer is valid only when all its entries are nonnegative; otherwise the minimization is over a simplex and may attain its minimum on a boundary. Also, the covariance version is the risk minimizer only for unbiased errors. For biased errors, the relevant quadratic matrix is \(M=\Sigma+bb^\top\), not \(\Sigma\).

What Fitted Stacking and Selection Do—and Do Not—Guarantee

The covariance-optimal formula is an oracle calculation: it assumes the population covariance matrix, or second-moment matrix, is known. Fitted stacking instead estimates weights, often by minimizing squared loss on validation or cross-fitted predictions. Its resulting weights are random because they depend on data. The risk of the fitted rule is therefore not automatically the minimum risk computed as if those weights were known in advance.

Selection is another feasible aggregation rule: each candidate corresponds to a vertex of the weight simplex. Consequently, the oracle minimum over the whole simplex cannot exceed the risk of the best fixed candidate. But a data-fitted stack or a data-driven selection rule need not achieve that oracle comparison on fresh data. Estimation noise, limited validation data, and dependence among candidate errors can affect realized performance.

Common Mistakes

Ignoring covariance. Adding variances alone is correct for independent errors, not in general. Include twice each pairwise covariance in the risk of an average.
Calling variance-optimal weights risk-optimal under bias. The covariance formula omits the squared combined bias. Use the second-moment matrix \(M=\Sigma+bb^\top\) for unrestricted affine risk minimization.
Assuming an average must beat the best member. The first worked example shows equal averaging can have higher risk than the best single predictor when variances differ substantially.
Treating validation performance as a guarantee. Fitted weights minimize a criterion on the data used to fit them. Their performance on new observations must be assessed separately.

Key takeaway: Compare aggregation rules using the risk of their combined error, not a diversity score or a rule of thumb. Equal averaging, covariance-optimal weights, fitted stacking, and single-member selection answer different questions and can rank differently when errors are dependent or biased.

Check Your Understanding

Use the risk formulas and distinctions above to answer these questions.

For three independent, unbiased errors with variances 1, 4, and 9, why can equal averaging have higher risk than selecting the first predictor?
How does a positive covariance between two unbiased prediction errors enter the risk of their equal average?
Why does the inverse-variance weighting rule not generally minimize squared-error risk when prediction errors have nonzero means?
When does the unconstrained covariance-optimal formula give weights that are not permissible for a nonnegative stack?
What does fitted stacking minimize, and why is its validation risk not a guarantee of its test risk?