Risk Decomposition for Linear Stacking

Why the Stack’s Errors Must Be Considered Jointly

A linear stack combines predictions from several fitted models. Its squared-error risk is not determined just by the separate risks of those models: the same prediction errors can reinforce one another or cancel, depending on their dependence and on the weights. The relevant object is therefore the joint distribution of the members’ prediction errors.

We derive the population risk for a fixed set of weights and show exactly where error means and covariances enter. Throughout, assess performance on a fresh test pair $(X,Y)$, drawn from the target population. If the fitted member predictors were trained on data $D$, regard them as fixed conditional on $D$. The calculations then describe conditional test risk; they can be averaged over training data afterward.

Definition: Let $f_1,\ldots,f_M$ be fixed prediction functions and let $w=(w_1,\ldots,w_M)^\top$ be a fixed vector of weights satisfying $\sum_{j=1}^M w_j=1$. The linear stack is $f_w(X)=\sum_{j=1}^M w_j f_j(X)$. Define member $j$'s prediction error by $e_j=f_j(X)-Y$, the error vector by $e=(e_1,\ldots,e_M)^\top$, and the population squared-error risk by $R(w)=E[(f_w(X)-Y)^2]$. Assume $E[e_j^2]<\infty$ for each $j$.

The sum-to-one condition makes the stack’s error an ordinary weighted combination of the member errors:

f_w(X)-Y =\sum_{j=1}^M w_jf_j(X)-Y =\sum_{j=1}^M w_j\{f_j(X)-Y\} =w^\top e.

The weights need not be nonnegative. Thus this definition includes affine combinations that extrapolate beyond the range of the members’ predictions, as long as their weights sum to one.

Risk as Bias Plus Joint Error Variance

Let $\mu=E[e]$ be the vector of mean member errors, and let $\Sigma=\operatorname{Cov}(e)$ be their covariance matrix, with entries $\Sigma_{jk}=\operatorname{Cov}(e_j,e_k)$. Finite second moments ensure these quantities exist. The stack’s mean error is $w^\top\mu$; its variance depends on the full covariance matrix, not only on its diagonal.

Theorem: For a fixed weight vector satisfying $\mathbf{1}^\top w=1$, and member errors with finite second moments, $$ R(w)=(w^\top\mu)^2+w^\top\Sigma w. $$ Equivalently, $R(w)$ is the square of the stack’s mean error plus the variance of its error.

Proof. Since $f_w(X)-Y=w^\top e$, its expectation is $E[w^\top e]=w^\top\mu$, and its variance is $\operatorname{Var}(w^\top e)=w^\top\Sigma w$. For any square-integrable random variable $Z$, $E[Z^2]=\{E[Z]\}^2+\operatorname{Var}(Z)$. Applying this identity to $Z=w^\top e$ gives the result. $\square$

The first term is squared bias relative to the response $Y$, measured for the stack as a whole. It is not generally a weighted average of the members’ squared biases. The second term can be expanded to show the role of every member and every pair of members.

R(w) =\left(\sum_{j=1}^M w_j\mu_j\right)^2 +\sum_{j=1}^M w_j^2\operatorname{Var}(e_j) +2\sum_{1\le j<k\le M}w_jw_k\operatorname{Cov}(e_j,e_k).

This expansion follows directly from $w^\top\Sigma w=\sum_j\sum_k w_jw_k\Sigma_{jk}$: the diagonal terms give the individual error variances, while each off-diagonal covariance appears twice because $\Sigma_{jk}=\Sigma_{kj}$. The covariance contribution can be positive or negative. For negative weights, even the sign of an individual weighted covariance term depends on the product $w_jw_k$.

Worked Example: Two Members With Unequal Biases

Suppose two members have mean errors $\mu_1=0.2$ and $\mu_2=-0.1$, variances $0.25$ and $0.16$, and covariance $0.08$. Take $w_1=0.4$ and $w_2=0.6$. The weights sum to one, so the theorem applies.

The stack’s mean error is $0.4(0.2)+0.6(-0.1)=0.08-0.06=0.02$, and its squared bias is $0.02^2=0.0004$.

Its error variance is $0.4^2(0.25)+0.6^2(0.16)+2(0.4)(0.6)(0.08)$ $=0.16(0.25)+0.36(0.16)+0.48(0.08)$ $=0.0400+0.0576+0.0384=0.1360$.

Therefore, $R(w)=0.0004+0.1360=0.1364$. For comparison, the members’ individual risks are $0.2^2+0.25=0.29$ and $(-0.1)^2+0.16=0.17$. The stack has lower risk here, but that conclusion uses the errors’ covariance as well as their separate risks.

Reading the Covariance Matrix

The covariance matrix summarizes how member errors move together around their respective means. Its diagonal entries are error variances; its off-diagonal entries are error covariances. Positive covariance means that errors tend to be above or below their own means together. Negative covariance means deviations tend to move in opposite directions. Neither statement alone determines stack risk: the weights and the mean errors also matter.

The covariance contribution is always nonnegative as a whole, since $w^\top\Sigma w=\operatorname{Var}(w^\top e)\geq 0$. Individual cross-terms, however, need not be nonnegative. The following example evaluates all terms and illustrates why calculating only the diagonal part is insufficient.

Worked Example: Three Members and a Full Covariance Calculation

Consider mean errors $\mu=(0.1,0.3,-0.2)^\top$, weights $w=(0.5,0.3,0.2)^\top$, and covariance matrix

\Sigma= \begin{pmatrix} 0.09&0.03&-0.015\\ 0.03&0.16&0.02\\ -0.015&0.02&0.04 \end{pmatrix}.

The weights sum to $0.5+0.3+0.2=1$. The weighted mean error is $0.5(0.1)+0.3(0.3)+0.2(-0.2)=0.05+0.09-0.04=0.10$, so the squared bias is $0.01$.

The variance contribution, including all pairwise terms, is $0.5^2(0.09)+0.3^2(0.16)+0.2^2(0.04)$ $+2(0.5)(0.3)(0.03)+2(0.5)(0.2)(-0.015)+2(0.3)(0.2)(0.02)$.

The diagonal terms sum to $0.0225+0.0144+0.0016=0.0385$. The three covariance terms are $0.009$, $-0.003$, and $0.0024$, summing to $0.0084$. Thus $w^\top\Sigma w=0.0469$, and the risk is $R(w)=0.01+0.0469=0.0569$.

If the off-diagonal covariances had been omitted, the calculated risk would have been $0.01+0.0385=0.0485$, which is too small by $0.0084$. The negative covariance between members 1 and 3 reduces risk, but the other two covariances more than offset that reduction.

A useful consequence of the theorem is that the same marginal error variances can lead to very different stack risks when the errors’ dependence changes. This is why a diversity summary that ignores the direction and scale of the relevant covariance terms cannot, by itself, determine squared-error risk.

Worked Example: Same Marginal Risks, Different Dependence

Take two members with zero mean errors, error variance $0.25$ each, and weights $w_1=w_2=0.5$. First suppose their covariance is $0.20$. Then the stack’s risk is its error variance: $0.5^2(0.25)+0.5^2(0.25)+2(0.5)(0.5)(0.20)=0.0625+0.0625+0.10=0.225$.

Now keep the weights, means, and marginal variances the same, but suppose the covariance is $-0.20$. The risk becomes $0.0625+0.0625+2(0.5)(0.5)(-0.20)=0.125-0.10=0.025$.

Both covariance choices are mathematically compatible with the stated variances: for two variables with variance $0.25$, the covariance must lie between $-0.25$ and $0.25$, and both values lie in that range. The individual risks are $0.25$ in both cases, yet the stack risks differ substantially because the joint error behavior differs.

When the Weights Do Not Sum to One

The sum-to-one condition is substantive, not cosmetic. If a stack is defined as $f_w(X)=\sum_jw_jf_j(X)$ with $s=\sum_jw_j\ne1$, then its error is not $w^\top e$. Instead,

f_w(X)-Y=w^\top e+(s-1)Y.

Consequently, the member-error vector alone does not determine the risk of this stack. The response $Y$, its covariance with the member errors, and its mean must also be accounted for. In particular, defining $a=(w^\top,s-1)^\top$ and $u=(e^\top,Y)^\top$ gives the exact identity

R(w)=E[(a^\top u)^2] =\{a^\top E[u]\}^2+a^\top\operatorname{Cov}(u)a,

provided the response and member errors have finite second moments. This follows from the same squared-mean-plus-variance identity used above. For a stack with weights summing to one, the coefficient on $Y$ in this augmented expression is zero, recovering the earlier formula.

Common Pitfall and Practical Interpretation

Incorrect reasoning: “The stack’s risk is the weighted average of the members’ risks, so knowing each member’s risk is enough.” This omits both the pairwise covariance contributions and the fact that squared bias does not combine by taking a weighted average of member risks.

Correction: For weights summing to one, first form the weighted mean error $w^\top\mu$, then calculate the full quadratic form $w^\top\Sigma w$. The mean error determines the stack’s squared bias; all entries of the joint covariance matrix determine its error variance. Marginal risks alone generally provide neither quantity.

When predictors and weights are fixed conditional on training data $D$, the decomposition applies to their conditional test risk using the conditional means and covariances of the fresh test errors. If weights themselves vary with $D$, the formula can be applied for each fixed $D$ and then averaged over $D$; replacing that average with a formula using only average weights or an average covariance is not generally justified.

Key takeaway: For a sum-to-one linear stack, population squared-error risk is $(w^\top\mu)^2+w^\top\Sigma w$. The first term is squared weighted error bias; the second contains both each member’s error variance and every pairwise error covariance.

Check Your Understanding

Work through these questions by applying the risk decomposition, not by averaging member risks.

State the condition on $w$ that allows the stack’s prediction error to be written as $w^\top e$.
For two members, which terms in $w^\top\Sigma w$ come from the diagonal of $\Sigma$, and which come from the off-diagonal entries?
Can an individual covariance term in the expansion of $w^\top\Sigma w$ be negative even though the full variance contribution is nonnegative? Explain.
Why are member-error means needed in addition to the covariance matrix to calculate squared-error risk?
If the weights do not sum to one, what additional random quantity must be included to express stack risk using member errors?