Negative Weights, Extrapolation, and Variance Inflation

Why Negative Weights Change the Ensemble

A convex combination of two predictions lies between them: each weight is nonnegative and the weights sum to one. An unconstrained linear combination can instead assign a negative weight to one member and a weight greater than one to the other. The resulting prediction extrapolates beyond the two predictions rather than averaging them. This can be useful when the members’ biases point in a direction that a convex combination cannot correct, but extrapolation can also magnify variation and sensitivity to errors in the component predictions.

We examine these effects at a fixed input $x$, treating the fitted predictors as random through their training data or algorithmic randomness. Let $m(x)$ be the target regression function, and let $\hat f_1(x)$ and $\hat f_2(x)$ be two predictions. For a weight $a$, the sum-to-one ensemble is $\hat f_a(x)=a\hat f_1(x)+(1-a)\hat f_2(x)$. The coefficient $a$ is unrestricted unless a constraint is explicitly imposed.

Definition: At the fixed input $x$, define the member biases $b_j=E[\hat f_j(x)]-m(x)$, the centered errors $Z_j=\hat f_j(x)-E[\hat f_j(x)]$, and their covariance matrix $\Sigma=\begin{pmatrix}v_1&c\\c&v_2\end{pmatrix}$, where $v_j=\operatorname{Var}(\hat f_j(x))$ and $c=\operatorname{Cov}(\hat f_1(x),\hat f_2(x))$. The ensemble has weights $(a,1-a)$. It is a convex combination when $0\leq a\leq1$; otherwise it extrapolates.

The target here is the regression function, so the risk is $E[(\hat f_a(x)-m(x))^2]$. If the prediction target is a new response $Y$ with $E[Y\mid X=x]=m(x)$, and the fitted predictors are independent of that new response conditional on $x$, then $E[(\hat f_a(x)-Y)^2]$ adds the constant $\operatorname{Var}(Y\mid X=x)$. Thus the weight comparisons below are unchanged.

Bias Cancellation and the Sign of the Weight

Linearity of expectation gives the ensemble bias $b(a)=ab_1+(1-a)b_2=b_2+a(b_1-b_2)$. When $b_1\ne b_2$, the unique weight that sets this bias to zero is

a_0=-\frac{b_2}{b_1-b_2}, \qquad 1-a_0=\frac{b_1}{b_1-b_2}.

This is an algebraic bias-cancellation weight; it need not be negative, and it need not be a sensible choice under the full risk. If $b_1$ and $b_2$ have opposite signs, the zero lies between them and $a_0\in[0,1]$. If they have the same nonzero sign, no convex combination can have zero bias, so the zero lies outside $[0,1]$. It can be below zero or above one. More precisely, when $b_1\ne b_2$, $a_0<0$ exactly when $-b_2/(b_1-b_2)<0$, equivalently $b_2(b_1-b_2)>0$. If $b_1=b_2$, changing $a$ does not change the bias: it is impossible to cancel a common nonzero bias, while if both are zero every weight is unbiased.

Worked Example: Same-Sign Biases Can Give a Weight Above One

Suppose $b_1=1$ and $b_2=2$. Then $a_0=-2/(1-2)=2$, so the weights are $2$ and $-1$. The resulting bias is $2(1)+(-1)(2)=0$. The weight on member 1 is positive and greater than one; the negative weight is on member 2.

For comparison, take $b_1=2$ and $b_2=1$. Now $a_0=-1/(2-1)=-1$, giving weights $-1$ and $2$. The bias is $(-1)(2)+2(1)=0$. Both pairs have same-sign biases, but the negative coefficient occurs on a different member. The sign follows from the formula, not from the fact that the biases share a sign.

Variance of an Extrapolating Pair

Centering each member’s prediction separates the ensemble’s mean from its random fluctuation. Its centered error is $aZ_1+(1-a)Z_2$. Expanding its variance gives the following exact identity.

Result: For any real $a$, the ensemble variance at $x$ is $$ V(a)=a^2v_1+(1-a)^2v_2+2a(1-a)c. $$ The squared-error risk against $m(x)$ is $R(a)=b(a)^2+V(a)$.

To verify the variance formula, use $\operatorname{Var}(U+W)=\operatorname{Var}(U)+\operatorname{Var}(W)+2\operatorname{Cov}(U,W)$, with $U=aZ_1$ and $W=(1-a)Z_2$. This yields the displayed expression. For the risk identity, write $\hat f_a-m=b(a)+aZ_1+(1-a)Z_2$. The centered terms have mean zero, so the cross term with $b(a)$ has expectation zero, leaving $b(a)^2+V(a)$. $\square$

It is useful to expose how the variance changes with $a$. Set $D=v_1+v_2-2c=\operatorname{Var}(Z_1-Z_2)$, which is nonnegative. Expanding the variance identity around member 2 gives $V(a)=v_2+2a(c-v_2)+a^2D$, and therefore $V(a)-V(0)=2a(c-v_2)+a^2D$. If $D>0$, the quadratic term eventually dominates as $|a|$ grows, so sufficiently extreme extrapolation has large variance. But the difference from $V(0)$ also contains a linear term, and the variance at a particular negative weight need not exceed $V(0)$. The covariance matters.

Worked Example: Bias Cancellation Can Carry a Large Variance Cost

Use biases $b_1=2$, $b_2=1$, and independent centered prediction errors with $v_1=v_2=1$ and $c=0$. The bias-cancelling weight is $a=-1$, so the ensemble prediction uses weights $(-1,2)$. Its bias is zero, while its variance is $V(-1)=(-1)^2(1)+2^2(1)+2(-1)(2)(0)=5$. Its risk is therefore 5.

Member 1 alone has risk $2^2+1=5$, and member 2 alone has risk $1^2+1=2$. The convex equal-weight ensemble has bias $1.5$ and variance $(0.5)^2+(0.5)^2=0.5$, so its risk is $1.5^2+0.5=2.75$. Here extrapolation removes bias but raises variance enough that its total risk is greater than that of the convex equal-weight ensemble and member 2. Bias cancellation by itself does not establish a risk improvement.

Covariance and Stability Conditions

The covariance matrix $\Sigma$ is positive semidefinite because it is a covariance matrix. For $w=(a,1-a)^\mathsf{T}$, the ensemble variance is $w^\mathsf{T}\Sigma w$. If the eigenvalues of $\Sigma$ are $0\leq\lambda_{\min}\leq\lambda_{\max}$, then $\lambda_{\min}\|w\|_2^2\leq V(a)\leq\lambda_{\max}\|w\|_2^2$. These bounds follow by expressing $w$ in an orthonormal eigenbasis: its quadratic form is the sum of each eigenvalue times the squared coordinate in that direction. In particular, if $\lambda_{\min}>0$, increasing the Euclidean size of the weight vector necessarily increases this lower bound on variance. If $\Sigma$ is singular, the lower bound may be zero, so large weights alone do not force a large variance.

For sum-to-one weights, $\|w\|_1=|a|+|1-a|$ equals 1 on the convex interval and exceeds 1 outside it. This gives a direct perturbation stability bound. Suppose each member prediction is changed by at most $\epsilon$ at $x$. The ensemble change is bounded by $|a||\delta_1|+|1-a||\delta_2|\leq (|a|+|1-a|)\epsilon$. Thus a convex combination cannot magnify the worst-case component perturbation beyond $\epsilon$, whereas extrapolating weights can. This is a deterministic sensitivity statement, distinct from a probabilistic variance statement.

Worked Example: Correlation Changes the Variance Cost

Let $v_1=v_2=1$ and $c=0.99$. The covariance matrix is $\Sigma=\begin{pmatrix}1&0.99\\0.99&1\end{pmatrix}$. Its eigenvalues are $1.99$ and $0.01$, so it is positive definite. For weights $(-1,2)$, the variance is $1+4+2(-1)(2)(0.99)=5-3.96=1.04$. With the same marginal variances but zero covariance, that variance would be $1+4=5$. The negative weight does not automatically produce variance 5: the positive covariance substantially changes the result.

The weight norm is $\|w\|_1=3$. If each component prediction is perturbed by at most $\epsilon$, the ensemble can change by as much as $3\epsilon$ under the bound above. For convex equal weights, the corresponding bound is $\epsilon$. In this example, high positive correlation moderates the variance of this particular combination, but it does not remove its larger worst-case perturbation bound.

Stability takeaway: For two members, $D=\operatorname{Var}(Z_1-Z_2)$ controls the quadratic growth of variance as the weights move away from the convex range. A positive-definite error covariance matrix also gives a positive lower bound through its smallest eigenvalue. Neither condition says that every negative weight has greater variance than every convex weight; the actual covariance and the chosen coefficient determine that comparison.

Common Mistakes

Assuming same-sign biases always give a negative first weight. They imply that the bias-cancelling weight is outside $[0,1]$, not which side. For $b_1=1,b_2=2$, $a_0=2$; for $b_1=2,b_2=1$, $a_0=-1$. Check $-b_2/(b_1-b_2)$.
Claiming every negative weight increases variance. The variance difference includes the covariance-dependent linear term $2a(c-v_2)$. Evaluate the full quadratic form rather than relying on the sign of $a$.
Treating an unbiased ensemble as necessarily better. Risk is bias squared plus variance. A reduction in bias can be outweighed by an increase in variance, as the worked calculation demonstrates.
Confusing variance with sensitivity to perturbations. Covariance determines the variance for a specified random-error model; the absolute weight sum gives a worst-case bound for specified component perturbations. They answer different questions.

Check Your Understanding

Use the bias and covariance formulas to answer these questions.

For $b_1=3$ and $b_2=1$, find the bias-cancelling weight $a_0$ and both ensemble weights. Which coefficient is negative?
For what values of $b_1,b_2$ does the formula for a bias-cancelling weight fail to give a unique value? What happens to the bias as $a$ varies in that case?
Given $v_1=2$, $v_2=3$, and $c=1$, calculate $V(a)$ at $a=-1$ and compare it with $V(0)$.
Why does positive definiteness of $\Sigma$ give a lower variance bound for large weight vectors, and why might that argument fail for singular $\Sigma$?
If both component predictions can be perturbed by at most $\epsilon$, what bound applies to an ensemble with weights $(-2,3)$?