Exact Risk Change When Adding a Predictor

Risk Change From a Prediction Update

A predictor can be individually useful and still raise an ensemble’s risk when assigned an unsuitable weight. The relevant question is not just whether the new predictor is accurate, but how its contribution lines up with the ensemble’s current errors and how large that contribution is. Squared-error risk makes this comparison exact: the change is a quadratic function of the weight.

Let \((X,Y)\) have a fixed evaluation distribution, and suppose the predictions being compared have finite second moments. Write \(f(X)\) for the current ensemble prediction and \(h(X)\) for a proposed change in prediction. Giving that change weight \(\alpha\) produces \(f_\alpha(X)=f(X)+\alpha h(X)\). This notation covers directly adding a predictor and, with a different choice of \(h\), blending in a replacement predictor.

All expectations and covariances below are under the same evaluation distribution. That common distribution matters: comparing risks calculated under different distributions would not isolate the effect of changing the predictor.

The Exact Risk-Change Identity

After the update, the error is \(e_\alpha=Y-f_\alpha(X)=e-\alpha h\). Squaring this error and taking expectations gives the risk difference directly.

Proof. Since \(e_\alpha=e-\alpha h\), expansion gives \(e_\alpha^2=e^2-2\alpha eh+\alpha^2h^2\). Each term has a finite expectation: \(e\) and \(h\) have finite second moments, so \(E[|eh|]\leq \sqrt{E[e^2]E[h^2]}<\infty\) by Cauchy–Schwarz. Taking expectations and subtracting \(R(f)=E[e^2]\) proves the first identity. The second follows from \(E[eh]=\operatorname{Cov}(e,h)+E[e]E[h]\). \(\square\)

The identity separates the contribution’s size, \(E[h^2]\), from its alignment with current error, \(E[eh]\). If current error is positive where \(h\) is positive, for example, adding a positive amount of \(h\) pushes the prediction in the direction that reduces that error. If \(h\) tends to have the opposite sign, the update can instead move predictions further from the target.

When both \(e\) and \(h\) have mean zero, the alignment term simplifies to \(\operatorname{Cov}(e,h)\), and \(E[h^2]=\operatorname{Var}(h)\). But covariance by itself is not generally enough: when either mean is nonzero, the product of means also contributes to \(E[eh]\), and the update’s second moment includes any nonzero mean of \(h\).

Convex Blending and Candidate Error Covariance

A common alternative is to blend the current ensemble with a candidate predictor \(g(X)\), assigning the candidate weight \(\alpha\) and scaling the current ensemble weight to \(1-\alpha\). This is not the same operation as leaving the old weights unchanged and appending a term. For the blend \(f_\alpha=(1-\alpha)f+\alpha g\), the update direction in the identity is \(h=g-f\).

Write \(u=Y-g(X)\) for the candidate’s error. Then \(h=e-u\), and the blended error is \((1-\alpha)e+\alpha u\). Substitution gives an exact formula involving both prediction errors:

The second form shows how covariance between the current and candidate errors enters the decision. For a positive blend weight and \(E[(e-u)^2]>0\), blending reduces risk exactly when $$ 2\{E[e^2]-E[eu]\}>\alpha E[(e-u)^2]. $$ Under the stated zero-mean conditions, \(E[eu]=\operatorname{Cov}(e,u)\). Without those conditions, replace the covariance by \(\operatorname{Cov}(e,u)+E[e]E[u]\); covariance alone does not give the exact criterion.

Choosing a Weight Along a Fixed Direction

The exact change also identifies the best unconstrained scalar weight for a fixed update direction. This is a one-dimensional quadratic calculation; it does not claim that the chosen direction is itself optimal among all possible predictors.

Derivation. For \(M>0\), complete the square: $$ R(f_\alpha)-R(f)=M\alpha^2-2C\alpha =M\left(\alpha-\frac{C}{M}\right)^2-\frac{C^2}{M}. $$ The squared term is smallest at \(\alpha=C/M\), giving the stated minimum. The minimum is strictly negative when \(C\neq 0\). If a strictly positive weight is required, an improving positive weight exists precisely when \(C>0\); if a strictly negative weight is required, one exists precisely when \(C<0\). \(\square\)

This result explains why selecting a weight and deciding whether to include a predictor are related but distinct questions. For example, a direction with \(C>0\) can improve risk at its optimal positive weight while worsening risk at a sufficiently large positive weight. If the permitted weights are restricted, the unconstrained optimum need not be permitted; the exact risk-change identity remains the criterion for any chosen value.

Common Pitfall and Practical Interpretation

A common mistake is to conclude that positive covariance between the current error and a new contribution guarantees improvement. The exact condition shows why this is incomplete: the covariance must be large enough relative to the specified weight and the contribution’s second moment. A large contribution can overshoot, as the previous calculation demonstrates. Likewise, for nonzero-mean errors, covariance is not the whole alignment term; the product of the means must be included.

Another source of error is to use the right formula for the wrong update. Appending \(\alpha g\) while keeping the current prediction fixed uses \(h=g\). Blending with \(g\) while reducing the current ensemble’s weight uses \(h=g-f\). These updates generally have different risk changes because they move predictions in different directions. State the updated prediction first, then identify \(h\), before applying the identity.

The identity is exact for population risk under the chosen evaluation distribution. In practice, its expectations may be estimated from evaluation observations, but estimates do not turn a population identity into a guarantee about the true risk. The key conceptual check remains the same: assess the alignment of the update with current error and compare it with the update’s size at the weight actually being considered.