Risk Penalties From Weight–Error Dependence

Why Weight–Error Dependence Changes Risk

An ensemble combines predictions, but its risk depends on more than the average weight assigned to each member. If weights vary across prediction cases, the weights may systematically be large when particular errors are large, or small when they are large. That dependence changes the squared error of the aggregate. It can increase risk, but it can also reduce it.

The distinction is between fixed-weight aggregation, where the same weight vector is used for every case, and adaptive-weight aggregation, where the weight vector is random and statistically related to the prediction errors being evaluated. The goal here is to derive the risk terms created by that relationship. We use squared prediction loss and do not require the individual errors to have mean zero.

Write \(M=E[ee^\top]\), the error second-moment matrix. For a fixed vector \(w\), the risk is

If the errors have mean zero, \(M\) is their covariance matrix. If not, it includes both covariance and mean-error contributions. Using the second-moment matrix keeps the fixed-weight formula valid without a centering assumption.

An Exact Decomposition for Random Weights

For adaptive weights, replacing \(W\) by its mean \(\mu=E[W]\) does not generally preserve risk. The exact difference has two parts: an interaction between the mean-weight aggregate error and the weight fluctuation, and a squared fluctuation term.

Proof. Since \(W=\mu+\Delta\), the aggregate error satisfies \(W^\top e=\mu^\top e+\Delta^\top e\). Squaring and taking expectations gives

Here \((\mu^\top e)(\Delta^\top e)=\mu^\top ee^\top\Delta\). The first term is \(R(\mu)\), and the last term is an expectation of a square, so it is nonnegative. Subtracting \(R(\mu)\) proves the risk-difference identity. \(\square\)

The interaction term is the part that a simple “random weights add variance” argument misses. It is not generally zero just because \(E[\Delta]=0\): it involves \(ee^\top\Delta\), so the relevant dependence is between the weights and products of errors. The interaction may offset some or all of the nonnegative fluctuation term.

When Random Weights Are Independent of Evaluation Errors

Independence is a useful special case, but it is stronger than merely having mean-zero weight fluctuations. If \(W\) is independent of \(e\), then \(E[ee^\top\Delta]=E[ee^\top]E[\Delta]=0\). The interaction term vanishes, leaving only the nonnegative cost of weight variability around \(\mu\).

Proof. Independence and \(E[\Delta]=0\) imply \(E[ee^\top\Delta]=E[ee^\top]E[\Delta]=0\), so the exact decomposition reduces to \(R(W)=\mu^\top M\mu+E[\Delta^\top ee^\top\Delta]\). By independence, the second term is \(E[\Delta^\top M\Delta]\). Writing the scalar as a trace and using \(E[\Delta\Delta^\top]=\operatorname{Var}(W)\) gives

For positive semidefinite matrices \(M\) and \(\operatorname{Var}(W)\), this trace is nonnegative: it equals \(\operatorname{tr}(M^{1/2}\operatorname{Var}(W)M^{1/2})\), the trace of a positive semidefinite matrix. This proves the result. \(\square\)

Adaptive Weights Can Also Reduce Risk

The word “penalty” can be misleading if it suggests that dependence must increase risk. The squared fluctuation term is always nonnegative, but the interaction can be negative and larger in magnitude. In that case adaptive weighting improves on the fixed mean-weight aggregate. The next example uses error information from the same evaluation case, so it illustrates the algebra rather than a generally implementable prediction rule.

What the Dependence Is—and Is Not

The exact decomposition applies to the joint distribution of the weights and evaluation errors. It does not require weights to be calculated directly from the true errors. For example, weights may be estimated from training or validation predictions, while the errors of interest are on new cases. Whether the interaction vanishes depends on the relationship between those random weights and the new-case errors, not simply on the fact that the weights were estimated in advance.

Conditioning makes this distinction precise. For any random \(W\) and \(e\) with finite squared risk,

This follows by conditioning \((W^\top e)^2\) on \(W\), which is then fixed inside the conditional expectation. If \(E[ee^\top\mid W]\) varies with \(W\), replacing it by the unconditional matrix \(M\) is generally invalid. If \(W\) and \(e\) are independent, the conditional second moment is \(M\), and the independent-weight result applies.

A practical analysis should therefore state what information determines the weights and which cases define the risk. Weights chosen using error estimates may be independent of future evaluation errors under a suitable sampling design, or they may remain dependent because of shared data, model fitting, or distributional structure. The risk identity describes the consequences without assuming one answer in advance.

Common Mistake

• Wrong reasoning: “The random weights have mean \(\mu\), so their risk equals the fixed-weight risk \(\mu^\top M\mu\).” This silently replaces the random product \(W^\top ee^\top W\) by a product of separate expectations.
• Correction: Mean weights determine the fixed-weight baseline, not the adaptive risk. The exact difference includes \(2\mu^\top E[ee^\top\Delta]\) and \(E[(\Delta^\top e)^2]\). Independence makes the first term zero; dependence need not. The net change can be positive or negative.