Squared-Loss Decomposition for Aggregated Regression

Why Squared Loss Gives an Exact Aggregation Identity

When several regression predictors are combined, their weighted mean is itself a prediction. Under squared loss, its performance has a particularly precise relationship to the performances of the component predictors: the difference is exactly the weighted variance of their predictions. This is stronger than a general statement that averaging can help. It identifies the amount of the improvement, and the predictions for which the improvement occurs.

The identity is pointwise: it holds for every response value, every input, and every set of nonnegative weights that sum to one. It therefore does not require a probability model, independent predictors, or a correctly specified regression function. Probability and sampling assumptions enter only when we use the identity to compare expected risks.

Definition of the Aggregate and Its Prediction Variance

Let \(f_1,\ldots,f_M\) be real-valued regression predictors, evaluated at an input \(x\). Let \(w_1,\ldots,w_M\) be weights satisfying \(w_j\geq 0\) and \(\sum_{j=1}^M w_j=1\). The weights may be fixed in advance or selected using data; the algebra below applies once their values are given.

The term “variance” here refers to the variance of the finite distribution that places mass \(w_j\) on prediction \(f_j(x)\). In particular, it is nonnegative and is zero exactly when every predictor with positive weight gives the same prediction at \(x\). It is not the variance of the response \(Y\), nor does it describe uncertainty in the outcome.

The Pointwise Squared-Loss Identity

For a response value \(y\), squared loss for a prediction \(a\) is \((y-a)^2\). The central identity separates the weighted average of the component losses into the loss of the aggregate and a nonnegative dispersion term.

To prove the identity, write each prediction as its weighted mean plus its deviation from that mean. Expanding the squared error produces a cross term, but that term vanishes because the weighted deviations sum to zero.

The equality also shows exactly when the aggregate improves on the weighted component losses. Since \(V_w(x)\geq 0\), the loss of \(\bar f(x)\) is never larger than the weighted mean component loss at that same input and response. The improvement is strict if and only if the positively weighted predictions are not all equal.

Empirical and Population Risk

The pointwise result immediately yields an empirical identity. For observations \((X_i,Y_i)\), \(i=1,\ldots,n\), define the empirical squared-error risk of a predictor \(f\) by \(R_n(f)=n^{-1}\sum_{i=1}^n(Y_i-f(X_i))^2\). If the weights are the same across observations, the pointwise identity can be applied separately to each row and then averaged.

The same reasoning applies to expected risk. Let \((X,Y)\) have a joint distribution, and define \(R(f)=E[(Y-f(X))^2]\). Suppose \(E[Y^2]<\infty\) and \(E[f_j(X)^2]<\infty\) for each \(j\). Then each squared error has finite expectation, because \((Y-f_j(X))^2\leq 2Y^2+2f_j(X)^2\). The finite pointwise identity can therefore be integrated term by term.

For the strictness statement, the expectation of a nonnegative random variable is positive if and only if that variable is positive with positive probability. Thus, if the component predictions differ on a set of inputs that occurs with positive probability, and the differing predictions carry positive weight, the aggregate has strictly smaller risk than their weighted mean risk. This does not imply that the aggregate beats the best individual predictor: the weighted mean risk itself can exceed the risk of the best component.

Separating Prediction Dispersion from Response Noise

The reduction term concerns disagreement among the predictors, not the randomness of the response around its conditional mean. To make that distinction precise, suppose \(E[Y^2]<\infty\), and write \(m(X)=E[Y\mid X]\). For any square-integrable predictor \(g(X)\), conditioning on \(X\) gives $$ E[(Y-g(X))^2\mid X] = \operatorname{Var}(Y\mid X)+(m(X)-g(X))^2. $$

Indeed, write \(Y-g(X)=(Y-m(X))+(m(X)-g(X))\), square, and condition on \(X\). The cross term has conditional expectation zero because \(E[Y-m(X)\mid X]=0\); the squared first term has conditional expectation \(\operatorname{Var}(Y\mid X)\). The remaining term is fixed given \(X\).

Interpretation and a Common Mistake

This identity is useful whenever a regression procedure averages candidate predictions. It gives an exact diagnostic for the benefit attributable to averaging: at each input, inspect the weighted spread of the candidate predictions. Large disagreement creates a large gap between weighted component loss and aggregate loss; agreement leaves little or no gap. The identity itself does not say how to choose weights, guarantee a small risk, or establish that the aggregate is close to the best possible predictor.

If the weights are estimated from a training sample, the algebra remains valid for the realized weights and predictions. It can also be applied to a new test observation conditional on the training data. The exact comparison is between the aggregate loss and the weighted losses of the same component predictions, evaluated at the same response and input.

• Common mistake: treating the aggregate loss as the weighted mean component loss. That omits the prediction-variance term. The correct identity subtracts \(V_w(x)\) from the weighted component loss to obtain the aggregate loss.
• Common mistake: concluding that averaging always beats every component. The identity compares the aggregate with the weighted average of component losses, not with their minimum. For example, a poorly chosen set of weights can give a weighted mean risk above the risk of one component.
• Common mistake: interpreting prediction variance as response noise. It measures disagreement among the predictions at a fixed input. Conditional response variance is a different quantity and remains even when the aggregate prediction equals the conditional mean.