Regularization and Stability of Stacking Weights

Why Regularize Stacking Weights?

Stacking chooses weights by comparing candidate predictors on validation predictions, then combines their predictions using the fitted weights. When candidate predictors make similar predictions, the validation loss can be almost flat in some directions: quite different weight vectors may perform nearly equally on the observed data. In that situation, small changes in validation outcomes can produce large changes in the fitted weights. A penalty can stabilize the choice, but it may also move the ensemble away from the best-fitting mixture.

Here the weights are constrained to be nonnegative and sum to one. The regularizer we study is quadratic shrinkage toward uniform averaging. It favors balanced weights, while allowing the validation data to move the weights away from uniform when there is enough evidence to do so. The penalty parameter controls this tradeoff; its numerical meaning depends on the scale of the loss.

Because the weights sum to one, penalizing their squared norm is equivalent to shrinking toward uniform weights: \(\|w-u\|_2^2=\|w\|_2^2-1/M\). Thus a penalty on \(\|w\|_2^2\) and a penalty on squared distance from \(u\) give the same minimizer, up to an additive constant in the objective. The distance form makes the target of shrinkage explicit.

The simplex constraint matters. This penalty does not encourage sparse weights; among simplex vectors, it favors the most balanced vector, \(u\). If the validation loss supports a less balanced mixture, the fitted weights can still depart from \(u\). As \(\lambda\) grows, departures become more costly. At \(\lambda=0\), the solution is an unregularized constrained fit, which need not be unique. For \(\lambda>0\), the objective is strictly convex in \(w\), so the minimizer is unique.

What the Penalty Does to a Two-Predictor Mixture

This example isolates the direction of shrinkage, but it does not imply that a larger penalty always improves prediction on new data. The validation loss is only an estimate of predictive performance. Regularization is useful when reduced sensitivity and variability are worth the possible increase in approximation error.

A Perturbation Bound for the Fitted Weights

For fixed validation predictions \(A\), the quadratic penalty makes the objective strongly convex even if the candidate predictions are highly correlated. Strong convexity provides a direct guarantee: changing the validation responses by a small amount cannot change the fitted weights arbitrarily much.

Proof. Separate the objective into a common part \(g(w)=(1/(2n))\|Aw\|_2^2+(\lambda/2)\|w-u\|_2^2\) and a response-dependent linear term \(q_y^Tw\), where \(q_y=-A^Ty/n\). The gradient of \(g\) is \(\lambda\)-strongly monotone: for all \(v,w\), \( (\nabla g(v)-\nabla g(w))^T(v-w)=(1/n)\|A(v-w)\|_2^2+\lambda\|v-w\|_2^2\geq\lambda\|v-w\|_2^2.\)

A minimizer of a differentiable convex function over a convex set satisfies the variational inequality that its gradient has nonnegative inner product with every feasible displacement from that minimizer. Apply this to \(w\) with displacement \(w'-w\), and to \(w'\) with displacement \(w-w'\). Adding the resulting inequalities yields \( (\nabla g(w)-\nabla g(w'))^T(w-w')\leq(q_{y'}-q_y)^T(w-w').\) The left side is at least \(\lambda\|w-w'\|_2^2\). The Cauchy–Schwarz inequality bounds the right side above by \(\|q_{y'}-q_y\|_2\|w-w'\|_2\), and \(\|q_{y'}-q_y\|_2=\|A^T(y-y')/n\|_2\leq\|A\|_{\mathrm{op}}\|y-y'\|_2/n\). If \(w=w'\), the first claimed bound holds immediately; otherwise divide by \(\lambda\|w-w'\|_2\). Finally, \(\|A(w-w')\|_2\leq\|A\|_{\mathrm{op}}\|w-w'\|_2\), which gives the prediction bound. \(\square\)

The bound has two interpretable features. First, greater \(\lambda\) reduces sensitivity. Second, sensitivity also depends on \(\|A\|_{\mathrm{op}}\): changing the response matters more when the prediction columns collectively have a large scale. The penalty should therefore be interpreted relative to the scale of the loss and predictions, not as a scale-free tuning constant.

Stability and Weight Variability

The perturbation result can also be read as a variability bound. Suppose \(Y\) is a random validation-response vector, \(A\) is fixed, and the fitted weights are \(W=\hat w_\lambda(Y)\). Assume \(E\|Y\|_2^2<\infty\). For an independent copy \(Y'\), let \(W'=\hat w_\lambda(Y')\). The identity \(E\|W-E[W]\|_2^2=\frac12E\|W-W'\|_2^2\) follows by expanding the squared norm and using independence. Applying the perturbation bound to \(Y,Y'\) then gives \(E\|W-E[W]\|_2^2\leq\frac{\|A\|_{\mathrm{op}}^2}{n^2\lambda^2}\operatorname{tr}\{\operatorname{Cov}(Y)\}.\)

Here \(\operatorname{tr}\{\operatorname{Cov}(Y)\}=E\|Y-E[Y]\|_2^2\). Thus, conditional on the fixed prediction matrix, quadratic shrinkage controls the total variance of the fitted weight vector. This is a bound, not an assertion that the weight variance always equals a constant times \(1/\lambda^2\); the actual variance may be much smaller, and the matrix scale and response variability matter.

The Approximation-Bias Cost

Stability is not free: a penalty can pull the weights away from the best mixture for the population risk. The next result makes that cost explicit. Let \(R(w)\) be a finite, continuous, convex population risk on \(\Delta_M\), and define \(w^*\) as a minimizer of \(R\) on the simplex. Define \(w_\lambda\) as a minimizer of \(R(w)+(\lambda/2)\|w-u\|_2^2\). These minimizers exist because the simplex is compact and the objectives are continuous.

Proof. The lower bound follows because \(w^*\) minimizes \(R\) over the simplex and \(w_\lambda\in\Delta_M\). By the definition of \(w_\lambda\), its penalized objective is no greater than the value at the feasible point \(w^*\): \( R(w_\lambda)+\frac{\lambda}{2}\|w_\lambda-u\|_2^2\leq R(w^*)+\frac{\lambda}{2}\|w^*-u\|_2^2.\) Subtract \(R(w^*)\) and discard the nonnegative term \((\lambda/2)\|w_\lambda-u\|_2^2\) from the left. The result is the stated upper bound. \(\square\)

The bound is small when the best mixture is already close to uniform. It can be loose when the oracle weights are far from uniform, but it makes clear why the penalty target matters: quadratic shrinkage toward \(u\) is most benign when equal averaging is itself a good approximation.

Practical Interpretation and a Common Pitfall

Quadratic shrinkage is useful when validation data do not reliably distinguish among several similar predictors or when weights vary substantially across plausible validation samples. It discourages extreme reweighting without changing the allowed class of simplex mixtures. In practice, choose \(\lambda\) on a scale compatible with the loss and use a validation design that avoids evaluating weights on the same in-sample predictions used to fit the candidate predictors.

• Pitfall: treating the penalty as variance reduction with no cost. The incorrect reasoning is that more shrinkage always improves the ensemble because the fitted weights are more stable. Stability alone does not guarantee lower predictive risk; the population-risk result allows an excess-risk cost up to \((\lambda/2)\|w^*-u\|_2^2\).
• Correction: assess the stability benefit alongside the possible approximation bias. A large penalty is most defensible when uniform weights are already a reasonable mixture, or when the unregularized fit is so unstable that its apparent validation advantage is not trustworthy.
• Pitfall: assuming the same numerical \(\lambda\) has the same meaning for every problem. Multiplying the loss by a constant changes its scale relative to the penalty. Rescaling predictions or changing the loss normalization can therefore change the effective amount of shrinkage.

The key is not simply to make stacking weights smooth or balanced. It is to stabilize the weight contrasts that the validation data estimate weakly, while checking that the chosen shrinkage target does not impose too much approximation bias.