Optimism From Fitting Weights on Training Predictions

Why Training Risk Can Be Too Small

Stacking chooses weights to combine predictions from several candidate models. If those weights are chosen using the same responses on which the combined predictions are evaluated, the fitted combination can adapt to response noise: it can give more weight to candidates that happen to match those responses unusually well. Its training loss then reflects both predictive fit and this response-specific adaptation.

The issue is not that training loss is calculated incorrectly. It is that the fitted predictions and the responses used to evaluate them are dependent. To measure the resulting optimism, compare training loss with loss on an independent response replicate at the same prediction points. Throughout, losses are sums of squared errors; dividing both risks by the sample size gives the corresponding average-risk comparison.

Definition: Let $Y=(Y_1,\ldots,Y_n)^\top$ be the response vector, with $E[Y]=\mu$, and let $\hat\mu(Y)=(\hat\mu_1(Y),\ldots,\hat\mu_n(Y))^\top$ be the fitted prediction vector. The training loss is $T=\|Y-\hat\mu(Y)\|^2$. Let $Y^0$ be independent of $Y$, with $E[Y^0]=\mu$; the fresh-response loss is $Q=\|Y^0-\hat\mu(Y)\|^2$. The expected optimism is $E[Q]-E[T]$.

This comparison holds the fitted predictions fixed at the values learned from $Y$, then asks how those predictions perform against independent responses with the same mean. It isolates the cost of using the observed responses both to fit and to assess the predictions.

The Covariance Identity

The central quantity is the covariance between each fitted prediction and its corresponding response. A prediction that tends to rise when its observed response is unusually high has used information from that response; under squared loss, this dependence reduces the apparent training loss.

Theorem: Suppose $E\|Y-\mu\|^2<\infty$, $E\|Y^0-\mu\|^2<\infty$, and $E\|\hat\mu(Y)-\mu\|^2<\infty$. Suppose also that $Y^0$ is independent of $Y$, $E[Y^0]=E[Y]=\mu$, and $E\|Y^0-\mu\|^2=E\|Y-\mu\|^2$. Then $$ E[Q]-E[T]=2\sum_{i=1}^n\operatorname{Cov}(\hat\mu_i(Y),Y_i). $$ Equality of the full distributions of $Y^0$ and $Y$ is sufficient for the second-moment condition, but is not required.

Proof. Expand each squared norm around the common mean vector $\mu$: $$ Q=\|Y^0-\mu\|^2+\|\hat\mu(Y)-\mu\|^2 -2(Y^0-\mu)^\top(\hat\mu(Y)-\mu), $$ $$ T=\|Y-\mu\|^2+\|\hat\mu(Y)-\mu\|^2 -2(Y-\mu)^\top(\hat\mu(Y)-\mu). $$ By independence, $Y^0-\mu$ has mean zero and is independent of $\hat\mu(Y)-\mu$. Thus the expected cross term in the expansion of $Q$ is zero. The expected squared-norm terms involving $\hat\mu(Y)-\mu$ cancel when we subtract the two expectations. The assumed equality of centered second moments cancels the remaining squared-norm terms involving $Y^0$ and $Y$. What remains is $$ E[Q]-E[T] =2\sum_{i=1}^n E[(Y_i-\mu_i)(\hat\mu_i(Y)-\mu_i)] =2\sum_{i=1}^n\operatorname{Cov}(\hat\mu_i(Y),Y_i), $$ where the last equality uses $E[Y_i]=\mu_i$. All expectations exist by the stated second-moment assumptions. $\square$

The equal-second-moment condition matters: matching the means alone does not make the fresh and observed responses comparable in this identity. Without that condition, the same expansion gives $$ E[Q]-E[T] =E\|Y^0-\mu\|^2-E\|Y-\mu\|^2 +2\sum_{i=1}^n\operatorname{Cov}(\hat\mu_i(Y),Y_i). $$ The extra term is the difference in total response variance, not an effect of fitting.

Worked Examples

Worked Example: Fixed Weights Have No Response-Adaptation Optimism

Suppose the candidate prediction vectors $p_1,\ldots,p_K$ and weights $w_1,\ldots,w_K$ are fixed independently of $Y$. The fitted vector is $\hat\mu=\sum_{k=1}^K w_kp_k$. Assume $Y^0$ is an independent response vector with the same mean and total centered second moment as $Y$. What is the expected optimism?

For each coordinate $i$, $\hat\mu_i$ is fixed, so $\operatorname{Cov}(\hat\mu_i,Y_i)=0$. The covariance identity therefore gives $$ E[Q]-E[T]=2\sum_{i=1}^n 0=0. $$ The expected training and fresh-response losses agree. This does not say either loss is small; it says that fixed predictions do not gain an expected advantage from having adapted to the responses being evaluated.

Worked Example: Selecting Between Two Candidate Predictions

There is one response $Y\sim N(0,\sigma^2)$, and the two candidate predictions are the fixed values $-a$ and $a$, where $a>0$. Choose whichever candidate has smaller squared training loss. Calculate the expected optimism.

The selected prediction is $\hat\mu(Y)=a$ when $Y\geq 0$, and $\hat\mu(Y)=-a$ when $Y<0$. Hence $\hat\mu(Y)Y=a|Y|$. Symmetry gives $E[\hat\mu(Y)]=0$, so $$ \operatorname{Cov}(\hat\mu(Y),Y) =E[\hat\mu(Y)Y] =aE|Y| =a\sigma\sqrt{\frac{2}{\pi}}. $$ The covariance identity yields $$ E[Q]-E[T]=2a\sigma\sqrt{\frac{2}{\pi}}. $$ For a direct check, an independent $Y^0\sim N(0,\sigma^2)$ has fresh-response loss $$ E[(Y^0-\hat\mu(Y))^2]=\sigma^2+a^2, $$ because $Y^0$ is independent of the selected prediction, whose squared value is always $a^2$. The training loss is $(|Y|-a)^2$, so $$ E[T]=E[Y^2]+a^2-2aE|Y| =\sigma^2+a^2-2a\sigma\sqrt{\frac{2}{\pi}}. $$ Subtracting gives the same optimism. In the special case $a=\sigma$, the fresh loss is $2\sigma^2$, the training loss is $\bigl(2-2\sqrt{2/\pi}\bigr)\sigma^2$, and their difference is $2\sqrt{2/\pi}\,\sigma^2$.

This example makes the selection effect concrete: the chosen prediction is always the candidate closest to the observed response. It therefore tracks the response in the same direction, producing positive covariance and a lower training loss. The candidates themselves are fixed; it is the choice between them that adapts.

Worked Example: Unconstrained Linear Stacking and Prediction-Space Rank

Let $X$ be a fixed $n\times K$ matrix whose columns are candidate prediction vectors, and fit unconstrained stacking weights by least squares. Assume $X$ has rank $r$, $Y\sim N(\mu,\sigma^2 I_n)$, and $Y^0$ is an independent vector with the same distribution. Derive the expected optimism.

Least squares projects $Y$ onto the column space of $X$, so $\hat\mu=PY$, where $P$ is the orthogonal projection matrix onto that space. It satisfies $P=P^\top=P^2$ and $\operatorname{tr}(P)=r$. Since $Y=\mu+\varepsilon$, with $E[\varepsilon]=0$ and $\operatorname{Cov}(\varepsilon)=\sigma^2 I_n$, $$ \sum_{i=1}^n\operatorname{Cov}(\hat\mu_i,Y_i) =\sum_{i=1}^n\operatorname{Cov}((P\varepsilon)_i,\varepsilon_i) =\sigma^2\operatorname{tr}(P) =\sigma^2r. $$ Thus the identity gives $E[Q]-E[T]=2\sigma^2r$.

The result can also be checked from the losses. For training loss, $$ E[T]=E\|(I_n-P)Y\|^2 =\|(I_n-P)\mu\|^2+\sigma^2(n-r). $$ For fresh-response loss, independence of $Y^0$ and $Y$ gives $$ E[Q]=\|(I_n-P)\mu\|^2+n\sigma^2+r\sigma^2. $$ Indeed, the first two terms are the squared bias and fresh-response variance; the last is the variance of the fitted vector $PY$. Subtracting the training expression gives $2\sigma^2r$, as claimed.

What Adaptivity Does—and Does Not—Tell Us

The covariance identity applies to the complete fitted prediction rule. In stacking, that rule includes both the candidate predictions and the procedure that chooses their weights. If candidate predictions are fixed independently of the responses and the weights are also fixed, the covariance is zero. If weights are fitted from the responses, their choice can make the fitted values move with those responses, as the selection example shows.

In unconstrained linear stacking with fixed candidate predictions and least-squares weights, the projection rank gives an exact covariance sum. It is the dimension of the prediction space that matters in this calculation, not necessarily the raw number of columns: redundant candidate vectors do not increase the rank. For other weight-fitting procedures, including constrained weight choices, the fitted rule need not be a fixed linear projection. Its optimism is still governed by the covariance sum when the theorem’s conditions hold, but that sum need not equal $\sigma^2$ times the number of weights.

Common pitfall: “There are $K$ stacking weights, so the optimism must be $2K\sigma^2$.” This conclusion is not generally justified. The projection calculation establishes $2r\sigma^2$ only for fixed-design, unconstrained least squares with a linear projection of rank $r$ and homoskedastic independent normal noise. In a more general procedure, derive or estimate the dependence between each fitted prediction and its response; parameter count alone does not specify it.

Optimism is an expectation-level comparison, not a claim that every fitted ensemble performs worse on new responses than on its training responses. The realized difference can have either sign. The identity explains the average gap: positive response–prediction covariance increases expected optimism, while zero covariance removes this particular source of it. Reliable assessment of data-fitted weights must therefore account for the fact that the weights were chosen using the responses under evaluation.

Check Your Understanding

Use the definitions and covariance identity to answer the following.

Why does independence of $Y^0$ from $Y$ make the expected fresh-response cross term vanish?
Which additional condition beyond matching means is used to obtain the stated covariance identity?
In the two-candidate example, why is the selected prediction positively correlated with $Y$?
For unconstrained linear stacking with fixed candidate vectors, what feature of the prediction space determines the optimism?
Why can the number of stacking weights fail to determine optimism for a general data-adaptive fitting rule?

Optimism From Fitting Weights on Training Predictions

What You'll Learn