Sample Splitting and Cross-Fitting for Stacking

Why the Predictions Used to Fit Weights Matter

Stacking combines predictions from several fitted models by choosing weights to minimize a loss. The weights are themselves estimated from data, so the predictions used to estimate them matter: if each model's predictions were made on observations it had already fitted, those predictions can be unusually favorable. The resulting weights may favor members whose apparent accuracy reflects training fit rather than predictive performance.

A direct remedy is to separate the observations used to fit the member models from those used to estimate the weights. Cross-fitting uses the same principle repeatedly: each observation contributes a prediction from a member model trained without that observation. In both cases, the comparison supported by the data must match the models and risk target actually evaluated.

In a sample split, one part of the data fits the members and a separate validation part estimates the weights. Conditional on the fitting data, the member predictions are fixed, and the validation observations are independent draws from the target distribution. Cross-fitting instead partitions the observations into folds; the prediction for an observation is made by a member fitted without that observation's fold.

Independent Validation: A Risk Comparison

The validation criterion is useful because, for any fixed weight vector, it estimates the risk of the already-fitted members. But choosing weights by minimizing that criterion makes the chosen validation score optimistic. A risk comparison must account for this search. The following finite-candidate result gives one precise set of sufficient conditions.

Proof. Condition on \(D_{\mathrm{tr}}\). For each fixed \(w\), the \(m\) validation losses are independent, identically distributed, and lie in \([0,B]\), with mean \(R(w)\). Hoeffding's inequality therefore gives, for any \(t>0\), $$ P\left(\left|\widehat R_{\mathrm{val}}(w)-R(w)\right|>t\mid D_{\mathrm{tr}}\right)\leq 2\exp\left(-\frac{2mt^2}{B^2}\right). $$ Taking a union bound over the \(|\mathcal W_0|\) candidates shows that, with conditional probability at least \(1-\delta\), every candidate satisfies \(\left|\widehat R_{\mathrm{val}}(w)-R(w)\right|\leq t\), where \(t=B\sqrt{\log(2|\mathcal W_0|/\delta)/(2m)}\). On this simultaneous event, choose any risk minimizer \(w^\star\) in the finite set. The definition of \(\hat w\) and the two deviation bounds give $$ R(\hat w)\leq \widehat R_{\mathrm{val}}(\hat w)+t \leq \widehat R_{\mathrm{val}}(w^\star)+t \leq R(w^\star)+2t. $$ This proves the claim. Since the conditional failure probability is at most \(\delta\) for every training set, the same probability bound holds unconditionally. \(\square\)

The bound compares the selected weight with the best candidate in \(\mathcal W_0\), not automatically with every point in the simplex. A continuum of possible weights needs an additional uniform-convergence argument, or a finite candidate set with a suitable approximation guarantee. The bound also concerns the risk of the particular members fitted on \(D_{\mathrm{tr}}\); it does not silently change them to models fitted on more data.

Cross-Fitting and Its Risk Target

For \(K\) folds, write \(I_k\) for the indices in fold \(k\), with \(m_k=|I_k|\) and \(\sum_{k=1}^K m_k=n\). For each fold \(k\), fit every member using observations outside \(I_k\), and denote those predictors by \(g_{1,-k},\ldots,g_{J,-k}\). For \(i\in I_k\), form the out-of-fold prediction \(g_{w,-k}(X_i)=\sum_jw_jg_{j,-k}(X_i)\). The pooled cross-fitted criterion is $$ \widehat R_{\mathrm{CF}}(w)=\sum_{k=1}^K\frac{m_k}{n}\left\{\frac{1}{m_k}\sum_{i\in I_k}\ell(Y_i,g_{w,-k}(X_i))\right\}. $$

Its natural target is the weighted risk of the fold-specific fitted members, not necessarily the risk of a single model fitted on all \(n\) observations. For fixed \(w\), let \(R_k(w)\) be the risk on a fresh observation of the ensemble whose members were fitted outside fold \(k\), conditional on those fitted members. Set \(R_{\mathrm{CF}}(w)=\sum_k(m_k/n)R_k(w)\). This target can vary with the observed training folds.

Proof. Fix a fold \(k\). Conditional on the observations outside \(I_k\), the fold-\(k\) predictors are fixed. The observations in \(I_k\) are independent draws from the target distribution and are independent of those predictors. For each fixed \(w\in\mathcal W_0\), Hoeffding's inequality bounds the probability that the fold's average loss differs from \(R_k(w)\) by more than \(t_k\) by \(2\exp(-2m_kt_k^2/B^2)\). Choose $$ t_k=B\sqrt{\frac{\log(2K|\mathcal W_0|/\delta)}{2m_k}}. $$ The bound for one fold and one candidate is then \(\delta/(K|\mathcal W_0|)\). A union bound over every fold and candidate shows that, with probability at least \(1-\delta\), every fold average differs from its fold risk by at most \(t_k\). This union bound does not require the fold averages to be mutually independent.

On that simultaneous event, the pooled empirical criterion differs from \(R_{\mathrm{CF}}(w)\) by at most \(\sum_k(m_k/n)t_k\), for every candidate \(w\). Compare the selected \(\hat w\) with a minimizer of \(R_{\mathrm{CF}}\) in the finite set. The empirical minimizing property gives the same three-step comparison as in the validation proof: population risk of \(\hat w\) is at most its empirical risk plus the deviation bound; that empirical risk is no larger than the comparator's; and the comparator's empirical risk is at most its population risk plus the deviation bound. The total penalty is twice \(\sum_k(m_k/n)t_k\), as claimed. \(\square\)

For equal-sized folds, the penalty is \(2B\sqrt{\log(2K|\mathcal W_0|/\delta)/(2m)}\), where \(m=n/K\). The proof conditions separately on the training data for each fold and then uses a union bound. It does not incorrectly treat all out-of-fold losses as independent: the fitted models overlap in their training data, and different fold averages can be dependent.

When Cross-Fitted Weights Are Applied to Refit Models

In practice, one may estimate weights from out-of-fold predictions and then refit each member on all available development data before deployment. That can be sensible, but the cross-fitted risk bound alone does not establish the risk of the full-data refits. The fold-specific members were trained on fewer observations and differ across folds; the deployed members are different fitted predictors.

A sufficient way to make a comparison is to assume a transfer bound. Let \(R_{\mathrm{full}}(w)\) be the risk of the full-data-refitted ensemble, conditional on the full development data, evaluated on a fresh observation. If, for every \(w\in\mathcal W_0\), $$ |R_{\mathrm{full}}(w)-R_{\mathrm{CF}}(w)|\leq\Delta, $$ then the cross-fitting theorem implies $$ R_{\mathrm{full}}(\hat w)\leq \min_{w\in\mathcal W_0}R_{\mathrm{full}}(w) +2\sum_{k=1}^K\frac{m_k}{n}B\sqrt{\frac{\log(2K|\mathcal W_0|/\delta)}{2m_k}}+2\Delta. $$ Indeed, transfer from the full-data risk of \(\hat w\) to its cross-fitted risk costs at most \(\Delta\); the cross-fitted oracle comparison adds the theorem's penalty; and transferring the minimizing cross-fitted risk back to the full-data risk costs at most another \(\Delta\). The assumption may follow from appropriate stability or convergence properties of the fitting procedure, but it is not automatic.

Common Pitfall and Practical Implications

• Using fitted predictions as if they were held out. The error of a model on observations used to fit it need not estimate its fresh-observation risk. Weight fitting can exploit this optimism. Use an independent validation split or genuinely out-of-fold predictions.
• Calling the minimized validation score an unbiased risk estimate. Even when each fixed candidate's validation loss is unbiased conditional on the training data, selecting the smallest score creates a search effect. Use a separate evaluation sample or a justified uniform risk bound.
• Claiming cross-fitting proves the risk of full-data refits. Cross-fitted predictions evaluate fold-trained members. Transfer to full-data-refitted members requires an additional argument.
• Treating fold losses as independent. Their member fits share data. A valid argument can condition fold by fold and combine bounds by a union bound, as above; it must not assume unconditional independence without justification.

Sample splitting uses fewer observations to fit the members, while cross-fitting lets each observation contribute an out-of-fold loss and uses most observations for each fold-specific fit. Neither procedure removes the need to specify the risk target. The central check is whether the predictions used to estimate weights were produced without their outcomes, and whether the risk comparison concerns those same fitted predictors or a justified refit.