Nested Cross-Validation For Model Selection in Multi-Omics Integration

Why Model Selection Needs Its Own Evaluation

In multi-omics prediction, a modeling strategy is rarely fixed in advance. Analysts may compare integration strategies, feature-selection rules, regularization strengths, or other settings. Each choice can use information in the observed outcomes. If the same cross-validation results both select the best-performing procedure and estimate its performance, the estimate tends to be optimistic: random features of the data have helped determine which result is reported.

Nested cross-validation separates these tasks. An inner loop chooses among candidate procedures using only the current training data. An outer loop evaluates the procedure produced by that choice on held-out data. The target is the performance of the whole selection process, not the performance of a model whose settings were fixed using information from the held-out cases.

The independent unit for splitting must match the prediction target. For example, if predictions concern new patients, related samples from one patient must not be split across training and test data. If the target is prediction in a new cohort, random individual-level splitting may not answer the question; a cohort-aware design may be needed. The earlier material on choosing independent units and cohort-aware splits supplies this design principle.

Constructing the Two Loops

Let the \(n\) independent units be partitioned into \(K\) outer folds \(I_1,\ldots,I_K\). For outer fold \(k\), the outer test set is \(I_k\), and the outer training set contains all units not in \(I_k\). Let \(\lambda\) denote a candidate procedure: it may specify an integration strategy and settings such as a regularization strength. It is the complete candidate to be compared, not just a single numerical setting.

Within each outer training set, divide the data into inner folds. For each candidate \(\lambda\), fit it on each inner training portion and evaluate it on the corresponding inner validation portion. Aggregate those validation results according to a prespecified metric and selection rule. For a loss to be minimized, the choice can be written as:

Here \(\Lambda\) is the set of candidates, and \(\widehat{L}_{\mathrm{inner},k}(\lambda)\) is the inner-loop estimate of loss for candidate \(\lambda\) within outer training set \(k\). For a metric to be maximized, such as a prespecified discrimination measure, use the corresponding maximization rule. Ties require a rule stated in advance, such as choosing the simpler candidate.

After choosing \(\hat{\lambda}_k\), fit that procedure using the entire outer training set and predict outcomes for the outer test fold. Each unit therefore receives an outer prediction from a fit that did not use its outcome for tuning or fitting. Any data-adaptive operation used in a candidate procedure must respect the inner training and validation boundaries while candidates are compared. The later material on fold-local preprocessing and feature selection addresses those operations in detail; nesting does not make it acceptable to use validation outcomes during fitting.

What the Outer Results Estimate

Suppose the prediction target is a loss \(L\), with smaller values preferred. If \(L_k\) is the loss on outer fold \(k\), the unweighted mean of fold losses is:

This is a direct summary when folds have equal size. With unequal fold sizes and a per-observation loss, a size-weighted average is often appropriate:

where \(n_k\) is the number of observations in outer fold \(k\). For non-additive metrics, such as area under the receiver operating characteristic curve (AUC), the mean of fold-specific values and a metric computed from all outer predictions are not generally equivalent. In particular, pooling scores from separately fitted models can compare scores on different scales across folds. Choose and report the summary that matches the intended estimand; do not silently treat these summaries as interchangeable.

The outer estimate concerns the selection-and-fitting procedure trained on roughly the outer training-set size, not necessarily a final model trained on all \(n\) observations. Its relevance depends on the folds representing the intended deployment population and on the data-generating conditions being sufficiently similar. Cross-validation does not by itself establish transportability to a different population.

Why Reusing the Outer Results Creates Optimism

The outer folds protect the evaluation from decisions made inside each outer training set. But that protection is lost if the analyst uses the outer results to make another selection and then reports the selected result as unbiased. For example, comparing five integration strategies by their outer AUCs and reporting only the largest AUC uses outer performance to choose the winner. The reported maximum is favored partly because it may have benefited from favorable sampling variation.

The basic reason is the maximum's tendency to exceed the average when estimates vary. If \(Z_1,\ldots,Z_m\) are noisy estimates of equally good candidates, then:

This inequality does not quantify the amount of optimism, but it shows why selecting the largest noisy estimate is not a neutral reporting step. To estimate performance after choosing among strategies, include those strategies in the inner selection process and evaluate that selection process in the outer loop. Alternatively, use a genuinely untouched test set for the final evaluation. Outer-fold results can still compare prespecified procedures, but choosing a winner from those results changes what can be claimed about that winner's performance.

Assumptions, Diagnostics, and Alternatives

Nested cross-validation is useful only when its design matches the question. Its interpretation relies on several practical conditions:

• Splits respect independent units. Replicates, repeated measurements, or related samples that could share information must be kept together when the deployment target is a new independent unit.
• Outer folds represent the target population. If deployment is to new cohorts, sites, or time periods, folds should reflect that target rather than merely randomize individuals across all sources.
• Selection is contained in the inner loop. Candidate comparisons and data-adaptive choices must not use outer-test outcomes. The candidate set and metric should be specified without inspecting outer results to decide what to report.
• Each split permits fitting. Training folds must contain enough observations and relevant outcome classes for the chosen procedures and metrics. A fold with no positive cases, for example, cannot supply a meaningful sensitivity or AUC estimate.

Check fold sizes and outcome distributions before fitting. Record which candidate is selected in each outer fold; frequent changes can indicate unstable selection even when average performance looks adequate. Compare outer-fold metrics for unusual variation, and investigate whether it aligns with cohort, site, class composition, or fold size. Also verify that the inner loop is run separately within each outer training set. Running one global tuning exercise and carrying its chosen setting into every outer fold defeats the separation.

A single held-out test set is an alternative when there is enough data to reserve a representative test sample and the modeling decisions can be completed using the remaining training data. It offers a direct final evaluation but may be imprecise, especially with small samples. Non-nested cross-validation can be suitable for evaluating a fully prespecified procedure whose settings are not chosen from those same validation results. It does not, on its own, remove the optimism caused by selecting the best candidate using the same cross-validation scores later reported as performance.

Nested cross-validation also has limits. Its outer estimate is noisy in small samples, results depend on the split design, and its training sets are smaller than the full data set. Repeating the nested design can show sensitivity to random partitions, but repeated results are not independent replications of the study. Finally, no resampling design fixes a mismatch between the observed data and the deployment population.