Worked Example Of Nested Multi-Omics Evaluation

Why Nested Evaluation Is Needed

A multi-omics classifier can appear to predict well for two very different reasons: it may capture a reproducible relationship between the omics measurements and the outcome, or the analysis may have reused information from the subjects on whom it is evaluated. Reuse can happen indirectly. For example, scaling features using all subjects, selecting features before splitting, or choosing a tuning value after looking at test performance can all make an evaluation optimistic.

This tutorial works through one binary prediction task with two omics blocks. The target is an honest estimate of the performance of the complete fitted procedure, including its preprocessing and tuning—not just of a classifier considered in isolation. The outer test fold is kept untouched while all choices are made using the outer training data.

For the worked task, let \(Y_i\in\{0,1\}\) denote the binary outcome for subject \(i\), and let \(X_i^{(1)}\) and \(X_i^{(2)}\) denote that subject’s measurements in two omics blocks. The planned classifier concatenates the retained features from both blocks and fits a ridge-penalized logistic regression. For a feature vector \(x_i\), the model assigns probability \( \hat{p}_i = 1/(1+\exp[-(\hat{\beta}_0+x_i^\mathsf{T}\hat{\beta})]) \). The penalty parameter \(\lambda\) controls the amount of coefficient shrinkage. A larger \(\lambda\) applies stronger shrinkage. The candidates and evaluation metric are specified before inspecting outer-test results.

The assumptions behind interpreting the outer score are important. Subjects in the outer test fold must not have contributed to development decisions, directly or indirectly. The split should reflect the intended prediction setting: for a new-subject prediction task, all records from one subject must stay in the same fold. The held-out subjects should also represent the target population reasonably well. A random split does not measure performance in a different population or a future period if those differ from the sampled data.

Fold-Local Preprocessing and Inner Tuning

Suppose there are 54 hypothetical subjects, with two omics blocks measured on each subject. We reserve 6 subjects—3 with \(Y=1\) and 3 with \(Y=0\)—as the outer test fold. The remaining 48 subjects form the outer training set, with 24 in each outcome class. We use four stratified inner folds, each containing 6 positive and 6 negative subjects. Stratification helps keep both outcome classes represented in these particular folds; it does not remove the need to check that the split design suits the application.

For each inner split, the 36 inner-training subjects alone determine the preprocessing. A planned sequence might remove features failing a prespecified measurement-quality rule, impute missing values using feature-wise training medians, remove features with zero training variance, and standardize each retained feature using the inner-training mean and sample standard deviation. The resulting transformation is then applied to the 12 inner-validation subjects without refitting it. Any supervised feature screen must also be fitted anew using only the 36 inner-training subjects.

The same discipline applies to the outer evaluation. Once tuning is complete, preprocessing is refitted on all 48 outer-training subjects, and the selected model is fitted on those transformed data. Those fitted transformations and coefficients are then applied to the 6 outer-test subjects. The test measurements are inputs to prediction, but they do not determine imputation values, scaling constants, selected features, tuning choices, or model coefficients.

For a candidate \(\lambda\), fit the same two-block pipeline in each inner split and calculate its AUC on that split’s validation subjects. AUC is the proportion of positive-negative pairs in which the positive subject receives a higher predicted score, with tied pairs counted as one half. The inner selection rule here is to choose the \(\lambda\) with the highest mean of the four fold-specific AUCs. A prespecified tie rule—for example, choose the larger \(\lambda\) in an exact tie—prevents an unplanned choice after seeing results.

Refit and Evaluate Once on the Outer Fold

After choosing \(\lambda=1\), the selected pipeline is fitted again from the beginning using all 48 outer-training subjects. That means the training-only quality checks, imputation, variance checks, and scaling are recalculated on these 48 subjects, and the ridge logistic regression is refitted with \(\lambda=1\). The resulting model produces one predicted probability for each of the 6 outer-test subjects. No change to the pipeline is made in response to these predictions or the observed outer outcomes.

A short justification explains why this separation matters. Conditional on the outer-training data, the fitted pipeline and its predictions for future subjects are fixed. If the outer test subjects are independent of that training data and are sampled from the target population, the expected held-out AUC evaluates that particular fitted pipeline rather than a procedure selected to look good on those test outcomes. In contrast, choosing the best-looking candidate using the test AUC makes the evaluation data part of model selection. For candidate scores \(A_1,\ldots,A_K\), selecting \(\max_k A_k\) tends to give a value at least as large as the best candidate’s underlying expected score:

The inequality follows because \(\max_k A_k\geq A_j\) for every candidate \(j\); taking expectations and then choosing the candidate with the largest expectation gives the result. The outer fold protects against this selection effect only if it remains untouched through all choices.

Diagnostics, Alternatives, and Limitations

Before interpreting a score, audit the data flow at every split. Check that sample identifiers align across the two blocks and outcome labels, that no subject or repeated specimen crosses folds, and that each inner validation subject receives a prediction from a model that did not train on that subject. Confirm that every learned transformation is trained within the relevant training fold. A preprocessing object fitted once on all 54 subjects and then reused in cross-validation is a warning sign, even if the classifier itself was refitted correctly.

Also inspect whether each fold contains both outcome classes, whether missingness or failed assays cause exclusions that differ by outcome, and whether the chosen metric matches the prediction goal. AUC cannot be calculated in a validation fold with only one outcome class. If that occurs, revise the split plan or use a metric and design appropriate to the data; do not silently omit problematic folds. Large variation in inner-fold scores can indicate small-sample instability, influential subjects, or a split that is poorly matched to the study structure.

Nested evaluation differs from several common alternatives. A single development/test split with tuning performed entirely inside the development set can also protect the test set, but its result may be sensitive to which subjects happen to be held out. Ordinary cross-validation that uses the same folds both to select the best candidate and to report its best score is generally optimistic for the selected candidate. Nested cross-validation repeats the outer split-and-evaluate process across outer folds, giving multiple held-out predictions and a broader summary, but it is computationally more demanding and requires careful aggregation. It is not the same as evaluating on an independent external cohort, which tests transport to a separately sampled population.

A single outer fold has important limitations. It provides only one estimate, and with few held-out subjects its uncertainty can be substantial. The particular split can affect the result. If the intended use is prediction in a new hospital, future time period, or different population, a random subject-level split may not represent that use; the split strategy must instead mirror the intended deployment setting. Finally, honest evaluation does not guarantee a useful model: a fully leakage-free pipeline can still perform poorly or be unstable.

Common Pitfall