End-to-End Honest Evaluation of a Genomic Classifier

What an End-to-End Evaluation Must Estimate

A genomic classifier may have far more measured features than participants. In that setting, a high score on data used to select features, preprocess measurements, or tune a penalty can be misleading: some information from those data has already shaped the model. Honest evaluation therefore requires more than calculating a metric after fitting. It requires defining what future prediction is intended, respecting the data’s dependence structure, and keeping every model-selection decision inside the training data.

Suppose the intended use is to predict whether a new participant from the same target population has a specified disease, using that participant’s genomic measurements. The prediction target is performance on such future participants—not performance on new rows from the same family if the intended use is new families. If several participants are related, the split must reflect which kind of new unit the analysis aims to predict.

The evaluation depends on assumptions. The held-out groups should represent the target population; outcome labels and genomic measurements should be available under the stated prediction setting; and the analysis must not use information that would be unavailable at prediction time. Cross-validation does not itself guarantee these conditions. It estimates performance for the specified data-generating setting and modelling procedure, subject to the design and the observed sample.

Define the Pipeline Before Splitting

Consider a hypothetical genomic study with 12 participants from six families, with two related participants per family. Each participant has \(p=5{,}000\) gene-expression measurements and a binary disease label \(Y\), where \(Y=1\) denotes a case and \(Y=0\) a control. The intended target is prediction for a participant from a previously unseen family drawn from the same population. This small example is for illustrating calculations, not for recommending a sample size.

We use logistic ridge regression. Within each training set, each feature is standardized using that set’s feature means and standard deviations. A feature with zero training-set standard deviation would be removed using a rule fixed in advance. The intercept is not penalized. Candidate penalty values are \(\lambda \in \{0.01,0.1,1,10\}\), and the inner-fold selection criterion is mean log loss. The classification threshold is fixed at 0.50 before evaluation; it is not chosen by inspecting outer-fold outcomes.

Here \(\ell\) is the logistic log-likelihood for the current training data, and the penalty shrinks feature coefficients toward zero while leaving the intercept unpenalized. The standardization parameters, any feature filtering, and the fitted coefficients are recalculated from scratch within each training split. The same rule applies inside inner validation: validation participants do not contribute to the means, standard deviations, or model fit used to predict them.

For outer validation, the six families are assigned as intact groups to three folds, with two families (four participants) in each fold. Each outer training set thus contains the other four families. Within that training set, two-fold inner validation splits those four families into two groups of two families. Both levels keep relatives together. In an actual analysis, the folds should also be checked for adequate case and control representation; if the grouping structure makes that impossible, the limitation must be reported rather than ignored.

Run the Nested Analysis

For each outer fold, the analyst uses only its outer training participants to compare the candidate values of \(\lambda\). For each candidate, the entire preprocessing-and-fitting pipeline is run separately in each inner split, and predictions are made for that split’s held-out families. The mean inner validation log loss selects \(\lambda\); the selected pipeline is then refitted on the full outer training set and applied once to the outer held-out families. The outer outcomes are not used to choose the penalty or to alter the pipeline.

The reason for the two levels is that choosing the best-performing candidate and reporting that same validation performance confounds selection with evaluation. The inner scores guide selection; the outer scores evaluate the selection-and-fitting procedure on data not used for that selection. Thus the outer estimate concerns the complete specified procedure, including its tuning rule, rather than a penalty value chosen with knowledge of all outcomes.

Calculate and Report Outer-Fold Performance

The table gives hypothetical probabilities from the three outer-fold fits. Each family’s two participants appear in the same fold. A prediction is classified as a case when its probability is at least 0.50. The scores are held-out predictions: no participant’s outcome was used to fit the model that generated that participant’s score.

Accuracy is the fraction of all participants classified correctly. Sensitivity is the fraction of cases classified as cases; specificity is the fraction of controls classified as controls. These metrics depend on the threshold, which here was fixed at 0.50.

The area under the receiver operating characteristic curve (AUC) summarizes ranking across thresholds. With no tied scores, it is the proportion of case-control pairs in which the case receives the higher score. There are six cases and six controls, hence \(6\times6=36\) pairs. The case score 0.31 exceeds three control scores (0.12, 0.18, and 0.27); 0.44 exceeds five control scores; and each of 0.63, 0.71, 0.77, and 0.82 exceeds all six control scores. The total correctly ordered pairs are \(3+5+6+6+6+6=32\), so the pooled AUC is \(32/36=0.8889\). AUC has no units and does not identify a useful operating threshold by itself.

Pooling held-out predictions gives each participant one prediction from a model that did not train on that participant. However, predictions come from different outer-fold fits. A pooled AUC is most interpretable when those scores are comparable across folds; fold-specific AUCs and the fold assignment can help reveal whether that is plausible. Fold metrics are also useful diagnostics, but three folds in a small example provide little basis for estimating how much performance would vary across new samples.

Justification, Diagnostics, and Alternatives

The nested design separates selection from evaluation in a direct way. Let \(S\) denote the outer training data and \(V\) the outer held-out data. The selected tuning value is a function of \(S\), because it is selected using only inner splits of \(S\). The fitted model is also a function of \(S\). If \(V\) has not influenced any pipeline choice, predictions on \(V\) provide an evaluation not selected to look good on \(V\). By contrast, choosing a candidate using scores calculated on \(V\) makes the reported outer performance part of the selection process.

This argument depends on respecting the boundary. For example, selecting genes once using all 12 outcomes before making folds violates it: information from every outer held-out outcome has already affected the feature set. Computing standardization parameters from all participants also lets held-out measurements influence the fitted transformation. Even if outcome labels are not used for standardization, that procedure does not represent a pipeline trained only on available training data. Such operations must be learned within each training split.

Useful checks include confirming that no family occurs in more than one fold, that each participant receives exactly one outer prediction, and that preprocessing is refitted inside every inner and outer training set. Inspect inner selection results for penalties chosen at the edge of the candidate range, large differences between folds, or near-ties. Check outer-fold metrics for instability and verify that thresholded classifications agree with the recorded probabilities and threshold. If an outer fold lacks a case or a control, some metrics, including sensitivity or specificity, cannot be calculated for that fold.

Nested cross-validation is not the only design. A prespecified train-test split can provide an independent evaluation when there is enough data for both development and testing, but it may be inefficient and sensitive to the particular split. External validation tests a frozen pipeline on a separate dataset and can more directly assess transport to that dataset, though it may differ in population or measurement processes. Nested cross-validation uses the available data for both tuning and evaluation, but its estimate concerns models trained on outer-training subsets, not exactly a final model trained on every participant.

A common mistake is to report the best inner-fold score as if it were unbiased test performance. The reasoning is that the winning candidate performed best on validation data, so its score seems to summarize future accuracy. The correction is that the same scores helped choose the winner; that selection tends to favor candidates with unusually favorable validation results. Report the outer-fold results as the evaluation, and use inner results for selection only.